h3-blackwell-runtime/research/vortex_exact_attention/EXPERIMENT_LOG.md

Experiment Log

This is append-only. Projected ranges are design estimates, never achieved results.

2026-08-26: Phase 0 Import And Verification

Status: research_retained

Hypothesis: the retained Sage inventory is sufficient to import the exact observable contract without repeating prior experiments.

Actions:

• Verified public SageAttention commit d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5 exists.
• Rehashed ten retained contract, parity, timing, and NCU artifacts; all matched benchmarks/reference_inventory.json.
• Imported Q/K preparation, K smoothing, INT8 quantization, QK traversal, online-softmax, FP8 V/P, two-level PV accumulation, final conversion, and ragged-tail contracts into NUMERICAL_CONTRACT.md.
• Recorded missing production binary/checkpoint hashes and missing standalone canonical input tensors as blockers rather than fabricating provenance.
• Did not execute inference, modify dispatch, or touch the production service.

Decision: Phase 0 is verified as an imported specification. No kernel work is authorized until Phase 1 selects a structural state-ownership architecture.

Closed Prior Work

Work	Result	Preservation / reopening rule
P0 register caps	Three-CTA threshold at 168 registers caused catastrophic spills; no gain	Timing/JSON/NCU retained; exact candidate source missing and must not be claimed reproducible
P0 scope/lifetime narrowing	Fewer spills, approximately 0.06% gain	Reopen only with structural ownership change
P0 score aliasing	Approximately 0.01%, noise	Do not repeat unchanged
P0 early K prefetch	Approximately 0.30% slower	Reopen only with a different producer/consumer pipeline
P0 softmax interleaving	Neutral/slower	Do not repeat unchanged
P0 padded V shared memory	Same excessive wavefronts, no gain	Reopen only if staging instructions/dataflow change
P1 entry fusion	20.9-23.4% isolated; only 0.535-0.863% complete-block	Reuse partial implementation only inside an owned attention pipeline
P2 direct V preparation	39.48%, 4.17 ms; only 0.909% of block	Reuse partial implementation only inside an owned attention pipeline
P3 temporal pairing	Exact/sanitizer-clean; approximately 0.10%; private state remained live	Complete patch and launcher retained; new work must transfer/partition ownership
Hopper WGMMA retarget	CUDA 13 rejects WGMMA for sm_121a	Reopen only with changed ISA/toolchain support
SM121 UMMA INT8 QK	Public SM120/121 Blackwell MMA path does not expose a Sage-compatible UMMA INT8 operation	Retain supported INT8 mma.sync unless capability changes

P1/P2 source is partial and mixed in research/shared_cuda_experimental_worktree/. P3 source and launcher are complete under research/sage2_temporal_pair/. Exact P0 variant patches are unrecoverable; historical commands and environments must not be invented.

2026-08-26: Phase 1 State-Ownership Decision

Status: research_retained

Evaluated three architectures that permanently separate QK/softmax state from PV/output state: VEA-A Q64 split-D intra-CTA, VEA-B Q128 paired owners, and VEA-C two-CTA DSM ownership split. All retain strict tile order, exact FP8 score bytes, per-tile RO rescaling, two-level PV accumulation, and final conversion.

Decision: recommend VEA-B after register/shared/barrier/dual-pipe capability probes. Its projected 180-207 ms mainloop screening range has a credible sub-190 path without depending on unverified DSM. VEA-A is the fallback; VEA-C is conditional on cluster residency and DSM latency. These are projections only.

No kernel was implemented, no inference was run, and production dispatch and services were not modified.

Phase commits:

• Phase 0 contract import: c502842
• Phase 1 architecture decision: b842bad

2026-08-26: Phase 2A VEA-B Capability Decision

Status: research_retained

The corrected 320-thread VEA-B probes compile for SM121 with 96 registers/thread for handoff, 54 for QK role state, 138 for PV role state, and 139 for synthetic combined ownership. Ptxas and NCU report no spills, and 51.2 KiB shared memory permits one ten-warp CTA/SM.

The selected CUDA block-scope mbarrier handoff completes all 591 epochs with checksum 4,671,090, zero publication errors, deterministic repetition, 0.076800 ms barrier-only p50, and clean memcheck/racecheck. The faster inline named-barrier path is rejected because racecheck reports five hazards. INT8 and FP8 MMA role clocks overlap in 48/48 blocks. Canonical Q/K/V/output fixtures are self-contained and reload-verified; the output matches the locked Sage2 SHA-256.

The first full probe is invalid for payload conclusions because K/V slot bytes were indexed as one 16 KiB region instead of two 8 KiB regions. Deterministic repetition detected the bug. Corrected indexing passes checksum and sanitizer gates; the invalid 16.631 ms result is not hardware evidence.

Decision: VEA-B advances only to one isolated aligned-shape exact-attention prototype using mbarriers. No attention latency or speedup has been measured, and production dispatch remains unchanged.

2026-08-26: Phase 2B Fixed Aligned Numerical Prototype

Status: research_retained

The selected ten-warp Q128 x KV192 x D128 VEA-B kernel matches the recovered Sage2 checkpoint oracle for final BF16 output, QK, m, d, FP8 probability, rescale, FP16 PV, FP32 RO, and reciprocal values. The output SHA-256 is 75d4169c9ae0692b24a463a64b538f852acba5a5567404728c8af60c16fb8e1b. It repeats exactly for 1,000 launches.

Role-separated loops, ordered 8-column PV fragments, warp scheduling boundaries, and a per-lane shared retirement slot reduce the fast and capture kernels to 168 registers/thread with zero stack, spill traffic, and local bytes. The selected alias is protected by an explicit warp phase boundary; memcheck reports zero errors and racecheck reports zero hazards. Runtime and NCU show one ten-warp CTA/SM and zero local spilling requests. QK/PV clocks overlap across both epoch transitions.

The final batched stability run reports VEA-B p50 0.014426 ms, CV 0.87%, and public Sage2 p50 0.014871 ms, a fixed-shape prototype ratio of 1.031x. The 2,000-sample single-launch run reports 0.016288 ms versus 0.019840 ms, or 1.218x, with VEA-B CV 5.95%.

Rejected spill variants are retained in durable probe/build artifacts: combined ownership, sequential PV fragments, union role state, shared persistent RO, and setmaxnreg redistribution. The latter fails launch resource checks; shared persistent RO changes epoch-1 FP32 checkpoints.

Decision: Phase 2B passes only for the isolated aligned fixture. No ragged-tail, canonical-shape, H3 block, trajectory, resident-service, or production dispatch claim is made.