h3-blackwell-runtime/PERFORMANCE_ROADMAP.md

# H3 Performance Roadmap

This document is the canonical performance plan for the direct MiniMax H3
runtime on GB10/SM121, RTX PRO 6000/SM120, and B200/SM100. It separates measured
results from projections and requires numerical and visual gates before an
approximate optimization becomes a default.

## Executive Decision

The first component, bit-exact H3 modulation and residual-gate fusion on GB10,
is complete.

This is the best first engineering target because:

1. The current path spends about `39 ms` per representative block in modulation
   and residual gates, before counting the separate `25.8 ms` SwiGLU operation.
2. A Triton reference with explicit BF16 rounding already exists and has been
   validated bit-exact against eager H3 block behavior.
3. The optimization does not alter attention semantics, weights, precision, or
   scheduler behavior.
4. The same elementwise design applies to SM100, SM120, and SM121.
5. It provides a low-risk test of the profiling, fallback, parity, and deployment
   machinery required by later custom kernels.

The next component is NVFP4 scale, activation packing, GEMM, and epilogue
fusion. Component 1 established the numerical contracts and validation
infrastructure required to attempt it safely. A custom attention target follows
this component and should preserve SageAttention 2.2.0's validated numerical
policy while adopting zero-copy strided QKV input and SM-specific fused
execution. Sol sparsity must remain experimental until full latent, audio, and
visual quality gates pass.

Completed result: blocks 0, 24, and 49 are bit-exact and `3.6-3.9%` faster;
the canonical 12-step sampling run is checksum-identical and improves from
`310.11 s` to `301.05 s` (`3.0%`). Peak allocated memory falls by
`408,825,856` bytes. Evidence is in
`benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`.

Internal closeout summary:

> Vortex Component 1 is complete. We implemented bit-exact H3 modulation and
> residual-gate fusion on GB10, reducing canonical 12-step sampling from 310.11
> seconds to 301.05 seconds while retaining identical video and audio checksums.
> The fused runtime improves individual transformer blocks by 3.6-3.9%, reduces
> peak allocated memory by approximately 390 MiB, and passes 32 deployed
> numerical and integration tests. This establishes that Vortex can remove
> meaningful H3 execution overhead without changing the model's numerical
> program or generated output.

This is measured engineering evidence, not a projected optimization.

## Measured GB10 Baseline

The representative workload is 1344x768, 124 frames, 100 synthetic refined-text
tokens, and 37,810 packed tokens. Measurements use block 24 unless noted.

Warmed top-level sampling step:

| Backend | Step time | Relative to Sage2 |
| --- | ---: | ---: |
| SageAttention 2.2.0 | `25.414 s` | baseline |
| Sol BF16, tau 1.3 | `21.573 s` | `1.18x` |
| Sol residual-INT8 Q/K, tau 1.3 | `19.774 s` | `1.29x` |

The denoiser consumes effectively all warmed-step time. Packing takes about
`15 ms`; output transformation and the RES update together take about `1.4 ms`.

Representative Sage2 block:

| Component | P50 | Approximate block share |
| --- | ---: | ---: |
| Attention | `259.9 ms` | `50.5%` |
| NVFP4 QKV/out/FC1/FC2 projections | `151.1 ms` | `29.3%` |
| Modulation, residual gates, and SwiGLU | `64.9 ms` | `12.6%` |
| QKV and output layout materialization | `20.3 ms` | `4.0%` |
| Norm and RoPE | `19.7 ms` | `3.8%` |
| Full module forward | `514.9 ms` | baseline |

SageAttention 2.2.0 internals:

| Component | P50 |
| --- | ---: |
| Q/K INT8 quantization | `6.45 ms` |
| V FP8 quantization | `10.19 ms` |
| Attention forward | `239.00 ms` |
| K smoothing and framework estimate | `2.98 ms` |

Sol residual-INT8 Q/K internals at tau 1.3 with exact conditioning KV:

| Component | P50 |
| --- | ---: |
| K summary and residual quantization | `4.45 ms` |
| V summary | `2.45 ms` |
| Q quantization and routing threshold | `4.27 ms` |
| Total preparation | `11.53 ms` |
| Routed forward estimate | `114.88 ms` |
| Total | `126.42 ms` |

INT8 P-by-V is not useful on GB10: it increases total Sol attention time to
`138.65 ms`.

## Quality Boundary

Current Sol routing is a performance bound, not an accepted Sage2 replacement.

| Sol policy | Relative L2 versus Sage2 | Performance |
| --- | ---: | ---: |
| Fully dense, tau -100 | `0.0171` | `3.2x` slower than Sage2 |
| Exact conditioning KV, tau 0.8 | about `0.471` | about `1.12x` faster |
| Exact conditioning KV, tau 1.3 | about `0.626` | about `1.87x` faster in isolated attention |

No measured Sol setting is both faster than Sage2 and numerically Sage-like.
Exact conditioning sinks protect prompt/audio rows but do not remove sparse
video-attention error. A timestep or block hybrid therefore needs complete
latent, audio, lip-sync, and visual evaluation; attention-output timing alone
cannot approve it.

## Opportunity Ranking

The recoverable ranges below overlap and must not be added directly.

| Priority | Target | Measured cost | Plausible saving | Risk | Portability |
| ---: | --- | ---: | ---: | --- | --- |
| Complete | Fuse modulation and residual gates | `39.1 ms/block` projected scope | `3.6-3.9%` measured block gain | Low | Validated on SM121 |
| 1 | Fuse NVFP4 scale, pack, GEMM, and epilogue | `151.1 ms/block` projections | `30-75 ms/block` | Medium | Architecture-specific |
| 2 | Retain Vortex scale discovery | up to `21 ms/block` versus Comfy scale path | `3-9%` block gain already measured | Low | SM100/120/121, retune launch |
| 4 | Sage-compatible zero-copy dense attention | `259.9 ms/block` attention plus `20.3 ms` layout | `70-140 ms/block` | High | Separate SM100/120/121 policy |
| 5 | Fuse SwiGLU with FC1/FC2 boundaries | `25.8 ms/block` plus projection traffic | `5-15 ms/block` | Medium | SM100/120/121 |
| 6 | CUDA graphs or persistent denoiser execution | Not isolated yet | likely `1-5%` sampling | Medium | Shape-bucket-specific |
| 7 | Quality-gated sparse scheduling | up to `140 ms/block` attention bound | Unknown accepted saving | Very high | Backend-specific |

Native Sage NHD was tested and rejected as a standalone optimization. It was
numerically exact but did not materially improve median path time and had worse
tail latency. Eliminating layout cost requires fusion with QKV production,
RMSNorm/RoPE, or the attention kernel rather than changing Sage's layout flag.

## Implementation Phases

### Phase 1: Exact H3 Elementwise Fusion

Status: complete on GB10/SM121. Spark deployments enable
`H3_FUSED_ELEMENTWISE=1`; other deployments retain the eager default until
validated on their architecture.

Implement an opt-in direct-runtime path for:

1. RMSNorm output plus segmented AdaLN scale/shift modulation.
2. Attention residual gate/add.
3. MLP segmented scale/shift modulation.
4. MLP residual gate/add.
5. A request-layout segment-index cache reused by all 50 blocks and sampling
   steps.

Preserve explicit BF16 rounding boundaries. Keep the eager implementation as a
fallback until all gates pass.

The achieved `3.0%` sampling gain is materially below the original theoretical
`20-35 ms/block` opportunity. Before extending this fusion, profile:

1. Which modulation, gate, or add launches remain outside the fused path.
2. Whether tensor conversions or intermediate allocations remain.
3. Whether the fused kernels are launch-bound or bandwidth-bound.
4. Whether generated kernels perform unnecessary loads or stores.
5. Which remaining elementwise operations can move into exact NVFP4 GEMM
   epilogues.

The GB10 Nsight follow-up is complete. One warmed block has 53 kernel launches.
The fused path contains exactly two modulation and two residual gate/add
launches; no modulation, gate, or add launch remains outside it. Those four
kernels still consume about `20.48 ms`, showing that the residual is tensor
traffic rather than launch overhead. The four NVFP4 projections account for 32
launches across scale reduction/finalization, scalar conversions, block-scale
initialization, activation packing, and GEMM. Their current intermediate tensor
sizes sum to approximately `7.73 GB` per block when counted at each projection
boundary. See
`benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json`.

Success criteria:

- Bit-exact outputs at blocks 0, 24, and 49.
- Bit-exact one-step full-denoiser video and audio outputs.
- No change to a complete 12-step latent checksum.
- At least `5%` median block improvement or `3%` warmed sampling improvement.
- No peak-memory regression.

### Phase 2: Architecture Baselines

Capture the same component report on each architecture before writing a shared
kernel abstraction:

| Architecture | Required baseline |
| --- | --- |
| SM121 GB10 | Sage2 2.2, current Vortex scale path, 38K and 74K tokens |
| SM120 RTX PRO 6000 | SDPA, packaged Sage2 2.2, 38K and 74K tokens |
| SM100 B200 | SDPA, forced cuDNN SDPA, Sage2 if supported, 38K and 74K tokens |

Record actual kernel names, clocks, power, SM utilization, achieved bandwidth,
and achieved tensor throughput. Do not infer one architecture's policy from
another's result.

### Phase 3: NVFP4 Fused Projection Prototype

Status: active on GB10/SM121; isolated FC2 library scheduling is complete and
awaits production integration.

The first profile also exposed and fixed a native activation-packer layout bug:
the previous width-specific block-scale swizzle failed at the attention output's
7168-feature width. The replacement general 128-row by 4-scale-column mapping is
packed-bit exact at H3 widths 5376, 7168, and 14336, and all four linear outputs
now match Comfy exactly. Standalone native packing is not consistently faster,
so it remains prototype infrastructure rather than a deployed backend.

Component 2 execution order, revised from the measured Nsight profile:

1. Make QKV projection land directly in the attention backend's required
   layout, with Q/K normalization and RoPE operating there.
2. Make attention output land directly in the token-major layout consumed by
   the output projection.
3. Eliminate complete NVFP4 packed-activation and block-scale materialization
   through a CUTLASS/CuTe producer-consumer or persistent design.
4. Integrate reference-exact residual gates into output-projection and FC2
   epilogues.
5. Optimize the standalone packer only after the boundary-removing paths exist.
6. Recapture the complete block profile after every accepted boundary removal.

The first two layout items are complete for single-GPU Sage2 on GB10. Sage2
accepts the projection-strided NHD Q/K/V views directly, including in-place Q/K
normalization and RoPE, and emits contiguous NHD output that reshapes to
token-major rows without a copy. This removes three QKV copies and the attention
output copy without replacing the projection GEMM. Blocks 0, 24, and 49 and the
two-step and canonical 12-step trajectories are bit-exact. Canonical sampling
improves from the Component 1 baseline of `301.05 s` to `290.23 s` (`3.6%`),
with cumulative improvement of `6.4%` from the original `310.11 s` baseline.
The accepted Nsight recapture falls from 53 to 49 launches and contains none of
the four large BF16 layout-copy kernels.
Evidence is in
`benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json`.

The target NVFP4 dataflow is not another wrapper around
`scaled_mm_nvfp4`. Tensor-core GEMMs require exact packed-data and block-scale
layouts, so avoiding durable HBM intermediates likely requires an owned
CUTLASS/CuTe collective with tile-level producer-consumer execution.

Interface investigation confirms this boundary. Comfy Kitchen 0.2.31 allocates
complete QDATA, block-scale, and BF16 output tensors before invoking a cuBLAS
blockwise FP4 GEMM with fixed pointers. It cannot consume incremental tiles.
CUTLASS DSL 4.6.2's stock SM121 E2M1/E4M3 persistent kernel passes on GB10, but
its public mainloop also TMA-loads prebuilt A and SFA. The owned implementation
will follow `NVFP4_STREAMING_DESIGN.md`: preserve the mandatory global-scale
reduction, replace the A/SFA producer in one fixed tile, then measure multi-N
reuse before attempting full H3 shapes.

P0 is complete for the selected streamed roles. The owned CuTe bridge and
alpha-before-BF16 epilogue are bit-exact
for QKV, attention output, and FC1, proving E2M1, E4M3, and global-scale
interoperability. FC2 remains non-exact because its reference uses a different
reduction policy (`max_abs=16` after the correct epilogue). Resolve that policy
or retain an explicit FC2 cuBLAS fallback before the streaming A producer begins.

FC2 fallback is selected. The streamed backend will reject FC2 dispatch and use
the existing Comfy/cuBLAS implementation there; QKV, attention output, and FC1
advance to the P1 software A/SFA producer.

The fixed 128-row P1 producer-consumer checkpoint is complete. The stock
kernel's 32-thread DMA warp produces four rows per lane directly into staged
E2M1 A and E4M3 SFA shared memory; only B and SFB retain TMA loads. Every 128-K
tile has zero packed-data and scale-byte differences for real QKV,
attention-output, and FC1 activations, and all three complete GEMM outputs are
bit-exact after the fused alpha epilogue. The streamed kernel receives BF16 A
plus its tensor scale and has no global activation QDATA or SFA input.

The final arithmetic contract includes Comfy's `--use_fast_math` behavior:
encode scale must use PTX `rcp.approx.ftz.f32`. Correctly rounded division changes
FP4 midpoint decisions for real H3 values. CUTLASS DSL 4.6.2 also requires a
static contiguous destination view for vector FP4 stores. Evidence is in
`benchmarks/gb10-cute-p1-stream-a-summary.json`.

The P1 timing gate rejects direct per-output-N-CTA streaming. At 128 rows, the
exact streamed kernel is `9.14x` slower than the complete reference projection
for QKV, `12.98x` slower for attention output, and `12.44x` slower for FC1.
Producer overhead is linear at approximately `0.56-0.60 us` per `(N,K)` tile.
Break-even against the removed quantizer would require theoretical N reuse of
78, 28, and 102 tiles respectively, which cannot be implemented by retaining
duplicate accumulator state under the current register budget.

The next P2 prototype is a bounded global packed-tile ring or persistent work
queue. It must produce each `(M,K)` A tile once, expose it to all N consumers,
and recycle the slot, trading bounded global packed traffic for elimination of
the measured repeated BF16 conversion. Do not add canonical-shape runtime
dispatch for the rejected schedule. FC2 remains an explicit Comfy/cuBLAS
fallback. Evidence is in
`benchmarks/gb10-cute-p1-stream-a-timing-summary.json`.

The bounded-ring P2 checkpoint now uses caller-owned native QDATA/SFA buffers and
an allocation-free `_into` producer. A capacity sweep with one full-activation
scale selected 2048 rows. The ring occupies 6.19 MB for QKV/FC1 and 8.26 MB for
attention output. Measured chunk latency improves QKV by `10.7%`, attention
output by `14.4%`, and FC1 by `11.3%`; modeled canonical totals improve by
`10.1-12.5%`. Every packed byte, scale byte, and BF16 output matches the
reference. The 4096-row point regresses, confirming that larger buffers are not
monotonically better. Evidence is in
`benchmarks/gb10-cute-p2-ring-capacity-summary.json`.

Complete 37,810-row projection parity now passes for QKV, attention output, and
FC1 in blocks 0, 24, and 49. Every one of the 19 chunks, including the final
946-row tail, is bit-exact. QKV has a stable approximately 32 ms ring time and a
positive result across all three blocks. Attention-output timing is mixed. FC1
timing is not usable from the duplicate-model harness because multi-gigabyte
output pressure caused large variance and one container stop.

The opt-in QKV runtime gate is complete and rejects the current bounded-ring
schedule. An alternating baseline/ring benchmark inside one loaded block removes
the clock, allocator, input, and model-copy bias present in separate-process
profiles. Blocks 0, 24, and 49 remain bit-exact, but the 2048-row ring regresses
median block time by `0.81%`, `0.52%`, and `0.52%`, respectively. On block 24,
3072 rows regress by `1.00%`, 4096 by `0.24%`, 8192 by `4.80%`, and a 37888-row
full workspace by `14.49%`. The isolated projection win does not survive the
chunk launch and scheduler overhead. Keep the implementation opt-in and disabled;
do not spend trajectory-validation compute until a launch-fused work queue or a
different persistent scheduler passes this same block gate. Attention output and
FC1 remain experimental, and FC2 remains the explicit Comfy/cuBLAS fallback.
Evidence is in `benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`.

The post-optimization profile identifies Sage2's mainloop as the largest cost.
Canonical sampling is `288.93 s`; block 24 is
`468.22 ms`, and its production NHD Sage2 call is `258.47 ms`. The existing
SM89 mainloop alone takes `238.81 ms`, while Q/K and V quantization together take
`18.34 ms`. A direct Hopper WGMMA retarget was rejected by SM121 ptxas, and
CUTLASS SM120/121 UMMA does not support the INT8 QK operation needed for exact
Sage2 parity. Do not substitute Sage3's different FP4 algorithm under the exact
optimization contract.

The next implementable exact target was AdaLN modulation into NVFP4 production.
The MSA and MLP modulation passes cost `10.47 ms` and `10.60 ms` and materialize
their values in the RMSNorm buffers solely for QKV and FC1. The deployed fusion
recomputes the exact BF16 values inside Vortex scale/pack and retains Comfy
GEMMs. Complete producer bytes match in blocks 0, 24, and 49. Alternating block
medians improve by `0.28-0.79%`; warmed two-step and 12-step runs improve by
`0.52%` and `0.56%`, with bit-identical outputs. Spark deployments enable
`H3_NVFP4_MODULATE_FUSION=1`. Evidence is in
`benchmarks/gb10-nvfp4-modulate-fusion-summary.json`.

SwiGLU-to-FC2 producer fusion is complete. Recomputing the exact BF16 SiLU and
multiply boundaries inside Vortex scale/pack removes the complete intermediate
activation while retaining the reference Comfy FC2 GEMM. The real producer is
byte-exact and takes `21.93 ms`. Alternating blocks 0, 24, and 49 improve by
`2.14-2.26%`; the warmed canonical 12-step trajectory improves by `4.07%` with
identical video and audio tensors. Enable `H3_NVFP4_SWIGLU_FUSION=1` for Spark
single-GPU deployments. Evidence is in
`benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`.

The mandatory post-fusion profile supersedes the earlier approximately 515 ms
block distribution. With all production flags enabled, fresh block-24 timing is
`458.78 ms` median without Nsight and `465.78 ms` across the Nsight GPU span.
The 41 kernels have only `0.084 ms` total inter-kernel idle time, so CPU launch
gaps are not a block-level bottleneck. A complete warmed sampling step takes
`23.708 s`, contains 2,694 kernels, and has `12.995 ms` total inter-kernel idle
time.

The fresh block kernel-time distribution is:

| Component | Time | Share |
| --- | ---: | ---: |
| Sage2 preparation and mainloop | `268.22 ms` | `57.60%` |
| Four NVFP4 GEMMs | `125.11 ms` | `26.87%` |
| NVFP4 scale and packing | `41.53 ms` | `8.92%` |
| Norm and RoPE | `19.86 ms` | `4.27%` |
| Two residual gate/add kernels | `10.94 ms` | `2.35%` |

The one-step distribution independently matches these shares within 0.5
percentage points. The standalone modulation and SwiGLU kernels are gone.
Their remaining producer work is `13.04 ms` for modulated QKV/FC1 packing and
`23.04 ms` for SwiGLU-to-FC2 packing. The block still records 8,009,578,496
positive self-allocated bytes, but there are no large activation-layout copy
kernels or explicit CUDA memcpy operations. Remaining conversions are small
AdaLN table and scalar operations.

GB10 exposes no direct physical DRAM-byte counter. Nsight Compute measures
`320.999 GB` of L2-request traffic for the block. With cache control disabled,
L2 read/write miss sectors imply a `42.117 GB` off-chip request proxy: NVFP4
GEMMs account for `56.25%`, packing `18.25%`, Sage2 `15.09%`, norm/RoPE
`6.54%`, and gates `3.87%`. This separates the time bottleneck, Sage2, from the
off-chip traffic bottleneck, the NVFP4 GEMMs. Do not select a new kernel from
the old profile. See
`benchmarks/gb10-fully-fused-fresh-nsight-summary.json`.

The FC2 follow-up resolves that projection's `16.1x` off-chip amplification.
The production heuristic launches an undocumented-sentinel `_stream_k` kernel;
the retained documented configuration is cuBLASLt algorithm 70, tile 20,
stages 37, public split-K 1, reduction scheme 0, and zero workspace. Both
schedules request the same `25.664 GB` of operands, but split-K 1 raises L2 hit
rate from `53.32%` to `91.10%`, removes `9.853 GB` of L2 read misses, and raises
tensor-pipe activity from `25.01%` to `82.42%`. There is no material global
partial-accumulator or output-reduction traffic; the baseline instead spends
heavily in Stream-K synchronization polling and loses traversal locality.

The candidate is byte-exact and improves FC2 p50 from `53.618 ms` to
`15.636 ms`. Complete blocks 0, 24, and 49 improve by `8.16-8.88%`; two-step
and canonical 12-step trajectories improve by `7.50%` and `8.21%`, with exact
video and audio latents. The custom persistent-kernel branch is therefore
closed. Guarded production integration is complete behind
`H3_NVFP4_FC2_LT_SPLITK1=1`. Its direct 20-round block gate improves blocks
0/24/49 by `9.36%`, `8.16%`, and `7.86%`; the exact canonical 12-step gate
improves by `8.19%`
with 600 successful dispatches and zero fallback. Shape testing found two-BF16
differences at several nearby row counts, so only `M=37,810` is eligible and
all variable shapes retain Comfy. The hot service remains stopped until the
performance program is complete. See `research/fc2_nvfp4_scheduling/RESULTS.md` and
`benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json`.

The follow-up real block-24 Sage2 decomposition now selects the next exact
kernel experiment. Manual preparation plus the existing prequantized mainloop
is byte-exact against public SageAttention 2.2.0. Uninstrumented median phase
times are `2.37 ms` K smoothing, `7.63 ms` combined Q/K quantization, `10.61 ms`
combined V transpose/quantization, and `237.09 ms` for the fused mainloop.

The full-counter mainloop capture is profiler-perturbed to `258.92 ms`, so its
duration is not used as the baseline. Its ratios establish the bottleneck:

| Mainloop metric | Result |
| --- | ---: |
| Registers per thread | `255` |
| Dynamic shared memory per CTA | `32 KiB` |
| Achieved occupancy | `16.83%` / `8.08` warps per SM |
| Scheduler cycles with no eligible warp | `63.53%` |
| INT8 QK tensor-pipe utilization | `37.77%` of elapsed cycles |
| FP8 PV tensor-pipe utilization | `37.77%` of elapsed cycles |
| Combined tensor-pipe utilization | `75.54%` |
| Memory throughput | `31.61%` |
| L2 hit rate | `98.84%` |
| L2 request traffic | `161.50 GB` |
| Off-chip request proxy | `1.85 GB` |
| Excess shared-memory wavefronts | `7.68%` |

This explains why high SM activity did not imply a saturated tensor pipeline.
QK and PV alternate on separate INT and FP tensor sub-pipelines, while online
softmax, scaling, conversion, and synchronization occupy scalar pipelines.
Only `0.46` warps per scheduler are eligible on average. Fixed-latency
dependencies consume `2.01` of the `5.48` cycles between issued instructions,
and math-pipe throttle consumes another `1.24`; memory scoreboards are much
smaller. The 255-register footprint limits the kernel to two CTAs per SM, so it
cannot hide these dependencies. Q and KV tail sweeps each add less than 1 ms and
exclude tail scheduling as the primary target.

The source-identical classic SM89 P0 is complete and rejected. The exact
three-CTA cliff is 168 registers/thread, not approximately 170: 170 still has
two CTAs and `16.67%` theoretical occupancy, while 168 reaches `25%` at the cost
of `4.95 billion` local spill requests and `78.79%` no-eligible cycles. Caps from
240 through 170 never change residency and progressively worsen scheduler
eligibility.

Byte-exact source variants also missed the 3% gate. Narrowed scopes reduced
static spills from 44/44 to 12/12 bytes and dynamic spill requests by `73%`, but
improved interleaved latency by only `0.06%`. In-place score storage was
`+0.01%`; early K prefetch was `-0.30%`; independent softmax-chain interleaving
was `-0.02%`. The persistent 128-register FP32 output fragment plus exact QK or
instantaneous PV fragments prevents a source-only lifetime cleanup from
reaching the next residency tier.

The shared-memory follow-up is also closed. Source attribution maps all
`626,970,624` excessive wavefronts to four repeated V-staging `LDGSTS.128`
instructions. A 128-byte padded V stride raised dynamic shared memory from 32
to 40 KiB but left the excess count exactly unchanged and changed latency by
`-0.04%`. No complete-block benchmark or deployment was run because no mainloop
variant crossed 3%. Evidence is in
`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`, the per-variant
latency JSON files, and the P0 NCU reports.

Materializing the optional BF16 LoRA input while these fused producers pack
NVFP4 was also tested against the official Turbo-4 adapter. The isolated
1344x768, 124-frame trajectory was bit-exact but regressed from `131.11 s` to
`135.14 s` (`3.08%`). Reject this implementation: its extra BF16 writes cost
more than the removed standalone producer. Active LoRA must retain the existing
materialized fallback unless a future design consumes the producer values
without global-memory materialization. Evidence is in
`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`.

An exact residual-gate epilogue must preserve this numerical program:

1. Complete NVFP4 GEMM accumulation.
2. Convert to the reference BF16 GEMM output.
3. Apply the reference-equivalent gate multiplication and its BF16 boundary.
4. Add the BF16 residual using the reference operation order.
5. Round the final output to BF16.

Bit-exact GEMM output alone is insufficient. Validate randomized epilogues,
representative real blocks, all 50 blocks in one denoiser step, a two-step
trajectory, and the canonical 12-step trajectory before enabling an epilogue.

The direct-layout target extends naturally to distributed execution: QKV tiles
must be able to land in rank-destined Ulysses buffers, and attention output must
land in the output projection's local token-major layout without an intervening
transpose or contiguous copy.

Prototype behind `Nvfp4Linear` so model code remains unchanged. Compare:

1. Comfy Kitchen baseline.
2. Vortex scale plus Comfy pack/GEMM.
3. Fused scale and activation pack.
4. Fused scale, pack, GEMM, bias, and output epilogue.
5. Transformer Engine or CUTLASS reference where supported.

Gate every variant on packed activation identity, linear output error, block
output error, full denoiser-step error, and final media quality.

### Phase 4: Dense Sage-Compatible Attention

The first owned attention kernel should remain dense and target Sage2 behavior:

1. Consume strided BSHD views directly from fused QKV output.
2. Preserve Sage2 per-warp INT8 Q/K quantization and FP8 V policy.
3. Preserve its accumulation and K-smoothing behavior.
4. Fuse Q/K/V preparation with the attention launch where profitable.
5. Emit token-major output suitable for the NVFP4 output projection without an
   intermediate transpose/contiguous allocation.
6. Tune independently for SM100, SM120, and SM121.

The initial goal is `1.5x` attention speedup with bit-exact Sage2 output where
the operation order permits it; any nonzero tolerance must be declared and
quality-gated before implementation. A `2x` attention target is stretch.

### Phase 5: Sparse Hybrid Research

Only after the dense backend passes:

1. Keep early quality-sensitive sampling steps dense.
2. Keep first/last sensitive transformer blocks dense.
3. Force text, reference, and audio conditioning KV blocks exact.
4. Sweep tau by timestep using complete generated media, not random attention
   tensors alone.
5. Reject any policy that degrades prompt adherence, identity, lip sync, audio
   onset, or temporal stability.

## Performance Targets

### GB10 / SM121

The current two-step profile implies roughly `305 s` for a warmed 12-step
1344x768/124-frame sampling run.

| Target | Estimated sampling | Speedup |
| --- | ---: | ---: |
| Conservative | `215-240 s` | `1.3-1.4x` |
| Engineering | `170-195 s` | `1.6-1.8x` |
| Stretch | `130-145 s` | `2.1-2.3x` |
| Aggressive ceiling | about `120 s` | about `2.5x` |

Use `1.5x` as the commitment target, `1.8x` as the engineering target, and
`2.3x` as stretch. Do not plan around a `5x` end-to-end gain.

### RTX PRO 6000 / SM120

These projections have lower confidence because the measured RTX baseline uses
PyTorch SDPA rather than packaged Sage2.

| Workload | Current SDPA | Engineering target | Stretch target |
| --- | ---: | ---: | ---: |
| 1344x768, 124 frames | `97.94 s` | `45-60 s` | `35-45 s` |
| 1344x768, 243 frames | `311.02 s` | `120-170 s` | `90-120 s` |

Packaging and profiling Sage2 2.2 on SM120 is the first RTX action. Expected
overall improvement is `1.6-2.2x` at 124 frames and `1.8-2.6x` at 243 frames;
stretch ranges are `2.2-2.8x` and `2.6-3.5x` respectively. These are planning
ranges, not commitments.

### B200 / SM100

Measured B200 SDPA sampling is `79.72 s` at 124 frames and `257.86 s` at 243
frames, only `1.23x` and `1.21x` faster than RTX PRO 6000 despite much larger
theoretical FP4 and memory-bandwidth capability. No optimized B200 target is
assigned until component profiling confirms actual SDPA dispatch, NVFP4 kernel
selection, clocks, power, utilization, and achieved throughput.

The current B200 result measures this runtime path, not B200's hardware ceiling.

## Evidence

- `benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json`
- `benchmarks/gb10-37810-token-optimization-profile-seed440420.json`
- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
- `tools/profile_h3_block.py`
- `tools/profile_attention_components.py`
- `tools/profile_hybrid_attention.py`
- `tools/profile_sampling_stages.py`

Component timings use synchronized probes for attribution. End-to-end claims
must use separate uninstrumented runs. Resident services remained loaded during
GB10 profiling, so medians are preferred over means and final claims require an
isolated rerun.