h3-blackwell-runtime/research/fc2_nvfp4_scheduling/RESULTS.md

# FC2 NVFP4 Scheduling Results

Status: production accepted for the canonical descriptor shape behind
`H3_NVFP4_FC2_LT_SPLITK1`; all other shapes retain the existing fallback.

## Decision

The canonical GB10 FC2 slowdown is caused by the cuBLASLt heuristic selecting a
`_stream_k` kernel with poor traversal locality and heavy synchronization
polling. A documented public split-K-1 configuration is byte-exact and removes
the regression. Because this library candidate passed the complete validation
ladder, the custom persistent-kernel branch is closed.

The retained candidate is:

| Field | Value |
| --- | ---: |
| Algorithm ID | `70` |
| Tile ID | `20` |
| Stages ID | `37` |
| Split-K | `1` |
| Reduction scheme | `0` |
| Custom option / CTA swizzle | `0 / 0` |
| Required workspace | `0 bytes` |
| Supplied workspace | `64 MiB` sweep; `0 bytes` block/trajectory gates |

The baseline heuristic's negative split-K value is an undocumented library
sentinel. CUDA 13 has no documented public attribute that directly selects
Stream-K, so the sentinel itself is not interpreted as a public Stream-K
control. The launched baseline symbol does end in `_stream_k`.

## Canonical Operation

- Logical GEMM: `M=37,810, N=5,376, K=14,336`.
- Descriptor GEMM after accepted producer padding: `M=37,824`.
- Inputs: packed NVFP4 E2M1 activation and weight with vector-16 UE4M3 scales.
- Accumulation/scalars: FP32; output: BF16.
- Accepted activation producer: `vortex_native_quantize_swiglu_nvfp4`.
- Image: `sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`.

## Timing

The 20-round isolated sweep alternated AB/BA order in one loaded model.

| Measurement | Baseline | Split-K-1 | Result |
| --- | ---: | ---: | ---: |
| FC2 p50 | `53.618 ms` | `15.636 ms` | `3.43x` |
| Dense throughput p50 | `108.70 TFLOP/s` | `372.74 TFLOP/s` | `3.43x` |
| Producer + FC2 p50 | `76.737 ms` | `38.823 ms` | `49.41%` faster |

The negative-sentinel candidate reproduced the production-equivalent baseline
within noise. Explicit split-K factors `2`, `4`, `8`, and `16` were rejected by
`cublasLtMatmulAlgoCheck` for the tested configuration space.

Complete 20-round block gates replaced only
`block.mlp.fc2.forward_swiglu`:

| Block | Baseline p50 | Candidate p50 | Improvement |
| ---: | ---: | ---: | ---: |
| 0 | `457.891 ms` | `420.536 ms` | `8.16%` |
| 24 | `459.928 ms` | `420.243 ms` | `8.63%` |
| 49 | `458.054 ms` | `417.364 ms` | `8.88%` |

All block outputs were byte-exact against both the paired baseline and retained
traversal.

| Trajectory | Baseline | Candidate | Improvement | Correctness |
| --- | ---: | ---: | ---: | --- |
| Two-step | `45.962 s` | `42.516 s` | `7.50%` | video/audio BF16 exact |
| Canonical 12-step | `278.201 s` | `255.371 s` | `8.21%` | video/audio BF16 exact |

Trajectory values are single paired runs in baseline-then-candidate
order, not repeated medians or AB/BA trials. Their approximately `38 ms` per-FC2 savings agree
with the alternating isolated and complete-block gates, but the precise
end-to-end percentages retain run-order uncertainty.

The 12-step saving of `22.829 s` matches approximately 600 FC2 invocations
multiplied by the isolated roughly `38 ms` saving.

## Production Integration

The packaged path lives in `src/h3_blackwell_runtime/fc2_lt.py` and is called
from `Nvfp4Linear.forward_swiglu` only after the accepted native producer. It
requires the FC2 role, unsharded `K=14,336/N=5,376`, BF16 output, no bias or
active LoRA, GB10/SM121, CUDA 13.0, cuBLAS 13.1 headers, cuBLASLt 13.0 runtime,
logical `M=37,810`, and packed
`M=37,824`. A shape-specific `cublasLtMatmulAlgoCheck` must pass with zero
workspace. Load, check, or execution failure falls back through the already
produced NVFP4 bytes without quantizing twice; strict mode is available through
`H3_NVFP4_FC2_LT_STRICT=1`. Production startup prepares the extension and
caches its measured runtime compatibility result before inference.

The 20-round production-method gate exercised `Nvfp4Linear.forward_swiglu`
directly and recorded 23 successful dispatches with zero fallbacks per block:

| Block | Baseline p50 | Production p50 | Improvement |
| ---: | ---: | ---: | ---: |
| 0 | `464.153 ms` | `420.714 ms` | `9.36%` |
| 24 | `460.184 ms` | `422.656 ms` | `8.16%` |
| 49 | `456.183 ms` | `420.335 ms` | `7.86%` |

Production-method trajectories remained exact:

| Trajectory | Baseline | Production | Improvement | Dispatch |
| --- | ---: | ---: | ---: | --- |
| Two-step | `47.301 s` | `43.728 s` | `7.55%` | `100/100`, zero fallback |
| Canonical 12-step | `286.431 s` | `262.979 s` | `8.19%` | `600/600`, zero fallback |

The row-shape gate prevents unsafe generalization. Algorithm 70 was exact for
the tested `M=37,774`, `37,809`, `37,810`, `37,838`, `37,966`, and `38,222`
inputs, but differed in two BF16 elements at each of `M=37,711`, `37,725`, and
`37,742` despite passing `AlgoCheck`. Production therefore intentionally keeps
the canonical-only guard rather than inferring exactness from nearby shapes.
The retained shape artifact records compile-time CUDA/cuBLAS versions but
predates runtime-version fields in `build_info()`. The final extension separately
measured CUDA runtime `13000` and cuBLASLt runtime `130000`; the shape artifact
alone is not runtime-ABI evidence.

## Traffic Attribution

The one-pass distinct-data footprint is:

| Category | Bytes |
| --- | ---: |
| Activation QDATA + scales | `305,012,736` |
| Weight QDATA + scales | `43,352,064` |
| Padded BF16 output | `406,683,648` |
| Total | `755,048,448` |

Both schedules issue the same nominal operand requests:

| Requested category | Bytes | Geometric reuse |
| --- | ---: | ---: |
| Weight QDATA + scales | `12,832,210,944` | `296x` |
| Activation QDATA + scales | `12,832,210,944` | about `42.1x` |
| Total L2 operand reads | `25,664,424,960` | unchanged |

The schedule does not eliminate CTA-level rereading. It changes whether those
rereads remain cache-resident:

| NCU metric | Baseline `_stream_k` | Split-K-1 |
| --- | ---: | ---: |
| Main-kernel duration | `55.057 ms` | `17.031 ms` |
| Combined L2 hit rate | `53.32%` | `91.10%` |
| L2 read-miss bytes | `11.762 GB` | `1.909 GB` |
| L2 miss-byte proxy | `12.169 GB` | `2.316 GB` |
| Sysmem traffic proxy | `12.176 GB` | `2.327 GB` |
| Off-chip amplification | `16.12x` | `3.07x` |
| SM throughput | `25.11%` | `81.79%` |
| Tensor-pipe active share | `25.01%` | `82.42%` |
| Eligible warps/scheduler | `0.073` | `0.220` |
| Issue rate | `0.057` | `0.168` |

GB10 does not expose the usual discrete-GPU DRAM byte counters in these
captures. L2 misses and the reported sysmem fill/write sectors are used as the
off-chip proxy.

The `9.853 GB` reduction in L2 read misses is avoided operand rereading. NCU
aggregates the two TMA input descriptors, so it cannot defensibly assign exact
miss-byte totals separately to weights and activations. The request geometry
does prove that each input family accounts for `12.832 GB` of requested reads.

There is no material global partial-accumulator or output-reduction traffic:

- Candidate workspace is zero.
- Neither kernel issues global atomics or global reduction operations.
- No auxiliary reduction kernel is launched.
- Candidate writes exactly one padded BF16 output; baseline writes only
  `73,728` bytes more.
- Baseline-only local stack traffic is about `22.35 MB` at L1 and almost none
  reaches sysmem.

The baseline's dominant scheduler stall is sleeping. Source-correlated samples
land in `NANOSLEEP.SYNCS` polling around synchronization phase checks. Launch
resources are otherwise the same: 12,432 CTAs, 384 threads/CTA, 168 registers
per thread, 89,088 bytes allocated shared memory, and 25% theoretical
occupancy. The gain therefore comes from traversal locality and reduced
synchronization waiting, not occupancy or a reduction workspace.

## Evidence

- `benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-block-gate-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-trajectory-12step-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-baseline-profile-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-splitk1-profile-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.csv`
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv`
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep`
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep`
- `benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json`
- `benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json`
- `src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp`

The Spark hot runtime remains stopped while performance work continues.
Resident-service validation is intentionally deferred until the performance
program is finished.