h3-blackwell-runtime/research/fc2_nvfp4_scheduling/RESULTS.md

FC2 NVFP4 Scheduling Results

Status: research retained; validated as a canonical-shape production-integration candidate, but not wired into production dispatch.

Decision

The canonical GB10 FC2 slowdown is caused by the cuBLASLt heuristic selecting a _stream_k kernel with poor traversal locality and heavy synchronization polling. A documented public split-K-1 configuration is byte-exact and removes the regression. Because this library candidate passed the complete validation ladder, the custom persistent-kernel branch is closed.

The retained candidate is:

Field	Value
Algorithm ID	70
Tile ID	20
Stages ID	37
Split-K	1
Reduction scheme	0
Custom option / CTA swizzle	0 / 0
Required workspace	0 bytes
Supplied workspace	64 MiB sweep; 0 bytes block/trajectory gates

The baseline heuristic's negative split-K value is an undocumented library sentinel. CUDA 13 has no documented public attribute that directly selects Stream-K, so the sentinel itself is not interpreted as a public Stream-K control. The launched baseline symbol does end in _stream_k.

Canonical Operation

• Logical GEMM: M=37,810, N=5,376, K=14,336.
• Descriptor GEMM after accepted producer padding: M=37,824.
• Inputs: packed NVFP4 E2M1 activation and weight with vector-16 UE4M3 scales.
• Accumulation/scalars: FP32; output: BF16.
• Accepted activation producer: vortex_native_quantize_swiglu_nvfp4.
• Image: sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83.

Timing

The 20-round isolated sweep alternated AB/BA order in one loaded model.

Measurement	Baseline	Split-K-1	Result
FC2 p50	53.618 ms	15.636 ms	3.43x
Dense throughput p50	108.70 TFLOP/s	372.74 TFLOP/s	3.43x
Producer + FC2 p50	76.737 ms	38.823 ms	49.41% faster

The negative-sentinel candidate reproduced the production-equivalent baseline within noise. Explicit split-K factors 2, 4, 8, and 16 were rejected by cublasLtMatmulAlgoCheck for the tested configuration space.

Complete 20-round block gates replaced only block.mlp.fc2.forward_swiglu:

Block	Baseline p50	Candidate p50	Improvement
0	457.891 ms	420.536 ms	8.16%
24	459.928 ms	420.243 ms	8.63%
49	458.054 ms	417.364 ms	8.88%

All block outputs were byte-exact against both the paired baseline and retained traversal.

Trajectory	Baseline	Candidate	Improvement	Correctness
Two-step	45.962 s	42.516 s	7.50%	video/audio BF16 exact
Canonical 12-step	278.201 s	255.371 s	8.21%	video/audio BF16 exact

Trajectory values are single paired runs in baseline-then-candidate order, not repeated medians. Their approximately 38 ms per-FC2 savings agree with the alternating isolated and complete-block gates, but the precise end-to-end percentages retain run-order uncertainty.

The 12-step saving of 22.829 s matches approximately 600 FC2 invocations multiplied by the isolated roughly 38 ms saving.

Traffic Attribution

The one-pass distinct-data footprint is:

Category	Bytes
Activation QDATA + scales	305,012,736
Weight QDATA + scales	43,352,064
Padded BF16 output	406,683,648
Total	755,048,448

Both schedules issue the same nominal operand requests:

Requested category	Bytes	Geometric reuse
Weight QDATA + scales	12,832,210,944	296x
Activation QDATA + scales	12,832,210,944	about 42.1x
Total L2 operand reads	25,664,424,960	unchanged

The schedule does not eliminate CTA-level rereading. It changes whether those rereads remain cache-resident:

NCU metric	Baseline _stream_k	Split-K-1
Main-kernel duration	55.057 ms	17.031 ms
Combined L2 hit rate	53.32%	91.10%
L2 read-miss bytes	11.762 GB	1.909 GB
L2 miss-byte proxy	12.169 GB	2.316 GB
Sysmem traffic proxy	12.176 GB	2.327 GB
Off-chip amplification	16.12x	3.07x
SM throughput	25.11%	81.79%
Tensor-pipe active share	25.01%	82.42%
Eligible warps/scheduler	0.073	0.220
Issue rate	0.057	0.168

GB10 does not expose the usual discrete-GPU DRAM byte counters in these captures. L2 misses and the reported sysmem fill/write sectors are used as the off-chip proxy.

The 9.853 GB reduction in L2 read misses is avoided operand rereading. NCU aggregates the two TMA input descriptors, so it cannot defensibly assign exact miss-byte totals separately to weights and activations. The request geometry does prove that each input family accounts for 12.832 GB of requested reads.

There is no material global partial-accumulator or output-reduction traffic:

• Candidate workspace is zero.
• Neither kernel issues global atomics or global reduction operations.
• No auxiliary reduction kernel is launched.
• Candidate writes exactly one padded BF16 output; baseline writes only 73,728 bytes more.
• Baseline-only local stack traffic is about 22.35 MB at L1 and almost none reaches sysmem.

The baseline's dominant scheduler stall is sleeping. Source-correlated samples land in NANOSLEEP.SYNCS polling around synchronization phase checks. Launch resources are otherwise the same: 12,432 CTAs, 384 threads/CTA, 168 registers per thread, 89,088 bytes allocated shared memory, and 25% theoretical occupancy. The gain therefore comes from traversal locality and reduced synchronization waiting, not occupancy or a reduction workspace.

Evidence

• benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json
• benchmarks/gb10-fc2-nvfp4-block-gate-20260825.json
• benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json
• benchmarks/gb10-fc2-nvfp4-trajectory-12step-20260825.json
• benchmarks/gb10-fc2-nvfp4-baseline-profile-20260825.json
• benchmarks/gb10-fc2-nvfp4-splitk1-profile-20260825.json
• benchmarks/gb10-fc2-nvfp4-baseline-20260825.csv
• benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv
• benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep
• benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep

Production source and configuration were not changed by this study. The candidate is validated only for the canonical descriptor shape and current GB10/CUDA-library combination. Promotion still requires shape-specific AlgoCheck with a safe fallback, integration behind the existing Nvfp4Linear boundary, broader shape tests, and deployment validation.