200 lines
8.7 KiB
Markdown
200 lines
8.7 KiB
Markdown
# FC2 NVFP4 Scheduling Results
|
|
|
|
Status: production accepted for the canonical descriptor shape behind
|
|
`H3_NVFP4_FC2_LT_SPLITK1`; all other shapes retain the existing fallback.
|
|
|
|
## Decision
|
|
|
|
The canonical GB10 FC2 slowdown is caused by the cuBLASLt heuristic selecting a
|
|
`_stream_k` kernel with poor traversal locality and heavy synchronization
|
|
polling. A documented public split-K-1 configuration is byte-exact and removes
|
|
the regression. Because this library candidate passed the complete validation
|
|
ladder, the custom persistent-kernel branch is closed.
|
|
|
|
The retained candidate is:
|
|
|
|
| Field | Value |
|
|
| --- | ---: |
|
|
| Algorithm ID | `70` |
|
|
| Tile ID | `20` |
|
|
| Stages ID | `37` |
|
|
| Split-K | `1` |
|
|
| Reduction scheme | `0` |
|
|
| Custom option / CTA swizzle | `0 / 0` |
|
|
| Required workspace | `0 bytes` |
|
|
| Supplied workspace | `64 MiB` sweep; `0 bytes` block/trajectory gates |
|
|
|
|
The baseline heuristic's negative split-K value is an undocumented library
|
|
sentinel. CUDA 13 has no documented public attribute that directly selects
|
|
Stream-K, so the sentinel itself is not interpreted as a public Stream-K
|
|
control. The launched baseline symbol does end in `_stream_k`.
|
|
|
|
## Canonical Operation
|
|
|
|
- Logical GEMM: `M=37,810, N=5,376, K=14,336`.
|
|
- Descriptor GEMM after accepted producer padding: `M=37,824`.
|
|
- Inputs: packed NVFP4 E2M1 activation and weight with vector-16 UE4M3 scales.
|
|
- Accumulation/scalars: FP32; output: BF16.
|
|
- Accepted activation producer: `vortex_native_quantize_swiglu_nvfp4`.
|
|
- Image: `sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`.
|
|
|
|
## Timing
|
|
|
|
The 20-round isolated sweep alternated AB/BA order in one loaded model.
|
|
|
|
| Measurement | Baseline | Split-K-1 | Result |
|
|
| --- | ---: | ---: | ---: |
|
|
| FC2 p50 | `53.618 ms` | `15.636 ms` | `3.43x` |
|
|
| Dense throughput p50 | `108.70 TFLOP/s` | `372.74 TFLOP/s` | `3.43x` |
|
|
| Producer + FC2 p50 | `76.737 ms` | `38.823 ms` | `49.41%` faster |
|
|
|
|
The negative-sentinel candidate reproduced the production-equivalent baseline
|
|
within noise. Explicit split-K factors `2`, `4`, `8`, and `16` were rejected by
|
|
`cublasLtMatmulAlgoCheck` for the tested configuration space.
|
|
|
|
Complete 20-round block gates replaced only
|
|
`block.mlp.fc2.forward_swiglu`:
|
|
|
|
| Block | Baseline p50 | Candidate p50 | Improvement |
|
|
| ---: | ---: | ---: | ---: |
|
|
| 0 | `457.891 ms` | `420.536 ms` | `8.16%` |
|
|
| 24 | `459.928 ms` | `420.243 ms` | `8.63%` |
|
|
| 49 | `458.054 ms` | `417.364 ms` | `8.88%` |
|
|
|
|
All block outputs were byte-exact against both the paired baseline and retained
|
|
traversal.
|
|
|
|
| Trajectory | Baseline | Candidate | Improvement | Correctness |
|
|
| --- | ---: | ---: | ---: | --- |
|
|
| Two-step | `45.962 s` | `42.516 s` | `7.50%` | video/audio BF16 exact |
|
|
| Canonical 12-step | `278.201 s` | `255.371 s` | `8.21%` | video/audio BF16 exact |
|
|
|
|
Trajectory values are single paired runs in baseline-then-candidate
|
|
order, not repeated medians or AB/BA trials. Their approximately `38 ms` per-FC2 savings agree
|
|
with the alternating isolated and complete-block gates, but the precise
|
|
end-to-end percentages retain run-order uncertainty.
|
|
|
|
The 12-step saving of `22.829 s` matches approximately 600 FC2 invocations
|
|
multiplied by the isolated roughly `38 ms` saving.
|
|
|
|
## Production Integration
|
|
|
|
The packaged path lives in `src/h3_blackwell_runtime/fc2_lt.py` and is called
|
|
from `Nvfp4Linear.forward_swiglu` only after the accepted native producer. It
|
|
requires the FC2 role, unsharded `K=14,336/N=5,376`, BF16 output, no bias or
|
|
active LoRA, GB10/SM121, CUDA 13.0, cuBLAS 13.1 headers, cuBLASLt 13.0 runtime,
|
|
logical `M=37,810`, and packed
|
|
`M=37,824`. A shape-specific `cublasLtMatmulAlgoCheck` must pass with zero
|
|
workspace. Load, check, or execution failure falls back through the already
|
|
produced NVFP4 bytes without quantizing twice; strict mode is available through
|
|
`H3_NVFP4_FC2_LT_STRICT=1`. Production startup prepares the extension and
|
|
caches its measured runtime compatibility result before inference.
|
|
|
|
The 20-round production-method gate exercised `Nvfp4Linear.forward_swiglu`
|
|
directly and recorded 23 successful dispatches with zero fallbacks per block:
|
|
|
|
| Block | Baseline p50 | Production p50 | Improvement |
|
|
| ---: | ---: | ---: | ---: |
|
|
| 0 | `464.153 ms` | `420.714 ms` | `9.36%` |
|
|
| 24 | `460.184 ms` | `422.656 ms` | `8.16%` |
|
|
| 49 | `456.183 ms` | `420.335 ms` | `7.86%` |
|
|
|
|
Production-method trajectories remained exact:
|
|
|
|
| Trajectory | Baseline | Production | Improvement | Dispatch |
|
|
| --- | ---: | ---: | ---: | --- |
|
|
| Two-step | `47.301 s` | `43.728 s` | `7.55%` | `100/100`, zero fallback |
|
|
| Canonical 12-step | `286.431 s` | `262.979 s` | `8.19%` | `600/600`, zero fallback |
|
|
|
|
The row-shape gate prevents unsafe generalization. Algorithm 70 was exact for
|
|
the tested `M=37,774`, `37,809`, `37,810`, `37,838`, `37,966`, and `38,222`
|
|
inputs, but differed in two BF16 elements at each of `M=37,711`, `37,725`, and
|
|
`37,742` despite passing `AlgoCheck`. Production therefore intentionally keeps
|
|
the canonical-only guard rather than inferring exactness from nearby shapes.
|
|
The retained shape artifact records compile-time CUDA/cuBLAS versions but
|
|
predates runtime-version fields in `build_info()`. The final extension separately
|
|
measured CUDA runtime `13000` and cuBLASLt runtime `130000`; the shape artifact
|
|
alone is not runtime-ABI evidence.
|
|
|
|
## Traffic Attribution
|
|
|
|
The one-pass distinct-data footprint is:
|
|
|
|
| Category | Bytes |
|
|
| --- | ---: |
|
|
| Activation QDATA + scales | `305,012,736` |
|
|
| Weight QDATA + scales | `43,352,064` |
|
|
| Padded BF16 output | `406,683,648` |
|
|
| Total | `755,048,448` |
|
|
|
|
Both schedules issue the same nominal operand requests:
|
|
|
|
| Requested category | Bytes | Geometric reuse |
|
|
| --- | ---: | ---: |
|
|
| Weight QDATA + scales | `12,832,210,944` | `296x` |
|
|
| Activation QDATA + scales | `12,832,210,944` | about `42.1x` |
|
|
| Total L2 operand reads | `25,664,424,960` | unchanged |
|
|
|
|
The schedule does not eliminate CTA-level rereading. It changes whether those
|
|
rereads remain cache-resident:
|
|
|
|
| NCU metric | Baseline `_stream_k` | Split-K-1 |
|
|
| --- | ---: | ---: |
|
|
| Main-kernel duration | `55.057 ms` | `17.031 ms` |
|
|
| Combined L2 hit rate | `53.32%` | `91.10%` |
|
|
| L2 read-miss bytes | `11.762 GB` | `1.909 GB` |
|
|
| L2 miss-byte proxy | `12.169 GB` | `2.316 GB` |
|
|
| Sysmem traffic proxy | `12.176 GB` | `2.327 GB` |
|
|
| Off-chip amplification | `16.12x` | `3.07x` |
|
|
| SM throughput | `25.11%` | `81.79%` |
|
|
| Tensor-pipe active share | `25.01%` | `82.42%` |
|
|
| Eligible warps/scheduler | `0.073` | `0.220` |
|
|
| Issue rate | `0.057` | `0.168` |
|
|
|
|
GB10 does not expose the usual discrete-GPU DRAM byte counters in these
|
|
captures. L2 misses and the reported sysmem fill/write sectors are used as the
|
|
off-chip proxy.
|
|
|
|
The `9.853 GB` reduction in L2 read misses is avoided operand rereading. NCU
|
|
aggregates the two TMA input descriptors, so it cannot defensibly assign exact
|
|
miss-byte totals separately to weights and activations. The request geometry
|
|
does prove that each input family accounts for `12.832 GB` of requested reads.
|
|
|
|
There is no material global partial-accumulator or output-reduction traffic:
|
|
|
|
- Candidate workspace is zero.
|
|
- Neither kernel issues global atomics or global reduction operations.
|
|
- No auxiliary reduction kernel is launched.
|
|
- Candidate writes exactly one padded BF16 output; baseline writes only
|
|
`73,728` bytes more.
|
|
- Baseline-only local stack traffic is about `22.35 MB` at L1 and almost none
|
|
reaches sysmem.
|
|
|
|
The baseline's dominant scheduler stall is sleeping. Source-correlated samples
|
|
land in `NANOSLEEP.SYNCS` polling around synchronization phase checks. Launch
|
|
resources are otherwise the same: 12,432 CTAs, 384 threads/CTA, 168 registers
|
|
per thread, 89,088 bytes allocated shared memory, and 25% theoretical
|
|
occupancy. The gain therefore comes from traversal locality and reduced
|
|
synchronization waiting, not occupancy or a reduction workspace.
|
|
|
|
## Evidence
|
|
|
|
- `benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-block-gate-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-trajectory-12step-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-profile-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-profile-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.csv`
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv`
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep`
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep`
|
|
- `benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json`
|
|
- `benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json`
|
|
- `src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp`
|
|
|
|
The Spark hot runtime remains stopped while performance work continues.
|
|
Resident-service validation is intentionally deferred until the performance
|
|
program is finished.
|