159 lines
6.5 KiB
Markdown
159 lines
6.5 KiB
Markdown
|
# FC2 NVFP4 Scheduling Results
|
||
|
|
|
||
|
|
Status: research retained; validated as a canonical-shape production-integration
|
||
|
|
candidate, but not wired into production dispatch.
|
||
|
|
|
||
|
|
## Decision
|
||
|
|
|
||
|
|
The canonical GB10 FC2 slowdown is caused by the cuBLASLt heuristic selecting a
|
||
|
|
`_stream_k` kernel with poor traversal locality and heavy synchronization
|
||
|
|
polling. A documented public split-K-1 configuration is byte-exact and removes
|
||
|
|
the regression. Because this library candidate passed the complete validation
|
||
|
|
ladder, the custom persistent-kernel branch is closed.
|
||
|
|
|
||
|
|
The retained candidate is:
|
||
|
|
|
||
|
|
| Field | Value |
|
||
|
|
| --- | ---: |
|
||
|
|
| Algorithm ID | `70` |
|
||
|
|
| Tile ID | `20` |
|
||
|
|
| Stages ID | `37` |
|
||
|
|
| Split-K | `1` |
|
||
|
|
| Reduction scheme | `0` |
|
||
|
|
| Custom option / CTA swizzle | `0 / 0` |
|
||
|
|
| Required workspace | `0 bytes` |
|
||
|
|
| Supplied workspace | `64 MiB` sweep; `0 bytes` block/trajectory gates |
|
||
|
|
|
||
|
|
The baseline heuristic's negative split-K value is an undocumented library
|
||
|
|
sentinel. CUDA 13 has no documented public attribute that directly selects
|
||
|
|
Stream-K, so the sentinel itself is not interpreted as a public Stream-K
|
||
|
|
control. The launched baseline symbol does end in `_stream_k`.
|
||
|
|
|
||
|
|
## Canonical Operation
|
||
|
|
|
||
|
|
- Logical GEMM: `M=37,810, N=5,376, K=14,336`.
|
||
|
|
- Descriptor GEMM after accepted producer padding: `M=37,824`.
|
||
|
|
- Inputs: packed NVFP4 E2M1 activation and weight with vector-16 UE4M3 scales.
|
||
|
|
- Accumulation/scalars: FP32; output: BF16.
|
||
|
|
- Accepted activation producer: `vortex_native_quantize_swiglu_nvfp4`.
|
||
|
|
- Image: `sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`.
|
||
|
|
|
||
|
|
## Timing
|
||
|
|
|
||
|
|
The 20-round isolated sweep alternated AB/BA order in one loaded model.
|
||
|
|
|
||
|
|
| Measurement | Baseline | Split-K-1 | Result |
|
||
|
|
| --- | ---: | ---: | ---: |
|
||
|
|
| FC2 p50 | `53.618 ms` | `15.636 ms` | `3.43x` |
|
||
|
|
| Dense throughput p50 | `108.70 TFLOP/s` | `372.74 TFLOP/s` | `3.43x` |
|
||
|
|
| Producer + FC2 p50 | `76.737 ms` | `38.823 ms` | `49.41%` faster |
|
||
|
|
|
||
|
|
The negative-sentinel candidate reproduced the production-equivalent baseline
|
||
|
|
within noise. Explicit split-K factors `2`, `4`, `8`, and `16` were rejected by
|
||
|
|
`cublasLtMatmulAlgoCheck` for the tested configuration space.
|
||
|
|
|
||
|
|
Complete 20-round block gates replaced only
|
||
|
|
`block.mlp.fc2.forward_swiglu`:
|
||
|
|
|
||
|
|
| Block | Baseline p50 | Candidate p50 | Improvement |
|
||
|
|
| ---: | ---: | ---: | ---: |
|
||
|
|
| 0 | `457.891 ms` | `420.536 ms` | `8.16%` |
|
||
|
|
| 24 | `459.928 ms` | `420.243 ms` | `8.63%` |
|
||
|
|
| 49 | `458.054 ms` | `417.364 ms` | `8.88%` |
|
||
|
|
|
||
|
|
All block outputs were byte-exact against both the paired baseline and retained
|
||
|
|
traversal.
|
||
|
|
|
||
|
|
| Trajectory | Baseline | Candidate | Improvement | Correctness |
|
||
|
|
| --- | ---: | ---: | ---: | --- |
|
||
|
|
| Two-step | `45.962 s` | `42.516 s` | `7.50%` | video/audio BF16 exact |
|
||
|
|
| Canonical 12-step | `278.201 s` | `255.371 s` | `8.21%` | video/audio BF16 exact |
|
||
|
|
|
||
|
|
Trajectory values are single paired runs in baseline-then-candidate
|
||
|
|
order, not repeated medians. Their approximately `38 ms` per-FC2 savings agree
|
||
|
|
with the alternating isolated and complete-block gates, but the precise
|
||
|
|
end-to-end percentages retain run-order uncertainty.
|
||
|
|
|
||
|
|
The 12-step saving of `22.829 s` matches approximately 600 FC2 invocations
|
||
|
|
multiplied by the isolated roughly `38 ms` saving.
|
||
|
|
|
||
|
|
## Traffic Attribution
|
||
|
|
|
||
|
|
The one-pass distinct-data footprint is:
|
||
|
|
|
||
|
|
| Category | Bytes |
|
||
|
|
| --- | ---: |
|
||
|
|
| Activation QDATA + scales | `305,012,736` |
|
||
|
|
| Weight QDATA + scales | `43,352,064` |
|
||
|
|
| Padded BF16 output | `406,683,648` |
|
||
|
|
| Total | `755,048,448` |
|
||
|
|
|
||
|
|
Both schedules issue the same nominal operand requests:
|
||
|
|
|
||
|
|
| Requested category | Bytes | Geometric reuse |
|
||
|
|
| --- | ---: | ---: |
|
||
|
|
| Weight QDATA + scales | `12,832,210,944` | `296x` |
|
||
|
|
| Activation QDATA + scales | `12,832,210,944` | about `42.1x` |
|
||
|
|
| Total L2 operand reads | `25,664,424,960` | unchanged |
|
||
|
|
|
||
|
|
The schedule does not eliminate CTA-level rereading. It changes whether those
|
||
|
|
rereads remain cache-resident:
|
||
|
|
|
||
|
|
| NCU metric | Baseline `_stream_k` | Split-K-1 |
|
||
|
|
| --- | ---: | ---: |
|
||
|
|
| Main-kernel duration | `55.057 ms` | `17.031 ms` |
|
||
|
|
| Combined L2 hit rate | `53.32%` | `91.10%` |
|
||
|
|
| L2 read-miss bytes | `11.762 GB` | `1.909 GB` |
|
||
|
|
| L2 miss-byte proxy | `12.169 GB` | `2.316 GB` |
|
||
|
|
| Sysmem traffic proxy | `12.176 GB` | `2.327 GB` |
|
||
|
|
| Off-chip amplification | `16.12x` | `3.07x` |
|
||
|
|
| SM throughput | `25.11%` | `81.79%` |
|
||
|
|
| Tensor-pipe active share | `25.01%` | `82.42%` |
|
||
|
|
| Eligible warps/scheduler | `0.073` | `0.220` |
|
||
|
|
| Issue rate | `0.057` | `0.168` |
|
||
|
|
|
||
|
|
GB10 does not expose the usual discrete-GPU DRAM byte counters in these
|
||
|
|
captures. L2 misses and the reported sysmem fill/write sectors are used as the
|
||
|
|
off-chip proxy.
|
||
|
|
|
||
|
|
The `9.853 GB` reduction in L2 read misses is avoided operand rereading. NCU
|
||
|
|
aggregates the two TMA input descriptors, so it cannot defensibly assign exact
|
||
|
|
miss-byte totals separately to weights and activations. The request geometry
|
||
|
|
does prove that each input family accounts for `12.832 GB` of requested reads.
|
||
|
|
|
||
|
|
There is no material global partial-accumulator or output-reduction traffic:
|
||
|
|
|
||
|
|
- Candidate workspace is zero.
|
||
|
|
- Neither kernel issues global atomics or global reduction operations.
|
||
|
|
- No auxiliary reduction kernel is launched.
|
||
|
|
- Candidate writes exactly one padded BF16 output; baseline writes only
|
||
|
|
`73,728` bytes more.
|
||
|
|
- Baseline-only local stack traffic is about `22.35 MB` at L1 and almost none
|
||
|
|
reaches sysmem.
|
||
|
|
|
||
|
|
The baseline's dominant scheduler stall is sleeping. Source-correlated samples
|
||
|
|
land in `NANOSLEEP.SYNCS` polling around synchronization phase checks. Launch
|
||
|
|
resources are otherwise the same: 12,432 CTAs, 384 threads/CTA, 168 registers
|
||
|
|
per thread, 89,088 bytes allocated shared memory, and 25% theoretical
|
||
|
|
occupancy. The gain therefore comes from traversal locality and reduced
|
||
|
|
synchronization waiting, not occupancy or a reduction workspace.
|
||
|
|
|
||
|
|
## Evidence
|
||
|
|
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-block-gate-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-trajectory-12step-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-profile-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-profile-20260825.json`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.csv`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep`
|
||
|
|
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep`
|
||
|
|
|
||
|
|
Production source and configuration were not changed by this study. The
|
||
|
|
candidate is validated only for the canonical descriptor shape and current
|
||
|
|
GB10/CUDA-library combination. Promotion still requires shape-specific
|
||
|
|
`AlgoCheck` with a safe fallback, integration behind the existing
|
||
|
|
`Nvfp4Linear` boundary, broader shape tests, and deployment validation.
|