h3-blackwell-runtime/NVFP4_MODULATE_FUSION_DESIGN.md

NVFP4 Modulation Producer Fusion

Target

Fuse the two per-block AdaLN modulation operations into the exact NVFP4 activation scale and packing producer for attention QKV and MLP FC1. Retain the current Comfy Kitchen GEMMs and all BF16 numerical boundaries.

At the canonical 37,810-token shape, block 24 spends a median 10.47 ms on MSA modulation and 10.60 ms on MLP modulation. Production already mutates the two RMSNorm outputs in place, so this fusion targets memory traffic rather than a separate allocation.

Dataflow

Current:

1. Read normalized BF16 input and write modulated BF16 output.
2. Read modulated output to compute the global NVFP4 activation scale.
3. Read modulated output again to produce packed E2M1 data and E4M3 scales.
4. Execute the unchanged Comfy NVFP4 GEMM.

Prototype:

1. Read normalized input, reproduce the reference modulation BF16 boundary, and reduce its absolute maximum without materializing the modulated tensor.
2. Read normalized input again, reproduce the same BF16 values, and write exact QDATA/SFA directly into caller-owned buffers.
3. Execute the unchanged Comfy NVFP4 GEMM.

Exactness Contract

• Preserve segment-specific shift and scale selection.
• Preserve (x * (scale + 1)) + shift operation order and BF16 rounding.
• Preserve Vortex's approximate-reciprocal NVFP4 encoding behavior.
• Produce byte-identical QDATA and E4M3 block scales.
• Produce bit-identical QKV/FC1 outputs before changing block dispatch.

Gate

1. Adversarial modulation and quantization tests.
2. Exact packed bytes and linear outputs for blocks 0, 24, and 49.
3. Alternating module-forward benchmark in one loaded block.
4. Full 50-block step and two-step trajectory only after all three blocks gain.
5. Canonical 12-step trajectory before enabling the path by default.

Result

The producer is byte-exact for complete 37,810-row block 0, 24, and 49 inputs. It takes 6.02 ms for block 24 QKV input. Integrated block medians improve by 0.28-0.79%, a warmed two-step trajectory improves by 0.52%, and the warmed canonical 12-step run improves from 285.996 s to 284.408 s (0.56%). Video and audio tensors are bit-identical. Spark deployments enable the path with H3_NVFP4_MODULATE_FUSION=1.