daniel/h3-blackwell-runtime
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
h3-blackwell-runtime/NVFP4_SWIGLU_FUSION_DESIGN.md
Daniel Maddern ea14b98544 Preserve Vortex performance research history
2026-08-25 20:30:22 +07:00

1.2 KiB
Raw Permalink Blame History

NVFP4 SwiGLU Producer Fusion

Implementation

The H3 MLP now recomputes its exact BF16 SwiGLU values inside the Vortex global scale and NVFP4 pack passes instead of materializing the 37,810x14,336 BF16 activation. The FC2 Comfy Kitchen NVFP4 GEMM and its reduction order are unchanged.

The fused producer preserves both BF16 boundaries:

  1. silu(gate) rounded to BF16.
  2. The BF16 SiLU result multiplied by BF16 up and rounded to BF16.

It then preserves the established Vortex tensor-scale compatibility cast, E4M3 block scales, E2M1 encoding, approximate reciprocal, and Comfy physical scale layout.

Validation

  • Randomized BF16 tensor scale, QDATA, and SFA are byte-exact.
  • Real block-24 FC1 output is byte-exact; producer median is 21.93 ms.
  • Blocks 0, 24, and 49 are bit-exact and improve by 2.14-2.26%.
  • Warmed two-step video/audio tensors are bit-exact.
  • Warmed canonical 12-step tensors are bit-exact and improve from 289.14 s to 277.36 s, or 4.07%.

Spark single-GPU deployments enable the path with H3_NVFP4_SWIGLU_FUSION=1. Active FC2 LoRA, full-precision, pre-scale, autograd, and unsupported dtype/layout cases use the materialized fallback.