diff --git a/PARITY.md b/PARITY.md index a351d11..beb0643 100644 --- a/PARITY.md +++ b/PARITY.md @@ -73,6 +73,48 @@ Comfy input dtype, full-precision weight construction, SDPA backend selection, and residual/BF16 rounding boundaries. Do not run another full inference merely to rediscover this projection result. +## Qwen Layer-0 Arithmetic Audit After `4050afe` + +The follow-up statement audit covered every remaining active layer-0 arithmetic +boundary outside the already-fixed SDPA wrapper: full-precision NVFP4 dispatch, +weight/scales, `pre_quant_scale`, BF16 GEMM, activation layout, Q/K RMSNorm, +O projection/residual, post-attention norm, SiLU/gate/up product, down +projection, and final residual. + +**Result: no further source-proven active mismatch was found.** For the actual +checkpoint, Comfy and direct both use BF16 input, BF16 NVFP4 dequantization, +BF16 `F.linear`, no projection biases, identical pre-scale ordering, identical +MLP evaluation order, and ordinary BF16 residual additions. Explicit flattening +and `.contiguous()` in direct are not a demonstrated numerical difference for +the captured contiguous batch-one input. + +Do not modify `nvfp4.py` or Qwen MLP/residual code to chase the remaining +layer-0 delta. The next justified work is a bounded component experiment that +identifies the actual selected SDPA kernel/GQA fallback and verifies capture +consistency; it is not another full denoising or video run. + +### Bounded SDPA/Layer-0 Result + +The bounded replay was completed with the fresh projection-capture tensors. +For the reference's 17-token layer-0 attention, both paths have native cuDNN +GQA available (`flash=False`, `cudnn=True`, `efficient=False`). Therefore the +direct SDPA wrapper takes the same native-GQA branch Comfy takes. + +| Boundary | Mean absolute error | Maximum absolute error | +| --- | ---: | ---: | +| O-projected attention | `0.000244372` | `0.0161562` | +| First residual | `0.00026845` | `0.0213833` | +| Post-attention RMSNorm | `3.87714e-05` | `0.000426032` | +| MLP output | `0.000674194` | `0.0500984` | +| Layer-0 output | `0.000808998` | `0.0526199` | + +This confirms the residual error is already present in the captured projection +outputs and is amplified by the MLP. The NVFP4 and MLP code audits found no +remaining active source-level branch difference. Further edits to flattening, +contiguity, dequantization dtype, `pre_quant_scale`, SiLU ordering, or residual +adds would be speculative and are prohibited until a new code-path difference +is demonstrated. + ## Current Runtime Scope | Component | Implemented | Known limitation |