h3-blackwell-runtime/PARITY.md

# H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so
we do not repeat investigations. A source review is never recorded as a passed
numeric gate.

## Reference Runs And Immutable Artifacts

| Asset | Location on Spark | Contents | Status |
| --- | --- | --- | --- |
| Qwen trace | `/tmp/fl2va-qwen-trace` | `qwen_input_embeds.pt`, 50 decoder outputs in `qwen_layers/00.pt` through `49.pt`, and `qwen_layer50.pt` | Complete, immutable Comfy capture |
| Qwen layer-0 trace | `/tmp/fl2va-qwen0-capture` | `qwen0_norm1.pt`, attention, post-attention, norm2, MLP, output | Complete, immutable Comfy capture |
| H3 full trace | `/tmp/fl2va-full-capture` | assembled DiT input/output and `blocks/00.pt` through `49.pt` | Complete, immutable Comfy capture |
| Working-copy capture mirror | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture` | Qwen layer-0 and layer-50 assets plus H3 input/output | Present; do not treat as immutable if the working checkout is modified |
| Sampler reference | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference` | `initial.pt`, exact `sigmas`, `step_00.pt` through `step_11.pt` | Complete, immutable Comfy capture |

All captures above use the established prompt-only FL2VA reference:

```text
prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)
```

This reference is **not** the 72-frame cat preview. Do not overwrite it to
diagnose preview output.

## Completed Findings

| Boundary | Evidence | Result | Conclusion |
| --- | --- | --- | --- |
| Direct runtime build | GB10 CUDA 13 image, Torch `2.9.1+cu130`, committed SageAttention3 wheel | Repeated direct runs complete | Environment is operational; PyTorch's `sm_121` warning is stale metadata in this environment |
| H3 attention baseline | Comfy `--use-sage-attention` versus direct backend investigation | SageAttention2 (`sage2`) was established as the H3 correctness baseline | Use `sage2` for all H3 parity gates; Sage3 remains experimental |
| Qwen tokenizer | Direct raw-prompt tokenizer versus MiniMax H3 source contract | Raw text, no chat template, `add_special_tokens=False` | Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token `151643` |
| Qwen embedding loading, before fix | Direct versus Comfy input embedding capture | Example direct values were approximately `11` where Comfy was approximately `0.017`; downstream refined-text mean absolute delta `1.79263` | Root cause was omitted per-row `model.embed_tokens.weight_scale` |
| Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference |
| Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning |
| Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material |
| Qwen all-layer replay, after FP32 fix | `compare_qwen_layer_trace.py` replayed `/tmp/fl2va-qwen-trace` without new Comfy inference | Layers `00` through `49` and `qwen_layer50.pt` all reported mean/max `0` / `0` | **Passed: prompt-only Qwen conditioning is bit-exact for the immutable 17-token FL2VA reference.** |
| Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance |
| Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE |
| Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA |
| Qwen SDPA dispatch audit and fix | Actual Spark Comfy block audit found direct raw SDPA omitted Comfy's four-dimensional mask wrapper, backend priority, and masked-GQA fallback. Direct ported that policy without importing Comfy. | Replaying immutable layer-0 capture after the change: attention mean/max `0.000244372` / `0.0161562`; post-attention `0.00026845` / `0.0213833`; layer output `0.000808998` / `0.0526199`. Previous attention mean was about `0.0357`; previous layer-output mean about `0.1496`. | Active code mismatch identified and substantially repaired. The remaining layer output error is now downstream of projection/elementwise BF16 boundaries, not an unreviewed SDPA dispatcher difference. |
| Qwen NVFP4 metadata | Qwen layer-0 checkpoint sidecars inspected | Every Q/K/V/O and MLP projection has `full_precision_matrix_mult: true`; `o_proj` and `down_proj` also have `pre_quant_scale` | Direct was incorrectly quantizing activations for all Qwen projections |
| Qwen NVFP4 fix | Commit `0cad1db` | Direct honors `full_precision_matrix_mult` and loads/applies `pre_quant_scale`; Spark smoke passed finite expected-shape Q and O projections | Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required |
| Qwen layer-0 projections, after NVFP4 fix | Fresh Comfy projection capture at `/tmp/fl2va-qwen0-projections`; direct compared from captured input embedding and post-attention state | Q mean/max `6.6547e-05` / `0.00195485`; K `9.11704e-05` / `0.00241077`; V `6.0298e-05` / `0.00102112`; gate `0.000365695` / `0.0082469`; up `0.000335494` / `0.00792789`; activated `4.76882e-05` / `0.031621`; down `0.000557594` / `0.0749016` | The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; `down_proj` is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case. |
| Qwen projection code audit, post-fix | Statement-level comparison against actual Spark Comfy commit `43cb4fffc89bba20ab7bd61467a36d0339338dab`, Torch `2.9.1+cu130`, and `/h3-models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` | All 350 Qwen projection sidecars set `full_precision_matrix_mult: true`; exactly 100 (`o_proj` and `down_proj` in each layer) have BF16 `pre_quant_scale`; no Qwen projection has bias. Packed weight is contiguous uint8, block scale is native E4M3, tensor scale is scalar FP32. | Direct and Comfy agree on the active full-precision branch, NVFP4 scale association, logical shapes, no-bias behavior, pre-scale order, BF16 compute dtype, and use the same installed Comfy Kitchen NVFP4 dequantizer. Do not change projection math based on speculation. |
| H3 DiT block trace | `/tmp/fl2va-full-capture` | Block 0 was close, around mean absolute delta `0.00018`; small independent differences accumulated across blocks | Full 50-block strict parity remains failed/unresolved |
| Sampler initial state | Comfy `initial.pt` inspected against direct preview | Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero | Known direct mismatch; identical integer seed is not sampler parity |
| Preview output, before Qwen fixes | Dragon/locomotive prompt previews | Flower-like output | Confirmed incorrect conditioning had visible semantic impact |
| Preview output, after Qwen metadata fix | 72-frame cat, 12-step direct preview | Cat semantic structure appears, with persistent 32-pixel-like square artifacts | Conditioning improved; denoiser/sampler/VAE boundary remains non-parity |
| Video encoder | Direct ffmpeg raw RGB to H.264/yuv420p, no audio | Different from Comfy `CreateVideo`/`SaveVideo` | Not a valid latent-quality gate; cannot explain model-space corruption by itself |

## Qwen Projection Code Audit: Remaining Non-Active Differences

These are real implementation differences, but the audit established they are
not active numerical branches for the captured Qwen checkpoint. They must not
be presented as an explanation for the current `down_proj` delta without new
evidence.

| Direct location | Difference from Comfy | Active for current capture? |
| --- | --- | --- |
| `nvfp4.py:47-51` | Calls `.contiguous()` while Comfy preserves loaded storage/view identity. In particular Comfy treats E4M3 block scales as a dtype view, not a numeric cast. | No: checkpoint tensors are already contiguous and direct also uses a dtype view. |
| `nvfp4.py:75-76` | Always flattens and contiguates activation rows; Comfy full-precision `F.linear` retains N-D input/stride. | No known value effect: elements and BF16 dtype are unchanged. |
| `nvfp4.py:82` | Uses `packed_weight.dequantize().to(flat_x)` rather than an explicitly named Comfy compute-dtype dequantization call. | No: input and output compute dtype are BF16, and both call the same installed Kitchen layout/dequantizer. |
| `nvfp4.py:46,81` | Does not implement Comfy global full-precision overrides, disabled-format controls, forced casts, weight functions, or LoRA/offload dispatch. | No: every Qwen sidecar itself forces full precision, no projection has bias, and no patches/LoRAs are attached. |

**Audit conclusion:** projection source code contains cleanup/generalization gaps,
but no identified active branch mismatch explains the current captured values.
The next code audit must move outward to the exact Qwen block boundary:
Comfy input dtype, full-precision weight construction, SDPA backend selection,
and residual/BF16 rounding boundaries. Do not run another full inference merely
to rediscover this projection result.

## Qwen Layer-0 Arithmetic Audit After `4050afe`

The follow-up statement audit covered every remaining active layer-0 arithmetic
boundary outside the already-fixed SDPA wrapper: full-precision NVFP4 dispatch,
weight/scales, `pre_quant_scale`, BF16 GEMM, activation layout, Q/K RMSNorm,
O projection/residual, post-attention norm, SiLU/gate/up product, down
projection, and final residual.

**Result: no further source-proven active mismatch was found.** For the actual
checkpoint, Comfy and direct both use BF16 input, BF16 NVFP4 dequantization,
BF16 `F.linear`, no projection biases, identical pre-scale ordering, identical
MLP evaluation order, and ordinary BF16 residual additions. Explicit flattening
and `.contiguous()` in direct are not a demonstrated numerical difference for
the captured contiguous batch-one input.

Do not modify `nvfp4.py` or Qwen MLP/residual code to chase the remaining
layer-0 delta. The next justified work is a bounded component experiment that
identifies the actual selected SDPA kernel/GQA fallback and verifies capture
consistency; it is not another full denoising or video run.

### Bounded SDPA/Layer-0 Result

The bounded replay was completed with the fresh projection-capture tensors.
For the reference's 17-token layer-0 attention, both paths have native cuDNN
GQA available (`flash=False`, `cudnn=True`, `efficient=False`). Therefore the
direct SDPA wrapper takes the same native-GQA branch Comfy takes.

| Boundary | Mean absolute error | Maximum absolute error |
| --- | ---: | ---: |
| O-projected attention | `0.000244372` | `0.0161562` |
| First residual | `0.00026845` | `0.0213833` |
| Post-attention RMSNorm | `3.87714e-05` | `0.000426032` |
| MLP output | `0.000674194` | `0.0500984` |
| Layer-0 output | `0.000808998` | `0.0526199` |

This confirms the residual error is already present in the captured projection
outputs and is amplified by the MLP. The NVFP4 and MLP code audits found no
remaining active source-level branch difference. Further edits to flattening,
contiguity, dequantization dtype, `pre_quant_scale`, SiLU ordering, or residual
adds would be speculative and are prohibited until a new code-path difference
is demonstrated.

## Loaded-Comfy Projection Module Probe

The source audit was followed by a direct replay against tensors consumed and
produced by the actual loaded Comfy layer-0 modules. This proves the remaining
difference is present **before attention** and is not inferred from source.

| Item | Evidence |
| --- | --- |
| Capture location | `/tmp/fl2va-qwen0-loaded-modules` on Spark |
| QKV capture SHA-256 | `8998722e73ba24c40c09a86133ccdb22b708998a3e8b8fb4c4293227345f19db` |
| MLP capture SHA-256 | `e52ed70f4aa76f1ba26302b9955631d8f490fb6dc269d96a0ce3e1ea981e7ac2` |
| Loaded Comfy class | `comfy.ops.mixed_precision_ops.<locals>.MixedPrecisionOps.Linear` |
| Loaded Comfy weight | `comfy_kitchen.tensor.base.QuantizedTensor`, logical dtype BF16, `TensorCoreNVFP4Layout`, full precision enabled |
| Q/K/V dimensions | `(8192,5120)`, `(1024,5120)`, `(1024,5120)`; no pre-scale |
| Gate/up/down dimensions | `(25600,5120)`, `(25600,5120)`, `(5120,25600)`; only down has pre-scale |

Direct replay on the **identical loaded-Comfy module inputs**:

| Projection | Mean absolute error | Maximum absolute error |
| --- | ---: | ---: |
| Q | `6.11124e-05` | `0.0019514` |
| K | `8.47002e-05` | `0.00167805` |
| V | `5.4459e-05` | `0.00098893` |
| Gate | `0.000308973` | `0.0082469` |
| Up | `0.00028138` | `0.00792789` |
| Down, using Comfy's captured activated input | `0.000424157` | `0.0500984` |

**Conclusion:** same checkpoint tensors, same BF16 inputs, and the same named
Kitchen layout do not currently produce identical projection outputs through
direct `Nvfp4Linear` and Comfy's loaded `MixedPrecisionOps.Linear`. The next
investigation must capture/compare the effective `QuantizedTensor` parameters
and direct packed-weight wrapper at dispatch time. Do not revisit attention,
sampler, or MLP ordering until that exact module-dispatch difference is found.

### Effective Weight Root Cause And Resolution

The storage-safe dispatch probe found the active difference. Comfy's text
encoder calls Qwen with `dtype=torch.float32`; its full-precision NVFP4 branch
casts each projection to an effective FP32 matrix before `F.linear`. Direct had
constructed the Qwen encoder with BF16, causing its full-precision NVFP4 branch
to dequantize the same packed data into BF16.

The probe proved that packed qdata, tensor scale, and block scale are exact for
all Q/K/V/gate/up/down projections. With direct Qwen switched to FP32:

| Projection check | Result |
| --- | --- |
| Effective Q/K/V/gate/up/down FP32 weights | Bit-exact to Comfy |
| Q/K/V/gate/up linear outputs | Bit-exact to Comfy |
| Layer-0 O-projected attention | Mean/max `0` / `0` |
| Layer-0 post-attention residual | Mean/max `0` / `0` |
| Layer-0 post-attention RMSNorm | Mean/max `0` / `0` |
| Layer-0 MLP output | Mean/max `0` / `0` |
| Layer-0 final output | Mean/max `0` / `0` |

The special `down` dispatch sub-probe that supplied Comfy's already-activated
input directly to `F.linear` is intentionally invalid because it bypasses the
module's required `pre_quant_scale`; the full layer replay above is the valid
gate and is exact.

## Current Runtime Scope

| Component | Implemented | Known limitation |
| --- | --- | --- |
| Qwen text and vision conditioning | Yes | Text-only prompt parity is established; first/last keyframe vision conditioning is implemented, while arbitrary reference video/audio remains missing |
| Token refiner | Yes | Bit-exact from captured 5376-wide refiner input through both blocks and final RMSNorm; Qwen-to-refiner projection boundary is still not separately captured |
| T2VA/keyframe packer | Yes | Bit-exact for the coherent captured prompt-only DiT input; first-frame, last-frame, and first/last keyframe condition rows are implemented |
| H3 DiT backbone | Yes | Bit-exact through all 50 blocks from the coherent assembled FL2VA input; requires the standalone Comfy Kitchen fused Q/K RMSNorm + split-half RoPE operator |
| H3 final layer | Yes | Bit-exact final AdaLN, target-row modulation, and video/audio patch rows; Comfy materializes the AdaLN and output-head biases through BF16 |
| H3 DiT | Yes | Strict all-block numeric parity not achieved |
| Beta/RES sampler | Yes | H3 replay is bit-exact for all 12 captured sampler calls and RES update is bit-exact from captured denoised; direct raw-output to model-wrapper denoised conversion retains small BF16/state boundary deltas |
| Video VAE decoder | Yes | Direct VAE temporal assembly matches upstream after overlap fix. FP16 is the default Comfy-equivalent runtime path; cat benchmark VAE decode is `25.085s`. Use FP32 only for exact direct diagnostics. |
| Audio VAE/decode/mux | Yes | Direct decoder-only MiniMax H3 audio VAE returns stereo `32000 Hz` waveform and muxes with generated video. Native audio latent scaling is fixed. |
| End-to-end prompt-only FL2VA preview | Yes | Apples-to-apples warm cat benchmark is at Comfy parity: Comfy warm `150.26s`; direct warm after text conditioning `149.304s`; direct warm including text conditioning `151.465s`. |
| Full Ref2VA | No | Image keyframes, vision conditioning, VAE encode, generated audio, and muxing are implemented; arbitrary reference video/audio and identity/voice conditioning remain missing |

## Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete
unless the checkpoint, Comfy version, prompt, or backend changes.

1. **Distributed validation.** Run real NCCL parity and performance sweeps at
   2/4/6/8 GPUs; CPU transport and one-GPU real-checkpoint identity are complete.
2. **Full Ref2VA support.** Add arbitrary reference video/audio, identity and
   voice conditioning, and reference-path validation gates.

## Existing Tools And Their Intended Gate

| Gate | Tools |
| --- | --- |
| Qwen captures | `patch_comfy_qwen_output_capture.py`, `patch_comfy_qwen_layer_trace.py`, `patch_comfy_qwen_layer0_sublayers.py`, `patch_comfy_qwen_layer0_projections.py` |
| Qwen projection gate | `compare_qwen0_projections.py` |
| Qwen local diagnosis | `trace_qwen0_attention.py` needs repair before reuse; it currently passes an obsolete encoder argument |
| H3 block capture | `patch_comfy_h3_capture.py`, `patch_comfy_h3_block_capture.py`, `patch_comfy_h3_block0_sublayers.py` |
| H3 local diagnosis | `compare_block0_qkv.py`, `compare_attention_backends.py`, `compare_block0_mlp_projections.py`, `localize_block_sublayers.py`, `localize_block_mismatch.py` |
| Sampler | `patch_comfy_sampler_capture.py`, `compare_fl2va_steps.py` |
| VAE | `smoke_h3_vae_decoder.py`; a same-latent direct-versus-Comfy pixel comparator still needs to be added |

The `patch_comfy_*` tools modify `/opt/ComfyUI` and are only safe against the
restorable Comfy reference image. Capture provenance must be appended here at
the time a new capture is made.