h3-blackwell-runtime/PARITY.md

# H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so
we do not repeat investigations. A source review is never recorded as a passed
numeric gate.

## Reference Runs And Immutable Artifacts

| Asset | Location on Spark | Contents | Status |
| --- | --- | --- | --- |
| Qwen trace | `/tmp/fl2va-qwen-trace` | `qwen_input_embeds.pt`, 50 decoder outputs in `qwen_layers/00.pt` through `49.pt`, and `qwen_layer50.pt` | Complete, immutable Comfy capture |
| Qwen layer-0 trace | `/tmp/fl2va-qwen0-capture` | `qwen0_norm1.pt`, attention, post-attention, norm2, MLP, output | Complete, immutable Comfy capture |
| H3 full trace | `/tmp/fl2va-full-capture` | assembled DiT input/output and `blocks/00.pt` through `49.pt` | Complete, immutable Comfy capture |
| Working-copy capture mirror | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture` | Qwen layer-0 and layer-50 assets plus H3 input/output | Present; do not treat as immutable if the working checkout is modified |
| Sampler reference | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference` | `initial.pt`, exact `sigmas`, `step_00.pt` through `step_11.pt` | Complete, immutable Comfy capture |

All captures above use the established prompt-only FL2VA reference:

```text
prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)
```

This reference is **not** the 72-frame cat preview. Do not overwrite it to
diagnose preview output.

## Completed Findings

| Boundary | Evidence | Result | Conclusion |
| --- | --- | --- | --- |
| Direct runtime build | GB10 CUDA 13 image, Torch `2.9.1+cu130`, committed SageAttention3 wheel | Repeated direct runs complete | Environment is operational; PyTorch's `sm_121` warning is stale metadata in this environment |
| H3 attention baseline | Comfy `--use-sage-attention` versus direct backend investigation | SageAttention2 (`sage2`) was established as the H3 correctness baseline | Use `sage2` for all H3 parity gates; Sage3 remains experimental |
| Qwen tokenizer | Direct raw-prompt tokenizer versus MiniMax H3 source contract | Raw text, no chat template, `add_special_tokens=False` | Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token `151643` |
| Qwen embedding loading, before fix | Direct versus Comfy input embedding capture | Example direct values were approximately `11` where Comfy was approximately `0.017`; downstream refined-text mean absolute delta `1.79263` | Root cause was omitted per-row `model.embed_tokens.weight_scale` |
| Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference |
| Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning |
| Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material |
| Qwen all-layer replay, after FP32 fix | `compare_qwen_layer_trace.py` replayed `/tmp/fl2va-qwen-trace` without new Comfy inference | Layers `00` through `49` and `qwen_layer50.pt` all reported mean/max `0` / `0` | **Passed: prompt-only Qwen conditioning is bit-exact for the immutable 17-token FL2VA reference.** |
| Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance |
| Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE |
| Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA |
| Qwen SDPA dispatch audit and fix | Actual Spark Comfy block audit found direct raw SDPA omitted Comfy's four-dimensional mask wrapper, backend priority, and masked-GQA fallback. Direct ported that policy without importing Comfy. | Replaying immutable layer-0 capture after the change: attention mean/max `0.000244372` / `0.0161562`; post-attention `0.00026845` / `0.0213833`; layer output `0.000808998` / `0.0526199`. Previous attention mean was about `0.0357`; previous layer-output mean about `0.1496`. | Active code mismatch identified and substantially repaired. The remaining layer output error is now downstream of projection/elementwise BF16 boundaries, not an unreviewed SDPA dispatcher difference. |
| Qwen NVFP4 metadata | Qwen layer-0 checkpoint sidecars inspected | Every Q/K/V/O and MLP projection has `full_precision_matrix_mult: true`; `o_proj` and `down_proj` also have `pre_quant_scale` | Direct was incorrectly quantizing activations for all Qwen projections |
| Qwen NVFP4 fix | Commit `0cad1db` | Direct honors `full_precision_matrix_mult` and loads/applies `pre_quant_scale`; Spark smoke passed finite expected-shape Q and O projections | Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required |
| Qwen layer-0 projections, after NVFP4 fix | Fresh Comfy projection capture at `/tmp/fl2va-qwen0-projections`; direct compared from captured input embedding and post-attention state | Q mean/max `6.6547e-05` / `0.00195485`; K `9.11704e-05` / `0.00241077`; V `6.0298e-05` / `0.00102112`; gate `0.000365695` / `0.0082469`; up `0.000335494` / `0.00792789`; activated `4.76882e-05` / `0.031621`; down `0.000557594` / `0.0749016` | The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; `down_proj` is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case. |
| Qwen projection code audit, post-fix | Statement-level comparison against actual Spark Comfy commit `43cb4fffc89bba20ab7bd61467a36d0339338dab`, Torch `2.9.1+cu130`, and `/h3-models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` | All 350 Qwen projection sidecars set `full_precision_matrix_mult: true`; exactly 100 (`o_proj` and `down_proj` in each layer) have BF16 `pre_quant_scale`; no Qwen projection has bias. Packed weight is contiguous uint8, block scale is native E4M3, tensor scale is scalar FP32. | Direct and Comfy agree on the active full-precision branch, NVFP4 scale association, logical shapes, no-bias behavior, pre-scale order, BF16 compute dtype, and use the same installed Comfy Kitchen NVFP4 dequantizer. Do not change projection math based on speculation. |
| H3 DiT block trace | `/tmp/fl2va-full-capture` | Block 0 was close, around mean absolute delta `0.00018`; small independent differences accumulated across blocks | Full 50-block strict parity remains failed/unresolved |
| Sampler initial state | Comfy `initial.pt` inspected against direct preview | Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero | Known direct mismatch; identical integer seed is not sampler parity |
| Preview output, before Qwen fixes | Dragon/locomotive prompt previews | Flower-like output | Confirmed incorrect conditioning had visible semantic impact |
| Preview output, after Qwen metadata fix | 72-frame cat, 12-step direct preview | Cat semantic structure appears, with persistent 32-pixel-like square artifacts | Conditioning improved; denoiser/sampler/VAE boundary remains non-parity |
| Video encoder | Direct ffmpeg raw RGB to H.264/yuv420p, no audio | Different from Comfy `CreateVideo`/`SaveVideo` | Not a valid latent-quality gate; cannot explain model-space corruption by itself |

## Qwen Projection Code Audit: Remaining Non-Active Differences

These are real implementation differences, but the audit established they are
not active numerical branches for the captured Qwen checkpoint. They must not
be presented as an explanation for the current `down_proj` delta without new
evidence.

| Direct location | Difference from Comfy | Active for current capture? |
| --- | --- | --- |
| `nvfp4.py:47-51` | Calls `.contiguous()` while Comfy preserves loaded storage/view identity. In particular Comfy treats E4M3 block scales as a dtype view, not a numeric cast. | No: checkpoint tensors are already contiguous and direct also uses a dtype view. |
| `nvfp4.py:75-76` | Always flattens and contiguates activation rows; Comfy full-precision `F.linear` retains N-D input/stride. | No known value effect: elements and BF16 dtype are unchanged. |
| `nvfp4.py:82` | Uses `packed_weight.dequantize().to(flat_x)` rather than an explicitly named Comfy compute-dtype dequantization call. | No: input and output compute dtype are BF16, and both call the same installed Kitchen layout/dequantizer. |
| `nvfp4.py:46,81` | Does not implement Comfy global full-precision overrides, disabled-format controls, forced casts, weight functions, or LoRA/offload dispatch. | No: every Qwen sidecar itself forces full precision, no projection has bias, and no patches/LoRAs are attached. |

**Audit conclusion:** projection source code contains cleanup/generalization gaps,
but no identified active branch mismatch explains the current captured values.
The next code audit must move outward to the exact Qwen block boundary:
Comfy input dtype, full-precision weight construction, SDPA backend selection,
and residual/BF16 rounding boundaries. Do not run another full inference merely
to rediscover this projection result.

## Qwen Layer-0 Arithmetic Audit After `4050afe`

The follow-up statement audit covered every remaining active layer-0 arithmetic
boundary outside the already-fixed SDPA wrapper: full-precision NVFP4 dispatch,
weight/scales, `pre_quant_scale`, BF16 GEMM, activation layout, Q/K RMSNorm,
O projection/residual, post-attention norm, SiLU/gate/up product, down
projection, and final residual.

**Result: no further source-proven active mismatch was found.** For the actual
checkpoint, Comfy and direct both use BF16 input, BF16 NVFP4 dequantization,
BF16 `F.linear`, no projection biases, identical pre-scale ordering, identical
MLP evaluation order, and ordinary BF16 residual additions. Explicit flattening
and `.contiguous()` in direct are not a demonstrated numerical difference for
the captured contiguous batch-one input.

Do not modify `nvfp4.py` or Qwen MLP/residual code to chase the remaining
layer-0 delta. The next justified work is a bounded component experiment that
identifies the actual selected SDPA kernel/GQA fallback and verifies capture
consistency; it is not another full denoising or video run.

### Bounded SDPA/Layer-0 Result

The bounded replay was completed with the fresh projection-capture tensors.
For the reference's 17-token layer-0 attention, both paths have native cuDNN
GQA available (`flash=False`, `cudnn=True`, `efficient=False`). Therefore the
direct SDPA wrapper takes the same native-GQA branch Comfy takes.

| Boundary | Mean absolute error | Maximum absolute error |
| --- | ---: | ---: |
| O-projected attention | `0.000244372` | `0.0161562` |
| First residual | `0.00026845` | `0.0213833` |
| Post-attention RMSNorm | `3.87714e-05` | `0.000426032` |
| MLP output | `0.000674194` | `0.0500984` |
| Layer-0 output | `0.000808998` | `0.0526199` |

This confirms the residual error is already present in the captured projection
outputs and is amplified by the MLP. The NVFP4 and MLP code audits found no
remaining active source-level branch difference. Further edits to flattening,
contiguity, dequantization dtype, `pre_quant_scale`, SiLU ordering, or residual
adds would be speculative and are prohibited until a new code-path difference
is demonstrated.

## Loaded-Comfy Projection Module Probe

The source audit was followed by a direct replay against tensors consumed and
produced by the actual loaded Comfy layer-0 modules. This proves the remaining
difference is present **before attention** and is not inferred from source.

| Item | Evidence |
| --- | --- |
| Capture location | `/tmp/fl2va-qwen0-loaded-modules` on Spark |
| QKV capture SHA-256 | `8998722e73ba24c40c09a86133ccdb22b708998a3e8b8fb4c4293227345f19db` |
| MLP capture SHA-256 | `e52ed70f4aa76f1ba26302b9955631d8f490fb6dc269d96a0ce3e1ea981e7ac2` |
| Loaded Comfy class | `comfy.ops.mixed_precision_ops.<locals>.MixedPrecisionOps.Linear` |
| Loaded Comfy weight | `comfy_kitchen.tensor.base.QuantizedTensor`, logical dtype BF16, `TensorCoreNVFP4Layout`, full precision enabled |
| Q/K/V dimensions | `(8192,5120)`, `(1024,5120)`, `(1024,5120)`; no pre-scale |
| Gate/up/down dimensions | `(25600,5120)`, `(25600,5120)`, `(5120,25600)`; only down has pre-scale |

Direct replay on the **identical loaded-Comfy module inputs**:

| Projection | Mean absolute error | Maximum absolute error |
| --- | ---: | ---: |
| Q | `6.11124e-05` | `0.0019514` |
| K | `8.47002e-05` | `0.00167805` |
| V | `5.4459e-05` | `0.00098893` |
| Gate | `0.000308973` | `0.0082469` |
| Up | `0.00028138` | `0.00792789` |
| Down, using Comfy's captured activated input | `0.000424157` | `0.0500984` |

**Conclusion:** same checkpoint tensors, same BF16 inputs, and the same named
Kitchen layout do not currently produce identical projection outputs through
direct `Nvfp4Linear` and Comfy's loaded `MixedPrecisionOps.Linear`. The next
investigation must capture/compare the effective `QuantizedTensor` parameters
and direct packed-weight wrapper at dispatch time. Do not revisit attention,
sampler, or MLP ordering until that exact module-dispatch difference is found.

### Effective Weight Root Cause And Resolution

The storage-safe dispatch probe found the active difference. Comfy's text
encoder calls Qwen with `dtype=torch.float32`; its full-precision NVFP4 branch
casts each projection to an effective FP32 matrix before `F.linear`. Direct had
constructed the Qwen encoder with BF16, causing its full-precision NVFP4 branch
to dequantize the same packed data into BF16.

The probe proved that packed qdata, tensor scale, and block scale are exact for
all Q/K/V/gate/up/down projections. With direct Qwen switched to FP32:

| Projection check | Result |
| --- | --- |
| Effective Q/K/V/gate/up/down FP32 weights | Bit-exact to Comfy |
| Q/K/V/gate/up linear outputs | Bit-exact to Comfy |
| Layer-0 O-projected attention | Mean/max `0` / `0` |
| Layer-0 post-attention residual | Mean/max `0` / `0` |
| Layer-0 post-attention RMSNorm | Mean/max `0` / `0` |
| Layer-0 MLP output | Mean/max `0` / `0` |
| Layer-0 final output | Mean/max `0` / `0` |

The special `down` dispatch sub-probe that supplied Comfy's already-activated
input directly to `F.linear` is intentionally invalid because it bypasses the
module's required `pre_quant_scale`; the full layer replay above is the valid
gate and is exact.

## Current Runtime Scope

| Component | Implemented | Known limitation |
| --- | --- | --- |
| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags |
| Token refiner | Yes | Architecture ported; no completed direct-versus-Comfy refiner tensor comparison |
| Prompt-only FL2VA packer | Yes | No keyframe/reference condition rows; all preview work is text-only FL2VA |
| H3 DiT | Yes | Strict all-block numeric parity not achieved |
| Beta/RES sampler | Yes | Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state |
| Video VAE decoder | Yes | No direct-versus-Comfy same-latent pixel comparison yet |
| Audio VAE/decode/mux | No | Preview intentionally produces video only |
| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented |

## Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete
unless the checkpoint, Comfy version, prompt, or backend changes.

1. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner,
   block 0, block 1, and final refined states once, then compare direct using
   the matched Qwen layer-50 tensor.
2. **H3 assembled input comparison.** Use the existing FL2VA input capture to
   compare text rows, audio/video rows, positions, times, and segments. This
   isolates packing from DiT error.
3. **H3 first-divergence repair.** Reuse the existing block trace. Start at
   block 0 sublayers, repair the first mismatch, then use Comfy block output as
   the next direct input to distinguish local error from accumulation. Existing
   evidence says block 0 is close but accumulation is unresolved.
4. **Final rows and video latent.** Compare final packed video rows and
   `unpatchify_video` output with Comfy before entering the sampler.
5. **Sampler replay using Comfy state.** Feed `initial.pt` and captured
   `sigmas` directly to the sampler. Compare video and audio denoised/update
   states for all 12 saved steps. Do not test seed equality until this passes.
6. **VAE pixel gate.** Decode the identical captured final video latent in
   direct and Comfy and compare pixels before ffmpeg. This decides whether any
   residual grid comes from decoder behavior.
7. **End-to-end FL2VA preview.** Generate the reference dragon only after
   gates 1-8 pass. Compare its raw final latent first, then video.
8. **Feature/performance work.** Only then add audio, Ref2VA/reference paths,
    Sage3, CUDA graphs, and multi-GPU execution.

## Existing Tools And Their Intended Gate

| Gate | Tools |
| --- | --- |
| Qwen captures | `patch_comfy_qwen_output_capture.py`, `patch_comfy_qwen_layer_trace.py`, `patch_comfy_qwen_layer0_sublayers.py`, `patch_comfy_qwen_layer0_projections.py` |
| Qwen projection gate | `compare_qwen0_projections.py` |
| Qwen local diagnosis | `trace_qwen0_attention.py` needs repair before reuse; it currently passes an obsolete encoder argument |
| H3 block capture | `patch_comfy_h3_capture.py`, `patch_comfy_h3_block_capture.py`, `patch_comfy_h3_block0_sublayers.py` |
| H3 local diagnosis | `compare_block0_qkv.py`, `compare_attention_backends.py`, `compare_block0_mlp_projections.py`, `localize_block_sublayers.py`, `localize_block_mismatch.py` |
| Sampler | `patch_comfy_sampler_capture.py`, `compare_fl2va_steps.py` |
| VAE | `smoke_h3_vae_decoder.py`; a same-latent direct-versus-Comfy pixel comparator still needs to be added |

The `patch_comfy_*` tools modify `/opt/ComfyUI` and are only safe against the
restorable Comfy reference image. Capture provenance must be appended here at
the time a new capture is made.