h3-blackwell-runtime/PARITY.md

# H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so
we do not repeat investigations. A source review is never recorded as a passed
numeric gate.

## Reference Runs And Immutable Artifacts

| Asset | Location on Spark | Contents | Status |
| --- | --- | --- | --- |
| Qwen trace | `/tmp/fl2va-qwen-trace` | `qwen_input_embeds.pt`, 50 decoder outputs in `qwen_layers/00.pt` through `49.pt`, and `qwen_layer50.pt` | Complete, immutable Comfy capture |
| Qwen layer-0 trace | `/tmp/fl2va-qwen0-capture` | `qwen0_norm1.pt`, attention, post-attention, norm2, MLP, output | Complete, immutable Comfy capture |
| H3 full trace | `/tmp/fl2va-full-capture` | assembled DiT input/output and `blocks/00.pt` through `49.pt` | Complete, immutable Comfy capture |
| Working-copy capture mirror | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture` | Qwen layer-0 and layer-50 assets plus H3 input/output | Present; do not treat as immutable if the working checkout is modified |
| Sampler reference | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference` | `initial.pt`, exact `sigmas`, `step_00.pt` through `step_11.pt` | Complete, immutable Comfy capture |

All captures above use the established prompt-only FL2VA reference:

```text
prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)
```

This reference is **not** the 72-frame cat preview. Do not overwrite it to
diagnose preview output.

## Completed Findings

| Boundary | Evidence | Result | Conclusion |
| --- | --- | --- | --- |
| Direct runtime build | GB10 CUDA 13 image, Torch `2.9.1+cu130`, committed SageAttention3 wheel | Repeated direct runs complete | Environment is operational; PyTorch's `sm_121` warning is stale metadata in this environment |
| H3 attention baseline | Comfy `--use-sage-attention` versus direct backend investigation | SageAttention2 (`sage2`) was established as the H3 correctness baseline | Use `sage2` for all H3 parity gates; Sage3 remains experimental |
| Qwen tokenizer | Direct raw-prompt tokenizer versus MiniMax H3 source contract | Raw text, no chat template, `add_special_tokens=False` | Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token `151643` |
| Qwen embedding loading, before fix | Direct versus Comfy input embedding capture | Example direct values were approximately `11` where Comfy was approximately `0.017`; downstream refined-text mean absolute delta `1.79263` | Root cause was omitted per-row `model.embed_tokens.weight_scale` |
| Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference |
| Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning |
| Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material |
| Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance |
| Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE |
| Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA |
| Qwen SDPA dispatch audit and fix | Actual Spark Comfy block audit found direct raw SDPA omitted Comfy's four-dimensional mask wrapper, backend priority, and masked-GQA fallback. Direct ported that policy without importing Comfy. | Replaying immutable layer-0 capture after the change: attention mean/max `0.000244372` / `0.0161562`; post-attention `0.00026845` / `0.0213833`; layer output `0.000808998` / `0.0526199`. Previous attention mean was about `0.0357`; previous layer-output mean about `0.1496`. | Active code mismatch identified and substantially repaired. The remaining layer output error is now downstream of projection/elementwise BF16 boundaries, not an unreviewed SDPA dispatcher difference. |
| Qwen NVFP4 metadata | Qwen layer-0 checkpoint sidecars inspected | Every Q/K/V/O and MLP projection has `full_precision_matrix_mult: true`; `o_proj` and `down_proj` also have `pre_quant_scale` | Direct was incorrectly quantizing activations for all Qwen projections |
| Qwen NVFP4 fix | Commit `0cad1db` | Direct honors `full_precision_matrix_mult` and loads/applies `pre_quant_scale`; Spark smoke passed finite expected-shape Q and O projections | Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required |
| Qwen layer-0 projections, after NVFP4 fix | Fresh Comfy projection capture at `/tmp/fl2va-qwen0-projections`; direct compared from captured input embedding and post-attention state | Q mean/max `6.6547e-05` / `0.00195485`; K `9.11704e-05` / `0.00241077`; V `6.0298e-05` / `0.00102112`; gate `0.000365695` / `0.0082469`; up `0.000335494` / `0.00792789`; activated `4.76882e-05` / `0.031621`; down `0.000557594` / `0.0749016` | The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; `down_proj` is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case. |
| Qwen projection code audit, post-fix | Statement-level comparison against actual Spark Comfy commit `43cb4fffc89bba20ab7bd61467a36d0339338dab`, Torch `2.9.1+cu130`, and `/h3-models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` | All 350 Qwen projection sidecars set `full_precision_matrix_mult: true`; exactly 100 (`o_proj` and `down_proj` in each layer) have BF16 `pre_quant_scale`; no Qwen projection has bias. Packed weight is contiguous uint8, block scale is native E4M3, tensor scale is scalar FP32. | Direct and Comfy agree on the active full-precision branch, NVFP4 scale association, logical shapes, no-bias behavior, pre-scale order, BF16 compute dtype, and use the same installed Comfy Kitchen NVFP4 dequantizer. Do not change projection math based on speculation. |
| H3 DiT block trace | `/tmp/fl2va-full-capture` | Block 0 was close, around mean absolute delta `0.00018`; small independent differences accumulated across blocks | Full 50-block strict parity remains failed/unresolved |
| Sampler initial state | Comfy `initial.pt` inspected against direct preview | Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero | Known direct mismatch; identical integer seed is not sampler parity |
| Preview output, before Qwen fixes | Dragon/locomotive prompt previews | Flower-like output | Confirmed incorrect conditioning had visible semantic impact |
| Preview output, after Qwen metadata fix | 72-frame cat, 12-step direct preview | Cat semantic structure appears, with persistent 32-pixel-like square artifacts | Conditioning improved; denoiser/sampler/VAE boundary remains non-parity |
| Video encoder | Direct ffmpeg raw RGB to H.264/yuv420p, no audio | Different from Comfy `CreateVideo`/`SaveVideo` | Not a valid latent-quality gate; cannot explain model-space corruption by itself |

## Qwen Projection Code Audit: Remaining Non-Active Differences

These are real implementation differences, but the audit established they are
not active numerical branches for the captured Qwen checkpoint. They must not
be presented as an explanation for the current `down_proj` delta without new
evidence.

| Direct location | Difference from Comfy | Active for current capture? |
| --- | --- | --- |
| `nvfp4.py:47-51` | Calls `.contiguous()` while Comfy preserves loaded storage/view identity. In particular Comfy treats E4M3 block scales as a dtype view, not a numeric cast. | No: checkpoint tensors are already contiguous and direct also uses a dtype view. |
| `nvfp4.py:75-76` | Always flattens and contiguates activation rows; Comfy full-precision `F.linear` retains N-D input/stride. | No known value effect: elements and BF16 dtype are unchanged. |
| `nvfp4.py:82` | Uses `packed_weight.dequantize().to(flat_x)` rather than an explicitly named Comfy compute-dtype dequantization call. | No: input and output compute dtype are BF16, and both call the same installed Kitchen layout/dequantizer. |
| `nvfp4.py:46,81` | Does not implement Comfy global full-precision overrides, disabled-format controls, forced casts, weight functions, or LoRA/offload dispatch. | No: every Qwen sidecar itself forces full precision, no projection has bias, and no patches/LoRAs are attached. |

**Audit conclusion:** projection source code contains cleanup/generalization gaps,
but no identified active branch mismatch explains the current captured values.
The next code audit must move outward to the exact Qwen block boundary:
Comfy input dtype, full-precision weight construction, SDPA backend selection,
and residual/BF16 rounding boundaries. Do not run another full inference merely
to rediscover this projection result.

## Qwen Layer-0 Arithmetic Audit After `4050afe`

The follow-up statement audit covered every remaining active layer-0 arithmetic
boundary outside the already-fixed SDPA wrapper: full-precision NVFP4 dispatch,
weight/scales, `pre_quant_scale`, BF16 GEMM, activation layout, Q/K RMSNorm,
O projection/residual, post-attention norm, SiLU/gate/up product, down
projection, and final residual.

**Result: no further source-proven active mismatch was found.** For the actual
checkpoint, Comfy and direct both use BF16 input, BF16 NVFP4 dequantization,
BF16 `F.linear`, no projection biases, identical pre-scale ordering, identical
MLP evaluation order, and ordinary BF16 residual additions. Explicit flattening
and `.contiguous()` in direct are not a demonstrated numerical difference for
the captured contiguous batch-one input.

Do not modify `nvfp4.py` or Qwen MLP/residual code to chase the remaining
layer-0 delta. The next justified work is a bounded component experiment that
identifies the actual selected SDPA kernel/GQA fallback and verifies capture
consistency; it is not another full denoising or video run.

### Bounded SDPA/Layer-0 Result

The bounded replay was completed with the fresh projection-capture tensors.
For the reference's 17-token layer-0 attention, both paths have native cuDNN
GQA available (`flash=False`, `cudnn=True`, `efficient=False`). Therefore the
direct SDPA wrapper takes the same native-GQA branch Comfy takes.

| Boundary | Mean absolute error | Maximum absolute error |
| --- | ---: | ---: |
| O-projected attention | `0.000244372` | `0.0161562` |
| First residual | `0.00026845` | `0.0213833` |
| Post-attention RMSNorm | `3.87714e-05` | `0.000426032` |
| MLP output | `0.000674194` | `0.0500984` |
| Layer-0 output | `0.000808998` | `0.0526199` |

This confirms the residual error is already present in the captured projection
outputs and is amplified by the MLP. The NVFP4 and MLP code audits found no
remaining active source-level branch difference. Further edits to flattening,
contiguity, dequantization dtype, `pre_quant_scale`, SiLU ordering, or residual
adds would be speculative and are prohibited until a new code-path difference
is demonstrated.

## Current Runtime Scope

| Component | Implemented | Known limitation |
| --- | --- | --- |
| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags |
| Token refiner | Yes | Architecture ported; no completed direct-versus-Comfy refiner tensor comparison |
| Prompt-only FL2VA packer | Yes | No keyframe/reference condition rows; all preview work is text-only FL2VA |
| H3 DiT | Yes | Strict all-block numeric parity not achieved |
| Beta/RES sampler | Yes | Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state |
| Video VAE decoder | Yes | No direct-versus-Comfy same-latent pixel comparison yet |
| Audio VAE/decode/mux | No | Preview intentionally produces video only |
| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented |

## Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete
unless the checkpoint, Comfy version, prompt, or backend changes.

1. **Qwen all-layer replay after `0cad1db`.** The layer-0 projection capture
   was completed: errors are recorded above. Replay the existing 50 layer
   tensors in `/tmp/fl2va-qwen-trace`; report first failing layer and mean/max
   error. Do not generate another video before this is recorded.
2. **Qwen projection precision follow-up if layer replay fails at layer 0.**
   Do not repeat the capture. Use `/tmp/fl2va-qwen0-projections` to isolate the
   remaining `down_proj` max error `0.0749016` and its BF16/dequantization
   boundary.
3. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner,
   block 0, block 1, and final refined states once, then compare direct using
   the matched Qwen layer-50 tensor.
4. **H3 assembled input comparison.** Use the existing FL2VA input capture to
   compare text rows, audio/video rows, positions, times, and segments. This
   isolates packing from DiT error.
5. **H3 first-divergence repair.** Reuse the existing block trace. Start at
   block 0 sublayers, repair the first mismatch, then use Comfy block output as
   the next direct input to distinguish local error from accumulation. Existing
   evidence says block 0 is close but accumulation is unresolved.
6. **Final rows and video latent.** Compare final packed video rows and
   `unpatchify_video` output with Comfy before entering the sampler.
7. **Sampler replay using Comfy state.** Feed `initial.pt` and captured
   `sigmas` directly to the sampler. Compare video and audio denoised/update
   states for all 12 saved steps. Do not test seed equality until this passes.
8. **VAE pixel gate.** Decode the identical captured final video latent in
   direct and Comfy and compare pixels before ffmpeg. This decides whether any
   residual grid comes from decoder behavior.
9. **End-to-end FL2VA preview.** Generate the reference dragon only after
   gates 1-8 pass. Compare its raw final latent first, then video.
10. **Feature/performance work.** Only then add audio, Ref2VA/reference paths,
    Sage3, CUDA graphs, and multi-GPU execution.

## Existing Tools And Their Intended Gate

| Gate | Tools |
| --- | --- |
| Qwen captures | `patch_comfy_qwen_output_capture.py`, `patch_comfy_qwen_layer_trace.py`, `patch_comfy_qwen_layer0_sublayers.py`, `patch_comfy_qwen_layer0_projections.py` |
| Qwen projection gate | `compare_qwen0_projections.py` |
| Qwen local diagnosis | `trace_qwen0_attention.py` needs repair before reuse; it currently passes an obsolete encoder argument |
| H3 block capture | `patch_comfy_h3_capture.py`, `patch_comfy_h3_block_capture.py`, `patch_comfy_h3_block0_sublayers.py` |
| H3 local diagnosis | `compare_block0_qkv.py`, `compare_attention_backends.py`, `compare_block0_mlp_projections.py`, `localize_block_sublayers.py`, `localize_block_mismatch.py` |
| Sampler | `patch_comfy_sampler_capture.py`, `compare_fl2va_steps.py` |
| VAE | `smoke_h3_vae_decoder.py`; a same-latent direct-versus-Comfy pixel comparator still needs to be added |

The `patch_comfy_*` tools modify `/opt/ComfyUI` and are only safe against the
restorable Comfy reference image. Capture provenance must be appended here at
the time a new capture is made.