h3-blackwell-runtime/PARITY.md

# H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so
we do not repeat investigations. A source review is never recorded as a passed
numeric gate.

## Reference Runs And Immutable Artifacts

| Asset | Location on Spark | Contents | Status |
| --- | --- | --- | --- |
| Qwen trace | `/tmp/fl2va-qwen-trace` | `qwen_input_embeds.pt`, 50 decoder outputs in `qwen_layers/00.pt` through `49.pt`, and `qwen_layer50.pt` | Complete, immutable Comfy capture |
| Qwen layer-0 trace | `/tmp/fl2va-qwen0-capture` | `qwen0_norm1.pt`, attention, post-attention, norm2, MLP, output | Complete, immutable Comfy capture |
| H3 full trace | `/tmp/fl2va-full-capture` | assembled DiT input/output and `blocks/00.pt` through `49.pt` | Complete, immutable Comfy capture |
| Working-copy capture mirror | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture` | Qwen layer-0 and layer-50 assets plus H3 input/output | Present; do not treat as immutable if the working checkout is modified |
| Sampler reference | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference` | `initial.pt`, exact `sigmas`, `step_00.pt` through `step_11.pt` | Complete, immutable Comfy capture |

All captures above use the established prompt-only FL2VA reference:

```text
prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)
```

This reference is **not** the 72-frame cat preview. Do not overwrite it to
diagnose preview output.

## Completed Findings

| Boundary | Evidence | Result | Conclusion |
| --- | --- | --- | --- |
| Direct runtime build | GB10 CUDA 13 image, Torch `2.9.1+cu130`, committed SageAttention3 wheel | Repeated direct runs complete | Environment is operational; PyTorch's `sm_121` warning is stale metadata in this environment |
| H3 attention baseline | Comfy `--use-sage-attention` versus direct backend investigation | SageAttention2 (`sage2`) was established as the H3 correctness baseline | Use `sage2` for all H3 parity gates; Sage3 remains experimental |
| Qwen tokenizer | Direct raw-prompt tokenizer versus MiniMax H3 source contract | Raw text, no chat template, `add_special_tokens=False` | Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token `151643` |
| Qwen embedding loading, before fix | Direct versus Comfy input embedding capture | Example direct values were approximately `11` where Comfy was approximately `0.017`; downstream refined-text mean absolute delta `1.79263` | Root cause was omitted per-row `model.embed_tokens.weight_scale` |
| Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference |
| Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning |
| Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material |
| Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance |
| Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE |
| Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA |
| Qwen NVFP4 metadata | Qwen layer-0 checkpoint sidecars inspected | Every Q/K/V/O and MLP projection has `full_precision_matrix_mult: true`; `o_proj` and `down_proj` also have `pre_quant_scale` | Direct was incorrectly quantizing activations for all Qwen projections |
| Qwen NVFP4 fix | Commit `0cad1db` | Direct honors `full_precision_matrix_mult` and loads/applies `pre_quant_scale`; Spark smoke passed finite expected-shape Q and O projections | Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required |
| Qwen layer-0 projections, after NVFP4 fix | Fresh Comfy projection capture at `/tmp/fl2va-qwen0-projections`; direct compared from captured input embedding and post-attention state | Q mean/max `6.6547e-05` / `0.00195485`; K `9.11704e-05` / `0.00241077`; V `6.0298e-05` / `0.00102112`; gate `0.000365695` / `0.0082469`; up `0.000335494` / `0.00792789`; activated `4.76882e-05` / `0.031621`; down `0.000557594` / `0.0749016` | The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; `down_proj` is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case. |
| H3 DiT block trace | `/tmp/fl2va-full-capture` | Block 0 was close, around mean absolute delta `0.00018`; small independent differences accumulated across blocks | Full 50-block strict parity remains failed/unresolved |
| Sampler initial state | Comfy `initial.pt` inspected against direct preview | Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero | Known direct mismatch; identical integer seed is not sampler parity |
| Preview output, before Qwen fixes | Dragon/locomotive prompt previews | Flower-like output | Confirmed incorrect conditioning had visible semantic impact |
| Preview output, after Qwen metadata fix | 72-frame cat, 12-step direct preview | Cat semantic structure appears, with persistent 32-pixel-like square artifacts | Conditioning improved; denoiser/sampler/VAE boundary remains non-parity |
| Video encoder | Direct ffmpeg raw RGB to H.264/yuv420p, no audio | Different from Comfy `CreateVideo`/`SaveVideo` | Not a valid latent-quality gate; cannot explain model-space corruption by itself |

## Current Runtime Scope

| Component | Implemented | Known limitation |
| --- | --- | --- |
| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags |
| Token refiner | Yes | Architecture ported; no completed direct-versus-Comfy refiner tensor comparison |
| Prompt-only FL2VA packer | Yes | No keyframe/reference condition rows; all preview work is text-only FL2VA |
| H3 DiT | Yes | Strict all-block numeric parity not achieved |
| Beta/RES sampler | Yes | Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state |
| Video VAE decoder | Yes | No direct-versus-Comfy same-latent pixel comparison yet |
| Audio VAE/decode/mux | No | Preview intentionally produces video only |
| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented |

## Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete
unless the checkpoint, Comfy version, prompt, or backend changes.

1. **Qwen all-layer replay after `0cad1db`.** The layer-0 projection capture
   was completed: errors are recorded above. Replay the existing 50 layer
   tensors in `/tmp/fl2va-qwen-trace`; report first failing layer and mean/max
   error. Do not generate another video before this is recorded.
2. **Qwen projection precision follow-up if layer replay fails at layer 0.**
   Do not repeat the capture. Use `/tmp/fl2va-qwen0-projections` to isolate the
   remaining `down_proj` max error `0.0749016` and its BF16/dequantization
   boundary.
3. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner,
   block 0, block 1, and final refined states once, then compare direct using
   the matched Qwen layer-50 tensor.
4. **H3 assembled input comparison.** Use the existing FL2VA input capture to
   compare text rows, audio/video rows, positions, times, and segments. This
   isolates packing from DiT error.
5. **H3 first-divergence repair.** Reuse the existing block trace. Start at
   block 0 sublayers, repair the first mismatch, then use Comfy block output as
   the next direct input to distinguish local error from accumulation. Existing
   evidence says block 0 is close but accumulation is unresolved.
6. **Final rows and video latent.** Compare final packed video rows and
   `unpatchify_video` output with Comfy before entering the sampler.
7. **Sampler replay using Comfy state.** Feed `initial.pt` and captured
   `sigmas` directly to the sampler. Compare video and audio denoised/update
   states for all 12 saved steps. Do not test seed equality until this passes.
8. **VAE pixel gate.** Decode the identical captured final video latent in
   direct and Comfy and compare pixels before ffmpeg. This decides whether any
   residual grid comes from decoder behavior.
9. **End-to-end FL2VA preview.** Generate the reference dragon only after
   gates 1-8 pass. Compare its raw final latent first, then video.
10. **Feature/performance work.** Only then add audio, Ref2VA/reference paths,
    Sage3, CUDA graphs, and multi-GPU execution.

## Existing Tools And Their Intended Gate

| Gate | Tools |
| --- | --- |
| Qwen captures | `patch_comfy_qwen_output_capture.py`, `patch_comfy_qwen_layer_trace.py`, `patch_comfy_qwen_layer0_sublayers.py`, `patch_comfy_qwen_layer0_projections.py` |
| Qwen projection gate | `compare_qwen0_projections.py` |
| Qwen local diagnosis | `trace_qwen0_attention.py` needs repair before reuse; it currently passes an obsolete encoder argument |
| H3 block capture | `patch_comfy_h3_capture.py`, `patch_comfy_h3_block_capture.py`, `patch_comfy_h3_block0_sublayers.py` |
| H3 local diagnosis | `compare_block0_qkv.py`, `compare_attention_backends.py`, `compare_block0_mlp_projections.py`, `localize_block_sublayers.py`, `localize_block_mismatch.py` |
| Sampler | `patch_comfy_sampler_capture.py`, `compare_fl2va_steps.py` |
| VAE | `smoke_h3_vae_decoder.py`; a same-latent direct-versus-Comfy pixel comparator still needs to be added |

The `patch_comfy_*` tools modify `/opt/ComfyUI` and are only safe against the
restorable Comfy reference image. Capture provenance must be appended here at
the time a new capture is made.