From eaca0034e4a6dbe290c9a8c6ce6540b59e6a609b Mon Sep 17 00:00:00 2001 From: Daniel Maddern Date: Wed, 12 Aug 2026 21:50:25 +0700 Subject: [PATCH] Record H3 parity evidence --- PARITY.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 PARITY.md diff --git a/PARITY.md b/PARITY.md new file mode 100644 index 0000000..3ad69d3 --- /dev/null +++ b/PARITY.md @@ -0,0 +1,113 @@ +# H3 Parity Evidence Ledger + +ComfyUI is the oracle. This ledger records completed work and measured facts so +we do not repeat investigations. A source review is never recorded as a passed +numeric gate. + +## Reference Runs And Immutable Artifacts + +| Asset | Location on Spark | Contents | Status | +| --- | --- | --- | --- | +| Qwen trace | `/tmp/fl2va-qwen-trace` | `qwen_input_embeds.pt`, 50 decoder outputs in `qwen_layers/00.pt` through `49.pt`, and `qwen_layer50.pt` | Complete, immutable Comfy capture | +| Qwen layer-0 trace | `/tmp/fl2va-qwen0-capture` | `qwen0_norm1.pt`, attention, post-attention, norm2, MLP, output | Complete, immutable Comfy capture | +| H3 full trace | `/tmp/fl2va-full-capture` | assembled DiT input/output and `blocks/00.pt` through `49.pt` | Complete, immutable Comfy capture | +| Working-copy capture mirror | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture` | Qwen layer-0 and layer-50 assets plus H3 input/output | Present; do not treat as immutable if the working checkout is modified | +| Sampler reference | `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference` | `initial.pt`, exact `sigmas`, `step_00.pt` through `step_11.pt` | Complete, immutable Comfy capture | + +All captures above use the established prompt-only FL2VA reference: + +```text +prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour. +size: 320x192, length: 22 frames +seed: 440204 +sampler: beta / res_multistep / 12 steps +Comfy attention baseline: --use-sage-attention (SageAttention2) +``` + +This reference is **not** the 72-frame cat preview. Do not overwrite it to +diagnose preview output. + +## Completed Findings + +| Boundary | Evidence | Result | Conclusion | +| --- | --- | --- | --- | +| Direct runtime build | GB10 CUDA 13 image, Torch `2.9.1+cu130`, committed SageAttention3 wheel | Repeated direct runs complete | Environment is operational; PyTorch's `sm_121` warning is stale metadata in this environment | +| H3 attention baseline | Comfy `--use-sage-attention` versus direct backend investigation | SageAttention2 (`sage2`) was established as the H3 correctness baseline | Use `sage2` for all H3 parity gates; Sage3 remains experimental | +| Qwen tokenizer | Direct raw-prompt tokenizer versus MiniMax H3 source contract | Raw text, no chat template, `add_special_tokens=False` | Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token `151643` | +| Qwen embedding loading, before fix | Direct versus Comfy input embedding capture | Example direct values were approximately `11` where Comfy was approximately `0.017`; downstream refined-text mean absolute delta `1.79263` | Root cause was omitted per-row `model.embed_tokens.weight_scale` | +| Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference | +| Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning | +| Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material | +| Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance | +| Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE | +| Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA | +| Qwen NVFP4 metadata | Qwen layer-0 checkpoint sidecars inspected | Every Q/K/V/O and MLP projection has `full_precision_matrix_mult: true`; `o_proj` and `down_proj` also have `pre_quant_scale` | Direct was incorrectly quantizing activations for all Qwen projections | +| Qwen NVFP4 fix | Commit `0cad1db` | Direct honors `full_precision_matrix_mult` and loads/applies `pre_quant_scale`; Spark smoke passed finite expected-shape Q and O projections | Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required | +| H3 DiT block trace | `/tmp/fl2va-full-capture` | Block 0 was close, around mean absolute delta `0.00018`; small independent differences accumulated across blocks | Full 50-block strict parity remains failed/unresolved | +| Sampler initial state | Comfy `initial.pt` inspected against direct preview | Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero | Known direct mismatch; identical integer seed is not sampler parity | +| Preview output, before Qwen fixes | Dragon/locomotive prompt previews | Flower-like output | Confirmed incorrect conditioning had visible semantic impact | +| Preview output, after Qwen metadata fix | 72-frame cat, 12-step direct preview | Cat semantic structure appears, with persistent 32-pixel-like square artifacts | Conditioning improved; denoiser/sampler/VAE boundary remains non-parity | +| Video encoder | Direct ffmpeg raw RGB to H.264/yuv420p, no audio | Different from Comfy `CreateVideo`/`SaveVideo` | Not a valid latent-quality gate; cannot explain model-space corruption by itself | + +## Current Runtime Scope + +| Component | Implemented | Known limitation | +| --- | --- | --- | +| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags | +| Token refiner | Yes | Architecture ported; no completed direct-versus-Comfy refiner tensor comparison | +| Prompt-only FL2VA packer | Yes | No keyframe/reference condition rows; all preview work is text-only FL2VA | +| H3 DiT | Yes | Strict all-block numeric parity not achieved | +| Beta/RES sampler | Yes | Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state | +| Video VAE decoder | Yes | No direct-versus-Comfy same-latent pixel comparison yet | +| Audio VAE/decode/mux | No | Preview intentionally produces video only | +| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented | + +## Remaining Gates, In Dependency Order + +Only these are outstanding. Do not recapture or revisit rows marked complete +unless the checkpoint, Comfy version, prompt, or backend changes. + +1. **Qwen projection replay after `0cad1db`.** Use the existing immutable + layer-0 capture. Compare direct Q/K/V, O, gate, up, activated, and down + against Comfy. This is the first unresolved boundary after the completed + embedding/RMSNorm/backend investigations. +2. **Qwen all-layer replay after the projection gate passes.** Replay the + existing 50 layer tensors in `/tmp/fl2va-qwen-trace`; report first failing + layer and mean/max error. Do not generate another video before this passes. +3. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner, + block 0, block 1, and final refined states once, then compare direct using + the matched Qwen layer-50 tensor. +4. **H3 assembled input comparison.** Use the existing FL2VA input capture to + compare text rows, audio/video rows, positions, times, and segments. This + isolates packing from DiT error. +5. **H3 first-divergence repair.** Reuse the existing block trace. Start at + block 0 sublayers, repair the first mismatch, then use Comfy block output as + the next direct input to distinguish local error from accumulation. Existing + evidence says block 0 is close but accumulation is unresolved. +6. **Final rows and video latent.** Compare final packed video rows and + `unpatchify_video` output with Comfy before entering the sampler. +7. **Sampler replay using Comfy state.** Feed `initial.pt` and captured + `sigmas` directly to the sampler. Compare video and audio denoised/update + states for all 12 saved steps. Do not test seed equality until this passes. +8. **VAE pixel gate.** Decode the identical captured final video latent in + direct and Comfy and compare pixels before ffmpeg. This decides whether any + residual grid comes from decoder behavior. +9. **End-to-end FL2VA preview.** Generate the reference dragon only after + gates 1-8 pass. Compare its raw final latent first, then video. +10. **Feature/performance work.** Only then add audio, Ref2VA/reference paths, + Sage3, CUDA graphs, and multi-GPU execution. + +## Existing Tools And Their Intended Gate + +| Gate | Tools | +| --- | --- | +| Qwen captures | `patch_comfy_qwen_output_capture.py`, `patch_comfy_qwen_layer_trace.py`, `patch_comfy_qwen_layer0_sublayers.py` | +| Qwen local diagnosis | `trace_qwen0_attention.py` needs repair before reuse; it currently passes an obsolete encoder argument | +| H3 block capture | `patch_comfy_h3_capture.py`, `patch_comfy_h3_block_capture.py`, `patch_comfy_h3_block0_sublayers.py` | +| H3 local diagnosis | `compare_block0_qkv.py`, `compare_attention_backends.py`, `compare_block0_mlp_projections.py`, `localize_block_sublayers.py`, `localize_block_mismatch.py` | +| Sampler | `patch_comfy_sampler_capture.py`, `compare_fl2va_steps.py` | +| VAE | `smoke_h3_vae_decoder.py`; a same-latent direct-versus-Comfy pixel comparator still needs to be added | + +The `patch_comfy_*` tools modify `/opt/ComfyUI` and are only safe against the +restorable Comfy reference image. Capture provenance must be appended here at +the time a new capture is made.