h3-blackwell-runtime/PARITY.md

H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so we do not repeat investigations. A source review is never recorded as a passed numeric gate.

Reference Runs And Immutable Artifacts

Asset	Location on Spark	Contents	Status
Qwen trace	/tmp/fl2va-qwen-trace	qwen_input_embeds.pt, 50 decoder outputs in qwen_layers/00.pt through 49.pt, and qwen_layer50.pt	Complete, immutable Comfy capture
Qwen layer-0 trace	/tmp/fl2va-qwen0-capture	qwen0_norm1.pt, attention, post-attention, norm2, MLP, output	Complete, immutable Comfy capture
H3 full trace	/tmp/fl2va-full-capture	assembled DiT input/output and blocks/00.pt through 49.pt	Complete, immutable Comfy capture
Working-copy capture mirror	/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture	Qwen layer-0 and layer-50 assets plus H3 input/output	Present; do not treat as immutable if the working checkout is modified
Sampler reference	/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference	initial.pt, exact sigmas, step_00.pt through step_11.pt	Complete, immutable Comfy capture

All captures above use the established prompt-only FL2VA reference:

prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)

This reference is not the 72-frame cat preview. Do not overwrite it to diagnose preview output.

Completed Findings

Boundary	Evidence	Result	Conclusion
Direct runtime build	GB10 CUDA 13 image, Torch 2.9.1+cu130, committed SageAttention3 wheel	Repeated direct runs complete	Environment is operational; PyTorch's sm_121 warning is stale metadata in this environment
H3 attention baseline	Comfy --use-sage-attention versus direct backend investigation	SageAttention2 (sage2) was established as the H3 correctness baseline	Use sage2 for all H3 parity gates; Sage3 remains experimental
Qwen tokenizer	Direct raw-prompt tokenizer versus MiniMax H3 source contract	Raw text, no chat template, add_special_tokens=False	Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token 151643
Qwen embedding loading, before fix	Direct versus Comfy input embedding capture	Example direct values were approximately 11 where Comfy was approximately 0.017; downstream refined-text mean absolute delta 1.79263	Root cause was omitted per-row model.embed_tokens.weight_scale
Qwen embedding loading, after fix	Direct rows multiplied by weight_scale in FP32	Input embeddings matched the Comfy capture exactly	Passed for the captured nonempty text-only reference
Qwen layer-50, before embedding fix	Direct versus Comfy layer-50 capture	Mean absolute delta 23.50596, max 38977.09375	Failed; invalidated direct text conditioning
Qwen layer-50, after embedding fix but before projection fix	Direct versus Comfy layer-50 capture	Mean absolute delta improved to about 5.63	Embedding issue fixed; remaining error was material
Qwen layer-0 normalization	Direct F.rms_norm versus qwen0_norm1.pt	Mean absolute delta about 8.96e-06	Passed to expected BF16-level tolerance
Qwen layer-0 attention, before projection fix	Direct versus qwen0_attention.pt	Mean absolute delta about 0.0357; layer output about 0.1496	First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE
Qwen attention backend	Comfy source inspection	Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2	Direct Qwen was changed to BF16 causal-mask SDPA with GQA
Qwen NVFP4 metadata	Qwen layer-0 checkpoint sidecars inspected	Every Q/K/V/O and MLP projection has full_precision_matrix_mult: true; o_proj and down_proj also have pre_quant_scale	Direct was incorrectly quantizing activations for all Qwen projections
Qwen NVFP4 fix	Commit 0cad1db	Direct honors full_precision_matrix_mult and loads/applies pre_quant_scale; Spark smoke passed finite expected-shape Q and O projections	Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required
Qwen layer-0 projections, after NVFP4 fix	Fresh Comfy projection capture at /tmp/fl2va-qwen0-projections; direct compared from captured input embedding and post-attention state	Q mean/max 6.6547e-05 / 0.00195485; K 9.11704e-05 / 0.00241077; V 6.0298e-05 / 0.00102112; gate 0.000365695 / 0.0082469; up 0.000335494 / 0.00792789; activated 4.76882e-05 / 0.031621; down 0.000557594 / 0.0749016	The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; down_proj is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case.
H3 DiT block trace	/tmp/fl2va-full-capture	Block 0 was close, around mean absolute delta 0.00018; small independent differences accumulated across blocks	Full 50-block strict parity remains failed/unresolved
Sampler initial state	Comfy initial.pt inspected against direct preview	Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero	Known direct mismatch; identical integer seed is not sampler parity
Preview output, before Qwen fixes	Dragon/locomotive prompt previews	Flower-like output	Confirmed incorrect conditioning had visible semantic impact
Preview output, after Qwen metadata fix	72-frame cat, 12-step direct preview	Cat semantic structure appears, with persistent 32-pixel-like square artifacts	Conditioning improved; denoiser/sampler/VAE boundary remains non-parity
Video encoder	Direct ffmpeg raw RGB to H.264/yuv420p, no audio	Different from Comfy CreateVideo/SaveVideo	Not a valid latent-quality gate; cannot explain model-space corruption by itself

Current Runtime Scope

Component	Implemented	Known limitation
Text-only Qwen	Yes	No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags
Token refiner	Yes	Architecture ported; no completed direct-versus-Comfy refiner tensor comparison
Prompt-only FL2VA packer	Yes	No keyframe/reference condition rows; all preview work is text-only FL2VA
H3 DiT	Yes	Strict all-block numeric parity not achieved
Beta/RES sampler	Yes	Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state
Video VAE decoder	Yes	No direct-versus-Comfy same-latent pixel comparison yet
Audio VAE/decode/mux	No	Preview intentionally produces video only
Full Ref2VA	No	References, vision conditioning, VAE encode, audio, and muxing are unimplemented

Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete unless the checkpoint, Comfy version, prompt, or backend changes.

1. Qwen all-layer replay after 0cad1db. The layer-0 projection capture was completed: errors are recorded above. Replay the existing 50 layer tensors in /tmp/fl2va-qwen-trace; report first failing layer and mean/max error. Do not generate another video before this is recorded.
2. Qwen projection precision follow-up if layer replay fails at layer 0. Do not repeat the capture. Use /tmp/fl2va-qwen0-projections to isolate the remaining down_proj max error 0.0749016 and its BF16/dequantization boundary.
3. Token-refiner capture and comparison. Capture exact Comfy pre-refiner, block 0, block 1, and final refined states once, then compare direct using the matched Qwen layer-50 tensor.
4. H3 assembled input comparison. Use the existing FL2VA input capture to compare text rows, audio/video rows, positions, times, and segments. This isolates packing from DiT error.
5. H3 first-divergence repair. Reuse the existing block trace. Start at block 0 sublayers, repair the first mismatch, then use Comfy block output as the next direct input to distinguish local error from accumulation. Existing evidence says block 0 is close but accumulation is unresolved.
6. Final rows and video latent. Compare final packed video rows and unpatchify_video output with Comfy before entering the sampler.
7. Sampler replay using Comfy state. Feed initial.pt and captured sigmas directly to the sampler. Compare video and audio denoised/update states for all 12 saved steps. Do not test seed equality until this passes.
8. VAE pixel gate. Decode the identical captured final video latent in direct and Comfy and compare pixels before ffmpeg. This decides whether any residual grid comes from decoder behavior.
9. End-to-end FL2VA preview. Generate the reference dragon only after gates 1-8 pass. Compare its raw final latent first, then video.
10. Feature/performance work. Only then add audio, Ref2VA/reference paths, Sage3, CUDA graphs, and multi-GPU execution.

Existing Tools And Their Intended Gate

Gate	Tools
Qwen captures	patch_comfy_qwen_output_capture.py, patch_comfy_qwen_layer_trace.py, patch_comfy_qwen_layer0_sublayers.py, patch_comfy_qwen_layer0_projections.py
Qwen projection gate	compare_qwen0_projections.py
Qwen local diagnosis	trace_qwen0_attention.py needs repair before reuse; it currently passes an obsolete encoder argument
H3 block capture	patch_comfy_h3_capture.py, patch_comfy_h3_block_capture.py, patch_comfy_h3_block0_sublayers.py
H3 local diagnosis	compare_block0_qkv.py, compare_attention_backends.py, compare_block0_mlp_projections.py, localize_block_sublayers.py, localize_block_mismatch.py
Sampler	patch_comfy_sampler_capture.py, compare_fl2va_steps.py
VAE	smoke_h3_vae_decoder.py; a same-latent direct-versus-Comfy pixel comparator still needs to be added

The patch_comfy_* tools modify /opt/ComfyUI and are only safe against the restorable Comfy reference image. Capture provenance must be appended here at the time a new capture is made.