h3-blackwell-runtime/PARITY.md

H3 Parity Evidence Ledger

ComfyUI is the oracle. This ledger records completed work and measured facts so we do not repeat investigations. A source review is never recorded as a passed numeric gate.

Reference Runs And Immutable Artifacts

Asset	Location on Spark	Contents	Status
Qwen trace	/tmp/fl2va-qwen-trace	qwen_input_embeds.pt, 50 decoder outputs in qwen_layers/00.pt through 49.pt, and qwen_layer50.pt	Complete, immutable Comfy capture
Qwen layer-0 trace	/tmp/fl2va-qwen0-capture	qwen0_norm1.pt, attention, post-attention, norm2, MLP, output	Complete, immutable Comfy capture
H3 full trace	/tmp/fl2va-full-capture	assembled DiT input/output and blocks/00.pt through 49.pt	Complete, immutable Comfy capture
Working-copy capture mirror	/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/capture	Qwen layer-0 and layer-50 assets plus H3 input/output	Present; do not treat as immutable if the working checkout is modified
Sampler reference	/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference	initial.pt, exact sigmas, step_00.pt through step_11.pt	Complete, immutable Comfy capture

All captures above use the established prompt-only FL2VA reference:

prompt: A brass-and-paper dragon flies above a rain-washed old city at blue hour.
size: 320x192, length: 22 frames
seed: 440204
sampler: beta / res_multistep / 12 steps
Comfy attention baseline: --use-sage-attention (SageAttention2)

This reference is not the 72-frame cat preview. Do not overwrite it to diagnose preview output.

Completed Findings

Boundary	Evidence	Result	Conclusion
Direct runtime build	GB10 CUDA 13 image, Torch 2.9.1+cu130, committed SageAttention3 wheel	Repeated direct runs complete	Environment is operational; PyTorch's sm_121 warning is stale metadata in this environment
H3 attention baseline	Comfy --use-sage-attention versus direct backend investigation	SageAttention2 (sage2) was established as the H3 correctness baseline	Use sage2 for all H3 parity gates; Sage3 remains experimental
Qwen tokenizer	Direct raw-prompt tokenizer versus MiniMax H3 source contract	Raw text, no chat template, add_special_tokens=False	Correct for nonempty text-only prompts; empty prompt differs: direct substitutes a space while Comfy injects token 151643
Qwen embedding loading, before fix	Direct versus Comfy input embedding capture	Example direct values were approximately 11 where Comfy was approximately 0.017; downstream refined-text mean absolute delta 1.79263	Root cause was omitted per-row model.embed_tokens.weight_scale
Qwen embedding loading, after fix	Direct rows multiplied by weight_scale in FP32	Input embeddings matched the Comfy capture exactly	Passed for the captured nonempty text-only reference
Qwen layer-50, before embedding fix	Direct versus Comfy layer-50 capture	Mean absolute delta 23.50596, max 38977.09375	Failed; invalidated direct text conditioning
Qwen layer-50, after embedding fix but before projection fix	Direct versus Comfy layer-50 capture	Mean absolute delta improved to about 5.63	Embedding issue fixed; remaining error was material
Qwen all-layer replay, after FP32 fix	compare_qwen_layer_trace.py replayed /tmp/fl2va-qwen-trace without new Comfy inference	Layers 00 through 49 and qwen_layer50.pt all reported mean/max 0 / 0	Passed: prompt-only Qwen conditioning is bit-exact for the immutable 17-token FL2VA reference.
Qwen layer-0 normalization	Direct F.rms_norm versus qwen0_norm1.pt	Mean absolute delta about 8.96e-06	Passed to expected BF16-level tolerance
Qwen layer-0 attention, before projection fix	Direct versus qwen0_attention.pt	Mean absolute delta about 0.0357; layer output about 0.1496	First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE
Qwen attention backend	Comfy source inspection	Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2	Direct Qwen was changed to BF16 causal-mask SDPA with GQA
Qwen SDPA dispatch audit and fix	Actual Spark Comfy block audit found direct raw SDPA omitted Comfy's four-dimensional mask wrapper, backend priority, and masked-GQA fallback. Direct ported that policy without importing Comfy.	Replaying immutable layer-0 capture after the change: attention mean/max 0.000244372 / 0.0161562; post-attention 0.00026845 / 0.0213833; layer output 0.000808998 / 0.0526199. Previous attention mean was about 0.0357; previous layer-output mean about 0.1496.	Active code mismatch identified and substantially repaired. The remaining layer output error is now downstream of projection/elementwise BF16 boundaries, not an unreviewed SDPA dispatcher difference.
Qwen NVFP4 metadata	Qwen layer-0 checkpoint sidecars inspected	Every Q/K/V/O and MLP projection has full_precision_matrix_mult: true; o_proj and down_proj also have pre_quant_scale	Direct was incorrectly quantizing activations for all Qwen projections
Qwen NVFP4 fix	Commit 0cad1db	Direct honors full_precision_matrix_mult and loads/applies pre_quant_scale; Spark smoke passed finite expected-shape Q and O projections	Structural fix passed; post-fix numeric QKV/MLP/layer-50 comparisons are still required
Qwen layer-0 projections, after NVFP4 fix	Fresh Comfy projection capture at /tmp/fl2va-qwen0-projections; direct compared from captured input embedding and post-attention state	Q mean/max 6.6547e-05 / 0.00195485; K 9.11704e-05 / 0.00241077; V 6.0298e-05 / 0.00102112; gate 0.000365695 / 0.0082469; up 0.000335494 / 0.00792789; activated 4.76882e-05 / 0.031621; down 0.000557594 / 0.0749016	The full-precision metadata fix removed the previous large projection failure. Projection parity is close but not strict; down_proj is the largest remaining layer-0 projection boundary. Preserve these numbers and do not recapture this case.
Qwen projection code audit, post-fix	Statement-level comparison against actual Spark Comfy commit 43cb4fffc89bba20ab7bd61467a36d0339338dab, Torch 2.9.1+cu130, and /h3-models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors	All 350 Qwen projection sidecars set full_precision_matrix_mult: true; exactly 100 (o_proj and down_proj in each layer) have BF16 pre_quant_scale; no Qwen projection has bias. Packed weight is contiguous uint8, block scale is native E4M3, tensor scale is scalar FP32.	Direct and Comfy agree on the active full-precision branch, NVFP4 scale association, logical shapes, no-bias behavior, pre-scale order, BF16 compute dtype, and use the same installed Comfy Kitchen NVFP4 dequantizer. Do not change projection math based on speculation.
H3 DiT block trace	/tmp/fl2va-full-capture	Block 0 was close, around mean absolute delta 0.00018; small independent differences accumulated across blocks	Full 50-block strict parity remains failed/unresolved
Sampler initial state	Comfy initial.pt inspected against direct preview	Comfy carries joint AV state; direct preview initializes video noise only and leaves audio zero	Known direct mismatch; identical integer seed is not sampler parity
Preview output, before Qwen fixes	Dragon/locomotive prompt previews	Flower-like output	Confirmed incorrect conditioning had visible semantic impact
Preview output, after Qwen metadata fix	72-frame cat, 12-step direct preview	Cat semantic structure appears, with persistent 32-pixel-like square artifacts	Conditioning improved; denoiser/sampler/VAE boundary remains non-parity
Video encoder	Direct ffmpeg raw RGB to H.264/yuv420p, no audio	Different from Comfy CreateVideo/SaveVideo	Not a valid latent-quality gate; cannot explain model-space corruption by itself

Qwen Projection Code Audit: Remaining Non-Active Differences

These are real implementation differences, but the audit established they are not active numerical branches for the captured Qwen checkpoint. They must not be presented as an explanation for the current down_proj delta without new evidence.

Direct location	Difference from Comfy	Active for current capture?
nvfp4.py:47-51	Calls .contiguous() while Comfy preserves loaded storage/view identity. In particular Comfy treats E4M3 block scales as a dtype view, not a numeric cast.	No: checkpoint tensors are already contiguous and direct also uses a dtype view.
nvfp4.py:75-76	Always flattens and contiguates activation rows; Comfy full-precision F.linear retains N-D input/stride.	No known value effect: elements and BF16 dtype are unchanged.
nvfp4.py:82	Uses packed_weight.dequantize().to(flat_x) rather than an explicitly named Comfy compute-dtype dequantization call.	No: input and output compute dtype are BF16, and both call the same installed Kitchen layout/dequantizer.
nvfp4.py:46,81	Does not implement Comfy global full-precision overrides, disabled-format controls, forced casts, weight functions, or LoRA/offload dispatch.	No: every Qwen sidecar itself forces full precision, no projection has bias, and no patches/LoRAs are attached.

Audit conclusion: projection source code contains cleanup/generalization gaps, but no identified active branch mismatch explains the current captured values. The next code audit must move outward to the exact Qwen block boundary: Comfy input dtype, full-precision weight construction, SDPA backend selection, and residual/BF16 rounding boundaries. Do not run another full inference merely to rediscover this projection result.

Qwen Layer-0 Arithmetic Audit After 4050afe

The follow-up statement audit covered every remaining active layer-0 arithmetic boundary outside the already-fixed SDPA wrapper: full-precision NVFP4 dispatch, weight/scales, pre_quant_scale, BF16 GEMM, activation layout, Q/K RMSNorm, O projection/residual, post-attention norm, SiLU/gate/up product, down projection, and final residual.

Result: no further source-proven active mismatch was found. For the actual checkpoint, Comfy and direct both use BF16 input, BF16 NVFP4 dequantization, BF16 F.linear, no projection biases, identical pre-scale ordering, identical MLP evaluation order, and ordinary BF16 residual additions. Explicit flattening and .contiguous() in direct are not a demonstrated numerical difference for the captured contiguous batch-one input.

Do not modify nvfp4.py or Qwen MLP/residual code to chase the remaining layer-0 delta. The next justified work is a bounded component experiment that identifies the actual selected SDPA kernel/GQA fallback and verifies capture consistency; it is not another full denoising or video run.

Bounded SDPA/Layer-0 Result

The bounded replay was completed with the fresh projection-capture tensors. For the reference's 17-token layer-0 attention, both paths have native cuDNN GQA available (flash=False, cudnn=True, efficient=False). Therefore the direct SDPA wrapper takes the same native-GQA branch Comfy takes.

Boundary	Mean absolute error	Maximum absolute error
O-projected attention	0.000244372	0.0161562
First residual	0.00026845	0.0213833
Post-attention RMSNorm	3.87714e-05	0.000426032
MLP output	0.000674194	0.0500984
Layer-0 output	0.000808998	0.0526199

This confirms the residual error is already present in the captured projection outputs and is amplified by the MLP. The NVFP4 and MLP code audits found no remaining active source-level branch difference. Further edits to flattening, contiguity, dequantization dtype, pre_quant_scale, SiLU ordering, or residual adds would be speculative and are prohibited until a new code-path difference is demonstrated.

Loaded-Comfy Projection Module Probe

The source audit was followed by a direct replay against tensors consumed and produced by the actual loaded Comfy layer-0 modules. This proves the remaining difference is present before attention and is not inferred from source.

Item	Evidence
Capture location	/tmp/fl2va-qwen0-loaded-modules on Spark
QKV capture SHA-256	8998722e73ba24c40c09a86133ccdb22b708998a3e8b8fb4c4293227345f19db
MLP capture SHA-256	e52ed70f4aa76f1ba26302b9955631d8f490fb6dc269d96a0ce3e1ea981e7ac2
Loaded Comfy class	comfy.ops.mixed_precision_ops.<locals>.MixedPrecisionOps.Linear
Loaded Comfy weight	comfy_kitchen.tensor.base.QuantizedTensor, logical dtype BF16, TensorCoreNVFP4Layout, full precision enabled
Q/K/V dimensions	(8192,5120), (1024,5120), (1024,5120); no pre-scale
Gate/up/down dimensions	(25600,5120), (25600,5120), (5120,25600); only down has pre-scale

Direct replay on the identical loaded-Comfy module inputs:

Projection	Mean absolute error	Maximum absolute error
Q	6.11124e-05	0.0019514
K	8.47002e-05	0.00167805
V	5.4459e-05	0.00098893
Gate	0.000308973	0.0082469
Up	0.00028138	0.00792789
Down, using Comfy's captured activated input	0.000424157	0.0500984

Conclusion: same checkpoint tensors, same BF16 inputs, and the same named Kitchen layout do not currently produce identical projection outputs through direct Nvfp4Linear and Comfy's loaded MixedPrecisionOps.Linear. The next investigation must capture/compare the effective QuantizedTensor parameters and direct packed-weight wrapper at dispatch time. Do not revisit attention, sampler, or MLP ordering until that exact module-dispatch difference is found.

Effective Weight Root Cause And Resolution

The storage-safe dispatch probe found the active difference. Comfy's text encoder calls Qwen with dtype=torch.float32; its full-precision NVFP4 branch casts each projection to an effective FP32 matrix before F.linear. Direct had constructed the Qwen encoder with BF16, causing its full-precision NVFP4 branch to dequantize the same packed data into BF16.

The probe proved that packed qdata, tensor scale, and block scale are exact for all Q/K/V/gate/up/down projections. With direct Qwen switched to FP32:

Projection check	Result
Effective Q/K/V/gate/up/down FP32 weights	Bit-exact to Comfy
Q/K/V/gate/up linear outputs	Bit-exact to Comfy
Layer-0 O-projected attention	Mean/max 0 / 0
Layer-0 post-attention residual	Mean/max 0 / 0
Layer-0 post-attention RMSNorm	Mean/max 0 / 0
Layer-0 MLP output	Mean/max 0 / 0
Layer-0 final output	Mean/max 0 / 0

The special down dispatch sub-probe that supplied Comfy's already-activated input directly to F.linear is intentionally invalid because it bypasses the module's required pre_quant_scale; the full layer replay above is the valid gate and is exact.

Current Runtime Scope

Component	Implemented	Known limitation
Text-only Qwen	Yes	No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags
Token refiner	Yes	Bit-exact from captured 5376-wide refiner input through both blocks and final RMSNorm; Qwen-to-refiner projection boundary is still not separately captured
Prompt-only FL2VA packer	Yes	Bit-exact for the coherent captured text-only FL2VA DiT input; no keyframe/reference condition rows
H3 DiT	Yes	Strict all-block numeric parity not achieved
Beta/RES sampler	Yes	Exact sigma/state/update parity not demonstrated; direct preview has wrong initial audio state
Video VAE decoder	Yes	No direct-versus-Comfy same-latent pixel comparison yet
Audio VAE/decode/mux	No	Preview intentionally produces video only
Full Ref2VA	No	References, vision conditioning, VAE encode, audio, and muxing are unimplemented

Remaining Gates, In Dependency Order

Only these are outstanding. Do not recapture or revisit rows marked complete unless the checkpoint, Comfy version, prompt, or backend changes.

1. H3 first-divergence repair. Reuse the existing block trace. Start at block 0 sublayers, repair the first mismatch, then use Comfy block output as the next direct input to distinguish local error from accumulation. Existing evidence says block 0 is close but accumulation is unresolved.
2. Final rows and video latent. Compare final packed video rows and unpatchify_video output with Comfy before entering the sampler.
3. Sampler replay using Comfy state. Feed initial.pt and captured sigmas directly to the sampler. Compare video and audio denoised/update states for all 12 saved steps. Do not test seed equality until this passes.
4. VAE pixel gate. Decode the identical captured final video latent in direct and Comfy and compare pixels before ffmpeg. This decides whether any residual grid comes from decoder behavior.
5. End-to-end FL2VA preview. Generate the reference dragon only after gates 1-8 pass. Compare its raw final latent first, then video.
6. Feature/performance work. Only then add audio, Ref2VA/reference paths, Sage3, CUDA graphs, and multi-GPU execution.

Existing Tools And Their Intended Gate

Gate	Tools
Qwen captures	patch_comfy_qwen_output_capture.py, patch_comfy_qwen_layer_trace.py, patch_comfy_qwen_layer0_sublayers.py, patch_comfy_qwen_layer0_projections.py
Qwen projection gate	compare_qwen0_projections.py
Qwen local diagnosis	trace_qwen0_attention.py needs repair before reuse; it currently passes an obsolete encoder argument
H3 block capture	patch_comfy_h3_capture.py, patch_comfy_h3_block_capture.py, patch_comfy_h3_block0_sublayers.py
H3 local diagnosis	compare_block0_qkv.py, compare_attention_backends.py, compare_block0_mlp_projections.py, localize_block_sublayers.py, localize_block_mismatch.py
Sampler	patch_comfy_sampler_capture.py, compare_fl2va_steps.py
VAE	smoke_h3_vae_decoder.py; a same-latent direct-versus-Comfy pixel comparator still needs to be added

The patch_comfy_* tools modify /opt/ComfyUI and are only safe against the restorable Comfy reference image. Capture provenance must be appended here at the time a new capture is made.