From abc651fadbf5113f6656c6963aca3655c325a865 Mon Sep 17 00:00:00 2001 From: Daniel Maddern Date: Thu, 13 Aug 2026 00:05:32 +0700 Subject: [PATCH] Verify full Qwen layer trace --- PARITY.md | 25 +++++++++---------------- tools/compare_qwen_layer_trace.py | 27 +++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 16 deletions(-) create mode 100644 tools/compare_qwen_layer_trace.py diff --git a/PARITY.md b/PARITY.md index 5b08590..e75feaf 100644 --- a/PARITY.md +++ b/PARITY.md @@ -38,6 +38,7 @@ diagnose preview output. | Qwen embedding loading, after fix | Direct rows multiplied by `weight_scale` in FP32 | Input embeddings matched the Comfy capture exactly | Passed for the captured nonempty text-only reference | | Qwen layer-50, before embedding fix | Direct versus Comfy layer-50 capture | Mean absolute delta `23.50596`, max `38977.09375` | Failed; invalidated direct text conditioning | | Qwen layer-50, after embedding fix but before projection fix | Direct versus Comfy layer-50 capture | Mean absolute delta improved to about `5.63` | Embedding issue fixed; remaining error was material | +| Qwen all-layer replay, after FP32 fix | `compare_qwen_layer_trace.py` replayed `/tmp/fl2va-qwen-trace` without new Comfy inference | Layers `00` through `49` and `qwen_layer50.pt` all reported mean/max `0` / `0` | **Passed: prompt-only Qwen conditioning is bit-exact for the immutable 17-token FL2VA reference.** | | Qwen layer-0 normalization | Direct `F.rms_norm` versus `qwen0_norm1.pt` | Mean absolute delta about `8.96e-06` | Passed to expected BF16-level tolerance | | Qwen layer-0 attention, before projection fix | Direct versus `qwen0_attention.pt` | Mean absolute delta about `0.0357`; layer output about `0.1496` | First material Qwen divergence was at projection/attention boundary, not tokenization, embeddings, RMSNorm, or RoPE | | Qwen attention backend | Comfy source inspection | Qwen chooses Comfy small-input attention, i.e. SDPA path, not Sage2 | Direct Qwen was changed to BF16 causal-mask SDPA with GQA | @@ -193,35 +194,27 @@ gate and is exact. Only these are outstanding. Do not recapture or revisit rows marked complete unless the checkpoint, Comfy version, prompt, or backend changes. -1. **Qwen all-layer replay after `0cad1db`.** The layer-0 projection capture - was completed: errors are recorded above. Replay the existing 50 layer - tensors in `/tmp/fl2va-qwen-trace`; report first failing layer and mean/max - error. Do not generate another video before this is recorded. -2. **Qwen projection precision follow-up if layer replay fails at layer 0.** - Do not repeat the capture. Use `/tmp/fl2va-qwen0-projections` to isolate the - remaining `down_proj` max error `0.0749016` and its BF16/dequantization - boundary. -3. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner, +1. **Token-refiner capture and comparison.** Capture exact Comfy pre-refiner, block 0, block 1, and final refined states once, then compare direct using the matched Qwen layer-50 tensor. -4. **H3 assembled input comparison.** Use the existing FL2VA input capture to +2. **H3 assembled input comparison.** Use the existing FL2VA input capture to compare text rows, audio/video rows, positions, times, and segments. This isolates packing from DiT error. -5. **H3 first-divergence repair.** Reuse the existing block trace. Start at +3. **H3 first-divergence repair.** Reuse the existing block trace. Start at block 0 sublayers, repair the first mismatch, then use Comfy block output as the next direct input to distinguish local error from accumulation. Existing evidence says block 0 is close but accumulation is unresolved. -6. **Final rows and video latent.** Compare final packed video rows and +4. **Final rows and video latent.** Compare final packed video rows and `unpatchify_video` output with Comfy before entering the sampler. -7. **Sampler replay using Comfy state.** Feed `initial.pt` and captured +5. **Sampler replay using Comfy state.** Feed `initial.pt` and captured `sigmas` directly to the sampler. Compare video and audio denoised/update states for all 12 saved steps. Do not test seed equality until this passes. -8. **VAE pixel gate.** Decode the identical captured final video latent in +6. **VAE pixel gate.** Decode the identical captured final video latent in direct and Comfy and compare pixels before ffmpeg. This decides whether any residual grid comes from decoder behavior. -9. **End-to-end FL2VA preview.** Generate the reference dragon only after +7. **End-to-end FL2VA preview.** Generate the reference dragon only after gates 1-8 pass. Compare its raw final latent first, then video. -10. **Feature/performance work.** Only then add audio, Ref2VA/reference paths, +8. **Feature/performance work.** Only then add audio, Ref2VA/reference paths, Sage3, CUDA graphs, and multi-GPU execution. ## Existing Tools And Their Intended Gate diff --git a/tools/compare_qwen_layer_trace.py b/tools/compare_qwen_layer_trace.py new file mode 100644 index 0000000..697cd59 --- /dev/null +++ b/tools/compare_qwen_layer_trace.py @@ -0,0 +1,27 @@ +"""Replay direct Qwen against an immutable per-layer Comfy trace.""" + +import argparse +from pathlib import Path + +import torch + +from h3_blackwell_runtime.qwen3vl_text import Qwen3VL32BTextEncoder + + +parser = argparse.ArgumentParser() +parser.add_argument("--trace-dir", type=Path, required=True) +parser.add_argument("--checkpoint", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors") +args = parser.parse_args() + +encoder = Qwen3VL32BTextEncoder(args.checkpoint).eval() +hidden = torch.load(args.trace_dir / "qwen_input_embeds.pt", map_location="cuda", weights_only=False).to(encoder.dtype) + +with torch.inference_mode(): + for index, layer in enumerate(encoder.layers): + hidden = layer(hidden) + expected = torch.load(args.trace_dir / "qwen_layers" / f"{index:02d}.pt", map_location="cuda", weights_only=False) + delta = (hidden.float() - expected.float()).abs() + print(f"layer={index:02d} max_abs={delta.max().item():.6g} mean_abs={delta.mean().item():.6g}") + expected = torch.load(args.trace_dir / "qwen_layer50.pt", map_location="cuda", weights_only=False) + delta = (hidden.float() - expected.float()).abs() + print(f"layer50 max_abs={delta.max().item():.6g} mean_abs={delta.mean().item():.6g}")