diff --git a/PLAN.md b/PLAN.md index c7c4432..4b9119c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -55,3 +55,47 @@ The direct runner must first match the model contract and output quality. Beatin - Validate denoiser outputs at each scheduler step, not only encoded video. - Record attention, GEMM, communication, VAE, and end-to-end timings separately. - Treat SageAttention3 as an experimental quality-gated kernel for H3. + +## 2026-08-13 VAE Debug Handoff + +Current saved latent and comparison assets live under: + +- Spark: `/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime` +- Share: `\\192.168.1.162\StoryStudioAssets\H3-output\h3-blackwell-runtime` + +Key assets: + +- Good sampled latent: `direct-cat-house-backflip-disco-shades-960x544-5s-latent.pt` +- Known-good same-latent Comfy/upstream VAE decode: `direct-cat-house-backflip-disco-shades-960x544-5s-upstream-comfy-tiled-decode.mp4` +- Current direct standalone FP32/Sage VAE decode candidate: `direct-cat-house-backflip-disco-shades-960x544-5s-direct-fp32-sage-vae-decode.mp4` + +What is proven: + +- The latent is good. Same latent decoded through Comfy/upstream VAE is visually clean. +- The sampler/model path is already exact against Comfy free-run parity for the fixed baseline. +- Direct VAE BF16/checkpoint-dtype loading was wrong. Loading direct VAE weights as FP32 reduced isolated decoder clip drift from roughly `mean=1.5e-3, max=1e-1` to roughly `mean=1e-6, max=6e-5`. +- Individual direct tiled VAE clips match upstream closely (`mean` around `4e-7`). +- Full direct `decode_temporal()` still differs from upstream at global frames `17, 34, 51, 68, 85, 102`, i.e. temporal join boundaries. +- Temporal assembly tracer shows pre-blend current chunks and overlap tails each match upstream, but blended join output differs hugely (`mean` around `0.15`, max around `5`) when comparing direct blend result to upstream blend result using their respective near-identical inputs. +- Direct and upstream `blend()` return identical results on the exact same inputs, so the remaining issue is likely an input/aliasing/dtype/shape subtlety at the temporal join, not the blend formula itself. + +Relevant debug tools committed: + +- `tools/decode_video_latent.py` +- `tools/compare_frame_dirs.py` +- `tools/compare_vae_decoder_clip.py` +- `tools/compare_vae_full_decode.py` +- `tools/compare_vae_tiled_clip.py` +- `tools/compare_vae_temporal_assembly.py` + +Next VAE debugging steps: + +- In `tools/compare_vae_temporal_assembly.py`, compare `prev_d` vs `prev_u` and `part_d` vs `part_u` after casting both pairs to a shared dtype and before blending. The current stats say they are close, but the blend of respective inputs explodes, which suggests a subtle shape/stride/dim broadcasting mismatch. +- Log `shape`, `stride`, `dtype`, `is_contiguous`, and `storage_offset` for `prev_*`, `part_*`, blend weights, and slices at `chunk_1_blended_at_17`. +- Try forcing `prev_d`, `prev_u`, `part_d`, and `part_u` to `.contiguous()` immediately before temporal blend in both direct and tracer paths. +- If that fixes it, patch direct `decode_temporal()` only. If not, compare exact selected slices (`prev[..., -9:, :, :]`, `part[..., :9, :, :]`) elementwise before and after multiplying weights. + +Backends/experiments to evaluate later: + +- Attention backend sweeps after VAE quality is fixed: `sage2`, `sage3`, `sdpa`, plus future FlashAttention-4/Sol-Attn/KJ candidates. +- Spectrum MiniMax H3 repo to inspect later: https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3