Document H3 audio VAE path

This commit is contained in:
Daniel Maddern 2026-08-14 13:33:08 +07:00
parent 4696a3e72c
commit c5cf42850b

31
PLAN.md
View file

@ -99,3 +99,34 @@ Backends/experiments to evaluate later:
- Attention backend sweeps after VAE quality is fixed: `sage2`, `sage3`, `sdpa`, plus future FlashAttention-4/Sol-Attn/KJ candidates.
- Spectrum MiniMax H3 repo to inspect later: https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3
## H3 Audio VAE Notes
Comfy source references:
- `comfy/ldm/minimax/audio_vae.py`: `MiniMaxH3AudioVAE`
- `comfy_extras/nodes_minimax_h3.py`: AV latent creation and audio guide encoding
- `comfy_extras/nodes_lt_audio.py`: `LTXVAudioVAEDecode`, which also handles nested AV latents and calls `audio_vae.decode(audio_latent)`
- `comfy_extras/nodes_audio.py`: generic audio save/preview and normalization helpers
Local checkpoint:
- Spark: `/home/daniel/aeon-spark-test/h3/comfy-models/vae/minimax_h3_audio_vae_fp32.safetensors`
- Runtime mount: `/vae/minimax_h3_audio_vae_fp32.safetensors`
- Shape inspection: 917 keys; top-level names include `dec_in_proj.*`, `decoder.*`, `encoder.*`, `latents_mean`, `latents_std`, `mean_proj.*`, `logs_proj.*`, and `pre_block.*`.
Architecture summary:
- This is not similar to the video ViT3D VAE. It is a DAC-lineage waveform encoder plus BigVGAN decoder.
- Latents are normalized `[B, 32, 2, T]`: 32 channels, 2 stereo channels, 40 latent frames/sec.
- Decoder output is stereo waveform `[B, 2, L]` at 32 kHz.
- One audio latent frame equals 800 samples (`32000 / 40`).
- Comfy decode path for nested H3 AV latent is effectively: select nested audio tensor, call `audio_vae.decode(audio_latent)`, then return waveform with sample rate 32000. VideoHelperSuite/Comfy audio save nodes use ffmpeg or audio helpers to save/mux.
Direct runtime implementation path:
- Change `sample_video_res_multistep()` to return both final video and final audio latents, or add a `sample_av_res_multistep()` wrapper that preserves the existing video-only API.
- Port `MiniMaxH3AudioVAE` from Comfy into a standalone `audio_vae_decoder.py`, starting decoder-only if we only need generated audio.
- Load `/vae/minimax_h3_audio_vae_fp32.safetensors` strictly in FP32, mirroring the video VAE precision lesson.
- Add a latent-only decode tool that accepts saved audio latent `[1,32,2,T]`, writes WAV/FLAC at 32 kHz, and optionally muxes with MP4 using ffmpeg.
- Validate first by capturing or decoding the same final audio latent through Comfy and direct, comparing waveform tensors before muxing.