Document H3 audio VAE path
This commit is contained in:
parent
4696a3e72c
commit
c5cf42850b
1 changed files with 31 additions and 0 deletions
31
PLAN.md
31
PLAN.md
|
|
@ -99,3 +99,34 @@ Backends/experiments to evaluate later:
|
||||||
|
|
||||||
- Attention backend sweeps after VAE quality is fixed: `sage2`, `sage3`, `sdpa`, plus future FlashAttention-4/Sol-Attn/KJ candidates.
|
- Attention backend sweeps after VAE quality is fixed: `sage2`, `sage3`, `sdpa`, plus future FlashAttention-4/Sol-Attn/KJ candidates.
|
||||||
- Spectrum MiniMax H3 repo to inspect later: https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3
|
- Spectrum MiniMax H3 repo to inspect later: https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3
|
||||||
|
|
||||||
|
## H3 Audio VAE Notes
|
||||||
|
|
||||||
|
Comfy source references:
|
||||||
|
|
||||||
|
- `comfy/ldm/minimax/audio_vae.py`: `MiniMaxH3AudioVAE`
|
||||||
|
- `comfy_extras/nodes_minimax_h3.py`: AV latent creation and audio guide encoding
|
||||||
|
- `comfy_extras/nodes_lt_audio.py`: `LTXVAudioVAEDecode`, which also handles nested AV latents and calls `audio_vae.decode(audio_latent)`
|
||||||
|
- `comfy_extras/nodes_audio.py`: generic audio save/preview and normalization helpers
|
||||||
|
|
||||||
|
Local checkpoint:
|
||||||
|
|
||||||
|
- Spark: `/home/daniel/aeon-spark-test/h3/comfy-models/vae/minimax_h3_audio_vae_fp32.safetensors`
|
||||||
|
- Runtime mount: `/vae/minimax_h3_audio_vae_fp32.safetensors`
|
||||||
|
- Shape inspection: 917 keys; top-level names include `dec_in_proj.*`, `decoder.*`, `encoder.*`, `latents_mean`, `latents_std`, `mean_proj.*`, `logs_proj.*`, and `pre_block.*`.
|
||||||
|
|
||||||
|
Architecture summary:
|
||||||
|
|
||||||
|
- This is not similar to the video ViT3D VAE. It is a DAC-lineage waveform encoder plus BigVGAN decoder.
|
||||||
|
- Latents are normalized `[B, 32, 2, T]`: 32 channels, 2 stereo channels, 40 latent frames/sec.
|
||||||
|
- Decoder output is stereo waveform `[B, 2, L]` at 32 kHz.
|
||||||
|
- One audio latent frame equals 800 samples (`32000 / 40`).
|
||||||
|
- Comfy decode path for nested H3 AV latent is effectively: select nested audio tensor, call `audio_vae.decode(audio_latent)`, then return waveform with sample rate 32000. VideoHelperSuite/Comfy audio save nodes use ffmpeg or audio helpers to save/mux.
|
||||||
|
|
||||||
|
Direct runtime implementation path:
|
||||||
|
|
||||||
|
- Change `sample_video_res_multistep()` to return both final video and final audio latents, or add a `sample_av_res_multistep()` wrapper that preserves the existing video-only API.
|
||||||
|
- Port `MiniMaxH3AudioVAE` from Comfy into a standalone `audio_vae_decoder.py`, starting decoder-only if we only need generated audio.
|
||||||
|
- Load `/vae/minimax_h3_audio_vae_fp32.safetensors` strictly in FP32, mirroring the video VAE precision lesson.
|
||||||
|
- Add a latent-only decode tool that accepts saved audio latent `[1,32,2,T]`, writes WAV/FLAC at 32 kHz, and optionally muxes with MP4 using ffmpeg.
|
||||||
|
- Validate first by capturing or decoding the same final audio latent through Comfy and direct, comparing waveform tensors before muxing.
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue