From c5cf42850b9c358cb3db9d4c7cb9f79c15e862f3 Mon Sep 17 00:00:00 2001 From: Daniel Maddern Date: Fri, 14 Aug 2026 13:33:08 +0700 Subject: [PATCH] Document H3 audio VAE path --- PLAN.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/PLAN.md b/PLAN.md index 4b9119c..2db1fbd 100644 --- a/PLAN.md +++ b/PLAN.md @@ -99,3 +99,34 @@ Backends/experiments to evaluate later: - Attention backend sweeps after VAE quality is fixed: `sage2`, `sage3`, `sdpa`, plus future FlashAttention-4/Sol-Attn/KJ candidates. - Spectrum MiniMax H3 repo to inspect later: https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3 + +## H3 Audio VAE Notes + +Comfy source references: + +- `comfy/ldm/minimax/audio_vae.py`: `MiniMaxH3AudioVAE` +- `comfy_extras/nodes_minimax_h3.py`: AV latent creation and audio guide encoding +- `comfy_extras/nodes_lt_audio.py`: `LTXVAudioVAEDecode`, which also handles nested AV latents and calls `audio_vae.decode(audio_latent)` +- `comfy_extras/nodes_audio.py`: generic audio save/preview and normalization helpers + +Local checkpoint: + +- Spark: `/home/daniel/aeon-spark-test/h3/comfy-models/vae/minimax_h3_audio_vae_fp32.safetensors` +- Runtime mount: `/vae/minimax_h3_audio_vae_fp32.safetensors` +- Shape inspection: 917 keys; top-level names include `dec_in_proj.*`, `decoder.*`, `encoder.*`, `latents_mean`, `latents_std`, `mean_proj.*`, `logs_proj.*`, and `pre_block.*`. + +Architecture summary: + +- This is not similar to the video ViT3D VAE. It is a DAC-lineage waveform encoder plus BigVGAN decoder. +- Latents are normalized `[B, 32, 2, T]`: 32 channels, 2 stereo channels, 40 latent frames/sec. +- Decoder output is stereo waveform `[B, 2, L]` at 32 kHz. +- One audio latent frame equals 800 samples (`32000 / 40`). +- Comfy decode path for nested H3 AV latent is effectively: select nested audio tensor, call `audio_vae.decode(audio_latent)`, then return waveform with sample rate 32000. VideoHelperSuite/Comfy audio save nodes use ffmpeg or audio helpers to save/mux. + +Direct runtime implementation path: + +- Change `sample_video_res_multistep()` to return both final video and final audio latents, or add a `sample_av_res_multistep()` wrapper that preserves the existing video-only API. +- Port `MiniMaxH3AudioVAE` from Comfy into a standalone `audio_vae_decoder.py`, starting decoder-only if we only need generated audio. +- Load `/vae/minimax_h3_audio_vae_fp32.safetensors` strictly in FP32, mirroring the video VAE precision lesson. +- Add a latent-only decode tool that accepts saved audio latent `[1,32,2,T]`, writes WAV/FLAC at 32 kHz, and optionally muxes with MP4 using ffmpeg. +- Validate first by capturing or decoding the same final audio latent through Comfy and direct, comparing waveform tensors before muxing.