daniel/h3-blackwell-runtime
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
h3-blackwell-runtime/AUDIO_BOUNDARY_INVESTIGATION.md
Daniel Maddern bd92baeb46 Add distributed H3 execution and validation
2026-08-22 14:09:45 +07:00

5.5 KiB
Raw Blame History

H3 Startup Audio Investigation

Investigation date: 2026-08-21

Reproduction Cases

  • Affected dialogue: base 12-step, Sage2, seed 440420, tagged dialogue.
  • Clean control: base 12-step, Sage2, seed 440421, immediate nightclub music.
  • Prompt-format control: affected dialogue prompt and seed with only the two <d>[English]...</d> spans replaced by quoted speech.

All diagnostic assets are under /home/daniel/StoryStudioAssets/H3-output/h3-baselines.

Findings

  1. The affected transient is already present in the retained lossless WAV. AAC encoding and MP4 muxing are not the source.
  2. The AudioVAE is not the primary source. Moving the affected first four audio latent frames to frame 40 in an otherwise near-silent latent carrier creates a similarly strong event at 1.0s: -18.44 dBFS peak versus -19.12 dBFS when the same frames are placed at the start.
  3. AudioVAE boundary context changes the exact waveform, but the event remains. This makes the decoder a secondary shaper rather than the origin.
  4. Zero normalized latents are not silence. The official AudioVAE decodes them to approximately -26.17 dBFS RMS in the first 100ms, so zero replacement is not a valid repair.
  5. Repeating affected latent frame 4 produces near-silence (-56.02 dBFS RMS), but unconditional replacement is unsafe because valid music begins in the same first four frames in the clean control.
  6. There is no evidence for end-to-start wraparound in this sample. First/last four-frame latent cosine is -0.122; first/last 100ms PCM correlation is 0.006.
  7. In the tagged-dialogue denoising trace, the unwanted onset is near silence through step 6 and begins growing materially at step 7 (audio sigma 0.751). It reaches -18.94 dBFS peak and -34.23 dBFS RMS in the final first 100ms.
  8. Clean immediate music is strongly predicted from step 1. This distinguishes legitimate onset generation from the late-forming dialogue artifact.
  9. Replacing only tagged dialogue with quoted speech suppresses the final first 100ms by about 21.3 dB peak and 15.9 dB RMS. The quoted result remains below -40.28 dBFS peak and retains two later non-silent speech regions.

Current Diagnosis

The startup artifact is encoded into the first four generated audio latent frames during late denoising. Dialogue markup is a strong trigger for the reproduced seed. The AudioVAE renders and contextually shapes the event but does not create it independently.

This does not prove that markup is the only trigger or that quoted dialogue is universally clean. It does establish markup as a repeatable trigger for this prompt family. Automatic prompt transformation still requires subjective speech, word-accuracy, and lip-sync review.

Ten-Seed Prompt-Format Sweep

A matched sweep used seeds 440420 through 440429, Sage2, base 12-step beta/RES sampling, and identical dialogue semantics. Each seed generated one tagged and one quoted audio latent and lossless waveform.

  • Quoted speech reduced first-100ms peak level for all 10 seeds.
  • Median peak reduction was 21.55 dB; mean was 20.11 dB.
  • Median RMS reduction was 17.50 dB; mean was 17.93 dB.
  • Tagged speech exceeded -40 dBFS peak in the first 100ms for 9/10 seeds.
  • Quoted speech exceeded that threshold for 0/10 seeds.
  • Tagged speech became active within 100ms for 9/10 seeds; quoted speech did so for 0/10 seeds.
  • A greater-than-10dB boundary decay occurred for 8/10 tagged cases and 0/10 quoted cases.
  • Simple first-frame and first-four-frame latent magnitude/delta features overlap between groups and cannot safely detect the artifact by themselves.

The complete report and paired WAV/latent files are in h3-baselines/audio-dialogue-format-sweep. A subjective listening pass on 2026-08-21 judged all ten quoted WAVs good. Quoted dialogue is therefore the project default; full-video lip-sync validation remains pending.

Diagnostic Assets

  • audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.latent.pt
  • audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.wav
  • audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt
  • audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav
  • audio-diagnostic-dialogue-quoted-base12-sage2-seed440420.wav
  • audio-diagnostic-affected-dialogue-sage2-denoise-trace.pt
  • audio-diagnostic-affected-dialogue-sage2-denoise-trace.json
  • audio-diagnostic-clean-nightclub-sage2-denoise-trace.pt
  • audio-diagnostic-clean-nightclub-sage2-denoise-trace.json
  • audio-diagnostic-dialogue-quoted-sage2-denoise-trace.pt
  • audio-diagnostic-dialogue-quoted-sage2-denoise-trace.json
  • audio-vae-boundary-probes/report.json

Next Experiments

  1. Generate selected full videos to compare lip-sync and prompt adherence with the new quoted-dialogue default.
  2. Add one ambience-only prompt and one immediate-impact sound prompt to prevent a detector from equating quiet starts with correctness.
  3. Compare tagged-versus-quoted Qwen conditioning and per-step first-four-frame denoiser outputs to localize the conditioning pathway.
  4. Prototype a selective late-step boundary re-denoise only after a reliable latent classifier exists.
  5. Reject unconditional trimming, fading, zeroing, or fixed-frame replacement.

Measurement Correction

FFmpeg's apsnr results previously recorded for attention-backend audio were inconsistent with direct decoded-PCM array comparisons and must not be used. Future audio comparisons must decode each stream to aligned float PCM and compute error metrics directly.