109 lines
5.5 KiB
Markdown
109 lines
5.5 KiB
Markdown
# H3 Startup Audio Investigation
|
|
|
|
Investigation date: 2026-08-21
|
|
|
|
## Reproduction Cases
|
|
|
|
- Affected dialogue: base 12-step, Sage2, seed `440420`, tagged dialogue.
|
|
- Clean control: base 12-step, Sage2, seed `440421`, immediate nightclub music.
|
|
- Prompt-format control: affected dialogue prompt and seed with only the two
|
|
`<d>[English]...</d>` spans replaced by quoted speech.
|
|
|
|
All diagnostic assets are under `/home/daniel/StoryStudioAssets/H3-output/h3-baselines`.
|
|
|
|
## Findings
|
|
|
|
1. The affected transient is already present in the retained lossless WAV. AAC
|
|
encoding and MP4 muxing are not the source.
|
|
2. The AudioVAE is not the primary source. Moving the affected first four audio
|
|
latent frames to frame 40 in an otherwise near-silent latent carrier creates
|
|
a similarly strong event at `1.0s`: `-18.44 dBFS` peak versus `-19.12 dBFS`
|
|
when the same frames are placed at the start.
|
|
3. AudioVAE boundary context changes the exact waveform, but the event remains.
|
|
This makes the decoder a secondary shaper rather than the origin.
|
|
4. Zero normalized latents are not silence. The official AudioVAE decodes them
|
|
to approximately `-26.17 dBFS` RMS in the first 100ms, so zero replacement is
|
|
not a valid repair.
|
|
5. Repeating affected latent frame 4 produces near-silence (`-56.02 dBFS` RMS),
|
|
but unconditional replacement is unsafe because valid music begins in the
|
|
same first four frames in the clean control.
|
|
6. There is no evidence for end-to-start wraparound in this sample. First/last
|
|
four-frame latent cosine is `-0.122`; first/last 100ms PCM correlation is
|
|
`0.006`.
|
|
7. In the tagged-dialogue denoising trace, the unwanted onset is near silence
|
|
through step 6 and begins growing materially at step 7 (`audio sigma 0.751`).
|
|
It reaches `-18.94 dBFS` peak and `-34.23 dBFS` RMS in the final first 100ms.
|
|
8. Clean immediate music is strongly predicted from step 1. This distinguishes
|
|
legitimate onset generation from the late-forming dialogue artifact.
|
|
9. Replacing only tagged dialogue with quoted speech suppresses the final first
|
|
100ms by about `21.3 dB` peak and `15.9 dB` RMS. The quoted result remains
|
|
below `-40.28 dBFS` peak and retains two later non-silent speech regions.
|
|
|
|
## Current Diagnosis
|
|
|
|
The startup artifact is encoded into the first four generated audio latent
|
|
frames during late denoising. Dialogue markup is a strong trigger for the
|
|
reproduced seed. The AudioVAE renders and contextually shapes the event but does
|
|
not create it independently.
|
|
|
|
This does not prove that markup is the only trigger or that quoted dialogue is
|
|
universally clean. It does establish markup as a repeatable trigger for this
|
|
prompt family. Automatic prompt transformation still requires subjective speech,
|
|
word-accuracy, and lip-sync review.
|
|
|
|
## Ten-Seed Prompt-Format Sweep
|
|
|
|
A matched sweep used seeds `440420` through `440429`, Sage2, base 12-step
|
|
beta/RES sampling, and identical dialogue semantics. Each seed generated one
|
|
tagged and one quoted audio latent and lossless waveform.
|
|
|
|
- Quoted speech reduced first-100ms peak level for all 10 seeds.
|
|
- Median peak reduction was `21.55 dB`; mean was `20.11 dB`.
|
|
- Median RMS reduction was `17.50 dB`; mean was `17.93 dB`.
|
|
- Tagged speech exceeded `-40 dBFS` peak in the first 100ms for 9/10 seeds.
|
|
- Quoted speech exceeded that threshold for 0/10 seeds.
|
|
- Tagged speech became active within 100ms for 9/10 seeds; quoted speech did so
|
|
for 0/10 seeds.
|
|
- A greater-than-10dB boundary decay occurred for 8/10 tagged cases and 0/10
|
|
quoted cases.
|
|
- Simple first-frame and first-four-frame latent magnitude/delta features overlap
|
|
between groups and cannot safely detect the artifact by themselves.
|
|
|
|
The complete report and paired WAV/latent files are in
|
|
`h3-baselines/audio-dialogue-format-sweep`. A subjective listening pass on
|
|
2026-08-21 judged all ten quoted WAVs good. Quoted dialogue is therefore the
|
|
project default; full-video lip-sync validation remains pending.
|
|
|
|
## Diagnostic Assets
|
|
|
|
- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.latent.pt`
|
|
- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.wav`
|
|
- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt`
|
|
- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav`
|
|
- `audio-diagnostic-dialogue-quoted-base12-sage2-seed440420.wav`
|
|
- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.pt`
|
|
- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.json`
|
|
- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.pt`
|
|
- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.json`
|
|
- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.pt`
|
|
- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.json`
|
|
- `audio-vae-boundary-probes/report.json`
|
|
|
|
## Next Experiments
|
|
|
|
1. Generate selected full videos to compare lip-sync and prompt adherence with
|
|
the new quoted-dialogue default.
|
|
2. Add one ambience-only prompt and one immediate-impact sound prompt to prevent
|
|
a detector from equating quiet starts with correctness.
|
|
3. Compare tagged-versus-quoted Qwen conditioning and per-step first-four-frame
|
|
denoiser outputs to localize the conditioning pathway.
|
|
4. Prototype a selective late-step boundary re-denoise only after a reliable
|
|
latent classifier exists.
|
|
5. Reject unconditional trimming, fading, zeroing, or fixed-frame replacement.
|
|
|
|
## Measurement Correction
|
|
|
|
FFmpeg's `apsnr` results previously recorded for attention-backend audio were
|
|
inconsistent with direct decoded-PCM array comparisons and must not be used.
|
|
Future audio comparisons must decode each stream to aligned float PCM and compute
|
|
error metrics directly.
|