daniel/h3-blackwell-runtime
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: daniel:master
Branches Tags
daniel:master
daniel:fl2va-direct-runtime
daniel:vae-decode-optimization
..
compare: daniel:vae-decode-optimization
Branches Tags
daniel:fl2va-direct-runtime
daniel:master
daniel:vae-decode-optimization

No commits in common. "master" and "vae-decode-optimization" have entirely different histories.
master ... vae-decode

83 changed files with 209 additions and 7815 deletions
Show stats Expand all files Collapse all files

5
.dockerignore
View file

@ -1,5 +0,0 @@
.git
.pytest_cache
artifacts
**/__pycache__
**/*.pyc

109
AUDIO_BOUNDARY_INVESTIGATION.md
View file

@ -1,109 +0,0 @@
# H3 Startup Audio Investigation
Investigation date: 2026-08-21
## Reproduction Cases
- Affected dialogue: base 12-step, Sage2, seed `440420`, tagged dialogue.
- Clean control: base 12-step, Sage2, seed `440421`, immediate nightclub music.
- Prompt-format control: affected dialogue prompt and seed with only the two
`<d>[English]...</d>` spans replaced by quoted speech.
All diagnostic assets are under `/home/daniel/StoryStudioAssets/H3-output/h3-baselines`.
## Findings
1. The affected transient is already present in the retained lossless WAV. AAC
encoding and MP4 muxing are not the source.
2. The AudioVAE is not the primary source. Moving the affected first four audio
latent frames to frame 40 in an otherwise near-silent latent carrier creates
a similarly strong event at `1.0s`: `-18.44 dBFS` peak versus `-19.12 dBFS`
when the same frames are placed at the start.
3. AudioVAE boundary context changes the exact waveform, but the event remains.
This makes the decoder a secondary shaper rather than the origin.
4. Zero normalized latents are not silence. The official AudioVAE decodes them
to approximately `-26.17 dBFS` RMS in the first 100ms, so zero replacement is
not a valid repair.
5. Repeating affected latent frame 4 produces near-silence (`-56.02 dBFS` RMS),
but unconditional replacement is unsafe because valid music begins in the
same first four frames in the clean control.
6. There is no evidence for end-to-start wraparound in this sample. First/last
four-frame latent cosine is `-0.122`; first/last 100ms PCM correlation is
`0.006`.
7. In the tagged-dialogue denoising trace, the unwanted onset is near silence
through step 6 and begins growing materially at step 7 (`audio sigma 0.751`).
It reaches `-18.94 dBFS` peak and `-34.23 dBFS` RMS in the final first 100ms.
8. Clean immediate music is strongly predicted from step 1. This distinguishes
legitimate onset generation from the late-forming dialogue artifact.
9. Replacing only tagged dialogue with quoted speech suppresses the final first
100ms by about `21.3 dB` peak and `15.9 dB` RMS. The quoted result remains
below `-40.28 dBFS` peak and retains two later non-silent speech regions.
## Current Diagnosis
The startup artifact is encoded into the first four generated audio latent
frames during late denoising. Dialogue markup is a strong trigger for the
reproduced seed. The AudioVAE renders and contextually shapes the event but does
not create it independently.
This does not prove that markup is the only trigger or that quoted dialogue is
universally clean. It does establish markup as a repeatable trigger for this
prompt family. Automatic prompt transformation still requires subjective speech,
word-accuracy, and lip-sync review.
## Ten-Seed Prompt-Format Sweep
A matched sweep used seeds `440420` through `440429`, Sage2, base 12-step
beta/RES sampling, and identical dialogue semantics. Each seed generated one
tagged and one quoted audio latent and lossless waveform.
- Quoted speech reduced first-100ms peak level for all 10 seeds.
- Median peak reduction was `21.55 dB`; mean was `20.11 dB`.
- Median RMS reduction was `17.50 dB`; mean was `17.93 dB`.
- Tagged speech exceeded `-40 dBFS` peak in the first 100ms for 9/10 seeds.
- Quoted speech exceeded that threshold for 0/10 seeds.
- Tagged speech became active within 100ms for 9/10 seeds; quoted speech did so
for 0/10 seeds.
- A greater-than-10dB boundary decay occurred for 8/10 tagged cases and 0/10
quoted cases.
- Simple first-frame and first-four-frame latent magnitude/delta features overlap
between groups and cannot safely detect the artifact by themselves.
The complete report and paired WAV/latent files are in
`h3-baselines/audio-dialogue-format-sweep`. A subjective listening pass on
2026-08-21 judged all ten quoted WAVs good. Quoted dialogue is therefore the
project default; full-video lip-sync validation remains pending.
## Diagnostic Assets
- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.latent.pt`
- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.wav`
- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt`
- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav`
- `audio-diagnostic-dialogue-quoted-base12-sage2-seed440420.wav`
- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.pt`
- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.json`
- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.pt`
- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.json`
- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.pt`
- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.json`
- `audio-vae-boundary-probes/report.json`
## Next Experiments
1. Generate selected full videos to compare lip-sync and prompt adherence with
the new quoted-dialogue default.
2. Add one ambience-only prompt and one immediate-impact sound prompt to prevent
a detector from equating quiet starts with correctness.
3. Compare tagged-versus-quoted Qwen conditioning and per-step first-four-frame
denoiser outputs to localize the conditioning pathway.
4. Prototype a selective late-step boundary re-denoise only after a reliable
latent classifier exists.
5. Reject unconditional trimming, fading, zeroing, or fixed-frame replacement.
## Measurement Correction
FFmpeg's `apsnr` results previously recorded for attention-backend audio were
inconsistent with direct decoded-PCM array comparisons and must not be used.
Future audio comparisons must decode each stream to aligned float PCM and compute
error metrics directly.

121
CURRENT_STATE.md
View file

@ -1,121 +0,0 @@
# H3 Runtime Current State
Status date: 2026-08-22
This document is the canonical snapshot of implemented scope and remaining work.
Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
## Implemented And Validated
- Single-GPU prompt-only T2VA with joint video/audio generation.
- First-frame I2VA, last-frame L2VA, and first/last FL2VA through the shared
keyframe-conditioning path.
- Qwen text and vision conditioning, token refinement, video VAE encoding, H3
packed denoising, beta/RES sampling, video/audio decoding, and final MP4 mux.
- Resident HTTP runtime with warmup, readiness reporting, request-level backend
selection, timing stages, optional latent saving, and diagnostic intermediates.
- SageAttention2 as the default quality backend.
- SDPA, forced cuDNN SDPA, FlashAttention-4, Sage3, Comfy Kitchen INT8, KJ Sage,
head-sliced, and Sol-Attn experimental backends.
- Official FL2VA Turbo 4-step and 8-step adapters.
- Optional resident H3-native latent upscaling.
- Experimental EasyCache and H3-Cache delta-reuse modes.
- Quoted dialogue as the project prompt default. In a matched 10-seed test,
quoted dialogue eliminated immediate first-100ms activity in all ten cases and
all ten quoted WAVs passed subjective review.
- Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
- Sequence-sharded 50-block execution and distributed final projection.
- True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
- Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
- Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames,
and 12 steps, including exact video and audio latent equality.
- Matched one-GPU RTX PRO 6000 Blackwell Server SDPA sampling averaged `28.50s`
over two runs versus `126.66s` for the same tensor runner on GB10 (`4.44x`).
RTX repeat variance was `0.49%` and checksums were identical between repeats.
- Two-GPU Ulysses SDPA speedup grows with sequence size: `1.21x` at
864x480/141 frames, `1.55x` at 1344x768/124 frames, and `1.72x` at
1344x768/243 frames. The tested cards have no NVLink; P2P read/write is
available and NCCL uses `P2P/CUMEM`.
## Primary Missing Scope
### Full Ref2VA
- Arbitrary reference image, video, and audio inputs.
- Reference-audio encoder and reference soundtrack conditioning.
- Reference identity/voice blocks in the standalone packer.
- Ref2VA position, modality, and scheduling contracts.
- Direct-versus-Comfy full Ref2VA per-step and final-output parity benchmark.
### Explicit Task API
- Named `task` selection for T2VA, I2VA, L2VA, FL2VA, and Ref2VA.
- Mode-specific request schemas and incompatible-input validation.
- Intermediate keyframe anchors beyond the current first/last restriction.
### Distributed Execution
- Output parity above one rank; real two-rank NCCL transport is validated.
- 2/4/6/8-GPU topology and performance sweeps on one Blackwell machine.
- Distributed resident-service orchestration; the current launcher is batch
generation through `torchrun`.
- x86 SageAttention2 packaging; RunPod validation initially uses SDPA.
### Owned Performance Kernels
- H3-specific attention backend optimized for real GB10 tensor shapes.
- Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
- CUDA graph capture and shape buckets.
- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.
## Quality Work Remaining
- Generate full quoted-dialogue videos and validate wording, voice consistency,
speech timing, and lip-sync before closing the startup-audio work.
- Complete strict per-step LightX2V parity for Turbo adapters.
- Add real-adapter Turbo end-to-end fixtures.
- Add the optional target-resolution refinement stage after latent upscaling.
- Resolve or formally bound upscaler ringing, texture, chromatic-edge, and
identity changes.
- Run full-size cache threshold and quality sweeps before enabling caches for
production output.
- Keep Sage3, Sol-Attn, INT8, and other approximate backends quality-gated.
- Fix the inactive fused Sol QKV-layout path, which currently references an
undefined `qkv` value. The deployed native Sol layout does not use this path.
## Production Work Remaining
- Asynchronous jobs, queueing, progress, cancellation, and timeouts.
- Strict request validation, including Boolean fields and mode combinations.
- Input/output path sandboxing, request-size limits, authentication, and TLS.
- Configurable FPS, video codec, audio codec, sample rate, and media policy.
- Container healthcheck, restart policy, resource limits, durable structured
request logs, and runtime metrics.
- Batch generation and an intentional worker/concurrency model.
## Validation And Packaging Gaps
- GPU end-to-end fixtures for T2VA, I2VA, L2VA, and FL2VA.
- Full Ref2VA, AudioVAE waveform, cache, HTTP API, real Turbo, real upscaler,
attention-quality, CUDA-graph, and distributed tests.
- Reproducible local fixtures for parity evidence currently stored on Spark/SMB.
- Explicit package declarations/checks for NumPy, SciPy, Pillow, and FFmpeg.
- A standalone base image if removing the Comfy-derived image becomes a product
requirement; the current denoising path still intentionally uses Comfy Kitchen
kernels.
- Align Docker `H3_MODEL_PATH` and `RuntimeConfig`; the environment variable is
currently not consumed by the runtime default.
## Recommended Execution Order
1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
2. Correct the inactive fused Sol path.
3. Add explicit task schemas and automated single-GPU mode tests.
4. Implement full Ref2VA, including reference-audio encoding.
5. Build the H3-specific attention backend and CUDA graph buckets.
6. Harden the service API and operational deployment.
7. Complete RunPod NCCL validation and distributed scaling benchmarks.
The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is
implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still
blocked on an eight-GPU host. The other largest gap is standalone Ref2VA.

158
DISTRIBUTED.md
View file

@ -1,158 +0,0 @@
# Distributed H3 Execution
The runtime supports two single-node distributed denoiser modes:
- `ulysses`: token-sharded blocks with QKV sequence-to-head all-to-all and the
inverse head-to-sequence all-to-all around attention.
- `tensor`: token-sharded blocks plus true NVFP4 tensor parallelism across QKV,
attention output, MLP FC1, and MLP FC2.
Both modes preserve the native packed H3 sequence and reject empty token or head
partitions. They do not add semantic padding. Temporary collective padding used
by ragged row gathering is removed before model operations.
## Partition Contract
H3 has 56 attention heads of width 128. Head ownership is balanced and can be
ragged:
| GPUs | Heads per rank |
| --- | --- |
| 2 | 28, 28 |
| 4 | 14, 14, 14, 14 |
| 6 | 10, 10, 9, 9, 9, 9 |
| 8 | 7, 7, 7, 7, 7, 7, 7, 7 |
Token ranges use the same quotient/remainder partitioning. Segment boundaries
are clipped to each rank's token interval and rebased before AdaLN modulation
and residual gating.
In tensor mode, QKV output rows are selected by local head ownership. Attention
output and MLP FC2 input columns are sliced on NVFP4 alignment boundaries; each
rank computes a partial output and a ragged reduce-scatter sums and assigns token
rows. MLP FC1 selects corresponding local ranges from both the gate and value
halves. Projection bias is omitted from rank partials and added exactly once
after reduction.
## Launch
The launcher uses every visible GPU when the world size is omitted:
```bash
tools/run_distributed_t2va.sh ulysses
tools/run_distributed_t2va.sh tensor
```
Pass an explicit count and attention backend when needed:
```bash
tools/run_distributed_t2va.sh tensor 8 sdpa
```
Relevant environment variables:
- `H3_WORLD_SIZE`: fallback world size when no positional count is supplied.
- `H3_DISTRIBUTED_BENCHMARK`: benchmark JSON path.
- `H3_DISTRIBUTED_OUTPUT`: report and optional latent output directory.
- `H3_MODEL_PATH` and `H3_TEXT_ENCODER_PATH`: checkpoint paths.
- `H3_SAVE_LATENTS=0`: write reports without retaining large latent files.
Run every feasible target count on the current machine with:
```bash
H3_SAVE_LATENTS=0 tools/run_distributed_matrix.sh
```
The default matrix runs Ulysses and tensor modes at 1, 2, 4, 6, and 8 GPUs and
skips counts larger than the visible device count. Override the lists with
`H3_GPU_COUNTS` and `H3_DISTRIBUTED_MODES`.
The lower-level transport benchmark does not load H3 weights:
```bash
tools/run_ulysses_benchmark.sh
tools/run_ulysses_benchmark.sh 6 sdpa
```
## Validation
Automated Gloo tests cover:
- Ulysses transport identity at 2, 4, 6, and 8 ranks.
- Two-rank distributed SDPA parity.
- Ragged gather and reduce-scatter behavior.
- Distributed final-projection parity.
- TP attention and MLP parity at 2 and 6 ranks.
- Packed NVFP4 column- and row-shard layout preservation.
On GB10, the real 50-block NVFP4 checkpoint completed both one-rank distributed
paths at 864x480, 141 frames, 12 steps, seed 440420. Ulysses and tensor modes
produced identical video and audio tensors with zero maximum absolute error.
This validates integration and the world-size-one identity path, but it does not
replace multi-GPU NCCL parity testing.
## RunPod
The target is one eight-GPU RTX PRO 6000 Blackwell machine. Query current stock
using the guarded API v2 client:
```powershell
$env:RUNPOD_API_KEY = "..."
python .\tools\runpod_api.py catalog --count 8
```
After choosing an available data center, create the pod explicitly:
```powershell
python .\tools\runpod_api.py create --count 8 --datacenter US-XX-N --yes
```
Attach an existing network volume and inject an SSH key when required:
```powershell
python .\tools\runpod_api.py create --count 1 --datacenter EUR-IS-1 `
--network-volume VOLUME_ID --volume-mount-path /runpod-volume `
--ssh-public-key $HOME\.ssh\id_ed25519.pub --yes
```
The client defaults to the server-edition RTX PRO 6000 Blackwell and RunPod's
x86_64 CUDA 13.0, Torch 2.9.1 image. It creates persistent workspace storage and
enables SSH. Creation and termination require `--yes` to avoid accidental spend
or data loss. Use `get` to poll status and `terminate POD_ID --yes` when finished.
Transfer this checkout plus these two existing checkpoints to the pod workspace:
- `minimax_h3_fl2va_pruned_nvfp4.safetensors` (12,528,636,800 bytes)
- `qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` (15,687,142,551 bytes)
Install the project dependencies in the RunPod image, set the checkpoint path
variables, run `test_distributed.py`, then run the transport and generation
matrices. `Dockerfile.runpod` provides the equivalent reproducible custom image
once an x86_64 image builder and registry are available.
SageAttention2 is not currently packaged in the generic x86 image, so cloud
correctness and scaling start with SDPA. Sage2 can be measured after an x86
wheel is added without changing the distributed layout.
The first matched one-GPU Server Edition run completed on CUDA 13.2 and Torch
2.9.1+cu130. At 864x480, 141 frames, 12 steps, seed 440420, two SDPA sampling
runs took `28.43s` and `28.57s` (mean `28.50s`). The same tensor runner took
`126.66s` on GB10, making the RTX PRO 6000 `4.44x` faster. Peak allocated memory
during RTX sampling was 14,049,528,832 bytes. Both RTX repeats produced identical
checksums; cross-device latent parity was not tested because no latent was saved.
Two-GPU SDPA results show that Ulysses is faster than TP at all measured shapes.
Against one GPU, Ulysses reached `1.21x` at 864x480/141 frames, `1.55x` at
1344x768/124 frames, and `1.72x` at 1344x768/243 frames. The cards expose no
NVLink. `nvidia-smi topo -p2p r` and `-p2p w` report `OK` in both directions,
and NCCL 2.27.7 selected `P2P/CUMEM` with GDR enabled and zero NVLS channels.
RunPod allocations varied between cross-NUMA `SYS` and same-NUMA `NODE`
topologies, so topology must be recorded with every scaling result.
## Remaining GPU Gates
- Real NCCL identity at 4, 6, and 8 GPUs; two-GPU transport is complete.
- Distributed-versus-single latent parity above one rank.
- Full 2/4/6/8 timing, transport, and memory reports; the one-GPU SDPA baseline
is complete.
- Quality comparison after selecting an x86 attention backend.

24
Dockerfile.runpod
View file

@ -1,24 +0,0 @@
# x86_64 CUDA 13 image for RunPod Blackwell distributed validation.
FROM pytorch/pytorch:2.9.1-cuda13.0-cudnn9-devel
ARG COMFY_KITCHEN_VERSION=0.2.31
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ffmpeg git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /opt/h3-blackwell-runtime
COPY . .
RUN python -m pip install --no-cache-dir \
"comfy-kitchen==${COMFY_KITCHEN_VERSION}" \
"fastsafetensors>=0.1.10" \
"numpy>=2.0" \
"pillow>=11" \
"safetensors>=0.5.0" \
"scipy>=1.14" \
"transformers>=4.51,<5"
ENV PYTHONPATH=/opt/h3-blackwell-runtime/src
ENTRYPOINT []
CMD ["bash"]

10
Dockerfile.spark
View file

@ -11,16 +11,10 @@ COPY wheels/sageattn3-*.whl /tmp/wheels/
RUN python -m pip install --no-cache-dir --no-deps /tmp/wheels/sageattn3-*.whl \
&& rm -rf /tmp/wheels
RUN python -m pip install --no-cache-dir --no-deps comfy-kitchen==0.2.31
RUN python -m pip install --no-cache-dir --no-deps comfy-kitchen==0.2.28
RUN python -m pip install --no-cache-dir "fastsafetensors>=0.1.10"
# Official CuTeDSL FlashAttention-4 beta with CUDA 13 Blackwell support.
RUN python -m pip install --no-cache-dir --pre \
"flash-attn-4[cu13]==4.0.0b27" \
"nvidia-cutlass-dsl[cu13]==4.6.2" \
"quack-kernels==0.6.4"
RUN python -m pip uninstall -y pynvml \
&& python -m pip install --no-cache-dir nvidia-ml-py
@ -29,7 +23,7 @@ RUN git clone https://github.com/Saganaki22/ComfyUI-sol-attn.git /opt/ComfyUI-so
&& git checkout ${SOL_ATTN_COMMIT}
RUN python -m pip install --no-cache-dir --no-deps -e . \
&& python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); assert hasattr(comfy_kitchen, 'int8_attention'); assert hasattr(comfy_kitchen, 'int8_attention_is_available'); print(torch.__version__, torch.version.cuda)"
&& python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); print(torch.__version__, torch.version.cuda)"
ENV H3_MODEL_PATH=/models/minimax_h3_ref2va_pruned_nvfp4.safetensors
ENV PYTHONPATH=/opt/ComfyUI-sol-attn

71
FLASH4.md
View file

@ -1,71 +0,0 @@
# FlashAttention-4
The hot runtime exposes the official FlashAttention-4 CuTeDSL Blackwell kernel
as the strict `flash4` attention backend. It does not silently fall back to
SageAttention or SDPA.
## Versions
- `flash-attn-4==4.0.0b27`
- `nvidia-cutlass-dsl==4.6.2` with the CUDA 13 extra
- `quack-kernels==0.6.4`
- Upstream: <https://github.com/Dao-AILab/flash-attention>
- License: BSD-3-Clause
The package is a beta release. The Spark image pins these versions rather than
tracking the latest prerelease.
## API
Select Flash4 per request:
```json
{
"prompt": "A simple cinematic scene.",
"output": "/output/flash4-example.mp4",
"width": 960,
"height": 544,
"frames": 124,
"steps": 12,
"attention": "flash4"
}
```
H3's main attention blocks call Flash4 in native
`[batch, sequence, heads, head_dim]` layout. The adapter requires CUDA, matching
FP16/BF16 Q/K/V tensors, matching shapes, and H3's head dimension of 128. Any
violation raises an error instead of changing kernels.
## GB10 Validation
Flash4 dispatches its SM120 forward implementation on the GB10, which PyTorch
reports as compute capability 12.1.
| Validation | Result |
| --- | --- |
| BF16 `[1,257,8,128]` vs SDPA | max abs `0.001953125`, mean abs `5.04e-05` |
| BF16 `[1,20480,56,128]` vs SDPA | max abs `0.000244141`, mean abs `1.96e-08` |
| Tiny joint AV H3 smoke | passed, no fallback |
| 28 runtime contracts | passed |
Matched base 12-step talking benchmark at `960x544x124`, seed `440410`:
| Backend | Sampling | Relative to Flash4 |
| --- | ---: | ---: |
| Sage2 | `114.67s` | `-19.4%` |
| Flash4 | `142.23s` | baseline |
| PyTorch SDPA | `144.40s` | `+1.5%` |
Flash4 is therefore close to SDPA and materially slower than Sage2 for this H3
workload on GB10. Its value is as another exact-style attention and audio-quality
comparison, not as the current speed default.
The matched Flash4 proof contains H.264 video and stereo 32 kHz AAC audio:
```text
/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-flash4-960x544.mp4
```
The current CuTeDSL package emits a one-time `AuxData` JIT argument warning.
Compilation and inference still complete successfully, but the warning should
be rechecked when upgrading FlashAttention-4 or CUTLASS DSL.

149
H3_LATENT_UPSCALER.md
View file

@ -1,149 +0,0 @@
# H3 Native Latent Upscaler
The direct runtime can apply LBH-123-AI's learned 3D upscaler to a saved,
normalized MiniMax H3 video latent without a ComfyUI dependency. This is an
experimental spatial upscale. It preserves latent time and does not interpolate
frames.
Upstream implementation and checkpoint:
- <https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler>
- <https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler>
- Revision: `13ccf95d85d120bdbc92c05b1247a6e147bf54bf`
- FP16 SHA-256: `043e5a48e161610ef6c3ea974645220354d06fa618abca15f76d084812eb55c2`
- License: Apache-2.0
## Usage
The hot service supports the complete generation, latent upscale, video/audio
decode, and mux process in one request:
```json
{
"prompt": "A simple cinematic scene.",
"output": "/output/h3-native-upscale/example.mp4",
"width": 960,
"height": 544,
"frames": 124,
"turbo": "4step",
"upscale": 2.0,
"mux_audio": true,
"keep_intermediates": false
}
```
The response reports `source_width`, `source_height`, final `width` and
`height`, and a `latent_upscaled` timing stage. H3 audio bypasses the spatial
upscaler and is decoded and muxed normally.
Only the final muxed MP4 is retained by default. Set `keep_intermediates` to
`true` only when a separate WAV and video-only MP4 are needed for diagnostics;
`save_latent` remains a separate opt-in field.
For offline diagnostics, save the normalized latent during generation by
setting `save_latent`, then run:
```bash
python tools/upscale_video_latent.py \
--latent /output/source.pt \
--model /upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors \
--output /output/source-h3latent2x.pt \
--scale 2 \
--precision float16
python tools/decode_video_latent.py \
--latent /output/source-h3latent2x.pt \
--output /output/source-h3latent2x.mp4 \
--vae-dtype float16 \
--vae-tile-size 256
```
The upscaled state retains other source metadata and any audio latent. Its
`width` and `height` fields are updated from the resulting H3 latent dimensions.
## GB10 Validation
Validated on 2026-08-20 with the Posly hero image, Turbo 4-step v1.1, Sage2,
seed `440408`, and 124 frames:
| Stage | Result |
| --- | --- |
| Source | `960x544`, latent `[1,24,37,34,60]` |
| Learned 3D upscale | `10.94s`, latent `[1,24,37,68,120]` |
| Checkpoint load | `1.36s` |
| Tiled FP16 decode command | `137.67s`, `1920x1088` H.264 |
| Source vs downscaled result | SSIM `0.924986` |
The learned stage adds visible edge, face, and food detail. It also produces a
harsher illustrated texture, ringing/chromatic edges in some high-contrast
areas, and small facial changes. It is not fidelity-safe for protected UI,
text, QR codes, or identity-critical material without visual review.
The upstream example follows latent upscaling with a second target-resolution
H3 sampling pass using sigmas `0.9035, 0.6316, 0.3158, 0.0`. That refinement
was deliberately excluded from this first test so the upscaler itself could be
evaluated. It adds generation cost and another opportunity to alter content.
## Proofs
Spark output directory:
```text
/home/daniel/StoryStudioAssets/H3-output/posly/upscale-h3-native/
```
Key files:
- `01-posly-hero-turbo4-source-960x544.mp4`
- `01-posly-hero-turbo4-h3latent2x-1920x1088.mp4`
- `source-frame60-right-crop-lanczos.png`
- `h3latent2x-frame60-right-crop.png`
This result is promising for cinematic footage but does not replace the proven
LTX 2.3 chain yet. Run a matched LTX 2.3 comparison and a carefully bounded H3
refinement test before selecting a default production path.
## Integrated T2V Validation
The single-request path was validated with a matched 5.17-second talking-head
T2V pair at seed `440410`. Both requests sampled at `960x544`, applied the
resident 2x upscaler, decoded at `1920x1088`, and muxed native H3 audio into the
only retained artifact.
| Variant | Sampling | Latent upscale | VAE decode | Wall time |
| --- | ---: | ---: | ---: | ---: |
| Turbo 4-step v1.1 | `56.54s` | `7.52s` | `82.74s` | `163.02s` |
| Turbo 8-step v1.0 | `113.41s` | `7.28s` | `83.69s` | `220.06s` |
| Base 12-step beta/RES | `114.67s` | `7.26s` | `83.34s` | `221.66s` |
Both outputs contain H.264 video at 24 fps and stereo AAC audio at 32 kHz. The
4-step audio measured `-31.4 dB` mean / `-10.9 dB` peak; the 8-step audio
measured `-23.0 dB` mean / `-5.0 dB` peak; and the base 12-step audio measured
`-22.4 dB` mean / `-5.2 dB` peak.
```text
/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-h3native2x-1920x1088.mp4
/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-turbo8-h3native2x-1920x1088.mp4
/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-h3native2x-1920x1088.mp4
```
### Attention Audio Isolation
A matched base 12-step run used direct PyTorch SDPA instead of Sage2. Upscaling
was disabled, but that does not affect the audio comparison because the learned
upscaler runs after joint AV sampling and only receives the separated video
latent.
| Attention | Sampling | Final size | Audio level | Wall time |
| --- | ---: | ---: | ---: | ---: |
| Sage2 | `114.67s` | `1920x1088` after post-sample 2x | `-22.4 dB` mean / `-5.2 dB` peak | `221.66s` |
| PyTorch SDPA | `144.40s` | `960x544`, no upscale | `-22.5 dB` mean / `-2.3 dB` peak | `171.21s` |
The SDPA comparison file is:
```text
/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-sdpa-960x544.mp4
```
Signal levels alone cannot establish dialogue quality or lip sync; the matched
clips require listening review to determine whether Sage2 is the cause.

167
H3_PROMPT_GUIDE.md
View file

@ -1,167 +0,0 @@
# H3 Prompt Guide
Project guidance for MiniMax H3 audiovisual prompts. This document combines
MiniMax's official prompt format with observations from our direct-runtime and
ComfyUI tests.
## Core Principle
Treat an H3 prompt as an audiovisual timeline, not as a loose prose image
description. For T2VA and keyframe modes, use these exact fields in this order:
```text
integrated_multimodal_description: ...
overall_soundscape: ...
non_diegetic_music: ...
```
- `integrated_multimodal_description` contains shots, visible actions,
speakers, exact dialogue, singing, and synchronized diegetic events.
- `overall_soundscape` contains ambience, Foley, and non-verbal human sounds
such as breathing, laughter, and panting.
- `non_diegetic_music` contains only music heard by the audience rather than
the characters. Use `N/A` when no score is wanted.
T2VA begins directly with these fields. I2VA, FL2VA, and L2VA add the official
picture-alignment instruction before them. Ref2VA uses a different six-section
format and should follow the official reference guide.
## Dialogue Syntax
Give every speaking character a stable speaker ID such as `(S1)` or `(S2)`.
Describe the speaker, voice, action, and delivery outside the dialogue block.
Use quoted speech as the project default and state the language outside it:
```text
The woman with a low, clear voice and measured pace (S1) says in English in a
flat, matter-of-fact delivery: "The meeting starts at three."
```
MiniMax's official guide recommends `<d>[Language]...</d>`. Do not use that
markup by default in this project. In a matched 10-seed Sage2 sweep, tagged
dialogue produced immediate first-100ms activity in 9/10 cases, while quoted
dialogue produced it in 0/10. All ten quoted WAVs passed subjective listening.
See [`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md).
Rules:
- Preserve the dialogue wording and punctuation exactly.
- Specify the language explicitly in prose, such as `says in English`.
- Describe pitch, timbre, pace, volume, accent, and emotional restraint only
when useful.
- Keep the same speaker ID across shots.
- Use a compound ID such as `(S1,S2)` only when speakers vocalize together.
- Reserve `<d>[Language]...</d>` for controlled compatibility experiments until
the upstream startup-audio defect is resolved.
- Describe visible text explicitly without reusing dialogue syntax, for example
`a sign visibly reads MEETING ROOM`.
- For voiceover, use `says in an off-screen voiceover` and state that the
visible character's lips remain completely closed.
## Lip Sync And End State
Describe speech as a physical action and specify what happens when it ends:
```text
Her mouth movements naturally synchronize with each spoken word. Immediately
after the final word, her lips meet and her jaw ceases speaking motion. She
remains silent through the final frame.
```
This is especially important in short clips. Without an explicit end state, H3
may continue mouth motion or add a non-verbal vocal reaction after the line.
For multiple speakers, state whose lips move and whose remain still during each
line and pause.
## Avoiding Unwanted Laughter
Laughter is a non-verbal human sound, so control it in `overall_soundscape`, not
inside the dialogue block:
```text
overall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum.
No laughter, chuckling, giggling, sighing, gasping, audible breathing, filler
sounds, audience reaction, or other voices.
```
Also reinforce the visible end state in the shot description: closed lips,
stopped jaw motion, neutral expression, and silence through the final frame.
These exclusions are project guidance, not a guaranteed negative-prompt
mechanism. H3 jointly generates video and audio, so unwanted sounds remain
probabilistic.
Avoid positive cues that can conflict with the exclusion. In our first dialogue
test, `She smiles slightly` repeatedly became laughter across Sage2, PyTorch
SDPA, and FlashAttention-4. Since the behavior survived attention changes, the
prompt was the stronger common cause. Use observable neutral behavior instead:
```text
She maintains a neutral, composed, closed-mouth expression.
```
## Short Dialogue Tests
For a five-second diagnostic clip:
- Use one visible speaker and one short sentence.
- Use one static shot with no cuts.
- Choose emotionally neutral wording.
- Request a flat or matter-of-fact delivery.
- Specify the post-dialogue closed-mouth state.
- Permit only a simple, positive acoustic bed such as dry room tone.
- Set `non_diegetic_music: N/A`.
Do not use `overall_soundscape: N/A` for a speaking test. The official guide
reserves it for complete silence throughout the video.
## Reusable Neutral T2VA Prompt
```text
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static
medium close-up frames exactly one adult woman seated alone in an otherwise
empty, quiet cafe. Warm natural window light falls evenly across her face. She
maintains a neutral, composed, closed-mouth expression and looks steadily
toward the camera. The woman with a low, clear voice and slow, even speaking
pace (S1) physically speaks once in a flat, matter-of-fact delivery:
"The meeting starts at three." Her mouth movements naturally
synchronize with each spoken word. Immediately after the final word, her lips
meet and her jaw ceases speaking motion. She remains silent and maintains the
same neutral expression through the final frame. The camera remains completely
static with no cuts.
overall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum.
No laughter, chuckling, giggling, smiling vocalization, sighing, gasping,
audible breathing, filler sounds, audience reaction, or other voices.
non_diegetic_music: N/A
```
## Prompt Checklist
- Correct task mode selected.
- Exact field names and order preserved.
- Every shot has observable visual and audible events.
- Every speaker has a stable ID.
- Dialogue uses quoted speech.
- Spoken language, delivery, and voice are stated outside the quotation.
- `<d>[Language]...</d>` is avoided unless explicitly testing official syntax.
- Lip motion and post-speech closure are explicit.
- Ambience and non-verbal sounds are in `overall_soundscape`.
- Music is isolated in `non_diegetic_music`.
- Script length fits the requested duration.
- Positive emotional cues do not conflict with audio exclusions.
## Official Sources
- Base prompt guide:
<https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md>
- Reference prompt guide:
<https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md>
- Official prompt-writing skill:
<https://github.com/MiniMax-AI/MiniMax-H3/tree/main/.agents/skills/h3-prompt-writing>
- Official model repository:
<https://huggingface.co/MiniMaxAI/MiniMax-H3>

15
PARITY.md
View file

@ -180,9 +180,9 @@ gate and is exact.
| Component | Implemented | Known limitation |
| --- | --- | --- |
| Qwen text and vision conditioning | Yes | Text-only prompt parity is established; first/last keyframe vision conditioning is implemented, while arbitrary reference video/audio remains missing |
| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags |
| Token refiner | Yes | Bit-exact from captured 5376-wide refiner input through both blocks and final RMSNorm; Qwen-to-refiner projection boundary is still not separately captured |
| T2VA/keyframe packer | Yes | Bit-exact for the coherent captured prompt-only DiT input; first-frame, last-frame, and first/last keyframe condition rows are implemented |
| Prompt-only FL2VA packer | Yes | Bit-exact for the coherent captured text-only FL2VA DiT input; no keyframe/reference condition rows |
| H3 DiT backbone | Yes | Bit-exact through all 50 blocks from the coherent assembled FL2VA input; requires the standalone Comfy Kitchen fused Q/K RMSNorm + split-half RoPE operator |
| H3 final layer | Yes | Bit-exact final AdaLN, target-row modulation, and video/audio patch rows; Comfy materializes the AdaLN and output-head biases through BF16 |
| H3 DiT | Yes | Strict all-block numeric parity not achieved |
@ -190,17 +190,18 @@ gate and is exact.
| Video VAE decoder | Yes | Direct VAE temporal assembly matches upstream after overlap fix. FP16 is the default Comfy-equivalent runtime path; cat benchmark VAE decode is `25.085s`. Use FP32 only for exact direct diagnostics. |
| Audio VAE/decode/mux | Yes | Direct decoder-only MiniMax H3 audio VAE returns stereo `32000 Hz` waveform and muxes with generated video. Native audio latent scaling is fixed. |
| End-to-end prompt-only FL2VA preview | Yes | Apples-to-apples warm cat benchmark is at Comfy parity: Comfy warm `150.26s`; direct warm after text conditioning `149.304s`; direct warm including text conditioning `151.465s`. |
| Full Ref2VA | No | Image keyframes, vision conditioning, VAE encode, generated audio, and muxing are implemented; arbitrary reference video/audio and identity/voice conditioning remain missing |
| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented |
## Remaining Gates, In Dependency Order
Only these are outstanding. Do not recapture or revisit rows marked complete
unless the checkpoint, Comfy version, prompt, or backend changes.
1. **Distributed validation.** Run real NCCL parity and performance sweeps at
2/4/6/8 GPUs; CPU transport and one-GPU real-checkpoint identity are complete.
2. **Full Ref2VA support.** Add arbitrary reference video/audio, identity and
voice conditioning, and reference-path validation gates.
1. **Feature/performance work.** Prompt-only FL2VA is now closed against the
warm Comfy baseline. Optimize load/caching/sampling, then evaluate Sage3,
CUDA graphs, and multi-GPU execution.
2. **Full Ref2VA support.** Add references, vision conditioning, VAE encode,
and reference-path validation gates.
## Existing Tools And Their Intended Gate

37
PLAN.md
View file

@ -1,8 +1,5 @@
# H3 Blackwell Runtime Plan
Current implementation status is tracked in [`CURRENT_STATE.md`](CURRENT_STATE.md).
The dated handoffs below are retained as historical investigation records.
## Goal
Build a direct MiniMax H3 Ref2VA runtime for Blackwell and Grace Blackwell that consumes the current Comfy safetensors checkpoints while removing ComfyUI and Raylight from the denoising critical path.
@ -61,9 +58,7 @@ Prompt-only FL2VA is now at warm Comfy parity with the direct Sage2 baseline. Fe
3. Add exact memory/lifetime optimizations next: `kj_head_sliced` and `kj_chunked_ffn`. These must preserve the validated direct outputs before being kept.
4. Evaluate prior H3-tested attention candidates as standalone adapters: `sol_attn` and `kj_sage`.
- `kj_sage` is implemented as explicit SageAttention mode backends: `kj_sage_cuda`, `kj_sage_triton`, `kj_sage_fp8`, and `kj_sage_fp8pp`; all passed hot-runtime smoke tests.
- `sol_attn` is implemented through the pinned ComfyUI Triton source vendored
into the Spark image. The native QKV layout is deployed; the inactive fused
layout remains a known repair item.
- `sol_attn` is still blocked on locating/adding the standalone Sol-Attn source or package. It is not installed in the Spark image and is not present in this repository.
5. Evaluate approximate denoiser caches only after exact baselines are recorded: `easycache` and `h3_cache`.
- Initial direct cache modes are implemented as opt-in approximate sampler modes. They reuse cached denoised deltas and report skipped-step stats; full-size quality/threshold sweeps are still required before using them for production output.
6. Keep every backend explicit per run, with separate quality and timing records for sampling, VAE, audio, and end-to-end output.
@ -114,36 +109,6 @@ DGX Spark caveat:
- Record attention, GEMM, communication, VAE, and end-to-end timings separately.
- Treat SageAttention3 as an experimental quality-gated kernel for H3.
## 2026-08-15 Handoff
Current quality baseline:
- `sage2` is the correct visual path for the 960x544, 124-frame, 12-step cat prompt at seed `440407`.
- Fresh Spark hot-runtime Sage2/Vortex A/B asset: `\\192.168.1.162\StoryStudioAssets\H3-output\h3-blackwell-runtime\cat-sage2-vortex-5s-440407-ab.mp4`.
- `sol_attn` is faster but visually wrong for the same prompt/seed. Keep it experimental until localized and quality-gated.
- Sol-native BSHD layout is exact versus the old Sol path, so the visual issue is likely Sol's approximation/configuration, not the layout removal.
Tomorrow priority:
1. Diagnose `sol_attn` quality failure.
- Save per-step latents for `sage2` and `sol_attn` with identical prompt, seed, dimensions, and sampler settings.
- Measure per-step max/mean/RMSE/cosine divergence.
- Try hybrid schedules: early steps `sage2`, later steps `sol_attn`; also try Sage2 every Nth step.
- Sweep Sol params only after the divergence point is known: `H3_SOL_TAU`, `H3_SOL_THRESH_TYPE`, `H3_SOL_INT8_QK`, `H3_SOL_INT8_PV`.
2. Optimize the quality path (`sage2`).
- Use `sage2` as visual/numerical reference.
- Attack Q/K/V HND layout and Q/K RMS+RoPE first; a fused Q/K RMS+RoPE + Sage HND layout kernel is the most plausible exact win.
- Keep Sage3 experimental; previous measurements were slower and visually/numerically different.
3. Add durable hot-runtime request logging.
- Log `generate_start`, `generate_complete`, and `generate_failed` to stdout with output path, seed, attention, dimensions, and timings.
- This prevents losing timing data when a client disconnects before reading the synchronous response.
4. Check LTX 2.5 downloads.
- Detached Spark container: `ltx25-nvfp4-download`.
- Expected files under `/home/daniel/aeon-spark-test/h3/comfy-models`:
- `diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`
- `text_encoders/gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors`
- Official `Lightricks/LTX-2.5` does not list a matching NVFP4 text encoder; the int8 ConvRot Gemma4 encoder is the matching low-VRAM official text encoder.
## 2026-08-13 VAE Debug Handoff
Resolved on `vae-decode-optimization`: direct VAE temporal overlap constants now match upstream, audio decode/mux is implemented, and Comfy-equivalent FP16 video VAE is the default runtime path. The 960x544x124 cat benchmark now matches warm Comfy performance: Comfy `150.26s`, direct `149.304s` after text conditioning, direct VAE decode `25.085s`.

213
README.md
View file

@ -1,178 +1,73 @@
# H3 Blackwell Runtime
Direct MiniMax H3 audiovisual inference for NVIDIA Blackwell and Grace Blackwell.
The runtime consumes the current Comfy-format NVFP4 checkpoints while keeping
ComfyUI out of the inference critical path. ComfyUI remains the checkpoint and
correctness oracle.
Direct MiniMax H3 Ref2VA runtime research project. ComfyUI is the checkpoint and correctness oracle, not the target runtime.
## Current Status
## First Gate
Implemented and validated:
Inspect the mounted H3 NVFP4 safetensors headers before designing an importer:
- Prompt-only T2VA with jointly generated video and stereo audio.
- First-frame I2VA, last-frame L2VA, and first/last-frame FL2VA conditioning.
- Qwen3-VL text and image conditioning, token refinement, H3 packed denoising,
beta/RES sampling, video and audio VAE decode, H.264/AAC encoding, and muxing.
- Resident HTTP runtime with startup warmup, model reuse, request timing, and
request-level attention selection.
- SageAttention2 correctness default plus SDPA, forced cuDNN SDPA,
FlashAttention-4, Sage3, KJ Sage, Comfy Kitchen INT8, head-sliced, and Sol-Attn
experimental backends.
- Official FL2VA Turbo 4-step and 8-step adapters.
- Optional H3-native 3D latent upscaling.
- Opt-in EasyCache and H3-Cache experiments.
- Ragged Ulysses sequence parallelism and true NVFP4 tensor parallelism with
automatic 1/2/4/6/8-GPU launch tooling.
The main remaining feature gap is full arbitrary Ref2VA, especially reference
video/audio, identity, and voice conditioning. Multi-GPU code is CPU- and
one-GPU-validated; real 2/4/6/8-GPU NCCL scaling measurements are still pending.
See [`CURRENT_STATE.md`](CURRENT_STATE.md) for the canonical detailed status.
## Checkpoints
The default runtime paths are:
```text
/models/minimax_h3_fl2va_pruned_nvfp4.safetensors
/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors
/vae/ae.safetensors
/vae/mini_vae.safetensors
```powershell
python .\tools\inspect_safetensors.py /runpod-volume/ComfyUI/models/diffusion_models/minimax_h3_ref2va_pruned_nvfp4.safetensors
```
The denoiser and Qwen checkpoints use Comfy Kitchen NVFP4 layouts. Do not
convert or dequantize them during loading.
Write the output to `artifacts/checkpoints/` on the mounted volume. The result must identify packed weights, scales, and tensor naming before any kernel conversion work begins.
## Benchmark Contract
`benchmarks/ref2va-960x544-124f.json` is the single-GPU performance contract. Record direct-runner results as JSON and compare them with:
```powershell
python .\tools\compare_benchmark.py --result direct-result.json
```
## DGX Spark
`Dockerfile.spark` and `compose.spark.yml` provide the ARM64 CUDA 13 runtime used
on GB10. Build and start the resident service from the Spark checkout:
`Dockerfile.spark` and `compose.spark.yml` prepare an ARM64 GB10 development image using the existing AEON CUDA 13/SageAttention3 base. The compose target opens a shell only; it does not start inference.
### Forgejo Pulls From Spark
The Spark checkout uses Forgejo through the host's published local SSH port and a dedicated key:
```bash
cd /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime
git config core.sshCommand 'ssh -i ~/.ssh/id_ed25519_forgejo_h3 -o IdentitiesOnly=yes'
git remote set-url origin ssh://git@127.0.0.1:2222/daniel/h3-blackwell-runtime.git
git pull --ff-only origin master
```
The private key remains on Spark at `~/.ssh/id_ed25519_forgejo_h3`; only its public key is registered in Forgejo.
## Runtime Output
Generation and latent-decode tools are quiet by default: they suppress ffmpeg banners and only print compact JSON summaries. Use these flags when debugging:
- `--progress`: print per-step sampler timing in `tools/direct_t2v_preview.py`.
- `--profile-memory`: print memory checkpoints in `tools/direct_t2v_preview.py`.
- `--ffmpeg-loglevel info`: show ffmpeg details instead of the default `error` level.
- `--quiet`: suppress JSON summary lines.
- `--vae-dtype float16`: use Comfy-style FP16 video VAE decode in `tools/direct_t2v_preview.py` or `tools/decode_video_latent.py`; this is the default runtime path. Use `--vae-dtype float32` only for exact direct-path diagnostics. `tools/direct_t2v_preview.py` also accepts `H3_VAE_DTYPE`.
- `--vae-tile-size 256`: set the direct video VAE spatial tile size. `tools/direct_t2v_preview.py` also accepts `H3_VAE_TILE_SIZE`.
Standalone `tools/compare_*`, `tools/trace_*`, `tools/inspect_*`, and `tools/patch_comfy_*` scripts are debugging utilities and remain opt-in by being separate commands.
## Hot Runtime Service
`tools/serve_hot_runtime.py` keeps Qwen, H3, video VAE, and audio VAE resident in one process. Start the optional Spark service with:
```bash
docker compose -f compose.spark.yml build h3-hot-runtime
docker compose -f compose.spark.yml up -d h3-hot-runtime
curl http://127.0.0.1:8001/ready
```
The service listens on container port 8000 and Spark host port 8001. It keeps
Qwen, H3, both VAEs, Turbo adapters, and the optional latent upscaler resident.
SageAttention2 is the default because it matches the established Comfy quality
baseline.
Use `GET /ready` to confirm resident model readiness. Use `POST /generate` with JSON fields like `prompt`, `output`, `width`, `height`, `frames`, `steps`, `seed`, and optional `attention`. Supported request-level attention values are reported by `/ready`; switching attention does not reload model weights.
## HTTP API
Exact memory/lifetime options:
`GET /health` and `GET /ready` report readiness, loaded options, warmup results,
and attention backend status. `POST /generate` performs one serialized request:
- `attention: "kj_head_sliced"` slices attention heads and runs the slice backend from `H3_HEAD_SLICE_BACKEND` (`sage2` by default) with `H3_HEAD_SLICE_SIZE` heads per slice (`8` by default).
- `attention: "sol_attn"` routes eligible H3 attention calls through the pinned ComfyUI Sol-Attn Triton kernel vendored into the Spark image. Configure with `H3_SOL_TAU` (`1.3`), `H3_SOL_MIN_TOKENS` (`4096`), `H3_SOL_THRESH_TYPE` (`diag`), `H3_SOL_INT8_QK`, `H3_SOL_INT8_PV`, `H3_SOL_FALLBACK` (`sage2`), and `H3_SOL_STRICT`.
- `--mlp-chunks N` on `tools/serve_hot_runtime.py` or `tools/direct_t2v_preview.py` chunks H3 SwiGLU rows exactly to reduce peak activation memory. Default is `1` (disabled).
```bash
curl -X POST http://127.0.0.1:8001/generate \
-H 'Content-Type: application/json' \
-d '{
"prompt": "A quiet medium shot of a woman by a rain-streaked window. She says, \"We should leave before dawn.\"",
"output": "/output/h3-blackwell-runtime/example.mp4",
"width": 864,
"height": 480,
"frames": 141,
"steps": 12,
"seed": 440420,
"attention": "sage2"
}'
```
Approximate cache options are opt-in and must be quality-gated per prompt:
Optional request fields include `first_frame`, `last_frame`, `turbo`, `upscale`,
`mux_audio`, `save_latent`, `keep_intermediates`, `cache_mode`, and cache tuning
parameters. Keyframes accept an on-disk image path, base64 payload, or data URL.
The presence of first and last frames selects I2VA, L2VA, or FL2VA behavior; a
named task field is not yet exposed.
Set `turbo` to `"4step"` or `"8step"`; the service enforces the corresponding
step count and Turbo schedule. Set `upscale` to `2.0` to run the resident learned
latent upscaler before video decode. Turbo and denoiser caching cannot be
combined.
Quoted dialogue is the project prompt default. Tagged `<d>[English]...` dialogue
is a repeatable startup-audio trigger and should not be used as the default.
Never apply unconditional audio trimming or fading because valid sound can begin
at the first sample. See [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md) and
[`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md).
## Distributed Execution
Two batch-generation modes are available through `torchrun`:
- `ulysses`: token-sharded blocks with ragged sequence-to-head all-to-all around
attention.
- `tensor`: sequence-sharded residuals plus NVFP4-sharded QKV, attention output,
MLP FC1, and MLP FC2 projections.
Use all visible GPUs or pass an explicit world size:
```bash
tools/run_distributed_t2va.sh ulysses
tools/run_distributed_t2va.sh tensor 8 sdpa
H3_SAVE_LATENTS=0 tools/run_distributed_matrix.sh
```
The matrix runs feasible 1/2/4/6/8-GPU configurations and skips counts larger
than the visible device count. Six ranks use ragged head ownership
`[10, 10, 9, 9, 9, 9]`; no semantic token padding is introduced.
See [`DISTRIBUTED.md`](DISTRIBUTED.md) for the collective contracts, validation
evidence, environment variables, RunPod provisioning client, and x86 packaging
status.
## Validation
Run the contract suite with:
```bash
python -m unittest discover -s tests -v
```
The current suite has 38 passing tests. Distributed tests cover 2/4/6/8-rank
transport identity, ragged collectives, SDPA parity, final projection parity,
NVFP4 shard layout, and 2/6-rank TP attention/MLP math. On GB10, real-checkpoint
one-rank Ulysses and tensor paths produced exactly equal video and audio latents
at 864x480, 141 frames, 12 steps, seed 440420.
A matched one-GPU RunPod RTX PRO 6000 Blackwell Server SDPA test averaged
`28.50s` sampling over two runs versus `126.66s` for the same tensor runner on
GB10, a `4.44x` speedup. The two RTX runs were within `0.49%` and produced
identical checksums. See
`benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`.
Two-GPU Ulysses scaling improves as the packed sequence grows: `1.21x` at
864x480/141 frames, `1.55x` at 1344x768/124 frames, and `1.72x` at
1344x768/243 frames. These RTX PRO 6000 Server cards have no NVLink. CUDA P2P
read/write is available and NCCL selected `P2P/CUMEM`; topology varied between
`SYS` and `NODE` across RunPod allocations. See
`benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`.
Matched GB10 backend results and parity evidence are recorded in [`PLAN.md`](PLAN.md)
and [`PARITY.md`](PARITY.md). Standalone `tools/compare_*`, `tools/trace_*`,
`tools/inspect_*`, and `tools/patch_comfy_*` commands are diagnostic utilities,
not runtime startup requirements.
## Documentation
- [`CURRENT_STATE.md`](CURRENT_STATE.md): canonical implemented and missing scope.
- [`DISTRIBUTED.md`](DISTRIBUTED.md): Ulysses, TP, launchers, and RunPod workflow.
- [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md): audiovisual prompting contract.
- [`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md): startup
audio localization and prompt-format evidence.
- [`TURBO.md`](TURBO.md): official Turbo adapters and schedules.
- [`H3_LATENT_UPSCALER.md`](H3_LATENT_UPSCALER.md): learned latent upscaler.
- [`FLASH4.md`](FLASH4.md): FlashAttention-4 integration and benchmark.
- [`PARITY.md`](PARITY.md): direct-versus-Comfy evidence ledger.
- [`PLAN.md`](PLAN.md): historical investigation and future kernel plan.
## Known Gaps
- Full arbitrary Ref2VA reference video/audio and identity/voice conditioning.
- Explicit task schemas and stricter production request validation.
- Real multi-GPU NCCL parity and 1/2/4/6/8 scaling results.
- x86 SageAttention2 packaging for the generic RunPod image; SDPA is the initial
cloud validation backend.
- Production queueing, cancellation, authentication, TLS, metrics, and durable
job state.
- Full quality sweeps for approximate attention, cache, Turbo, and upscaler paths.
- `cache_mode: "easycache"` reuses cached denoised deltas while cumulative latent input change stays below `cache_threshold`.
- `cache_mode: "h3_cache"` reuses cached denoised deltas when the current per-step latent input change is below `cache_threshold`.
- Both modes accept `cache_start_percent`, `cache_end_percent`, and `cache_subsample_factor` in `POST /generate`; the CLI exposes equivalent `--cache-*` flags.

94
TURBO.md
View file

@ -1,94 +0,0 @@
# FL2VA Turbo
The hot runtime can keep both official LightX2V FL2VA Turbo adapters resident and switch them per request without rebuilding the packed NVFP4 base model.
## Variants
| API value | Official artifact | NFE | Video/audio shift | Schedule |
| --- | --- | ---: | --- | --- |
| `4step` | `minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors` | 4 | 6/3 | Uniform shifted training-Euler |
| `8step` | `minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors` | 8 | 12/3 | Uniform shifted training-Euler |
Downloaded from Hugging Face revision `ec01fa4c86263832faa0bd1d6d8f36a281eaabb2`:
| Artifact | SHA-256 |
| --- | --- |
| 4-step v1.1 Comfy BF16 | `449d80f301ac571622c72e28b8fd72a4b3681b7a8df8a92f17c8f6ec43f56558` |
| 8-step v1.0 Comfy BF16 | `2339acdf19bfe123f46b971ea35d367a84adb85de43627e1eceafa5a5b2b111e` |
The older unversioned 744 MB adapters remain on Spark but are not loaded by the hot service.
## Implementation
Each official file contains 624 tensors covering 208 linear targets: four projections in each of 50 denoiser blocks and two token-refiner blocks. The runtime validates the complete key set, dimensions, rank, and alpha while loading.
The adapters are applied dynamically:
```text
output = NVFP4_base(x) + strength * (alpha / rank) * B(A(x))
```
The BF16 branch receives the original activation. It does not receive the base checkpoint's `pre_quant_scale`-modified activation, and the LoRA delta is never merged into packed NVFP4 weights. Fused QKV uses rank/alpha 384; the other released projections use rank/alpha 128.
Turbo sampling maintains independent native video and audio states. The 4-step and 8-step variants use their own shifted sigma grids and positive data-ward training-Euler updates. The original beta/RES sampler remains unchanged when Turbo is disabled.
## Hot API
The compose service registers both adapters with repeated startup arguments:
```text
--turbo-lora 4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors
--turbo-lora 8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors
```
Select one per request:
```json
{
"prompt": "A calm restaurant at dusk.",
"output": "/output/example.mp4",
"width": 960,
"height": 544,
"frames": 124,
"seed": 440408,
"turbo": "4step"
}
```
The server chooses and enforces four or eight steps. Use `null` or `"none"` for the base path. Turbo and denoiser caching cannot be combined. `/ready` reports `available_turbos` and `current_turbo`.
## Matched GB10 Benchmark
All runs used the same Posly first frame, prompt, seed `440408`, 960x544 canvas, 124 frames, Sage2, FP16 tiled VAE, and no audio output.
| Mode | Steps | Sampling | Total wall | Speedup vs base 8-step | Sampling reduction |
| --- | ---: | ---: | ---: | ---: | ---: |
| Base beta/RES | 8 | 148.1 s | 179.3 s | 1.00x | baseline |
| Turbo v1.0 | 8 | 121.1 s | 152.3 s | 1.18x | 18.2% |
| Turbo v1.1 768p | 4 | 60.4 s | 91.2 s | 1.97x | 59.2% |
The 4-step run was 1.67x faster end-to-end than the 8-step Turbo run. These are observed hot-service request times, not isolated steady-state kernel timings; shape-specific compilation can affect the first request.
## Validation
- 22 unit/contract tests cover disabled exactness, dynamic branch arithmetic, adapter selection, both official sigma grids, and independent AV training-Euler updates.
- Both official files loaded with all 624 expected keys and all 208 target dimensions validated.
- Both variants generated matched 960x544x124 FL2VA proofs.
- Both variants completed joint AV smoke runs and produced stereo 32 kHz AAC muxed outputs.
- Base warmup remains bit-preserving while no adapter is selected.
Comparison outputs:
```text
\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo4-v1.1-seed440408-5s.mp4
\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo8-v1.0-seed440408-5s.mp4
\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo4-left-turbo8-right-seed440408.mp4
```
## Quality Notes
For the Posly restaurant source, both Turbo variants avoided the severe central lens flare produced by the base-model trials. The 4-step result appeared slightly more compositionally stable; the 8-step result showed more subject motion. Both still brightened the exterior over time, so Turbo does not by itself satisfy the fixed-exposure requirement.
The adapters were released against the BF16 H3 base. Dynamic use over the pruned NVFP4 checkpoint is operationally validated here but is not an upstream-certified pairing. A future strict parity gate should compare per-step denoiser outputs and final latents against the latest LightX2V BF16 reference.
SLA Turbo is intentionally excluded. It requires LightX2V's separate 85% dynamic sparse-attention path and should not be represented as equivalent to these dense variants.

50
benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json
View file

@ -1,50 +0,0 @@
{
"name": "audio-dialogue-format-sweep-sage2-seeds440420-440429",
"measured_at": "2026-08-21",
"hardware": "NVIDIA GB10",
"model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
"attention": "sage2",
"scheduler": "beta",
"sampler": "res_multistep",
"steps": 12,
"resolution": [864, 480],
"frames": 141,
"seeds": [440420, 440421, 440422, 440423, 440424, 440425, 440426, 440427, 440428, 440429],
"comparison": "Identical dialogue semantics with <d>[English]...</d> markup versus quoted speech",
"cases": 20,
"aggregate": {
"quoted_first_100ms_peak_reduction_db": {
"mean": 20.112007323193758,
"median": 21.550717420448272,
"minimum": 3.800458970207643,
"maximum": 27.93839234881301
},
"quoted_first_100ms_rms_reduction_db": {
"mean": 17.932329146978255,
"median": 17.504237701162523,
"minimum": 3.147146068931228,
"maximum": 27.10315527948587
},
"first_100ms_peak_above_minus_40_dbfs": {
"tagged": 9,
"quoted": 0
},
"first_activity_under_100ms": {
"tagged": 9,
"quoted": 0
},
"boundary_decay_above_10db": {
"tagged": 8,
"quoted": 0
}
},
"latent_feature_result": "Simple first-frame and first-block RMS/delta ranges overlap between tagged and quoted cases; no scalar latent threshold is justified.",
"subjective_audio_review": {
"reviewed_at": "2026-08-21",
"quoted_cases_reviewed": 10,
"result": "All ten quoted WAVs judged good"
},
"full_video_lip_sync_review_required": true,
"output_directory": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/audio-dialogue-format-sweep",
"full_report": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/audio-dialogue-format-sweep/report.json"
}

122
benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json
View file

@ -1,122 +0,0 @@
{
"device": "NVIDIA RTX PRO 6000 Blackwell Server Edition",
"attention": "sdpa",
"steps": 12,
"seed": 440420,
"hourly_usd": {
"one_gpu": 2.09,
"two_gpu": 4.18
},
"results": [
{
"resolution": [
864,
480
],
"frames": 141,
"video_seconds_at_24fps": 5.875,
"one_gpu": {
"mode": "tensor",
"sampling_seconds": 28.500914809992537,
"sampling_cost_usd": 0.016546364431356778
},
"two_gpu": {
"mode": "ulysses",
"sampling_seconds": 23.494563594460487,
"sampling_cost_usd": 0.027279798840234674,
"speedup": 1.2130855163750494,
"scaling_efficiency": 0.6065427581875247,
"cost_premium": 0.6486883842916646
}
},
{
"resolution": [
1344,
768
],
"frames": 124,
"video_seconds_at_24fps": 5.166666666666667,
"one_gpu": {
"mode": "tensor",
"sampling_seconds": 97.94385590963066,
"sampling_cost_usd": 0.05686184968086891
},
"two_gpu": {
"mode": "ulysses",
"sampling_seconds": 63.132430186495185,
"sampling_cost_usd": 0.07330376616098608,
"speedup": 1.5514032268408078,
"scaling_efficiency": 0.7757016134204039,
"cost_premium": 0.2891554983243718
}
},
{
"resolution": [
1344,
768
],
"frames": 243,
"video_seconds_at_24fps": 10.125,
"one_gpu": {
"mode": "tensor",
"sampling_seconds": 311.01632468774915,
"sampling_cost_usd": 0.18056225516594326
},
"two_gpu": {
"mode": "ulysses",
"sampling_seconds": 180.68109439313412,
"sampling_cost_usd": 0.2097908262675835,
"speedup": 1.7213551076407902,
"scaling_efficiency": 0.8606775538203951,
"cost_premium": 0.16187531040071534
}
}
],
"two_gpu_mode_comparison": [
{
"resolution": [
864,
480
],
"frames": 141,
"ulysses_sampling_seconds": 23.946259677002672,
"tensor_sampling_seconds": 28.037104761999217
},
{
"resolution": [
1344,
768
],
"frames": 124,
"ulysses_sampling_seconds": 66.31268315999478,
"tensor_sampling_seconds": 74.59926067800552
},
{
"resolution": [
1344,
768
],
"frames": 243,
"ulysses_sampling_seconds": 190.96160273600253,
"tensor_sampling_seconds": 203.40585646100226
}
],
"topology": {
"benchmark_host": "SYS between GPUs, separate NUMA nodes, no NVLink",
"diagnostic_host": "NODE between GPUs, same NUMA node, no NVLink",
"p2p_read": "OK in both directions",
"p2p_write": "OK in both directions",
"nccl_version": "2.27.7+cuda13.0",
"nccl_transport": "P2P/CUMEM",
"nccl_gdr": true,
"nccl_nvls_channels": 0,
"nccl_log": "/runpod-volume/h3-benchmarks/rtxpro6000-server-2gpu-nccl-transport.log"
},
"cost_scope": "Sampling wall time only. Model/Qwen loading, text conditioning, VAE decode, media encoding, muxing, and pod startup are excluded.",
"notes": [
"Two-GPU Ulysses timing in the primary results uses the better NODE topology diagnostic host.",
"The separate two-GPU mode comparison was measured on the SYS topology host.",
"Ulysses was faster than tensor parallelism at every tested shape.",
"Cross-mode and cross-device latent parity was not measured because latents were not retained."
]
}

46
benchmarks/rtxpro6000-server-2gpu-topology.txt
View file

@ -1,46 +0,0 @@
Diagnostic host topology (RunPod EUR-IS-1, CUDA 13.0)
$ nvidia-smi topo -m
GPU0 GPU1 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE SYS SYS 64-127,192-255 1 N/A
GPU1 NODE X SYS SYS 64-127,192-255 1 N/A
NIC0 SYS SYS X PIX
NIC1 SYS SYS PIX X
$ nvidia-smi topo -p2p r
GPU0 GPU1
GPU0 X OK
GPU1 OK X
$ nvidia-smi topo -p2p w
GPU0 GPU1
GPU0 X OK
GPU1 OK X
$ nvidia-smi nvlink -s
(no output; no NVLink devices or links exposed)
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,GRAPH,P2P,COLL summary:
- NCCL version 2.27.7+cuda13.0
- GPU path classified as PHB at 48.0 GB/s by NCCL topology
- 4 collective channels, 4 P2P channels, 2 P2P channels per peer
- 0 NVLS channels
- P2P chunksize 131072
- Check P2P Type isAllDirectP2p 1
- GPU 0 -> GPU 1 via P2P/CUMEM
- GPU 1 -> GPU 0 via P2P/CUMEM
- Connected all rings, PXN 0, GDR 1
The first benchmark allocation differed:
$ nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS 0-55,112-167 0 N/A
GPU1 SYS X 56-111,168-223 1 N/A
$ nvidia-smi nvlink -s
GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition
Device does not have or support Nvlink
GPU 1: NVIDIA RTX PRO 6000 Blackwell Server Edition
Device does not have or support Nvlink

58
benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json
View file

@ -1,58 +0,0 @@
{
"benchmark": "t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json",
"actual_attention": "sdpa",
"mode": "tensor",
"world_size": 1,
"resolution": [
864,
480
],
"frames": 141,
"steps": 12,
"seed": 440420,
"rtx_pro_6000_blackwell_server": {
"cloud": "RunPod Secure",
"data_center": "EUR-IS-1",
"gpu_memory_mib": 97887,
"driver": "595.91.07",
"host_cuda": "13.2",
"torch": "2.9.1+cu130",
"hourly_usd": 2.09,
"sampling_seconds": [
28.431582752993563,
28.57024686699151
],
"sampling_mean_seconds": 28.500914809992537,
"sampling_range_percent": 0.48652513409615622,
"model_load_seconds": [
4.332073616009438,
4.819102452020161
],
"conditioning_seconds": [
8.492386644007638,
8.663792312989244
],
"peak_sampling_allocated_bytes": 14049528832,
"checksums": [
98329.34375,
-345.9080505371094
],
"persistent_reports": [
"/runpod-volume/h3-benchmarks/rtxpro6000-server-1gpu-sdpa-864x480-141f-base12-seed440420.json",
"/runpod-volume/h3-benchmarks/rtxpro6000-server-1gpu-sdpa-864x480-141f-base12-seed440420-repeat2.json"
]
},
"gb10": {
"direct_sdpa_sampling_seconds": 125.3,
"same_tensor_runner_sampling_seconds": 126.66326454500086
},
"speedup": {
"versus_gb10_direct_sdpa": 4.396350111403066,
"versus_gb10_same_tensor_runner": 4.444182419737356
},
"notes": [
"Both RTX PRO 6000 runs produced identical checksums.",
"No latent was retained, so this is a performance and execution-stability gate rather than a cross-device numerical parity gate.",
"The billable pod was terminated after the repeat run; the network volume and reports were preserved."
]
}

349
benchmarks/t2va-dialogue-864x480-141f-base12-sdpa-seed440420.json
View file

@ -1,349 +0,0 @@
{
"name": "t2va-dialogue-864x480-141f-base12-sdpa-seed440420",
"measured_at": "2026-08-20",
"measurement": "Warm resident direct-runtime request including conditioning, sampling, video/audio decode, encoding, and mux",
"hardware": "NVIDIA GB10",
"runtime_image": "sha256:d842225c9c5036647ee5cab1fae2e093357cc8041bd8617e33a82c81d0bcc874",
"torch": "2.9.1+cu130",
"model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
"task": "t2va",
"resolution": [864, 480],
"frames": 141,
"fps": 24,
"duration_seconds": 5.875,
"steps": 12,
"scheduler": "beta",
"sampler": "res_multistep",
"seed": 440420,
"attention": "sdpa",
"turbo": null,
"upscale": null,
"mux_audio": true,
"keep_intermediates": false,
"vae_dtype": "float16",
"vae_tile_size": 256,
"mlp_chunks": 1,
"prompt": "integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static medium two-shot frames exactly two adults seated across from each other at a small table in a quiet, otherwise empty meeting room. A composed adult man sits on the left and a composed adult woman sits on the right. Both maintain neutral, attentive expressions. The man with a low, clear baritone voice and measured speaking pace (S1) looks toward the woman and says in a calm, matter-of-fact delivery: <d>[English] The north entrance closes at six.</d> During S1's line, only his lips and jaw move; the woman's lips remain completely closed. Immediately after his final word, his lips meet and his jaw ceases speaking motion. After a brief silent pause, the woman with a clear alto voice and measured speaking pace (S2) looks toward the man and replies in a calm, matter-of-fact delivery: <d>[English] Then we should leave by five thirty.</d> During S2's line, only her lips and jaw move; the man's lips remain completely closed. Immediately after her final word, her lips meet and her jaw ceases speaking motion. Both remain silent with neutral, closed-mouth expressions through the final frame. There is no overlapping speech. The camera remains completely static with no cuts.\n\noverall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum. Only S1 and S2 are audible, one at a time. No laughter, chuckling, giggling, smiling vocalization, sighing, gasping, audible breathing, filler sounds, audience reaction, narration, or other voices.\n\nnon_diegetic_music: N/A",
"timings_seconds": {
"latents_initialized": 0.018280818010680377,
"text_conditioned": 3.531974215002265,
"sampled": 125.30079188900709,
"vae_decoded": 23.84665890400356,
"pixels_cpu": 0.06607734999852255,
"raw_write": 2.75871228199685,
"video_encode": 0.39305945999512915,
"audio_decoded": 0.3400787889986532,
"audio_raw_write": 0.0008342489891219884,
"audio_encode": 0.034553833000245504,
"mux": 0.19705491400964092,
"request": 156.48807670301176,
"wall": 156.56338787100685
},
"throughput": {
"sampling_frames_per_second": 1.1252921699401506,
"wall_frames_per_second": 0.9005936950992043,
"sampling_realtime_factor": 21.327794364086314,
"wall_realtime_factor": 26.649087297192655
},
"cache": {
"mode": null,
"threshold": 0.0,
"skipped_steps": 0,
"rates": []
},
"output": {
"linux_path": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
"unc_path": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-baselines\\dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
"sha256": "2eb61334978b42bcec00ec68449f3f39139d9e24ef5ba70ef2240dc8bf7116e0",
"size_bytes": 429062,
"container_bit_rate": 584254,
"duration_seconds": 5.875,
"video": {
"codec": "h264",
"profile": "High",
"pixel_format": "yuv420p",
"width": 864,
"height": 480,
"fps": 24,
"frames": 141,
"bit_rate": 439001
},
"audio": {
"codec": "aac",
"profile": "LC",
"sample_rate": 32000,
"channels": 2,
"channel_layout": "stereo",
"bit_rate": 136335,
"mean_volume_db": -25.8,
"max_volume_db": -4.3,
"integrated_loudness_lufs": -21.18,
"true_peak_dbtp": -4.34,
"loudness_range_lu": 4.10,
"loudness_threshold_lufs": -33.94
}
},
"attention_backend_matrix": {
"reference_backend": "sdpa",
"comparison_note": "Video PSNR and decoded-PCM audio SNR measure output drift from SDPA, not subjective quality. Audio SNR is computed directly from aligned float PCM arrays; earlier FFmpeg apsnr results above 170 dB were invalid and have been replaced.",
"runs": {
"sdpa": {
"sampled_seconds": 125.30079188900709,
"request_seconds": 156.48807670301176,
"wall_seconds": 156.56338787100685,
"sampling_speedup_percent_vs_sdpa": 0.0,
"output_file": "dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
"sha256": "2eb61334978b42bcec00ec68449f3f39139d9e24ef5ba70ef2240dc8bf7116e0",
"size_bytes": 429062
},
"sage2": {
"sampled_seconds": 101.26120180900034,
"request_seconds": 132.59474182801205,
"wall_seconds": 132.66185540499282,
"sampling_speedup_percent_vs_sdpa": 19.185505,
"video_psnr_average_db_vs_sdpa": 22.4225,
"audio_pcm_snr_db_vs_sdpa": [2.515815, 2.492644],
"output_file": "dialogue-two-character-864x480-141f-base12-sage2-seed440420.mp4",
"sha256": "c3e0691586c4101987f240c35a79637965af75088731108037cd084be945ba9d",
"size_bytes": 433450
},
"cudnn_sdpa": {
"sampled_seconds": 125.72722270998929,
"request_seconds": 157.14237468996726,
"wall_seconds": 157.2081264879962,
"sampling_speedup_percent_vs_sdpa": -0.340326,
"video_psnr_average_db_vs_sdpa": 22.093143,
"audio_pcm_snr_db_vs_sdpa": [2.056909, 2.162249],
"output_file": "dialogue-two-character-864x480-141f-base12-cudnn_sdpa-seed440420.mp4",
"sha256": "cd17833c12a99cd71d1c0a9137a1dd9feba03943980d6eb8fff296cd9f9529a6",
"size_bytes": 429636
},
"ck_int8": {
"sampled_seconds": 104.05008868798905,
"request_seconds": 135.57199566195777,
"wall_seconds": 135.62453711099806,
"sampling_speedup_percent_vs_sdpa": 16.959752,
"video_psnr_average_db_vs_sdpa": 22.989999,
"audio_pcm_snr_db_vs_sdpa": [2.308146, 2.398058],
"output_file": "dialogue-two-character-864x480-141f-base12-ck_int8-seed440420.mp4",
"sha256": "a54f9f3f2ba351d31b0b807646aea2740aae7c13c03d4eb7d44182aab3cf5618",
"size_bytes": 423190
},
"sage3": {
"sampled_seconds": 109.8837421490025,
"request_seconds": 142.34848491402227,
"wall_seconds": 142.41395058500348,
"sampling_speedup_percent_vs_sdpa": 12.304032,
"video_psnr_average_db_vs_sdpa": 19.63725,
"audio_pcm_snr_db_vs_sdpa": [-3.047541, -2.395828],
"output_file": "dialogue-two-character-864x480-141f-base12-sage3-seed440420.mp4",
"sha256": "3763ca4b79399037d997d5a37c795fdaf233aca88fdb90e122bf25450c75dddd",
"size_bytes": 596651
},
"sage3_mean": {
"sampled_seconds": 109.88214365398744,
"request_seconds": 141.45796999098093,
"wall_seconds": 141.51303354099218,
"sampling_speedup_percent_vs_sdpa": 12.305308,
"video_psnr_average_db_vs_sdpa": 19.63725,
"audio_pcm_snr_db_vs_sdpa": [-3.047541, -2.395828],
"output_file": "dialogue-two-character-864x480-141f-base12-sage3_mean-seed440420.mp4",
"sha256": "3763ca4b79399037d997d5a37c795fdaf233aca88fdb90e122bf25450c75dddd",
"size_bytes": 596651,
"identical_output_to": "sage3"
},
"kj_sage_cuda": {
"sampled_seconds": 113.68463072601298,
"request_seconds": 145.39383504101716,
"wall_seconds": 145.45757150900317,
"sampling_speedup_percent_vs_sdpa": 9.270621,
"video_psnr_average_db_vs_sdpa": 22.83642,
"audio_pcm_snr_db_vs_sdpa": [2.039297, 2.098551],
"output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_cuda-seed440420.mp4",
"sha256": "d85aaca677d2c678604b0aa2aba9bbf4921934bfc2970c20d0498b1762373ab5",
"size_bytes": 427568
},
"kj_sage_triton": {
"sampled_seconds": 118.28390433100867,
"request_seconds": 150.37281119299587,
"wall_seconds": 150.43365036998875,
"sampling_speedup_percent_vs_sdpa": 5.600034,
"video_psnr_average_db_vs_sdpa": 22.086796,
"audio_pcm_snr_db_vs_sdpa": [5.920034, 6.184134],
"output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_triton-seed440420.mp4",
"sha256": "9fe63aafa51aae8ecc0940ba75fbc13a978a50091fd8f0c336c2bb04e3658a54",
"size_bytes": 430038
},
"kj_sage_fp8": {
"sampled_seconds": 102.6086639950081,
"request_seconds": 134.1615118059999,
"wall_seconds": 134.2236167689989,
"sampling_speedup_percent_vs_sdpa": 18.110123,
"video_psnr_average_db_vs_sdpa": 22.814833,
"audio_pcm_snr_db_vs_sdpa": [3.442309, 3.542901],
"output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_fp8-seed440420.mp4",
"sha256": "8c5bc2a8bf7e496c2108194479fe60e67883dea0ab50e95f3ab4bac916274816",
"size_bytes": 420189
},
"kj_sage_fp8pp": {
"sampled_seconds": 103.24360375599645,
"request_seconds": 134.77945204998832,
"wall_seconds": 134.84802630099875,
"sampling_speedup_percent_vs_sdpa": 17.603391,
"video_psnr_average_db_vs_sdpa": 22.296461,
"audio_pcm_snr_db_vs_sdpa": [3.893655, 4.120454],
"output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_fp8pp-seed440420.mp4",
"sha256": "71643c7163e5dc16d64ac0c5d567bdd846f7944471eb0a97fd54f9a1990c24e9",
"size_bytes": 433672
},
"kj_head_sliced": {
"sampled_seconds": 103.97128305501246,
"request_seconds": 135.67128342803335,
"wall_seconds": 135.71258915099315,
"sampling_speedup_percent_vs_sdpa": 17.022645,
"video_psnr_average_db_vs_sdpa": 21.679726,
"audio_pcm_snr_db_vs_sdpa": [1.20378, 1.17592],
"output_file": "dialogue-two-character-864x480-141f-base12-kj_head_sliced-seed440420.mp4",
"sha256": "48ee4da6c0a91f18714973ca23e26f1844b302024c8111946772cbc9a4a4dd83",
"size_bytes": 421728
},
"sol_attn": {
"sampled_seconds": 96.45773334198748,
"request_seconds": 128.10418028896675,
"wall_seconds": 128.17140150099294,
"sampling_speedup_percent_vs_sdpa": 23.019055,
"video_psnr_average_db_vs_sdpa": 16.653003,
"audio_pcm_snr_db_vs_sdpa": [1.186243, 1.042278],
"output_file": "dialogue-two-character-864x480-141f-base12-sol_attn-seed440420.mp4",
"sha256": "8b999630210692301549d11d20d9319a6af5873a60d4cd735035e2ed79c469d6",
"size_bytes": 570667,
"note": "Experimental sparse Triton dispatch; eligible calls use sol_attn and shorter calls fall back."
},
"flash4": {
"sampled_seconds": 128.5334454880067,
"request_seconds": 160.91555801399227,
"wall_seconds": 160.96679471699463,
"sampling_speedup_percent_vs_sdpa": -2.579915,
"video_psnr_average_db_vs_sdpa": 21.238297,
"audio_pcm_snr_db_vs_sdpa": [3.230108, 3.821894],
"output_file": "dialogue-two-character-864x480-141f-base12-flash4-seed440420.mp4",
"sha256": "cfb8a046cb79f8448988e7392ca748c0a08a2a034a43f6ab55a9058faa1a72ef",
"size_bytes": 449687
}
}
},
"turbo_sage2_matrix": {
"reference": "Base 12-step Sage2 from attention_backend_matrix",
"runs": {
"4step": {
"adapter": "Official FL2VA Turbo 4-step v1.1 768p",
"steps": 4,
"attention": "sage2",
"sampled_seconds": 51.661242722009774,
"request_seconds": 83.01122188399313,
"wall_seconds": 83.06598260099418,
"sampling_speedup_percent_vs_base_sage2": 48.982195,
"wall_speedup_percent_vs_base_sage2": 37.38518,
"output_file": "dialogue-two-character-864x480-141f-turbo4step-sage2-seed440420.mp4",
"sha256": "b02c29a8732d88c7dcf8e85462afcbe3aa83328541ce1647995e7d0d4188744a",
"size_bytes": 455325,
"container_bit_rate": 620017,
"audio": {
"integrated_loudness_lufs": -31.78,
"true_peak_dbtp": -14.55,
"first_0_25_seconds_peak_dbfs": -39.394894,
"first_0_25_seconds_rms_dbfs": -57.733654,
"first_sample_dbfs": [-54.110104, -50.553467]
}
},
"8step": {
"adapter": "Official FL2VA Turbo 8-step v1.0",
"steps": 8,
"attention": "sage2",
"sampled_seconds": 102.70967868898879,
"request_seconds": 134.31820170898573,
"wall_seconds": 134.36813215899747,
"sampling_speedup_percent_vs_base_sage2": -1.430436,
"wall_speedup_percent_vs_base_sage2": -1.286185,
"output_file": "dialogue-two-character-864x480-141f-turbo8step-sage2-seed440420.mp4",
"sha256": "21f450848c2ded8c02054cc241c8080d55a17979022eadfd892f3ce5205833e6",
"size_bytes": 585250,
"container_bit_rate": 796936,
"audio": {
"integrated_loudness_lufs": -24.39,
"true_peak_dbtp": -7.18,
"first_0_25_seconds_peak_dbfs": -21.305752,
"first_0_25_seconds_rms_dbfs": -41.479687,
"first_sample_dbfs": [-35.859436, -39.793118]
}
}
}
},
"comfy_sdpa_comparison": {
"comfyui_version": "0.31.0",
"attention": "PyTorch attention without --use-sage-attention",
"prompt_id": "29c45a77-efdf-4cc8-8ae9-9b8a40823b41",
"execution_seconds": 152.767,
"wall_seconds": 153.03360149999207,
"wall_difference_from_direct_percent": -2.25454138353405,
"output": {
"linux_path": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/comfy-sdpa-dialogue-two-character-864x480-141f-base12-seed440420_00001_.mp4",
"unc_path": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-baselines\\comfy-sdpa-dialogue-two-character-864x480-141f-base12-seed440420_00001_.mp4",
"sha256": "f4510d6b8ac06b8f2cb22172454c6977a214e061dcc5539ace68e1cb70abc2ee",
"size_bytes": 443723,
"container_bit_rate": 604218,
"duration_seconds": 5.875,
"video": {
"codec": "h264",
"profile": "High",
"pixel_format": "yuv420p",
"width": 864,
"height": 480,
"fps": 24,
"frames": 141,
"bit_rate": 457452
},
"audio": {
"codec": "aac",
"profile": "LC",
"sample_rate": 32000,
"channels": 2,
"channel_layout": "stereo",
"bit_rate": 133407,
"mean_volume_db": -25.7,
"max_volume_db": -4.3,
"integrated_loudness_lufs": -21.12,
"true_peak_dbtp": -4.25,
"loudness_range_lu": 4.10,
"loudness_threshold_lufs": -33.88
}
},
"first_0_25_seconds": {
"direct": {
"peak_level_db": -19.429245,
"rms_level_db": -33.601728,
"max_sample_difference": 0.022571,
"peak_count": 2,
"nan_count": 0,
"inf_count": 0
},
"comfy": {
"peak_level_db": -19.261140,
"rms_level_db": -33.545443,
"max_sample_difference": 0.023026,
"peak_count": 2,
"nan_count": 0,
"inf_count": 0
},
"decoded_pcm_snr_db": {
"channel_1": 31.498066,
"channel_2": 29.897251
},
"conclusion": "No amplitude clipping. Direct and Comfy closely match during the first 250ms and both exhibit the startup artifact, but they are not numerically identical. The shared defect originates before runtime-specific audio encoding and mux."
},
"full_decoded_pcm_snr_db": {
"channel_1": 21.414368,
"channel_2": 26.592979
}
}
}

12
benchmarks/t2va-dialogue-quoted-1344x768-124f-base12-sdpa-seed440420.json
View file

@ -1,12 +0,0 @@
{
"name": "t2va-dialogue-quoted-1344x768-124f-base12-sdpa-seed440420",
"mode": "t2va",
"prompt": "Medium two-shot inside a quiet late-night cafe during rain. Warm practical lamps, realistic skin texture, shallow depth of field, restrained natural movement, stable faces, accurate lip sync. A tired woman in a dark green coat looks directly at the man across from her and says, \"I kept the porch light on every night.\" He pauses, meets her eyes, and says, \"I know. I just did not know how to come home.\" Soft rain against the windows, low room tone, faint ceramic cup movement, no music.",
"resolution": [
1344,
768
],
"frames": 124,
"steps": 12,
"seed": 440420
}

12
benchmarks/t2va-dialogue-quoted-1344x768-243f-base12-sdpa-seed440420.json
View file

@ -1,12 +0,0 @@
{
"name": "t2va-dialogue-quoted-1344x768-243f-base12-sdpa-seed440420",
"mode": "t2va",
"prompt": "Medium two-shot inside a quiet late-night cafe during rain. Warm practical lamps, realistic skin texture, shallow depth of field, restrained natural movement, stable faces, accurate lip sync. A tired woman in a dark green coat looks directly at the man across from her and says, \"I kept the porch light on every night.\" He pauses, meets her eyes, and says, \"I know. I just did not know how to come home.\" Soft rain against the windows, low room tone, faint ceramic cup movement, no music.",
"resolution": [
1344,
768
],
"frames": 243,
"steps": 12,
"seed": 440420
}

19
benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json
View file

@ -1,19 +0,0 @@
{
"name": "t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420",
"measured_at": "2026-08-21",
"hardware": "NVIDIA GB10",
"model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
"task": "t2va",
"resolution": [864, 480],
"frames": 141,
"fps": 24,
"duration_seconds": 5.875,
"steps": 12,
"scheduler": "beta",
"sampler": "res_multistep",
"seed": 440420,
"attention": "sage2",
"turbo": null,
"prompt_change": "Only the two <d>[English]...</d> spans were replaced with quoted dialogue.",
"prompt": "integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static medium two-shot frames exactly two adults seated across from each other at a small table in a quiet, otherwise empty meeting room. A composed adult man sits on the left and a composed adult woman sits on the right. Both maintain neutral, attentive expressions. The man with a low, clear baritone voice and measured speaking pace (S1) looks toward the woman and says in a calm, matter-of-fact delivery: \"The north entrance closes at six.\" During S1's line, only his lips and jaw move; the woman's lips remain completely closed. Immediately after his final word, his lips meet and his jaw ceases speaking motion. After a brief silent pause, the woman with a clear alto voice and measured speaking pace (S2) looks toward the man and replies in a calm, matter-of-fact delivery: \"Then we should leave by five thirty.\" During S2's line, only her lips and jaw move; the man's lips remain completely closed. Immediately after her final word, her lips meet and her jaw ceases speaking motion. Both remain silent with neutral, closed-mouth expressions through the final frame. There is no overlapping speech. The camera remains completely static with no cuts.\n\noverall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum. Only S1 and S2 are audible, one at a time. No laughter, chuckling, giggling, smiling vocalization, sighing, gasping, audible breathing, filler sounds, audience reaction, narration, or other voices.\n\nnon_diegetic_music: N/A"
}

12
benchmarks/t2va-distributed-smoke-256x256-9f-1step.json
View file

@ -1,12 +0,0 @@
{
"name": "t2va-distributed-smoke-256x256-9f-1step",
"mode": "t2va",
"prompt": "A paper windmill turns steadily on a plain table. Quiet room tone.",
"resolution": [
256,
256
],
"frames": 9,
"steps": 1,
"seed": 440420
}

25
benchmarks/t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421.json
View file

@ -1,25 +0,0 @@
{
"name": "t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421",
"measured_at": "2026-08-21",
"hardware": "NVIDIA GB10",
"model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
"task": "t2va",
"resolution": [864, 480],
"frames": 141,
"fps": 24,
"duration_seconds": 5.875,
"steps": 12,
"scheduler": "beta",
"sampler": "res_multistep",
"seed": 440421,
"attention": "sage2",
"turbo": null,
"prompt": "integrated_multimodal_description: 0.0-2.0s: Inside a packed underground nightclub, the camera glides low across a crowded dance floor toward a raised DJ booth. A clean four-on-the-floor kick and deep bassline begin immediately at 0.0s. Cyan and magenta strobes strike precisely on the beat while dancers move in synchronized rhythm. 2.0-4.0s: The camera sweeps around the DJ as she works the mixer, one hand adjusting a filter while the other raises toward the crowd. The house groove remains continuous and coherent; crisp hi-hats enter over the kick and bass. The crowd cheers naturally beneath the music. 4.0-5.875s: The camera pushes close to the mixer and then tilts up as the room erupts under a bright white strobe hit. The beat continues without interruption, ending on an energetic club moment. No dialogue. overall_soundscape: Loud but clean diegetic nightclub house music coming from the venue sound system, beginning exactly at the first frame, with a steady kick, deep controlled bass, crisp hi-hats, room reflections, dancing footsteps, and a lively crowd. No clipping, crackling, popping, startup noise, gibberish, or speech. non_diegetic_music: N/A",
"diagnostic_capture": {
"output_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.mp4",
"latent_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt",
"wav_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav",
"sampled_seconds": 104.5049012459931,
"wall_seconds": 135.91074104901054
}
}

46
compose.qwen38-vllm.yml
View file

@ -1,46 +0,0 @@
services:
qwen38-vllm:
image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest
container_name: qwen38-vllm
restart: unless-stopped
gpus: all
ipc: host
shm_size: 16g
network_mode: host
volumes:
- /home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4:/model:ro
- /home/daniel/aeon-spark-test/h3/vllm-cache:/root/.cache/vllm
entrypoint: ["vllm"]
command:
- serve
- /model
- --served-model-name
- qwen38
- --host
- 0.0.0.0
- --port
- "8002"
- --tensor-parallel-size
- "1"
- --gpu-memory-utilization
- "0.45"
- --max-model-len
- "262144"
- --max-num-seqs
- "4"
- --max-num-batched-tokens
- "8192"
- --enable-chunked-prefill
- --enable-prefix-caching
- --kv-cache-dtype
- fp8_e4m3
- --reasoning-parser
- qwen3
- --tool-call-parser
- qwen3_xml
- --enable-auto-tool-choice
- --distributed-executor-backend
- mp
- --speculative-config
- '{"method":"mtp","num_speculative_tokens":5}'
- --trust-remote-code

84
compose.spark-stack.yml
View file

@ -1,84 +0,0 @@
services:
h3-hot-runtime:
profiles: ["h3"]
image: h3-blackwell-runtime:dev
container_name: h3-hot-runtime
restart: unless-stopped
gpus: all
volumes:
- /home/daniel/aeon-spark-test/h3/comfy-models/diffusion_models:/models:ro
- /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
- /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
- /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
- /home/daniel/StoryStudioAssets/H3-output:/output
ports:
- "8001:8000"
environment:
H3_DISABLE_MMAP: "1"
H3_NVFP4_SCALE_BACKEND: "vortex"
H3_NVFP4_SCALE_VERSION: "1"
H3_SOL_QKV_LAYOUT: "native"
command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--warmup"]
qwen38-vllm:
profiles: ["qwen"]
image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest
container_name: qwen38-vllm
restart: unless-stopped
gpus: all
ipc: host
shm_size: 16g
network_mode: host
volumes:
- /home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4:/model:ro
- /home/daniel/aeon-spark-test/h3/vllm-cache:/root/.cache/vllm
entrypoint: ["vllm"]
command:
- serve
- /model
- --served-model-name
- qwen38
- --host
- 0.0.0.0
- --port
- "8002"
- --tensor-parallel-size
- "1"
- --gpu-memory-utilization
- "0.45"
- --max-model-len
- "262144"
- --max-num-seqs
- "4"
- --max-num-batched-tokens
- "8192"
- --enable-chunked-prefill
- --enable-prefix-caching
- --kv-cache-dtype
- fp8_e4m3
- --reasoning-parser
- qwen3
- --tool-call-parser
- qwen3_xml
- --enable-auto-tool-choice
- --distributed-executor-backend
- mp
- --speculative-config
- '{"method":"mtp","num_speculative_tokens":5}'
- --trust-remote-code
flux2-klein-comfy:
profiles: ["flux"]
image: ghcr.io/aeon-7/comfyui-aeon-spark:slim
container_name: flux2-klein-comfy
restart: unless-stopped
gpus: all
ipc: host
shm_size: 16g
volumes:
- /home/daniel/aeon-spark-test/workspace:/workspace/ComfyUI:rw
- /home/daniel/StoryStudioAssets/H3-output:/workspace/ComfyUI/output:rw
- /home/daniel/aeon-spark-test/extra_model_paths.yaml:/opt/ComfyUI/extra_model_paths.yaml:ro
- /home/daniel/comfy-data/models:/shared-models:ro
ports:
- "8192:8188"

6
compose.spark.yml
View file

@ -10,8 +10,6 @@ services:
- /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
- /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
- /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
- /home/daniel/aeon-spark-test/h3/models/MiniMax-H3-Turbo-FL2VA/latest:/turbo:ro
- /home/daniel/aeon-spark-test/h3/models/Minimax-H3-Latent-Upscaler:/latent-upscaler:ro
- /home/daniel/StoryStudioAssets/H3-output:/output
command: ["sleep", "infinity"]
h3-hot-runtime:
@ -25,8 +23,6 @@ services:
- /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
- /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
- /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
- /home/daniel/aeon-spark-test/h3/models/MiniMax-H3-Turbo-FL2VA/latest:/turbo:ro
- /home/daniel/aeon-spark-test/h3/models/Minimax-H3-Latent-Upscaler:/latent-upscaler:ro
- /home/daniel/StoryStudioAssets/H3-output:/output
ports:
- "8001:8000"
@ -35,4 +31,4 @@ services:
H3_NVFP4_SCALE_BACKEND: "vortex"
H3_NVFP4_SCALE_VERSION: "1"
H3_SOL_QKV_LAYOUT: "native"
command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--turbo-lora", "4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors", "--turbo-lora", "8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors", "--latent-upscaler", "/latent-upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors", "--warmup"]
command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sol_attn", "--warmup"]

5
pyproject.toml
View file

@ -4,15 +4,12 @@ version = "0.1.0"
description = "Direct MiniMax H3 Blackwell inference research runtime"
requires-python = ">=3.12"
dependencies = [
"comfy-kitchen==0.2.31",
"comfy-kitchen==0.2.28",
"fastsafetensors>=0.1.10",
"safetensors>=0.5.0",
"torch==2.9.1+cu130",
"transformers>=4.51,<5"
]
[project.optional-dependencies]
flash4 = ["flash-attn-4[cu13]==4.0.0b27"]
[tool.setuptools.packages.find]
where = ["src"]

120
src/h3_blackwell_runtime/attention.py
View file

@ -1,8 +1,6 @@
"""Direct H3 self-attention using packed NVFP4 linears and SageAttention3."""
import os
from typing import TYPE_CHECKING
import torch
import torch.nn.functional as functional
from torch import nn
@ -10,22 +8,17 @@ from torch import nn
from .checkpoint import H3Checkpoint
from .nvfp4 import Nvfp4Linear
if TYPE_CHECKING:
from .distributed import SequenceParallelContext
AVAILABLE_BACKENDS = ("sage2", "cudnn_sdpa", "ck_int8", "sdpa", "flash4", "sage3", "sage3_mean", "kj_sage_cuda", "kj_sage_triton", "kj_sage_fp8", "kj_sage_fp8pp", "kj_head_sliced", "sol_attn")
PLANNED_BACKENDS = ("easycache", "h3_cache", "kj_chunked_ffn")
DEFAULT_ATTENTION_BACKEND = os.getenv("H3_DEFAULT_ATTENTION", "sage2")
AVAILABLE_BACKENDS = ("sage2", "sdpa", "sage3", "sage3_mean", "kj_sage_cuda", "kj_sage_triton", "kj_sage_fp8", "kj_sage_fp8pp", "kj_head_sliced", "sol_attn")
PLANNED_BACKENDS = ("flash4", "easycache", "h3_cache", "kj_chunked_ffn")
DEFAULT_ATTENTION_BACKEND = os.getenv("H3_DEFAULT_ATTENTION", "sol_attn")
def attention_backend_status() -> dict[str, str]:
"""Report direct-runtime attention choices without importing ComfyUI nodes."""
status = {name: "available" for name in AVAILABLE_BACKENDS}
status.update({"cudnn_sdpa": "available: forced cuDNN SDPA with no backend fallback"})
status.update({"ck_int8": "available: approximate Comfy Kitchen INT8 Q/K/V attention"})
status.update({"sol_attn": "experimental: sparse Triton attention for eligible non-causal H3 attention calls; falls back below H3_SOL_MIN_TOKENS unless H3_SOL_STRICT=1"})
status.update({"flash4": "available: official FlashAttention-4 CuTeDSL Blackwell kernel (strict, no fallback)"})
status.update({"flash4": "planned: exact Blackwell kernel adapter"})
status.update({"easycache": "planned: approximate denoiser cache"})
status.update({"h3_cache": "planned: approximate H3-specific cache"})
status.update({"kj_chunked_ffn": "available: exact H3 MLP row chunking via H3_MLP_CHUNKS or runtime args"})
@ -90,23 +83,6 @@ def qkv_to_bshd(qkv: torch.Tensor, heads: int, head_dim: int) -> tuple[torch.Ten
)
def run_flash4_attention_bshd(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, is_causal: bool) -> torch.Tensor:
"""Run official FlashAttention-4 on `[batch, sequence, heads, dim]` tensors."""
if not q.is_cuda or not k.is_cuda or not v.is_cuda:
raise ValueError("FlashAttention-4 requires CUDA tensors")
if q.dtype not in {torch.float16, torch.bfloat16} or k.dtype != q.dtype or v.dtype != q.dtype:
raise ValueError("FlashAttention-4 requires matching FP16 or BF16 Q/K/V tensors")
if q.shape != k.shape or q.shape != v.shape:
raise ValueError("FlashAttention-4 requires matching Q/K/V shapes")
if q.shape[-1] != 128:
raise ValueError(f"H3 FlashAttention-4 requires head dim 128, got {q.shape[-1]}")
from flash_attn.cute import flash_attn_func
result = flash_attn_func(q.contiguous(), k.contiguous(), v.contiguous(), causal=is_causal)
return result[0] if isinstance(result, tuple) else result
def run_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, backend: str, is_causal: bool) -> torch.Tensor:
"""Run one `[batch, heads, sequence, dim]` attention operation."""
if backend == "sol_attn":
@ -125,21 +101,6 @@ def run_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, backend:
from sageattention import sageattn
return sageattn(q, k, v, is_causal=is_causal, tensor_layout="HND", smooth_k=False)
if backend == "flash4":
return run_flash4_attention_bshd(
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=is_causal,
).transpose(1, 2)
if backend == "cudnn_sdpa":
from torch.nn.attention import SDPBackend, sdpa_kernel
with sdpa_kernel([SDPBackend.CUDNN_ATTENTION]):
return functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal)
if backend == "ck_int8":
if is_causal:
raise ValueError("Comfy Kitchen INT8 attention does not support causal H3 attention")
import comfy_kitchen
return comfy_kitchen.int8_attention(q, k, v)
if backend == "sage3":
from sageattn3 import sageattn3_blackwell
@ -238,44 +199,17 @@ class H3SageAttention(nn.Module):
backend=backend,
)
def forward(
self,
x: torch.Tensor,
rope_rotation: torch.Tensor,
sequence_parallel: "SequenceParallelContext | None" = None,
tensor_parallel: "SequenceParallelContext | None" = None,
) -> torch.Tensor:
def forward(self, x: torch.Tensor, rope_rotation: torch.Tensor) -> torch.Tensor:
if x.ndim != 2:
raise ValueError("H3 attention expects `[sequence, hidden]` input.")
if sequence_parallel is not None and tensor_parallel is not None:
raise ValueError("choose Ulysses sequence parallelism or tensor parallelism, not both")
if tensor_parallel is not None:
return self._forward_tensor_parallel(x, rope_rotation, tensor_parallel)
sequence = x.shape[0]
inner = self.heads * self.head_dim
qkv = self.qkv_proj(x)
q, k, v = qkv.split(inner, dim=-1)
q, k, v = self.qkv_proj(x).split(inner, dim=-1)
q = q.view(1, sequence, self.heads, self.head_dim)
k = k.view(1, sequence, self.heads, self.head_dim)
v = v.view(1, sequence, self.heads, self.head_dim)
q, k = rms_rope_split_half_(q, k, rope_rotation, self.q_norm_weight, self.k_norm_weight, self.eps)
if sequence_parallel is not None:
q, k, v = sequence_parallel.seq_to_heads(q, k, v)
if self.backend == "sol_attn":
out = run_sol_attention_bshd(q, k, v, is_causal=False)
elif self.backend == "flash4":
out = run_flash4_attention_bshd(q, k, v, is_causal=False)
else:
out = run_attention(
q.transpose(1, 2).contiguous(),
k.transpose(1, 2).contiguous(),
v.transpose(1, 2).contiguous(),
backend=self.backend,
is_causal=False,
).transpose(1, 2)
local_out = sequence_parallel.heads_to_seq(out)
return self.out_proj(local_out.reshape(sequence, inner))
if self.backend == "sol_attn":
if os.getenv("H3_SOL_QKV_LAYOUT", "native").lower() == "fused":
q, k, v = qkv_to_bshd(qkv, self.heads, self.head_dim)
@ -283,9 +217,6 @@ class H3SageAttention(nn.Module):
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
out = run_sol_attention_bshd(q, k, v, is_causal=False)
return self.out_proj(out.reshape(sequence, inner).contiguous())
if self.backend == "flash4":
out = run_flash4_attention_bshd(q, k, v, is_causal=False)
return self.out_proj(out.reshape(sequence, inner).contiguous())
q = q.transpose(1, 2).contiguous()
k = k.transpose(1, 2).contiguous()
@ -293,42 +224,3 @@ class H3SageAttention(nn.Module):
out = run_attention(q, k, v, backend=self.backend, is_causal=False)
return self.out_proj(out.transpose(1, 2).reshape(sequence, inner).contiguous())
def _forward_tensor_parallel(
self,
local_x: torch.Tensor,
local_rotation: torch.Tensor,
context: "SequenceParallelContext",
) -> torch.Tensor:
"""Run local-head attention with column/row-parallel NVFP4 projections."""
local_sequence = local_x.shape[0]
full_x = context.all_gather_rows(local_x)
full_rotation = context.all_gather_rows(local_rotation[0]).unsqueeze(0)
inner = self.heads * self.head_dim
q, k, v = self.qkv_proj(full_x).split(inner, dim=-1)
q = q.view(1, context.sequence_length, self.heads, self.head_dim)
k = k.view(1, context.sequence_length, self.heads, self.head_dim)
v = v.view(1, context.sequence_length, self.heads, self.head_dim)
q, k = rms_rope_split_half_(
q, k, full_rotation, self.q_norm_weight, self.k_norm_weight, self.eps,
)
if self.backend == "sol_attn":
out = run_sol_attention_bshd(q, k, v, is_causal=False)
elif self.backend == "flash4":
out = run_flash4_attention_bshd(q, k, v, is_causal=False)
else:
out = run_attention(
q.transpose(1, 2).contiguous(),
k.transpose(1, 2).contiguous(),
v.transpose(1, 2).contiguous(),
backend=self.backend,
is_causal=False,
).transpose(1, 2)
partial = self.out_proj(out.reshape(context.sequence_length, inner).contiguous())
local_output = context.reduce_scatter_rows(partial)
bias = getattr(self, "tensor_parallel_output_bias", None)
if bias is not None:
local_output = local_output + bias.to(local_output)
if local_output.shape[0] != local_sequence:
raise RuntimeError("tensor-parallel attention returned the wrong local token count")
return local_output

27
src/h3_blackwell_runtime/backbone.py
View file

@ -2,7 +2,6 @@
import torch
from torch import nn
from typing import TYPE_CHECKING
from .adaln import H3CurveAdaLN
from .attention import DEFAULT_ATTENTION_BACKEND
@ -10,9 +9,6 @@ from .block import H3DiTBlock
from .checkpoint import H3Checkpoint
from .rope import h3_rope_rotation
if TYPE_CHECKING:
from .distributed import SequenceParallelContext
class H3DenoiserBackbone(nn.Module):
"""Execute H3 transformer blocks over an already packed Ref2VA hidden sequence."""
@ -39,29 +35,8 @@ class H3DenoiserBackbone(nn.Module):
timesteps: torch.Tensor,
position_ids: torch.Tensor,
segments: list[tuple[int, int, int]],
sequence_parallel: "SequenceParallelContext | None" = None,
tensor_parallel: "SequenceParallelContext | None" = None,
) -> torch.Tensor:
if sequence_parallel is not None and tensor_parallel is not None:
raise ValueError("choose Ulysses sequence parallelism or tensor parallelism, not both")
parallel = sequence_parallel or tensor_parallel
if parallel is not None:
if hidden.shape[0] != parallel.local_token_length:
raise ValueError(
f"local hidden length {hidden.shape[0]} does not match sequence-parallel "
f"partition {parallel.local_token_length}"
)
if position_ids.shape[0] != hidden.shape[0]:
raise ValueError("local position IDs must match local hidden rows")
segments = parallel.localize_segments(segments)
rotation = h3_rope_rotation(position_ids.to(hidden.device), self.inv_freq, hidden.dtype)
for block, adaln in zip(self.blocks, self.adaln, strict=True):
hidden = block(
hidden,
rotation,
*adaln(timesteps),
segments,
sequence_parallel,
tensor_parallel,
)
hidden = block(hidden, rotation, *adaln(timesteps), segments)
return hidden

29
src/h3_blackwell_runtime/block.py
View file

@ -2,15 +2,11 @@
import torch
from torch import nn
from typing import TYPE_CHECKING
from .attention import DEFAULT_ATTENTION_BACKEND, H3SageAttention, rms_norm
from .checkpoint import H3Checkpoint
from .nvfp4 import Nvfp4Linear
if TYPE_CHECKING:
from .distributed import SequenceParallelContext
def modulate_segments(
x: torch.Tensor,
@ -53,19 +49,7 @@ class H3SwiGLU(nn.Module):
checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype),
)
def forward(
self,
x: torch.Tensor,
tensor_parallel: "SequenceParallelContext | None" = None,
) -> torch.Tensor:
if tensor_parallel is not None:
full_x = tensor_parallel.all_gather_rows(x)
partial = self._forward_chunk(full_x)
local_output = tensor_parallel.reduce_scatter_rows(partial)
bias = getattr(self, "tensor_parallel_output_bias", None)
if bias is not None:
local_output = local_output + bias.to(local_output)
return local_output
def forward(self, x: torch.Tensor) -> torch.Tensor:
if self.chunks > 1 and x.shape[0] >= self.chunk_threshold:
return torch.cat([self._forward_chunk(chunk) for chunk in x.chunk(self.chunks, dim=0)], dim=0)
return self._forward_chunk(x)
@ -125,15 +109,8 @@ class H3DiTBlock(nn.Module):
scale_mlp: torch.Tensor,
gate_mlp: torch.Tensor,
segments: list[tuple[int, int, int]],
sequence_parallel: "SequenceParallelContext | None" = None,
tensor_parallel: "SequenceParallelContext | None" = None,
) -> torch.Tensor:
h = modulate_segments(rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, segments)
x = gate_segments(
x,
self.attention(h, rope_rotation, sequence_parallel, tensor_parallel),
gate_msa,
segments,
)
x = gate_segments(x, self.attention(h, rope_rotation), gate_msa, segments)
h = modulate_segments(rms_norm(x, self.norm2_weight, self.norm_eps), shift_mlp, scale_mlp, segments)
return gate_segments(x, self.mlp(h, tensor_parallel), gate_mlp, segments)
return gate_segments(x, self.mlp(h), gate_mlp, segments)

4
src/h3_blackwell_runtime/checkpoint.py
View file

@ -59,10 +59,6 @@ class H3Checkpoint:
value = checkpoint.get_tensor(name)
return value.to(dtype=dtype) if dtype is not None else value
def release_cache(self) -> None:
"""Release tensors retained by whole-file loading after modules are built."""
self._no_mmap_tensors = None
def nvfp4_linear(self, prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear:
names = ("comfy_quant", "weight", "weight_scale", "weight_scale_2", "bias", "pre_quant_scale")
tensors = {}

68
src/h3_blackwell_runtime/denoiser.py
View file

@ -2,16 +2,12 @@
import torch
from torch import nn
from typing import TYPE_CHECKING
from .attention import DEFAULT_ATTENTION_BACKEND
from .backbone import H3DenoiserBackbone
from .checkpoint import H3Checkpoint
from .final import H3FinalLayer
if TYPE_CHECKING:
from .distributed import SequenceParallelContext
class H3PackedDenoiser(nn.Module):
"""Run the H3 transformer once its Ref2VA payload has been packed into hidden rows."""
@ -39,67 +35,3 @@ class H3PackedDenoiser(nn.Module):
) -> tuple[torch.Tensor, torch.Tensor]:
hidden = self.backbone(hidden, timesteps, position_ids, segments)
return self.final_layer(hidden, timesteps, video_segment, audio_segment)
def forward_sequence_parallel(
self,
full_hidden: torch.Tensor,
timesteps: torch.Tensor,
full_position_ids: torch.Tensor,
segments: list[tuple[int, int, int]],
video_segment: tuple[int, int, int],
audio_segment: tuple[int, int, int],
context: "SequenceParallelContext",
) -> tuple[torch.Tensor, torch.Tensor]:
"""Run all 50 blocks with token-sharded activations and Ulysses attention."""
if full_hidden.shape[0] != context.sequence_length:
raise ValueError("packed hidden length does not match sequence-parallel context")
start, stop = context.local_token_range
local_hidden = full_hidden[start:stop].contiguous()
local_positions = full_position_ids[start:stop].contiguous()
del full_hidden, full_position_ids
local_hidden = self.backbone(
local_hidden,
timesteps,
local_positions,
segments,
sequence_parallel=context,
)
return self.final_layer.forward_sequence_parallel(
local_hidden,
timesteps,
video_segment,
audio_segment,
context,
)
def forward_tensor_parallel(
self,
full_hidden: torch.Tensor,
timesteps: torch.Tensor,
full_position_ids: torch.Tensor,
segments: list[tuple[int, int, int]],
video_segment: tuple[int, int, int],
audio_segment: tuple[int, int, int],
context: "SequenceParallelContext",
) -> tuple[torch.Tensor, torch.Tensor]:
"""Run TP-sharded NVFP4 linears with ragged sequence-sharded residuals."""
if full_hidden.shape[0] != context.sequence_length:
raise ValueError("packed hidden length does not match tensor-parallel context")
start, stop = context.local_token_range
local_hidden = full_hidden[start:stop].contiguous()
local_positions = full_position_ids[start:stop].contiguous()
del full_hidden, full_position_ids
local_hidden = self.backbone(
local_hidden,
timesteps,
local_positions,
segments,
tensor_parallel=context,
)
return self.final_layer.forward_sequence_parallel(
local_hidden,
timesteps,
video_segment,
audio_segment,
context,
)

324
src/h3_blackwell_runtime/distributed.py
View file

@ -1,324 +0,0 @@
"""Ragged Ulysses sequence-parallel transport for H3 inference."""
from __future__ import annotations
from dataclasses import dataclass
import torch
import torch.distributed as dist
def balanced_ranges(total: int, parts: int) -> tuple[tuple[int, int], ...]:
"""Split ``total`` ordered items into balanced contiguous non-empty ranges."""
if parts < 1:
raise ValueError("parts must be positive")
if total < parts:
raise ValueError(f"cannot split {total} items into {parts} non-empty ranges")
base, extra = divmod(total, parts)
lengths = [base + (rank < extra) for rank in range(parts)]
ranges = []
start = 0
for length in lengths:
stop = start + int(length)
ranges.append((start, stop))
start = stop
return tuple(ranges)
def range_lengths(ranges: tuple[tuple[int, int], ...]) -> tuple[int, ...]:
return tuple(stop - start for start, stop in ranges)
def localize_segments(
segments: list[tuple[int, int, int]],
shard_start: int,
shard_stop: int,
) -> list[tuple[int, int, int]]:
"""Clip global H3 AdaLN segments to one contiguous token shard."""
localized = []
for start, stop, row in segments:
local_start = max(start, shard_start)
local_stop = min(stop, shard_stop)
if local_start < local_stop:
localized.append((local_start - shard_start, local_stop - shard_start, row))
return localized
@dataclass(frozen=True)
class SequenceParallelContext:
"""One rank's ragged token and attention-head ownership."""
group: dist.ProcessGroup | None
rank: int
world_size: int
token_ranges: tuple[tuple[int, int], ...]
head_ranges: tuple[tuple[int, int], ...]
head_dim: int
@classmethod
def create(
cls,
sequence_length: int,
heads: int,
head_dim: int,
*,
group: dist.ProcessGroup | None = None,
) -> "SequenceParallelContext":
if not dist.is_initialized():
raise RuntimeError("torch.distributed process group is not initialized")
world_size = dist.get_world_size(group)
rank = dist.get_rank(group)
return cls(
group=group,
rank=rank,
world_size=world_size,
token_ranges=balanced_ranges(sequence_length, world_size),
head_ranges=balanced_ranges(heads, world_size),
head_dim=head_dim,
)
@property
def sequence_length(self) -> int:
return self.token_ranges[-1][1]
@property
def heads(self) -> int:
return self.head_ranges[-1][1]
@property
def token_lengths(self) -> tuple[int, ...]:
return range_lengths(self.token_ranges)
@property
def head_lengths(self) -> tuple[int, ...]:
return range_lengths(self.head_ranges)
@property
def local_token_range(self) -> tuple[int, int]:
return self.token_ranges[self.rank]
@property
def local_head_range(self) -> tuple[int, int]:
return self.head_ranges[self.rank]
@property
def local_token_length(self) -> int:
start, stop = self.local_token_range
return stop - start
@property
def local_head_count(self) -> int:
start, stop = self.local_head_range
return stop - start
def localize_segments(self, segments: list[tuple[int, int, int]]) -> list[tuple[int, int, int]]:
return localize_segments(segments, *self.local_token_range)
def seq_to_heads(
self,
q: torch.Tensor,
k: torch.Tensor,
v: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Exchange local tokens for full-sequence Q/K/V over locally owned heads.
Inputs use BSHD layout ``[1, local_tokens, all_heads, head_dim]``. Outputs
use ``[1, all_tokens, local_heads, head_dim]``.
"""
expected = (1, self.local_token_length, self.heads, self.head_dim)
if tuple(q.shape) != expected or tuple(k.shape) != expected or tuple(v.shape) != expected:
raise ValueError(
f"sequence-parallel Q/K/V must each have shape {expected}; "
f"got {tuple(q.shape)}, {tuple(k.shape)}, {tuple(v.shape)}"
)
if q.dtype != k.dtype or q.dtype != v.dtype or q.device != k.device or q.device != v.device:
raise ValueError("sequence-parallel Q/K/V must share dtype and device")
if self.world_size == 1:
return q, k, v
local_qkv = torch.stack((q[0], k[0], v[0]), dim=1)
send_chunks = [
local_qkv[:, :, start:stop, :].contiguous().view(-1)
for start, stop in self.head_ranges
]
input_splits = [chunk.numel() for chunk in send_chunks]
send = torch.cat(send_chunks)
output_splits = [
token_length * 3 * self.local_head_count * self.head_dim
for token_length in self.token_lengths
]
receive = torch.empty(sum(output_splits), dtype=q.dtype, device=q.device)
dist.all_to_all_single(
receive,
send,
output_split_sizes=output_splits,
input_split_sizes=input_splits,
group=self.group,
)
source_chunks = []
offset = 0
for token_length, count in zip(self.token_lengths, output_splits, strict=True):
source_chunks.append(
receive[offset : offset + count].view(
token_length, 3, self.local_head_count, self.head_dim,
)
)
offset += count
full_qkv = torch.cat(source_chunks, dim=0)
full_q, full_k, full_v = full_qkv.unbind(dim=1)
return full_q.unsqueeze(0), full_k.unsqueeze(0), full_v.unsqueeze(0)
def heads_to_seq(self, output: torch.Tensor) -> torch.Tensor:
"""Exchange full-sequence local-head output back to local-token all-head output.
Input is BSHD ``[1, all_tokens, local_heads, head_dim]``. The return value
is ``[local_tokens, all_heads, head_dim]``.
"""
expected = (1, self.sequence_length, self.local_head_count, self.head_dim)
if tuple(output.shape) != expected:
raise ValueError(f"sequence-parallel output must have shape {expected}, got {tuple(output.shape)}")
if self.world_size == 1:
return output[0]
output = output[0]
send_chunks = []
input_splits = []
token_offset = 0
for token_length in self.token_lengths:
chunk = output[token_offset : token_offset + token_length].contiguous().view(-1)
send_chunks.append(chunk)
input_splits.append(chunk.numel())
token_offset += token_length
send = torch.cat(send_chunks)
output_splits = [
self.local_token_length * head_length * self.head_dim
for head_length in self.head_lengths
]
receive = torch.empty(sum(output_splits), dtype=output.dtype, device=output.device)
dist.all_to_all_single(
receive,
send,
output_split_sizes=output_splits,
input_split_sizes=input_splits,
group=self.group,
)
head_chunks = []
offset = 0
for head_length, count in zip(self.head_lengths, output_splits, strict=True):
head_chunks.append(
receive[offset : offset + count].view(
self.local_token_length, head_length, self.head_dim,
)
)
offset += count
return torch.cat(head_chunks, dim=1).contiguous()
def target_intersection(self, target_start: int, target_stop: int) -> tuple[int, int]:
"""Return one global target span's bounds relative to this token shard."""
shard_start, shard_stop = self.local_token_range
start = max(target_start, shard_start)
stop = min(target_stop, shard_stop)
if start >= stop:
return (0, 0)
return (start - shard_start, stop - shard_start)
def target_counts(self, target_start: int, target_stop: int) -> tuple[int, ...]:
"""Return ordered target-row counts contributed by every token rank."""
counts = []
for shard_start, shard_stop in self.token_ranges:
counts.append(max(0, min(target_stop, shard_stop) - max(target_start, shard_start)))
return tuple(counts)
def all_gather_target_rows(
self,
local_rows: torch.Tensor,
target_start: int,
target_stop: int,
) -> torch.Tensor:
"""Gather a global target span's projected rows onto every rank.
Padding is transport-only and is removed before concatenation; it is never
exposed to attention or model semantics.
"""
counts = self.target_counts(target_start, target_stop)
if local_rows.ndim != 2:
raise ValueError("target rows must be rank-2 [rows, features]")
if local_rows.shape[0] != counts[self.rank]:
raise ValueError(
f"rank {self.rank} must contribute {counts[self.rank]} target rows, "
f"got {local_rows.shape[0]}"
)
if self.world_size == 1:
return local_rows
max_rows = max(counts)
padded = torch.zeros(
max_rows, local_rows.shape[1], dtype=local_rows.dtype, device=local_rows.device,
)
if local_rows.shape[0]:
padded[: local_rows.shape[0]].copy_(local_rows)
gathered = [torch.empty_like(padded) for _ in range(self.world_size)]
dist.all_gather(gathered, padded, group=self.group)
return torch.cat(
[rows[:count] for rows, count in zip(gathered, counts, strict=True) if count],
dim=0,
)
def all_gather_rows(self, local_rows: torch.Tensor) -> torch.Tensor:
"""Gather ragged token rows on every rank without exposing padding to the model."""
if local_rows.shape[0] != self.local_token_length:
raise ValueError(
f"rank {self.rank} must contribute {self.local_token_length} rows, "
f"got {local_rows.shape[0]}"
)
if self.world_size == 1:
return local_rows
max_rows = max(self.token_lengths)
padded = torch.zeros(
(max_rows, *local_rows.shape[1:]),
dtype=local_rows.dtype,
device=local_rows.device,
)
padded[: local_rows.shape[0]].copy_(local_rows)
gathered = [torch.empty_like(padded) for _ in range(self.world_size)]
dist.all_gather(gathered, padded, group=self.group)
return torch.cat(
[rows[:count] for rows, count in zip(gathered, self.token_lengths, strict=True)],
dim=0,
)
def reduce_scatter_rows(self, partial_full_rows: torch.Tensor) -> torch.Tensor:
"""Sum tensor-parallel partials and return this rank's ragged token rows."""
if partial_full_rows.shape[0] != self.sequence_length:
raise ValueError(
f"partial rows must cover sequence length {self.sequence_length}, "
f"got {partial_full_rows.shape[0]}"
)
if self.world_size == 1:
return partial_full_rows
trailing_shape = partial_full_rows.shape[1:]
row_width = partial_full_rows[0].numel()
send_chunks = []
input_splits = []
offset = 0
for token_length in self.token_lengths:
chunk = partial_full_rows[offset : offset + token_length].contiguous().view(-1)
send_chunks.append(chunk)
input_splits.append(chunk.numel())
offset += token_length
send = torch.cat(send_chunks)
output_splits = [self.local_token_length * row_width] * self.world_size
receive = torch.empty(sum(output_splits), dtype=send.dtype, device=send.device)
dist.all_to_all_single(
receive,
send,
output_split_sizes=output_splits,
input_split_sizes=input_splits,
group=self.group,
)
contributions = receive.view(self.world_size, self.local_token_length, *trailing_shape)
return contributions.sum(dim=0)

44
src/h3_blackwell_runtime/final.py
View file

@ -3,14 +3,10 @@
import torch
import torch.nn.functional as functional
from torch import nn
from typing import TYPE_CHECKING
from .attention import rms_norm
from .checkpoint import H3Checkpoint
if TYPE_CHECKING:
from .distributed import SequenceParallelContext
class H3FinalLayer(nn.Module):
def __init__(
@ -74,43 +70,3 @@ class H3FinalLayer(nn.Module):
functional.linear(video_hidden, self.video_weight, self.video_bias),
functional.linear(audio_hidden, self.audio_weight, self.audio_bias),
)
def forward_sequence_parallel(
self,
local_hidden: torch.Tensor,
timesteps: torch.Tensor,
video_segment: tuple[int, int, int],
audio_segment: tuple[int, int, int],
context: "SequenceParallelContext",
) -> tuple[torch.Tensor, torch.Tensor]:
"""Project local target intersections and gather compact AV rows on every rank."""
position = timesteps.float().clamp(0, 1) * (self.curve_table.shape[0] - 1)
lower = position.floor().long().clamp(max=self.curve_table.shape[0] - 2)
embedding = torch.lerp(
self.curve_table[lower],
self.curve_table[lower + 1],
(position - lower).unsqueeze(1),
)
shift, scale = functional.linear(embedding, self.adaln_weight, self.adaln_bias).chunk(2, dim=-1)
outputs = []
for segment, weight, bias in (
(video_segment, self.video_weight, self.video_bias),
(audio_segment, self.audio_weight, self.audio_bias),
):
global_start, global_stop, row = segment
local_start, local_stop = context.target_intersection(global_start, global_stop)
local_target = local_hidden[local_start:local_stop]
if local_target.shape[0]:
local_target = (
rms_norm(local_target, self.norm_weight, self.eps)
* (1.0 + scale[row])
+ shift[row]
).to(torch.float32)
local_output = functional.linear(local_target, weight, bias)
else:
local_output = torch.empty(
0, weight.shape[0], dtype=torch.float32, device=local_hidden.device,
)
outputs.append(context.all_gather_target_rows(local_output, global_start, global_stop))
return outputs[0], outputs[1]

163
src/h3_blackwell_runtime/latent_upscaler.py
View file

@ -1,163 +0,0 @@
"""MiniMax H3 learned latent upscaler inference.
Adapted from LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler (Apache-2.0).
"""
from pathlib import Path
import re
import torch
from safetensors.torch import load_file
from torch import nn
from torch.nn import functional as F
LATENTS_MEAN = (
0.858090341091156, -0.9606591463088989, 1.0661640167236328, -0.5090325474739075,
-0.2727581858634949, -1.3675414323806763, -0.2553254961967468, -0.26907554268836975,
-0.5376840829849243, -0.0464097298681736, 0.6657370328903198, 0.19690127670764923,
-0.5460608005523682, -0.4035342037677765, -0.23683024942874908, 0.25928452610969543,
-0.30133944749832153, 0.211341992020607, -1.1206848621368408, 0.3581933379173279,
-0.04225143790245056, 0.2604829967021942, 0.22864092886447906, 0.7056031823158264,
)
LATENTS_STD = (
1.2223774194717407, 1.2767263650894165, 1.6831774711608887, 1.7549455165863037,
1.5636216402053833, 2.194143533706665, 0.9653137922286987, 1.0569885969161987,
0.841948926448822, 0.7729952931404114, 1.8955937623977661, 0.946841835975647,
0.7996809482574463, 0.44988900423049927, 0.7197399735450745, 0.6936293244361877,
2.961095094680786, 2.7694199085235596, 3.0496184825897217, 2.1088054180145264,
3.276226282119751, 3.1627357006073, 2.2816812992095947, 2.6127843856811523,
)
def _normalization(channels: int) -> nn.GroupNorm:
return nn.GroupNorm(32, channels)
class ResBlockEmb3D(nn.Module):
def __init__(self, channels: int, emb_channels: int, dropout: float = 0.0):
super().__init__()
self.in_layers = nn.Sequential(
_normalization(channels),
nn.SiLU(),
nn.Conv3d(channels, channels, 3, padding=1),
)
self.emb_layers = nn.Sequential(nn.SiLU(), nn.Linear(emb_channels, 2 * channels))
self.out_norm = _normalization(channels)
self.out_layers = nn.Sequential(
nn.SiLU(),
nn.Dropout(p=dropout),
nn.Conv3d(channels, channels, 3, padding=1),
)
nn.init.zeros_(self.out_layers[-1].weight)
nn.init.zeros_(self.out_layers[-1].bias)
def forward(self, x: torch.Tensor, emb: torch.Tensor) -> torch.Tensor:
hidden = self.in_layers(x)
scale, shift = self.emb_layers(emb).to(hidden.dtype).chunk(2, dim=1)
hidden = self.out_norm(hidden) * (1 + scale[:, :, None, None, None]) + shift[:, :, None, None, None]
return x + self.out_layers(hidden)
class TemporalConv(nn.Module):
def __init__(self, channels: int, kernel_size: int = 5):
super().__init__()
self.norm = _normalization(channels)
self.dwconv = nn.Conv3d(
channels, channels, kernel_size=(kernel_size, 1, 1),
padding=(kernel_size // 2, 0, 0), groups=channels,
)
self.pwconv = nn.Conv3d(channels, channels, 1)
nn.init.zeros_(self.pwconv.weight)
nn.init.zeros_(self.pwconv.bias)
def forward(self, x: torch.Tensor) -> torch.Tensor:
hidden = self.dwconv(F.silu(self.norm(x)))
return x + self.pwconv(hidden)
class H3LatentResizer3D(nn.Module):
def __init__(
self,
in_channels: int = 24,
in_blocks: int = 12,
out_blocks: int = 12,
channels: int = 512,
dropout: float = 0.1,
temporal_every: int = 2,
temporal_kernel: int = 5,
):
super().__init__()
self.conv_in = nn.Conv3d(in_channels, channels, 3, padding=1)
embed_dim = 64
self.embed = nn.Sequential(nn.Linear(1, embed_dim), nn.SiLU(), nn.Linear(embed_dim, embed_dim))
self.in_blocks = self._make_blocks(in_blocks, channels, embed_dim, dropout, temporal_every, temporal_kernel)
self.out_blocks = self._make_blocks(out_blocks, channels, embed_dim, dropout, temporal_every, temporal_kernel)
self.norm_out = _normalization(channels)
self.conv_out = nn.Conv3d(channels, in_channels, 3, padding=1)
@staticmethod
def _make_blocks(count, channels, embed_dim, dropout, temporal_every, temporal_kernel):
blocks = nn.ModuleList()
for index in range(count):
blocks.append(ResBlockEmb3D(channels, embed_dim, dropout))
if temporal_every > 0 and index % temporal_every == 0:
blocks.append(TemporalConv(channels, temporal_kernel))
return blocks
def forward(self, x: torch.Tensor, *, scale: float, target_size: tuple[int, int, int]) -> torch.Tensor:
emb = self.embed(torch.tensor([[scale - 1]], dtype=x.dtype, device=x.device)).expand(x.shape[0], -1)
hidden = self.conv_in(x)
for block in self.in_blocks:
hidden = block(hidden, emb) if isinstance(block, ResBlockEmb3D) else block(hidden)
hidden = F.interpolate(hidden, size=target_size, mode="trilinear", align_corners=False)
for block in self.out_blocks:
hidden = block(hidden, emb) if isinstance(block, ResBlockEmb3D) else block(hidden)
return self.conv_out(F.silu(self.norm_out(hidden)))
def _checkpoint_config(state: dict[str, torch.Tensor]) -> dict:
in_ids = {int(match.group(1)) for key in state if (match := re.match(r"in_blocks\.(\d+)\.in_layers\.", key))}
out_ids = {int(match.group(1)) for key in state if (match := re.match(r"out_blocks\.(\d+)\.in_layers\.", key))}
temporal_keys = [key for key in state if key.endswith("dwconv.weight")]
conv_in = state["conv_in.weight"]
return {
"in_channels": conv_in.shape[1],
"in_blocks": len(in_ids),
"out_blocks": len(out_ids),
"channels": conv_in.shape[0],
"temporal_every": 2 if temporal_keys else 0,
"temporal_kernel": state[temporal_keys[0]].shape[2] if temporal_keys else 5,
}
def load_h3_latent_upscaler(
path: str | Path,
*,
device: str | torch.device = "cuda",
dtype: torch.dtype = torch.float16,
) -> H3LatentResizer3D:
state = load_file(str(path), device="cpu")
if any(key.startswith("upscaler.") for key in state):
state = {key.removeprefix("upscaler."): value for key, value in state.items() if key.startswith("upscaler.")}
model = H3LatentResizer3D(**_checkpoint_config(state))
model.load_state_dict(state, strict=True)
return model.to(device=device, dtype=dtype).eval().requires_grad_(False)
@torch.inference_mode()
def upscale_h3_latent(model: H3LatentResizer3D, latent: torch.Tensor, *, scale: float = 2.0) -> torch.Tensor:
if latent.ndim != 5 or latent.shape[1] != 24:
raise ValueError("H3 latent must have shape [B,24,T,H,W]")
if not 1.0 <= scale <= 4.0:
raise ValueError("scale must be between 1.0 and 4.0")
dtype = next(model.parameters()).dtype
device = next(model.parameters()).device
source = latent.to(device=device, dtype=dtype)
if scale == 1.0:
return source
mean = torch.tensor(LATENTS_MEAN, device=device, dtype=dtype).view(1, 24, 1, 1, 1)
std = torch.tensor(LATENTS_STD, device=device, dtype=dtype).view(1, 24, 1, 1, 1)
target_size = (source.shape[2], round(source.shape[3] * scale), round(source.shape[4] * scale))
result = model((source - mean) / std, scale=scale, target_size=target_size)
return result.mul_(std).add_(mean)

95
src/h3_blackwell_runtime/lora.py
View file

@ -1,95 +0,0 @@
"""Resident dynamic LoRA branches for the quantized H3 denoiser and refiner."""
from collections.abc import Iterator
import torch
import torch.nn.functional as functional
from safetensors import safe_open
from torch import nn
class LoraBranch(nn.Module):
def __init__(self, down: torch.Tensor, up: torch.Tensor, alpha: float):
super().__init__()
if down.ndim != 2 or up.ndim != 2 or down.shape[0] != up.shape[1]:
raise ValueError(f"Invalid LoRA shapes: down={tuple(down.shape)}, up={tuple(up.shape)}")
self.scale = float(alpha) / down.shape[0]
self.register_buffer("down", down.contiguous(), persistent=False)
self.register_buffer("up", up.contiguous(), persistent=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = x.to(self.down.dtype)
return functional.linear(functional.linear(x, self.down), self.up) * self.scale
class DynamicLoraMixin:
"""Small mixin for linears that can host resident request-selectable LoRAs."""
def _init_dynamic_lora(self) -> None:
self.lora_branches = nn.ModuleDict()
self.active_lora: str | None = None
self.lora_strength = 0.0
def add_lora(self, name: str, down: torch.Tensor, up: torch.Tensor, alpha: float) -> None:
if name in self.lora_branches:
raise ValueError(f"LoRA {name!r} is already attached")
if down.shape[1] != self.in_features or up.shape[0] != self.out_features:
raise ValueError(
f"LoRA {name!r} dimensions {tuple(down.shape)}, {tuple(up.shape)} do not match "
f"linear [{self.out_features}, {self.in_features}]"
)
self.lora_branches[name] = LoraBranch(down, up, alpha)
def set_lora(self, name: str | None, strength: float = 1.0) -> None:
if name is not None and name not in self.lora_branches:
raise ValueError(f"LoRA {name!r} is not attached")
self.active_lora = name
self.lora_strength = float(strength) if name is not None else 0.0
def _apply_lora(self, x: torch.Tensor, base: torch.Tensor) -> torch.Tensor:
if self.active_lora is None or self.lora_strength == 0.0:
return base
delta = self.lora_branches[self.active_lora](x)
return base + delta.to(base.dtype) * self.lora_strength
def iter_lora_targets(model: nn.Module, refiner: nn.Module) -> Iterator[tuple[str, DynamicLoraMixin]]:
for index, block in enumerate(model.backbone.blocks):
yield f"blocks.{index}.attn.qkv_proj", block.attention.qkv_proj
yield f"blocks.{index}.attn.out_proj", block.attention.out_proj
yield f"blocks.{index}.mlp.fc1", block.mlp.fc1
yield f"blocks.{index}.mlp.fc2", block.mlp.fc2
for index, block in enumerate(refiner.blocks):
yield f"token_refiner.blocks.{index}.attn.qkv_proj", block.qkv
yield f"token_refiner.blocks.{index}.attn.out_proj", block.out
yield f"token_refiner.blocks.{index}.mlp.fc1", block.fc1
yield f"token_refiner.blocks.{index}.mlp.fc2", block.fc2
def load_lora_adapter(model: nn.Module, refiner: nn.Module, name: str, path: str, device: str) -> int:
targets = list(iter_lora_targets(model, refiner))
expected = {
f"diffusion_model.{target}.{suffix}"
for target, _module in targets
for suffix in ("alpha", "lora_A.weight", "lora_B.weight")
}
with safe_open(path, framework="pt", device=device) as checkpoint:
actual = set(checkpoint.keys())
if actual != expected:
missing = sorted(expected - actual)[:8]
unexpected = sorted(actual - expected)[:8]
raise ValueError(f"LoRA key mismatch: missing={missing}, unexpected={unexpected}")
for target, module in targets:
prefix = f"diffusion_model.{target}"
module.add_lora(
name,
checkpoint.get_tensor(f"{prefix}.lora_A.weight"),
checkpoint.get_tensor(f"{prefix}.lora_B.weight"),
checkpoint.get_tensor(f"{prefix}.alpha").item(),
)
return len(targets)
def set_active_lora(model: nn.Module, refiner: nn.Module, name: str | None, strength: float = 1.0) -> None:
for _target, module in iter_lora_targets(model, refiner):
module.set_lora(name, strength)

13
src/h3_blackwell_runtime/nvfp4.py
View file

@ -7,8 +7,6 @@ import torch
import torch.nn.functional as functional
from torch import nn
from .lora import DynamicLoraMixin
@dataclass(frozen=True)
class Nvfp4LinearTensors:
@ -30,7 +28,7 @@ def parse_quant_sidecar(sidecar: torch.Tensor) -> dict:
return metadata
class Nvfp4Linear(DynamicLoraMixin, nn.Module):
class Nvfp4Linear(nn.Module):
"""Execute a packed Comfy NVFP4 linear with Comfy Kitchen's CUDA 13 kernel."""
def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16):
@ -51,7 +49,6 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module):
self.register_buffer("weight_scale_2", tensors.weight_scale_2.to(torch.float32).contiguous(), persistent=False)
self.register_buffer("bias", tensors.bias.contiguous() if tensors.bias is not None else None, persistent=False)
self.register_buffer("pre_quant_scale", tensors.pre_quant_scale.contiguous() if tensors.pre_quant_scale is not None else None, persistent=False)
self._init_dynamic_lora()
def _packed_weight(self):
from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
@ -76,7 +73,6 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module):
from comfy_kitchen.tensor import QuantizedTensor
original_shape = x.shape[:-1]
original_x = x
flat_x = x.reshape(-1, self.in_features).contiguous()
if self.pre_quant_scale is not None:
flat_x = flat_x * self.pre_quant_scale.to(flat_x)
@ -85,13 +81,12 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module):
if self.full_precision_matrix_mult:
weight = packed_weight.dequantize().to(flat_x)
output = functional.linear(flat_x, weight, bias)
else:
return output.reshape(*original_shape, self.out_features)
if x.dtype == torch.float32:
raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.")
packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout")
output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features]
base = output.reshape(*original_shape, self.out_features)
return self._apply_lora(original_x, base)
output = functional.linear(packed_x, packed_weight, bias)
return output[:flat_x.shape[0], :self.out_features].reshape(*original_shape, self.out_features)
def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear:

149
src/h3_blackwell_runtime/packing.py
View file

@ -34,13 +34,6 @@ def _axis(dim: int, area: float) -> torch.Tensor:
return (torch.arange(count, dtype=torch.float64) * ratio / count + (1 - ratio) / 2) * 32
def _frame_positions(height: int, width: int) -> torch.Tensor:
"""(t ignored) area-normalized (h, w) grid of one latent frame's 2x2-patch rows."""
area = math.sqrt(height * width)
ys, xs = torch.meshgrid(_axis(height, area), _axis(width, area), indexing="ij")
return torch.stack((ys.flatten(), xs.flatten()), dim=-1) # [frame_rows, 2]
def _video_positions(frames: int, height: int, width: int, offset: float) -> torch.Tensor:
area = math.sqrt(height * width)
ys, xs = torch.meshgrid(_axis(height, area), _axis(width, area), indexing="ij")
@ -52,18 +45,6 @@ def _video_positions(frames: int, height: int, width: int, offset: float) -> tor
return result.reshape(-1, 3)
def _cond_positions(frames: int, cond_t: float, height: int, width: int) -> torch.Tensor:
spatial = _frame_positions(height, width)
g = torch.empty(spatial.shape[0], 3, dtype=torch.float64)
g[:, 0] = cond_t
g[:, 1:] = spatial
return g
def _video_t_spans(n: int) -> list[float]:
return [FRAME_RESCALE * FRAME_PER_TOKEN[k % 5] for k in range(n)]
def _audio_positions(steps: int, offset: float, width: int, height: int) -> torch.Tensor:
area = math.sqrt(height * width)
x_axis = _axis(width, area)
@ -84,8 +65,6 @@ class H3PromptPacker:
self.text_weight = checkpoint.tensor("condition_proj.weight", dtype=torch.bfloat16)
self.text_bias = checkpoint.tensor("condition_proj.bias", dtype=torch.bfloat16)
VISUAL_COND_TIMESTEP = 0.999
def __call__(
self,
text: torch.Tensor,
@ -93,125 +72,35 @@ class H3PromptPacker:
audio: torch.Tensor,
sigma: float | torch.Tensor,
model_timesteps: torch.Tensor | None = None,
*,
text_token_tags: torch.Tensor | None = None,
cond_latents: list[torch.Tensor] | None = None,
cond_frame_indices: list[int] | None = None,
frame_count: int | None = None,
seed: int = 0,
) -> tuple[torch.Tensor, torch.Tensor, list[tuple[int, int, int]], torch.Tensor, tuple[int, int, int], tuple[int, int, int]]:
"""Build ``[text | cond | audio | video]`` packed rows for (fl2va) H3.
``text`` is the refined text span (width 5376 when already refined, 5120
for raw Qwen states); ``text_token_tags`` is the per-token DiT modality tag
(1=text, 0=video over vision pads). ``cond_latents`` are normalized keyframe
latents ``[1,24,1,H/16,W/16]`` and ``cond_frame_indices`` preserves each
keyframe's resolved first/last pixel index. They are spliced right after
the text as non-denoised cond rows with their own near-1 timestep. Returns
``(hidden, times, segments, positions, video_seg, audio_seg)`` where
``segments`` rows are ``t_row*3 + modality_tag``.
"""
) -> tuple[torch.Tensor, torch.Tensor, list[tuple[int, int, int]], tuple[int, int, int], tuple[int, int, int]]:
if text.shape[-1] == 5120:
text_rows = functional.linear(text[0].to(self.text_weight.dtype), self.text_weight, self.text_bias).to(torch.bfloat16)
elif text.shape[-1] == 5376:
text_rows = text[0].to(torch.bfloat16)
else:
raise ValueError("H3 text states must be Qwen 5120-wide or refined 5376-wide.")
text_length = text_rows.shape[0]
latent_t, latent_h, latent_w = video.shape[2], video.shape[-2], video.shape[-1]
frame_rows = (latent_h // 2) * (latent_w // 2)
video_rows = functional.linear(patchify_video(video.to(torch.bfloat16)).float(), self.video_weight, self.video_bias).to(torch.bfloat16)
audio_rows = functional.linear(pack_audio(audio.to(torch.bfloat16)).float(), self.audio_weight, self.audio_bias).to(torch.bfloat16)
cond_rows = None
cond_lengths = []
if cond_latents:
if cond_frame_indices is None or len(cond_frame_indices) != len(cond_latents):
raise ValueError("cond_frame_indices must match cond_latents")
cond_patches = []
# every cond video restarts the same CPU RNG stream (Comfy _cond_video_rows)
for idx, z in enumerate(cond_latents):
r = patchify_video(z.to(torch.float32))
if self.VISUAL_COND_TIMESTEP < 1.0:
gen = torch.Generator("cpu").manual_seed(int(seed))
noise = torch.randn(r.shape, generator=gen, dtype=torch.float32)
r = self.VISUAL_COND_TIMESTEP * r + (1.0 - self.VISUAL_COND_TIMESTEP) * noise.to(r.device)
cond_patches.append(r)
cond_lengths.append(r.shape[0])
cond_rows = functional.linear(torch.cat(cond_patches, dim=0), self.video_weight.to(torch.float32), self.video_bias.to(torch.float32)).to(torch.bfloat16)
text_length, audio_length = text_rows.shape[0], audio_rows.shape[0]
hidden = torch.cat((text_rows, audio_rows, video_rows))
if model_timesteps is None:
video_sigma = torch.as_tensor(sigma, device=text_rows.device, dtype=torch.float32).clamp(min=1e-6)
video_sigma = torch.as_tensor(sigma, device=hidden.device, dtype=torch.float32).clamp(min=1e-6)
base = video_sigma / (12.0 + video_sigma * (1.0 - 12.0))
audio_sigma = 3.0 * base / (1.0 + (3.0 - 1.0) * base)
video_time, audio_time = (1.0 - video_sigma).item(), (1.0 - audio_sigma).item()
unique_times = sorted({video_time, audio_time})
else:
times_override = model_timesteps.to(device=text_rows.device, dtype=torch.float32).flatten()
unique_override = sorted(times_override.tolist())
video_time, audio_time = unique_override[0], unique_override[-1]
has_vis_cond = cond_rows is not None
cond_time = max(video_time, self.VISUAL_COND_TIMESTEP)
unique_times = sorted({video_time, audio_time} | ({cond_time} if has_vis_cond else set()))
t_row = {value: index for index, value in enumerate(unique_times)}
times = torch.tensor(unique_times, device=text_rows.device, dtype=torch.float32)
# Assemble hidden in segment order: text | cond | audio | video.
parts = [text_rows]
offsets = [0]
for block in ((cond_rows, "cond"), (audio_rows, "audio"), (video_rows, "video")):
if block[0] is None:
continue
parts.append(block[0])
offsets.append(offsets[-1] + block[0].shape[0])
hidden = torch.cat(parts, dim=0)
audio_length = audio_rows.shape[0]
cond_length = cond_rows.shape[0] if cond_rows is not None else 0
# Positions: text rows, cond (first/last t anchors), audio, video.
text_positions = torch.stack((torch.arange(text_length, dtype=torch.float64), torch.zeros(text_length), torch.zeros(text_length)), dim=-1)
position_blocks = [text_positions]
if cond_rows is not None and cond_latents:
spans = _video_t_spans(latent_t)
cond_t_values = []
for pixel_index in cond_frame_indices:
if pixel_index == 0:
cond_t_values.append(float(text_length))
elif frame_count is not None and pixel_index == frame_count - 1:
cond_t_values.append(float(text_length) + sum(spans) - FRAME_RESCALE)
else:
raise ValueError("only first/last keyframe anchors are supported")
position_blocks.append(torch.cat([_cond_positions(frame_rows, cond_t, latent_h, latent_w) for cond_t in cond_t_values], dim=0))
position_blocks.append(_audio_positions(audio.shape[-1], float(text_length), latent_w, latent_h))
position_blocks.append(_video_positions(latent_t, latent_h, latent_w, float(text_length)))
positions = torch.cat(position_blocks, dim=0)
# mod_segments: (start, stop, t_row*3 + tag).
segments: list[tuple[int, int, int]] = []
if text_token_tags is not None:
tags = text_token_tags.view(-1).tolist()
row_base = t_row[video_time] * 3
run_start = 0
for i in range(1, text_length + 1):
if i == text_length or tags[i] != tags[run_start]:
segments.append((run_start, i, row_base + int(tags[run_start])))
run_start = i
else:
segments.append((0, text_length, t_row[video_time] * 3 + 1))
cursor_start = text_length
if cond_rows is not None:
for length in cond_lengths:
segments.append((cursor_start, cursor_start + length, t_row[cond_time] * 3 + 0))
cursor_start += length
segments.append((cursor_start, cursor_start + audio_length, t_row[audio_time] * 3 + 2))
cursor_start += audio_length
video_start = cursor_start
video_stop = video_start + video_rows.shape[0]
segments.append((video_start, video_stop, t_row[video_time] * 3 + 0))
video_segment = (video_start, video_stop, t_row[video_time])
audio_start = text_length + cond_length
audio_segment = (audio_start, audio_start + audio_length, t_row[audio_time])
return hidden, times, segments, positions, video_segment, audio_segment
times_override = model_timesteps.to(device=hidden.device, dtype=torch.float32).flatten()
if times_override.numel() not in (1, 2):
raise ValueError("Prompt-only H3 expects one or two model timesteps.")
unique_times = times_override.tolist()
video_time, audio_time = unique_times[0], unique_times[-1]
row = {value: index for index, value in enumerate(unique_times)}
video_row, audio_row = row[video_time] * 3, row[audio_time] * 3
times = torch.tensor(unique_times, device=hidden.device, dtype=torch.float32)
positions = torch.cat((torch.stack((torch.arange(text_length, dtype=torch.float64), torch.zeros(text_length), torch.zeros(text_length)), dim=-1), _audio_positions(audio.shape[-1], float(text_length), video.shape[-1], video.shape[-2]), _video_positions(video.shape[2], video.shape[-2], video.shape[-1], float(text_length))))
block_video_segment = (text_length + audio_length, hidden.shape[0], video_row)
block_audio_segment = (text_length, text_length + audio_length, audio_row + 2)
final_video_segment = (text_length + audio_length, hidden.shape[0], row[video_time])
final_audio_segment = (text_length, text_length + audio_length, row[audio_time])
return hidden, times, [(0, text_length, video_row + 1), block_audio_segment, block_video_segment], positions, final_video_segment, final_audio_segment

79
src/h3_blackwell_runtime/qwen3vl_text.py
View file

@ -42,24 +42,6 @@ class _RMSNorm(nn.Module):
return F.rms_norm(x, self.weight.shape, weight=self.weight.to(x), eps=self.eps)
def _apply_rope(query: torch.Tensor, key: torch.Tensor, freqs) -> tuple[torch.Tensor, torch.Tensor]:
"""Direct PyTorch port of Comfy's `apply_rope` fed `freqs=(cos, sin, neg_sin)`.
``query``/``key`` are ``[batch, heads, seq, head_dim]``; ``freqs`` broadcast over
that shape. This is the exact split-half kernel the reference uses for both the
plain text rope and the Qwen3-VL interleaved mrope.
"""
cosine, sine, negative_sine = freqs
split = query.shape[-1] // 2
q = query * cosine
q[..., :split].addcmul_(query[..., split:], negative_sine)
q[..., split:].addcmul_(query[..., :split], sine)
k = key * cosine
k[..., :split].addcmul_(key[..., split:], negative_sine)
k[..., split:].addcmul_(key[..., :split], sine)
return q.to(query.dtype), k.to(key.dtype)
def _rope(query: torch.Tensor, key: torch.Tensor, theta: float) -> tuple[torch.Tensor, torch.Tensor]:
"""Direct PyTorch port of Comfy's text-only `precompute_freqs_cis` / `apply_rope`."""
sequence, head_dim = query.shape[-2:]
@ -72,7 +54,16 @@ def _rope(query: torch.Tensor, key: torch.Tensor, theta: float) -> tuple[torch.T
sine = embedding.sin().unsqueeze(1)
negative_sine = -sine[..., sine.shape[-1] // 2 :]
sine = sine[..., : sine.shape[-1] // 2]
return _apply_rope(query, key, (cosine, sine, negative_sine))
query_output = query * cosine
split = query_output.shape[-1] // 2
query_output[..., :split].addcmul_(query[..., split:], negative_sine)
query_output[..., split:].addcmul_(query[..., :split], sine)
key_output = key * cosine
split = key_output.shape[-1] // 2
key_output[..., :split].addcmul_(key[..., split:], negative_sine)
key_output[..., split:].addcmul_(key[..., :split], sine)
return query_output.to(query.dtype), key_output.to(key.dtype)
def _qwen_attention(query: torch.Tensor, key: torch.Tensor, value: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
@ -114,7 +105,7 @@ class _Qwen3VLBlock(nn.Module):
self.up_proj = checkpoint.nvfp4_linear(f"{prefix}.mlp.up_proj", output_dtype=dtype)
self.down_proj = checkpoint.nvfp4_linear(f"{prefix}.mlp.down_proj", output_dtype=dtype)
def forward(self, hidden_states: torch.Tensor, freqs_cis=None) -> torch.Tensor:
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
residual = hidden_states
x = self.input_layernorm(hidden_states)
batch, sequence, _ = x.shape
@ -123,10 +114,7 @@ class _Qwen3VLBlock(nn.Module):
value = self.v_proj(x).view(batch, sequence, self.config.num_key_value_heads, self.config.head_dim).transpose(1, 2)
query = self.q_norm(query)
key = self.k_norm(key)
if freqs_cis is None:
query, key = _rope(query, key, self.config.rope_theta)
else:
query, key = _apply_rope(query, key, freqs_cis)
# Comfy selects its small-input SDPA path for Qwen, with an explicit causal mask.
causal_mask = torch.full(
(sequence, sequence),
@ -190,44 +178,6 @@ class Qwen3VL32BTextEncoder(nn.Module):
+ ", ".join(missing)
)
def _embed_rows(self, input_ids: torch.Tensor) -> torch.Tensor:
"""Scaled token embeds ``[batch, tokens, 5120]`` in fp32 (pre-decoder)."""
token_rows = F.embedding(input_ids.to(self.embed_tokens.device), self.embed_tokens).to(torch.float32)
token_scales = F.embedding(input_ids.to(self.embed_scale.device), self.embed_scale)
return (token_rows * token_scales).to(torch.bfloat16).to(torch.float32)
def _run_layers(
self,
hidden_states: torch.Tensor,
*,
position_ids: torch.Tensor | None = None,
visual_pos_masks: torch.Tensor | None = None,
deepstack_embeds: list[torch.Tensor] | None = None,
) -> torch.Tensor:
"""Run all 50 decoder blocks over an fp32 ``[batch, tokens, 5120]`` state.
With ``position_ids`` (``[3, seq]``) the Qwen3-VL interleaved mrope is used
instead of the plain 1D text rope; with ``visual_pos_masks`` +
``deepstack_embeds`` the three DeepStack features are added at the visual
positions of the first three decoder layers (Comfy ``Llama2_.forward``).
"""
hidden_states = hidden_states.to(self.dtype)
freqs_cis = None
if position_ids is not None:
from .qwen3vl_vision import mrope_freqs_cis
freqs_cis = mrope_freqs_cis(position_ids.to(hidden_states.device))
for layer_index, layer in enumerate(self.layers):
hidden_states = layer(hidden_states, freqs_cis)
if (
deepstack_embeds is not None
and visual_pos_masks is not None
and layer_index < len(deepstack_embeds)
):
mask = visual_pos_masks.to(hidden_states.device)
hidden_states[mask] = hidden_states[mask] + deepstack_embeds[layer_index].to(hidden_states)
return hidden_states
@torch.inference_mode()
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
"""Return unnormalized `[batch, tokens, 5120]` output after decoder layer 50."""
@ -235,7 +185,12 @@ class Qwen3VL32BTextEncoder(nn.Module):
raise ValueError(f"input_ids must have shape [batch, tokens], got {tuple(input_ids.shape)}")
if input_ids.numel() == 0:
raise ValueError("input_ids must contain at least one token")
return self._run_layers(self._embed_rows(input_ids))
token_rows = F.embedding(input_ids.to(self.embed_tokens.device), self.embed_tokens).to(torch.float32)
token_scales = F.embedding(input_ids.to(self.embed_scale.device), self.embed_scale)
hidden_states = (token_rows * token_scales).to(self.dtype)
for layer in self.layers:
hidden_states = layer(hidden_states)
return hidden_states
class Qwen3VLPromptConditioner:

762
src/h3_blackwell_runtime/qwen3vl_vision.py
View file

@ -1,762 +0,0 @@
"""Qwen3-VL vision conditioning for the direct H3 runtime (first/last keyframes).
Standalone port of the Comfy reference's Qwen3.5/Qwen3-VL visual stack and the
MiniMax H3 presentation mechanism, built only on ``torch``:
- ``Qwen35VisionModel`` (visual.pos_embed, visual.patch_embed, 27 visual.blocks
with 2D rotary attention, visual.merger) plus the three
``Qwen3VLDeepstackMerger`` (visual.deepstack_merger_list).
- ``process_qwen2vl_images`` image preprocessing (Qwen grid/resize/normalize).
- ``qwen2vl_mrope_position_ids`` (H3 mrope ids from ``embeds_info``),
``precompute_mrope_freqs_cis`` (interleaved Qwen3-VL text mrope), and
``token_tags_from_embeds_info`` (DiT per-token modality tags).
- Comfy ``Llama2_.forward`` DeepStack plumbing: per-decoder-layer additions at
visual positions for the first three decoder layers.
The visual tower is loaded from the same ``qwen3vl_32b_minimax_h3_nvfp4_awq``
safetensors the text encoder uses; its keys are all plain (bf16) tensors under
the ``visual.*`` prefix (verified at 1.1 GB, no quantized sub-tensors).
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from pathlib import Path
import torch
from torch.nn.attention import SDPBackend, sdpa_kernel
from safetensors import safe_open
from torch import nn
from torch.nn import functional as F
# H3 presentation sentinels (see upstream_qwen3vl.py / upstream_text.py).
VISION_START = 151652
VISION_END = 151653
IMAGE_EMBED_TOKEN = 151655
# Qwen3-VL-32B visual geometry (QWEN3VL_VISION in upstream_qwen3vl.py).
VISION_HIDDEN = 1152
VISION_INTERMEDIATE = 4304
VISION_DEPTH = 27
VISION_HEADS = 16
VISION_HEAD_DIM = VISION_HIDDEN // VISION_HEADS # 72
VISION_PATCH = 16
VISION_TEMPORAL = 2
VISION_MERGE = 2
VISION_POSITION_EMBEDS = 2304
DEEPSTACK_VISUAL_INDEXES = (8, 16, 24)
# Qwen3-VL text mrope geometry (Qwen3VL_32BConfig in llama.py).
TEXT_ROPE_DIMS = (24, 20, 20)
TEXT_ROPE_THETA = 5_000_000.0
TEXT_HEAD_DIM = 128
# Qwen image preprocessing policy (process_qwen2vl_images, H3 mean/std 0.5).
QWEN_IMAGE_MEAN = (0.5, 0.5, 0.5)
QWEN_IMAGE_STD = (0.5, 0.5, 0.5)
QWEN_MIN_PIXELS = 3136
QWEN_MAX_PIXELS = 12845056
def process_image(
image: torch.Tensor,
*,
min_pixels: int = QWEN_MIN_PIXELS,
max_pixels: int = QWEN_MAX_PIXELS,
patch_size: int = VISION_PATCH,
temporal_patch_size: int = VISION_TEMPORAL,
merge_size: int = VISION_MERGE,
image_mean: tuple[float, ...] = QWEN_IMAGE_MEAN,
image_std: tuple[float, ...] = QWEN_IMAGE_STD,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Qwen image preprocessing (process_qwen2vl_images): a single
``[1, H, W, 3]`` float image (``[0, 1]``) -> (flatten_patches
``[grid_h*grid_w, C*tp*ps*ps]``, image_grid_thw ``[1, grid_h, grid_w]``).
"""
if image.ndim != 4 or image.shape[0] != 1:
raise ValueError("process_image expects a single [1, H, W, 3] tensor")
batch, height, width, _ = image.shape
device = image.device
images = image.permute(0, 3, 1, 2)
img = images[0]
factor = patch_size * merge_size
h_bar = round(height / factor) * factor
w_bar = round(width / factor) * factor
if h_bar * w_bar > max_pixels:
beta = math.sqrt((height * width) / max_pixels)
h_bar = max(factor, math.floor(height / beta / factor) * factor)
w_bar = max(factor, math.floor(width / beta / factor) * factor)
elif h_bar * w_bar < min_pixels:
beta = math.sqrt(min_pixels / (height * width))
h_bar = math.ceil(height * beta / factor) * factor
w_bar = math.ceil(width * beta / factor) * factor
img_resized = F.interpolate(img.unsqueeze(0), size=(h_bar, w_bar), mode="bilinear", align_corners=False).squeeze(0)
normalized = img_resized.clone()
for c in range(3):
normalized[c] = (img_resized[c] - image_mean[c]) / image_std[c]
grid_h = h_bar // patch_size
grid_w = w_bar // patch_size
grid_thw = torch.tensor([[1, grid_h, grid_w]], device=device, dtype=torch.long)
pixel_values = normalized
channel = pixel_values.shape[0]
grid_t = 1
pixel_values = pixel_values.unsqueeze(0).repeat(2, 1, 1, 1)
patches = pixel_values.reshape(
grid_t,
temporal_patch_size,
channel,
grid_h // merge_size,
merge_size,
patch_size,
grid_w // merge_size,
merge_size,
patch_size,
)
patches = patches.permute(0, 3, 6, 4, 7, 2, 1, 5, 8)
flatten = patches.reshape(grid_t * grid_h * grid_w, channel * temporal_patch_size * patch_size * patch_size)
return flatten, grid_thw
def mrope_position_ids(embeds_info: list[dict], seq_len: int, device) -> torch.Tensor | None:
"""(T, H, W) mrope ids for a sequence with spliced visual blocks (reference
``qwen2vl_mrope_position_ids``). ``embeds_info`` entries carry
``index``/``size`` spans and an ``extra`` dict with a ``grid`` tensor
``[1, grid_h, grid_w]``. Returns ``[3, seq_len]`` or ``None``."""
position_ids = None
offset = 0
for e in embeds_info:
if e.get("type") != "image":
continue
extra = e.get("extra", None)
grid = extra["grid"] if isinstance(extra, dict) else extra
start = e.get("index")
if position_ids is None:
position_ids = torch.zeros((3, seq_len), device=device)
position_ids[:, :start] = torch.arange(0, start, device=device)
end = e.get("size") + start
len_max = int(grid.max()) // 2
start_next = len_max + start
position_ids[:, end:] = torch.arange(start_next + offset, start_next + (seq_len - end) + offset, device=device)
position_ids[0, start:end] = start + offset
max_d = int(grid[0][1]) // 2
position_ids[1, start:end] = (
torch.arange(start + offset, start + max_d + offset, device=device)
.unsqueeze(1)
.repeat(1, math.ceil((end - start) / max_d))
.flatten(0)[: end - start]
)
max_d = int(grid[0][2]) // 2
position_ids[2, start:end] = (
torch.arange(start + offset, start + max_d + offset, device=device)
.unsqueeze(0)
.repeat(math.ceil((end - start) / max_d), 1)
.flatten(0)[: end - start]
)
offset += len_max - (end - start)
return position_ids
def token_tags(seq_len: int, embeds_info: list[dict], device) -> torch.Tensor:
"""DiT per-token AdaLN tags: 0 (video) inside a visual block including the
flanking sentinels, 1 (text) elsewhere (reference
``token_tags_from_embeds_info``)."""
tags = torch.ones(seq_len, dtype=torch.long, device=device)
for e in embeds_info:
if e.get("type") == "image":
start = max(0, e["index"] - 1)
stop = e["index"] + e["size"] + 1
tags[start:stop] = 0
return tags
def mrope_freqs_cis(position_ids: torch.Tensor, *, theta: float = TEXT_ROPE_THETA, head_dim: int = TEXT_HEAD_DIM, rope_dims: tuple[int, ...] = TEXT_ROPE_DIMS) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Interleaved Qwen3-VL text mrope (rope_dims=(24,20,20)).
``position_ids`` is ``[3, seq]``; returns ``(cos, sin, neg_sin)`` matching
the reference ``precompute_freqs_cis`` + ``apply_rope`` convention (shape
``[1, seq, head_dim]``; the ``neg_sin`` entry is ``-sin[..., half:]``).
"""
if position_ids.shape[0] < 1:
position_ids = position_ids.unsqueeze(0)
inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=position_ids.device, dtype=torch.float32) / head_dim))
inv_freq_expanded = inv_freq[None, :, None].expand(position_ids.shape[0], -1, 1)
position_ids_expanded = position_ids[:, None, :].to(torch.float32)
freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)
freqs_inter = freqs[0].clone()
for axis_idx, offset in ((1, 1), (2, 2)):
length = rope_dims[axis_idx] * 3
idx = slice(offset, length, 3)
freqs_inter[..., idx] = freqs[axis_idx, ..., idx]
emb = torch.cat((freqs_inter, freqs_inter), dim=-1)
cos = emb.cos().unsqueeze(0)
sin = emb.sin().unsqueeze(0)
sine = sin[..., : sin.shape[-1] // 2]
negative_sine = -sin[..., sin.shape[-1] // 2 :]
return cos, sine, negative_sine
class _VisionPatchEmbed(nn.Module):
def __init__(self, weight: torch.Tensor, bias: torch.Tensor):
super().__init__()
self.register_buffer("weight", weight, persistent=False)
self.register_buffer("bias", bias, persistent=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
target = self.weight.dtype
x = x.view(-1, 3, VISION_TEMPORAL, VISION_PATCH, VISION_PATCH)
s = (VISION_TEMPORAL, VISION_PATCH, VISION_PATCH)
x = x.to(target)
if x.is_cuda and target in (torch.float16, torch.bfloat16):
# Match Comfy's NVIDIA Conv3d workaround dispatch exactly.
output = torch.cudnn_convolution(
x,
self.weight,
(0, 0, 0),
s,
(1, 1, 1),
1,
benchmark=False,
deterministic=False,
allow_tf32=True,
)
output += self.bias.view(1, -1, 1, 1, 1)
else:
output = F.conv3d(x, self.weight, self.bias, stride=s)
return output.view(-1, self.weight.shape[0])
class _VisionMLP(nn.Module):
def __init__(self, fc1_w: torch.Tensor, fc1_b: torch.Tensor, fc2_w: torch.Tensor, fc2_b: torch.Tensor):
super().__init__()
self.register_buffer("fc1_weight", fc1_w, persistent=False)
self.register_buffer("fc1_bias", fc1_b, persistent=False)
self.register_buffer("fc2_weight", fc2_w, persistent=False)
self.register_buffer("fc2_bias", fc2_b, persistent=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return F.linear(F.gelu(F.linear(x, self.fc1_weight, self.fc1_bias), approximate="tanh"), self.fc2_weight, self.fc2_bias)
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
x1 = x[..., : x.shape[-1] // 2]
x2 = x[..., x.shape[-1] // 2 :]
return torch.cat((-x2, x1), dim=-1)
def _apply_rope_vision(q: torch.Tensor, k: torch.Tensor, freqs) -> tuple[torch.Tensor, torch.Tensor]:
"""Reference ``apply_rope`` (split-half) fed ``freqs=(cos, sin, neg_sin)``.
The vision rotary is built from the doubled-angle embedding so that
``sin[..., half:]`` already equals ``-sin[..., :half]``; the caller passes it
through unchanged, making this the rotate-half equivalent and keeping q/k at
their native dtype.
"""
cos, sin, neg_sin = freqs
original_q = q
q = original_q * cos
split = q.shape[-1] // 2
q[..., :split].addcmul_(original_q[..., split:], neg_sin)
q[..., split:].addcmul_(original_q[..., :split], sin)
original_k = k
k = original_k * cos
k[..., :split].addcmul_(original_k[..., split:], neg_sin)
k[..., split:].addcmul_(original_k[..., :split], sin)
return q, k
class _VisionAttention(nn.Module):
def __init__(self, qkv_w: torch.Tensor, qkv_b: torch.Tensor, proj_w: torch.Tensor, proj_b: torch.Tensor, *, num_heads: int, head_dim: int):
super().__init__()
self.num_heads = num_heads
self.head_dim = head_dim
self.register_buffer("qkv_weight", qkv_w, persistent=False)
self.register_buffer("qkv_bias", qkv_b, persistent=False)
self.register_buffer("proj_weight", proj_w, persistent=False)
self.register_buffer("proj_bias", proj_b, persistent=False)
def forward(self, x: torch.Tensor, cu_seqlens: torch.Tensor, position_embeddings: torch.Tensor) -> torch.Tensor:
seq_length = x.shape[0]
# Cast qkv to a common dtype (the weights may be bf16 or fp32).
qkv_dtype = self.qkv_weight.dtype
qkv = F.linear(x.to(qkv_dtype), self.qkv_weight, self.qkv_bias)
query_states, key_states, value_states = (
qkv.reshape(seq_length, 3, self.num_heads, self.head_dim).permute(1, 0, 2, 3).unbind(0)
)
# RoPE in fp32 to avoid bf16 precision loss, then cast back.
orig_dtype = query_states.dtype
cos, sin, neg_sin = position_embeddings
query_states, key_states = _apply_rope_vision(
query_states.float(), key_states.float(),
(cos.float(), sin.float(), neg_sin.float()),
)
query_states, key_states = query_states.to(orig_dtype), key_states.to(orig_dtype)
lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
attn_outputs = []
for q, k, v in zip(
torch.split(query_states, lengths, dim=0),
torch.split(key_states, lengths, dim=0),
torch.split(value_states, lengths, dim=0),
):
with sdpa_kernel(
[
SDPBackend.FLASH_ATTENTION,
SDPBackend.CUDNN_ATTENTION,
SDPBackend.EFFICIENT_ATTENTION,
SDPBackend.MATH,
],
set_priority=True,
):
output = F.scaled_dot_product_attention(
q.transpose(0, 1).unsqueeze(0),
k.transpose(0, 1).unsqueeze(0),
v.transpose(0, 1).unsqueeze(0),
)
attn_outputs.append(output.transpose(1, 2).reshape(1, q.shape[0], -1))
attn_output = torch.cat(attn_outputs, dim=1)
attn_output = attn_output.reshape(seq_length, -1)
return F.linear(attn_output, self.proj_weight, self.proj_bias)
class _VisionBlock(nn.Module):
def __init__(self, norm1_w: torch.Tensor, norm1_b: torch.Tensor, attn: _VisionAttention, norm2_w: torch.Tensor, norm2_b: torch.Tensor, mlp: _VisionMLP):
super().__init__()
self.attn = attn
self.mlp = mlp
self.register_buffer("norm1_weight", norm1_w, persistent=False)
self.register_buffer("norm1_bias", norm1_b, persistent=False)
self.register_buffer("norm2_weight", norm2_w, persistent=False)
self.register_buffer("norm2_bias", norm2_b, persistent=False)
def forward(self, x: torch.Tensor, cu_seqlens: torch.Tensor, position_embeddings: torch.Tensor) -> torch.Tensor:
x = x + self.attn(F.layer_norm(x, (x.shape[-1],), weight=self.norm1_weight, bias=self.norm1_bias, eps=1e-6), cu_seqlens=cu_seqlens, position_embeddings=position_embeddings)
return x + self.mlp(F.layer_norm(x, (x.shape[-1],), weight=self.norm2_weight, bias=self.norm2_bias, eps=1e-6))
class _VisionPatchMerger(nn.Module):
"""Qwen3-VL spatial-merge projector (main or deepstack).
The main merger applies LayerNorm over ``hidden_size`` (1152) BEFORE the
2x2 spatial merge; the deepstack merger applies LayerNorm over
``merge_dim`` (4608) AFTER the merge. This is controlled by ``norm_dim``.
"""
def __init__(self, norm_w: torch.Tensor, norm_b: torch.Tensor, fc1_w: torch.Tensor, fc1_b: torch.Tensor, fc2_w: torch.Tensor, fc2_b: torch.Tensor, *, merge_size: int, out_hidden_size: int, norm_dim: int | None = None):
super().__init__()
self.merge_dim = VISION_HIDDEN * (merge_size ** 2)
# Default: norm_dim = merge_dim (deepstack style). Main merger overrides.
self.norm_dim = norm_dim if norm_dim is not None else self.merge_dim
self.register_buffer("norm_weight", norm_w, persistent=False)
self.register_buffer("norm_bias", norm_b, persistent=False)
self.register_buffer("fc1_weight", fc1_w, persistent=False)
self.register_buffer("fc1_bias", fc1_b, persistent=False)
self.register_buffer("fc2_weight", fc2_w, persistent=False)
self.register_buffer("fc2_bias", fc2_b, persistent=False)
self.out_hidden_size = out_hidden_size
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x is the existing 2x2-block-major [t*h*w, hidden] patch stream.
if self.norm_dim == self.merge_dim:
# DeepStack merger: group the existing block-major patch stream first.
x = x.view(-1, self.merge_dim)
x = F.layer_norm(x, (self.merge_dim,), weight=self.norm_weight, bias=self.norm_bias, eps=1e-6)
else:
# Main merger: per-patch LayerNorm over hidden, then group 2x2 into merge_dim.
x = F.layer_norm(x, (x.shape[-1],), weight=self.norm_weight, bias=self.norm_bias, eps=1e-6)
x = x.view(-1, self.merge_dim)
return F.linear(F.gelu(F.linear(x, self.fc1_weight, self.fc1_bias)), self.fc2_weight, self.fc2_bias)
def resize_keyframe(image: torch.Tensor, width: int, height: int, *, crop: str = "disabled") -> torch.Tensor:
"""Resize a ``[1, H, W, 3]`` float ``[0, 1]`` keyframe to the ``width x height``
canvas the way the Comfy reference does (``common_upscale(..., "lanczos", crop)``):
- ``crop="disabled"`` (first frame) stretches to the canvas.
- ``crop="center"`` (last frame) aspect-covers then center-crops.
Returns ``[1, height, width, 3]`` float ``[0, 1]``.
"""
if image.ndim != 4 or image.shape[0] != 1:
raise ValueError("resize_keyframe expects a single [1, H, W, 3] image")
samples = image[:, :, :, :3].movedim(-1, 1) # [1, 3, H, W]
if crop == "center":
old_h, old_w = samples.shape[-2], samples.shape[-1]
old_aspect = old_w / old_h
new_aspect = width / height
x = 0
y = 0
if old_aspect > new_aspect:
x = round((old_w - old_w * (new_aspect / old_aspect)) / 2)
elif old_aspect < new_aspect:
y = round((old_h - old_h * (old_aspect / new_aspect)) / 2)
samples = samples.narrow(-2, y, old_h - 2 * y).narrow(-1, x, old_w - 2 * x)
# F.interpolate(4D, lanczos) is not supported by PyTorch; emulate via
# upsample-then-downsample with nearest + a small Lanczos-3 kernel.
# (Simpler: just use PIL's LANCZOS via numpy for the single image.)
import numpy as np
from PIL import Image as PILImage
arr = samples[0].permute(1, 2, 0).cpu().numpy() # [H, W, 3]
arr = (arr * 255.0).astype(np.uint8)
img = PILImage.fromarray(arr, mode="RGB")
img = img.resize((width, height), PILImage.LANCZOS)
samples = torch.from_numpy(np.asarray(img).astype(np.float32) / 255.0).permute(2, 0, 1).unsqueeze(0)
return samples.clamp(0.0, 1.0).movedim(1, -1) # [1, H, W, 3]
def _text_run_ids(tokenizer, prompt: str) -> list[int]:
"""Token ids for a raw text run (``add_special_tokens=False``, no template)."""
raw_tokenizer = getattr(tokenizer, "tokenizer", None)
if raw_tokenizer is not None:
ids = raw_tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids
else:
ids = tokenizer(prompt, device="cpu")
return [int(t) for t in ids.reshape(-1).tolist()]
@dataclass
class Fl2vaPresentation:
"""Expanded first/last keyframe prompt for the H3 DiT.
Carries the Qwen layer-50 text states (with spliced vision pads), the per-token
DiT modality tags, the mrope ids, and the DiT-level keyframe anchors used to
build the cond rows.
"""
input_ids: torch.Tensor
text_states: torch.Tensor
text_token_tags: torch.Tensor
embeds_info: list[dict]
keyframes: list[dict]
frame_count: int
def build_fl2va_presentation(
prompt: str,
first_frame: torch.Tensor | None,
last_frame: torch.Tensor | None,
*,
width: int,
height: int,
frame_count: int,
tokenizer,
vision: Qwen3VL32BVision,
text_encoder: "Qwen3VL32BTextEncoder",
device,
) -> Fl2vaPresentation:
"""Build the fl2va presentation and run the Qwen text conditioner over it.
``first_frame`` / ``last_frame`` are ``[1, H, W, 3]`` ``[0, 1]`` images (already
aligned to the target canvas by the caller when needed). The presentation
sequence is ``[Picture1 <img> Picture2? <img> prompt]``: raw text ids with the
``VISION_START`` sentinel, a vision pad run, and ``VISION_END`` per keyframe,
followed by the raw prompt ids (no chat template).
"""
images: list[torch.Tensor] = []
keyframes: list[dict] = []
if first_frame is not None:
img = resize_keyframe(first_frame, width, height, crop="disabled")
images.append(img)
keyframes.append({"resolved_frame_index": 0, "image": img})
if last_frame is not None:
img = resize_keyframe(last_frame, width, height, crop="center")
images.append(img)
keyframes.append({"resolved_frame_index": frame_count - 1, "image": img})
# Build the entry list: (token_id/"text") runs and image placeholders.
entries: list = []
for i in range(len(images)):
entries.extend((tid, "text") for tid in _text_run_ids(tokenizer, f"<Picture {i + 1}>: "))
entries.append((VISION_START, "text"))
entries.append((i, "image"))
entries.append((VISION_END, "text"))
entries.extend((tid, "text") for tid in _text_run_ids(tokenizer, prompt))
if not any(kind == "text" for _, kind in entries):
entries = [(151643, "text")]
# Expand images through the vision tower up front (needs the vision tower); the
# merged output becomes the pad run that the token sequence points at.
vision_outputs: list[tuple[torch.Tensor, torch.Tensor, torch.Tensor, list[torch.Tensor], int]] = []
for i in range(len(images)):
flatten, grid = process_image(images[i].to(device))
merged, deepstack = vision(flatten, grid.to(device))
vision_outputs.append((flatten, grid, merged, deepstack, merged.shape[0]))
# Expand into a flat token list + embeds_info.
token_ids: list[int] = []
embeds_info: list[dict] = []
vision_index = 0
for tok, kind in entries:
if kind == "image":
flatten, grid, merged, deepstack, size = vision_outputs[vision_index]
size = int(size)
start = len(token_ids)
token_ids.extend([0] * size) # placeholder ids, overwritten by the merged rows
embeds_info.append({"type": "image", "index": start, "size": size, "extra": {"grid": grid, "deepstack": deepstack, "merged": merged}})
vision_index += 1
else:
token_ids.append(int(tok))
seq_len = len(token_ids)
input_ids = torch.tensor([int(t) for t in token_ids], dtype=torch.long, device=device).unsqueeze(0) # [1, seq]
# Build the Qwen hidden-state input: scaled token embeds, with the merged vision
# rows spliced over the pad positions.
base_embeds = text_encoder._embed_rows(input_ids) # [1, seq, 5120] fp32
visual_pos_masks = torch.zeros((1, seq_len), dtype=torch.bool, device=device)
for e in embeds_info:
merged = e["extra"]["merged"]
base_embeds[0, e["index"]: e["index"] + e["size"], :] = merged.to(base_embeds.dtype)
visual_pos_masks[0, e["index"]: e["index"] + e["size"]] = True
# DeepStack features: one concatenated tensor per vision-layer index, spanning all
# spliced blocks in sequence order.
merged_deepstack_per_index: dict = {}
for e in embeds_info:
for i, ds in enumerate(e["extra"]["deepstack"]):
merged_deepstack_per_index.setdefault(i, []).append(ds)
deepstack_embeds = [torch.cat(v, dim=0) for i, v in sorted(merged_deepstack_per_index.items())]
position_ids = mrope_position_ids(embeds_info, seq_len, device)
text_states = text_encoder._run_layers(
base_embeds.float(),
position_ids=position_ids,
visual_pos_masks=visual_pos_masks,
deepstack_embeds=deepstack_embeds,
)
tags = token_tags(seq_len, embeds_info, device)
return Fl2vaPresentation(
input_ids=input_ids,
text_states=text_states,
text_token_tags=tags,
embeds_info=embeds_info,
keyframes=keyframes,
frame_count=frame_count,
)
class Qwen3VL32BVision(nn.Module):
"""H3's Qwen3-VL-32B visual tower (+ DeepStack mergers), from the text-encoder safetensors."""
def __init__(self, checkpoint_path: str | Path, *, device, dtype: torch.dtype):
super().__init__()
self.device = device
self.dtype = dtype
self.spatial_merge_size = VISION_MERGE
self.spatial_merge_unit = VISION_MERGE * VISION_MERGE
self.patch_size = VISION_PATCH
self.hidden_size = VISION_HIDDEN
self.num_heads = VISION_HEADS
self.num_position_embeddings = VISION_POSITION_EMBEDS
self.num_grid_per_side = int(self.num_position_embeddings ** 0.5)
self.depth = VISION_DEPTH
self.deepstack_visual_indexes = list(DEEPSTACK_VISUAL_INDEXES)
self.out_hidden_size = 5120
checkpoint_path = str(checkpoint_path)
required = {
"visual.pos_embed.weight",
"visual.patch_embed.proj.weight",
"visual.patch_embed.proj.bias",
"visual.merger.norm.weight",
"visual.merger.norm.bias",
"visual.merger.linear_fc1.weight",
"visual.merger.linear_fc1.bias",
"visual.merger.linear_fc2.weight",
"visual.merger.linear_fc2.bias",
}
for i in range(self.depth):
required.update({
f"visual.blocks.{i}.norm1.weight",
f"visual.blocks.{i}.norm1.bias",
f"visual.blocks.{i}.attn.qkv.weight",
f"visual.blocks.{i}.attn.qkv.bias",
f"visual.blocks.{i}.attn.proj.weight",
f"visual.blocks.{i}.attn.proj.bias",
f"visual.blocks.{i}.norm2.weight",
f"visual.blocks.{i}.norm2.bias",
f"visual.blocks.{i}.mlp.linear_fc1.weight",
f"visual.blocks.{i}.mlp.linear_fc1.bias",
f"visual.blocks.{i}.mlp.linear_fc2.weight",
f"visual.blocks.{i}.mlp.linear_fc2.bias",
})
for i in range(len(self.deepstack_visual_indexes)):
required.update({
f"visual.deepstack_merger_list.{i}.norm.weight",
f"visual.deepstack_merger_list.{i}.norm.bias",
f"visual.deepstack_merger_list.{i}.linear_fc1.weight",
f"visual.deepstack_merger_list.{i}.linear_fc1.bias",
f"visual.deepstack_merger_list.{i}.linear_fc2.weight",
f"visual.deepstack_merger_list.{i}.linear_fc2.bias",
})
with safe_open(checkpoint_path, framework="pt", device=device) as checkpoint:
names = set(checkpoint.keys())
missing = sorted(required - names)
if missing:
raise ValueError("Not a MiniMax H3 Qwen3-VL vision checkpoint; missing: " + ", ".join(missing[:12]))
self._init_modules(
device,
dtype,
lambda name: checkpoint.get_tensor(name).to(device=device, dtype=dtype),
)
# Comfy's Embedding is not dynamically cast to the FP32 vision stream.
self.pos_embed = checkpoint.get_tensor("visual.pos_embed.weight").to(device=device)
def _init_modules(self, device, dtype, get) -> None:
self.register_buffer("pos_embed", get("visual.pos_embed.weight"), persistent=False)
self.patch_embed = _VisionPatchEmbed(get("visual.patch_embed.proj.weight"), get("visual.patch_embed.proj.bias"))
self.merger = _VisionPatchMerger(
get("visual.merger.norm.weight"), get("visual.merger.norm.bias"),
get("visual.merger.linear_fc1.weight"), get("visual.merger.linear_fc1.bias"),
get("visual.merger.linear_fc2.weight"), get("visual.merger.linear_fc2.bias"),
merge_size=self.spatial_merge_size, out_hidden_size=self.out_hidden_size,
norm_dim=VISION_HIDDEN, # main merger: LayerNorm over hidden before 2x2 merge
)
self.deepstack_merger_list = nn.ModuleList([
_VisionPatchMerger(
get(f"visual.deepstack_merger_list.{i}.norm.weight"), get(f"visual.deepstack_merger_list.{i}.norm.bias"),
get(f"visual.deepstack_merger_list.{i}.linear_fc1.weight"), get(f"visual.deepstack_merger_list.{i}.linear_fc1.bias"),
get(f"visual.deepstack_merger_list.{i}.linear_fc2.weight"), get(f"visual.deepstack_merger_list.{i}.linear_fc2.bias"),
merge_size=self.spatial_merge_size, out_hidden_size=self.out_hidden_size,
)
for i in range(len(self.deepstack_visual_indexes))
])
self.blocks = nn.ModuleList([
_VisionBlock(
get(f"visual.blocks.{i}.norm1.weight"), get(f"visual.blocks.{i}.norm1.bias"),
_VisionAttention(
get(f"visual.blocks.{i}.attn.qkv.weight"), get(f"visual.blocks.{i}.attn.qkv.bias"),
get(f"visual.blocks.{i}.attn.proj.weight"), get(f"visual.blocks.{i}.attn.proj.bias"),
num_heads=self.num_heads, head_dim=VISION_HEAD_DIM,
),
get(f"visual.blocks.{i}.norm2.weight"), get(f"visual.blocks.{i}.norm2.bias"),
_VisionMLP(
get(f"visual.blocks.{i}.mlp.linear_fc1.weight"), get(f"visual.blocks.{i}.mlp.linear_fc1.bias"),
get(f"visual.blocks.{i}.mlp.linear_fc2.weight"), get(f"visual.blocks.{i}.mlp.linear_fc2.bias"),
),
)
for i in range(self.depth)
])
self.rotary_pos_emb = _VisionRotary(VISION_HIDDEN // self.num_heads // 2, device, dtype)
def rot_pos_emb(self, grid_thw: torch.Tensor) -> torch.Tensor:
"""Reference ``rot_pos_emb``: (row, col) coords -> 2D rotary angles per token."""
merge_size = self.spatial_merge_size
grid_thw_list = grid_thw.tolist()
max_hw = max(max(h, w) for _, h, w in grid_thw_list)
freq_table = self.rotary_pos_emb(max_hw).to(grid_thw.device)
device = freq_table.device
total_tokens = sum(int(t * h * w) for t, h, w in grid_thw_list)
pos_ids = torch.empty((total_tokens, 2), dtype=torch.long, device=device)
offset = 0
for num_frames, height, width in grid_thw_list:
num_frames, height, width = int(num_frames), int(height), int(width)
merged_h, merged_w = height // merge_size, width // merge_size
block_rows = torch.arange(merged_h, device=device)
block_cols = torch.arange(merged_w, device=device)
intra_row = torch.arange(merge_size, device=device)
intra_col = torch.arange(merge_size, device=device)
row_idx = (block_rows[:, None, None, None] * merge_size + intra_row[None, None, :, None]).expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
col_idx = (block_cols[None, :, None, None] * merge_size + intra_col[None, None, None, :]).expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
coords = torch.stack((row_idx, col_idx), dim=-1)
if num_frames > 1:
coords = coords.repeat(num_frames, 1)
num_tokens = coords.shape[0]
pos_ids[offset:offset + num_tokens] = coords
offset += num_tokens
return freq_table[pos_ids].flatten(1)
def fast_pos_embed_interpolate(self, grid_thw: torch.Tensor) -> torch.Tensor:
"""Reference 4-tap bilinear interpolation of the learned 48x48 grid."""
grid_ts = [int(row[0]) for row in grid_thw.tolist()]
grid_hs = [int(row[1]) for row in grid_thw.tolist()]
grid_ws = [int(row[2]) for row in grid_thw.tolist()]
device = self.pos_embed.device
idx_list: list[list] = [[] for _ in range(4)]
weight_list: list[list] = [[] for _ in range(4)]
for t, h, w in zip(grid_ts, grid_hs, grid_ws):
h, w = int(h), int(w)
h_idxs = torch.linspace(0, self.num_grid_per_side - 1, h, device=device)
w_idxs = torch.linspace(0, self.num_grid_per_side - 1, w, device=device)
h_floor = h_idxs.int()
w_floor = w_idxs.int()
h_ceil = (h_idxs + 1).int().clamp(max=self.num_grid_per_side - 1)
w_ceil = (w_idxs + 1).int().clamp(max=self.num_grid_per_side - 1)
dh = h_idxs - h_floor
dw = w_idxs - w_floor
base_h = h_floor * self.num_grid_per_side
base_h_ceil = h_ceil * self.num_grid_per_side
indices = [
(base_h[None].T + w_floor[None]).flatten(),
(base_h[None].T + w_ceil[None]).flatten(),
(base_h_ceil[None].T + w_floor[None]).flatten(),
(base_h_ceil[None].T + w_ceil[None]).flatten(),
]
weights = [
((1 - dh)[None].T * (1 - dw)[None]).flatten(),
((1 - dh)[None].T * dw[None]).flatten(),
(dh[None].T * (1 - dw)[None]).flatten(),
(dh[None].T * dw[None]).flatten(),
]
for j in range(4):
idx_list[j].extend(indices[j].tolist())
weight_list[j].extend(weights[j].tolist())
idx_tensor = torch.tensor(idx_list, dtype=torch.long, device=device)
weight_tensor = torch.tensor(weight_list, dtype=self.pos_embed.dtype, device=device)
pos_embeds = self.pos_embed[idx_tensor] * weight_tensor[:, :, None]
patch_pos_embeds = pos_embeds[0] + pos_embeds[1] + pos_embeds[2] + pos_embeds[3]
patch_pos_embeds = patch_pos_embeds.split([h * w for h, w in zip(grid_hs, grid_ws)])
patch_pos_embeds_permute = []
merge_size = self.spatial_merge_size
for pos_embed, t, h, w in zip(patch_pos_embeds, grid_ts, grid_hs, grid_ws):
pos_embed = pos_embed.repeat(t, 1)
pos_embed = pos_embed.view(t, h // merge_size, merge_size, w // merge_size, merge_size, -1).permute(0, 1, 3, 2, 4, 5).flatten(0, 4)
patch_pos_embeds_permute.append(pos_embed)
return torch.cat(patch_pos_embeds_permute)
def forward(self, flatten_patches: torch.Tensor, grid_thw: torch.Tensor) -> tuple[torch.Tensor, list[torch.Tensor]]:
"""Run the visual tower -> (merged, deepstack)."""
x = self.patch_embed(flatten_patches.to(self.dtype).to(self.device))
x = x + self.fast_pos_embed_interpolate(grid_thw).to(x.device)
x = x.reshape(x.shape[0], -1)
rotary = self.rot_pos_emb(grid_thw.to(x.device)).to(x.device).reshape(x.shape[0], -1)
emb = torch.cat((rotary, rotary), dim=-1)
cos = emb.cos().unsqueeze(-2)
sin = emb.sin().unsqueeze(-2)
sin_split = sin.shape[-1] // 2
position_embeddings = (cos, sin[..., :sin_split], -sin[..., sin_split:])
cu_seqlens = F.pad(
torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(0, dtype=torch.int32),
(1, 0), value=0,
)
deepstack_features = []
for layer_num, block in enumerate(self.blocks):
x = block(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings)
# x: [t*h*w, hidden] (unmerged patches).
if layer_num in self.deepstack_visual_indexes:
deepstack_features.append(
self.deepstack_merger_list[self.deepstack_visual_indexes.index(layer_num)](x)
)
return self.merger(x), deepstack_features
class _VisionRotary(nn.Module):
def __init__(self, dim: int, device, dtype):
super().__init__()
# Comfy constructs this buffer on CPU, then moves the model to CUDA.
inv_freq = 1.0 / (10000.0 ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
self.register_buffer("inv_freq", inv_freq.to(device), persistent=False)
def forward(self, seqlen: int) -> torch.Tensor:
seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
return torch.outer(seq, self.inv_freq)

181
src/h3_blackwell_runtime/runtime.py
View file

@ -15,22 +15,12 @@ from .attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND
from .block import configure_mlp_chunking
from .checkpoint import H3Checkpoint
from .denoiser import H3PackedDenoiser
from .lora import load_lora_adapter, set_active_lora
from .latent_upscaler import load_h3_latent_upscaler, upscale_h3_latent
from .packing import H3PromptPacker
from .qwen3vl_text import Qwen3VLPromptConditioner
from .qwen3vl_vision import build_fl2va_presentation, Qwen3VL32BVision
from .sampler import sample_video_res_multistep, sample_video_turbo
from .sampler import sample_video_res_multistep
from .t2v import random_av_latents
from .token_refiner import H3TokenRefiner
from .vae_decoder import MiniMaxH3VideoVAE, dtype_from_name
from .vae_encoder import MiniMaxH3VideoVAEEncoder
TURBO_VARIANTS = {
"4step": {"steps": 4, "video_shift": 6.0},
"8step": {"steps": 8, "video_shift": 12.0},
}
@dataclass(frozen=True)
@ -45,8 +35,6 @@ class RuntimeConfig:
vae_tile_size: int = 256
mlp_chunks: int = 1
mlp_chunk_threshold: int = 4096
turbo_loras: tuple[tuple[str, str], ...] = ()
latent_upscaler_path: str | None = None
device: str = "cuda"
@ -66,25 +54,12 @@ def _refiner_attention_backend(attention: str) -> str:
return fallback if fallback in AVAILABLE_BACKENDS and fallback != "sol_attn" else "sage2"
def normalize_upscale(value) -> float | None:
if value is None or value == "" or value == "none" or value == 1 or value == 1.0:
return None
try:
scale = float(value)
except (TypeError, ValueError) as exc:
raise ValueError("upscale must be a number between 1.0 and 4.0") from exc
if not 1.0 < scale <= 4.0:
raise ValueError("upscale must be greater than 1.0 and at most 4.0")
return scale
class H3HotRuntime:
"""Keep all prompt-only H3 models resident for repeated requests."""
def __init__(self, config: RuntimeConfig):
self.config = config
self.attention = config.attention
self.turbo: str | None = None
self.loaded_at = time.time()
self.load_stages: list[dict] = []
@ -101,15 +76,6 @@ class H3HotRuntime:
"token_refiner_loaded",
lambda: H3TokenRefiner(self.checkpoint, attention_backend=_refiner_attention_backend(config.attention)).eval(),
)
self.turbo_loras = dict(config.turbo_loras)
unknown_turbos = sorted(set(self.turbo_loras) - set(TURBO_VARIANTS))
if unknown_turbos:
raise ValueError(f"Unknown Turbo variants: {unknown_turbos}")
for name, path in self.turbo_loras.items():
self._timed_load(
f"turbo_{name}_loaded",
lambda name=name, path=path: load_lora_adapter(self.model, self.refiner, name, path, config.device),
)
self.packer = H3PromptPacker(self.checkpoint)
self.video_vae = self._timed_load(
"video_vae_loaded",
@ -119,20 +85,6 @@ class H3HotRuntime:
"audio_vae_loaded",
lambda: MiniMaxH3AudioVAE.from_safetensors(config.audio_vae_path, device=config.device).eval(),
)
self.vae_encoder = self._timed_load(
"vae_encoder_loaded",
lambda: MiniMaxH3VideoVAEEncoder.from_safetensors(config.video_vae_path, device=config.device).eval(),
)
self.vision_tower = self._timed_load(
"vision_tower_loaded",
lambda: Qwen3VL32BVision(config.text_encoder_path, device=config.device, dtype=torch.float32),
)
self.latent_upscaler = None
if config.latent_upscaler_path is not None:
self.latent_upscaler = self._timed_load(
"latent_upscaler_loaded",
lambda: load_h3_latent_upscaler(config.latent_upscaler_path, device=config.device, dtype=torch.float16),
)
def _timed_load(self, stage: str, fn):
_sync()
@ -161,9 +113,6 @@ class H3HotRuntime:
"ready": True,
"initial_attention": self.config.attention,
"current_attention": self.attention,
"available_turbos": sorted(self.turbo_loras),
"current_turbo": self.turbo,
"latent_upscaler_loaded": self.latent_upscaler is not None,
"vae_dtype": self.config.vae_dtype,
"vae_tile_size": self.config.vae_tile_size,
"mlp_chunks": self.config.mlp_chunks,
@ -185,53 +134,6 @@ class H3HotRuntime:
block.attention_backend = _refiner_attention_backend(attention)
self.attention = attention
@torch.inference_mode()
def set_turbo(self, turbo: str | None) -> None:
turbo = None if turbo in {None, "", "none"} else turbo
if turbo is not None and turbo not in self.turbo_loras:
raise ValueError(f"Turbo variant {turbo!r} is not loaded")
if turbo == self.turbo:
return
set_active_lora(self.model, self.refiner, turbo)
self.turbo = turbo
def _build_fl2va(self, prompt: str, first_frame: torch.Tensor | None, last_frame: torch.Tensor | None, width: int, height: int, aligned_frames: int):
from .qwen3vl_vision import build_fl2va_presentation, resize_keyframe
device = self.config.device
first = self._image_to_uint8_nhwc(first_frame.to(device)).float() / 255.0 if first_frame is not None else None
last = self._image_to_uint8_nhwc(last_frame.to(device)).float() / 255.0 if last_frame is not None else None
presentation = build_fl2va_presentation(
prompt,
first,
last,
width=width,
height=height,
frame_count=aligned_frames,
tokenizer=self.conditioner.tokenizer,
vision=self.vision_tower,
text_encoder=self.conditioner.encoder,
device=device,
)
# resize each keyframe to the canvas and VAE-encode it (the DiT cond latent).
cond_latents = []
for kf in presentation.keyframes:
resized = resize_keyframe(kf["image"].to(device), width, height, crop="disabled" if kf["resolved_frame_index"] == 0 else "center")
pix = resized.movedim(-1, 1).to(device, dtype=torch.float32).mul(2.0).sub(1.0)
cond_latents.append(self.vae_encoder.encode(pix))
return presentation, cond_latents, aligned_frames
def _image_to_uint8_nhwc(self, img: torch.Tensor) -> torch.Tensor:
"""Normalize a [1,3,H,W] image in [0,1], [-1,1], or [0,255] to NHWC uint8."""
x = img.float()
if x.numel() == 0:
return x
if x.max() > 1.0:
x = x / 255.0 # already 0..255
elif x.min() < 0.0:
x = (x.clamp(-1, 1) + 1) * 0.5 # -1..1 -> 0..1
return (x.movedim(1, -1).clamp(0, 1) * 255).to(torch.uint8)
@torch.inference_mode()
def generate(
self,
@ -244,12 +146,7 @@ class H3HotRuntime:
steps: int,
seed: int,
attention: str | None = None,
turbo: str | None = None,
upscale: float | None = None,
first_frame: torch.Tensor | None = None,
last_frame: torch.Tensor | None = None,
mux_audio: bool = True,
keep_intermediates: bool = False,
ffmpeg_loglevel: str = "error",
save_latent: str | Path | None = None,
cache_mode: str | None = None,
@ -271,70 +168,36 @@ class H3HotRuntime:
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
requested_turbo = None if turbo in {None, "", "none"} else turbo
upscale_scale = normalize_upscale(upscale)
if upscale_scale is not None and self.latent_upscaler is None:
raise ValueError("H3 latent upscaler is not loaded")
if requested_turbo is not None:
if requested_turbo not in self.turbo_loras:
raise ValueError(f"Turbo variant {requested_turbo!r} is not loaded")
expected_steps = TURBO_VARIANTS[requested_turbo]["steps"]
if steps != expected_steps:
raise ValueError(f"Turbo {requested_turbo} requires exactly {expected_steps} steps")
if cache_mode not in {None, "", "disabled", "none"}:
raise ValueError("Turbo sampling does not support denoiser caching")
if attention is not None:
self.set_attention(attention)
self.set_turbo(requested_turbo)
video, audio, aligned_frames = timed(
"latents_initialized",
lambda: random_av_latents(width, height, frames, seed, device=self.config.device),
)
use_fl2va = first_frame is not None or last_frame is not None
if use_fl2va:
# Build the fl2va Qwen presentation, encode the keyframes to cond latents,
# and refine the text span (vision-aware).
presentation, cond_latents, frame_count = timed(
"fl2va_conditioned",
lambda: self._build_fl2va(prompt, first_frame, last_frame, width, height, aligned_frames),
)
text = timed("text_conditioned", lambda: self.refiner(presentation.text_states))
pack_kwargs = {
"text_token_tags": presentation.text_token_tags,
"cond_latents": cond_latents,
"cond_frame_indices": [kf["resolved_frame_index"] for kf in presentation.keyframes],
"frame_count": frame_count,
}
else:
text = timed("text_conditioned", lambda: self.refiner(self.conditioner(prompt)))
pack_kwargs = {}
if self.turbo is None:
sample = lambda: sample_video_res_multistep(
self.model, self.packer, text, video, audio, steps=steps, seed=seed,
return_audio=mux_audio, cache_mode=cache_mode, cache_threshold=cache_threshold,
cache_start_percent=cache_start_percent, cache_end_percent=cache_end_percent,
cache_subsample_factor=cache_subsample_factor, cache_stats=cache_stats, **pack_kwargs,
sampled = timed(
"sampled",
lambda: sample_video_res_multistep(
self.model,
self.packer,
text,
video,
audio,
steps=steps,
return_audio=mux_audio,
cache_mode=cache_mode,
cache_threshold=cache_threshold,
cache_start_percent=cache_start_percent,
cache_end_percent=cache_end_percent,
cache_subsample_factor=cache_subsample_factor,
cache_stats=cache_stats,
),
)
else:
sample = lambda: sample_video_turbo(
self.model, self.packer, text, video, audio, steps=steps,
video_shift=TURBO_VARIANTS[self.turbo]["video_shift"], seed=seed,
return_audio=mux_audio, **pack_kwargs,
)
sampled = timed("sampled", sample)
if mux_audio:
latent, audio_latent = sampled
else:
latent, audio_latent = sampled, None
source_width, source_height = width, height
if upscale_scale is not None:
latent = timed(
"latent_upscaled",
lambda: upscale_h3_latent(self.latent_upscaler, latent, scale=upscale_scale),
)
width = latent.shape[-1] * 16
height = latent.shape[-2] * 16
if save_latent is not None:
latent_path = Path(save_latent)
latent_path.parent.mkdir(parents=True, exist_ok=True)
@ -385,10 +248,7 @@ class H3HotRuntime:
timed("audio_encode", lambda: subprocess.run(_ffmpeg_command(ffmpeg_loglevel, "-y", "-f", "f32le", "-ar", "32000", "-ac", "2", "-i", str(audio_raw), str(audio_output)), check=True))
audio_raw.unlink()
timed("mux", lambda: subprocess.run(_ffmpeg_command(ffmpeg_loglevel, "-y", "-i", str(video_output), "-i", str(audio_output), "-c:v", "copy", "-c:a", "aac", "-shortest", str(output)), check=True))
if not keep_intermediates:
video_output.unlink()
audio_output.unlink()
audio_output = None
return {
"output": str(output),
@ -396,13 +256,8 @@ class H3HotRuntime:
"frames": aligned_frames,
"width": width,
"height": height,
"source_width": source_width,
"source_height": source_height,
"seed": seed,
"attention": self.attention,
"turbo": self.turbo,
"upscale": upscale_scale,
"keep_intermediates": keep_intermediates,
"vae_dtype": self.config.vae_dtype,
"vae_tile_size": self.config.vae_tile_size,
"stages": stages,

100
src/h3_blackwell_runtime/sampler.py
View file

@ -14,28 +14,13 @@ def shifted_sigma(base: torch.Tensor, shift: float) -> torch.Tensor:
def beta_sigmas(steps: int, *, device: torch.device | str, alpha: float = 0.6, beta: float = 0.6) -> torch.Tensor:
"""Comfy's discrete beta scheduler over H3's 1,000-entry shift-12 table."""
import numpy as np
from scipy.stats import beta as beta_distribution
timesteps = (torch.arange(1, 1001, 1) / 1000) * 1000
table = shifted_sigma(timesteps / 1000, 12.0)
fractions = 1.0 - np.linspace(0, 1, steps, endpoint=False)
indices = np.rint(beta_distribution.ppf(fractions, alpha, beta) * 999)
sigmas = []
last_index = -1
for index in indices:
if index != last_index:
sigmas.append(float(table[int(index)]))
last_index = index
return torch.FloatTensor([*sigmas, 0.0]).to(device)
def turbo_sigmas(steps: int, shift: float, *, device: torch.device | str) -> torch.Tensor:
"""Uniform shifted sigma grid used to train the distilled Turbo adapters."""
if steps < 1 or shift <= 0:
raise ValueError("Turbo steps and shift must be positive")
base = torch.linspace(1.0, 0.0, steps + 1, dtype=torch.float32, device="cpu")
return shifted_sigma(base, shift).to(device)
table = shifted_sigma(torch.arange(1, 1001, device=device, dtype=torch.float32) / 1000, 12.0)
fractions = 1.0 - torch.arange(steps, device=device, dtype=torch.float64).cpu().numpy() / steps
indices = torch.from_numpy((999 * beta_distribution.ppf(fractions, alpha, beta)).round().astype("int64")).to(device)
indices = torch.unique_consecutive(indices)
return torch.cat((table[indices], table.new_zeros(1)))
def res_multistep_update(x: torch.Tensor, denoised: torch.Tensor, sigma: torch.Tensor, sigma_down: torch.Tensor, old_denoised: torch.Tensor | None, old_sigma_down: torch.Tensor | None, previous_sigma: torch.Tensor | None) -> torch.Tensor:
@ -92,18 +77,12 @@ def sample_video_res_multistep(
model_timesteps: list[torch.Tensor] | tuple[torch.Tensor, ...] | None = None,
return_audio: bool = False,
progress: bool = False,
seed: int = 0,
text_token_tags: torch.Tensor | None = None,
cond_latents: list[torch.Tensor] | None = None,
cond_frame_indices: list[int] | None = None,
frame_count: int | None = None,
cache_mode: str | None = None,
cache_threshold: float = 0.0,
cache_start_percent: float = 0.0,
cache_end_percent: float = 1.0,
cache_subsample_factor: int = 2,
cache_stats: dict | None = None,
audio_step_trace: list[dict] | None = None,
) -> torch.Tensor:
"""Direct H3 beta/RES sampling with Comfy-equivalent joint AV carry semantics."""
sigmas = beta_sigmas(steps, device=video.device)
@ -129,7 +108,6 @@ def sample_video_res_multistep(
step_started = time.perf_counter()
previous_index = index - 1
sigma_down = sigmas[index]
audio_before = audio_carried if audio_step_trace is None else audio_carried.detach().cpu()
current_percent = previous_index / total_steps
can_cache = cache_mode is not None and cache_threshold > 0 and cache_start_percent <= current_percent <= cache_end_percent and cache["video_diff"] is not None
skipped = False
@ -158,18 +136,7 @@ def sample_video_res_multistep(
carry = sigma_audio / sigma
native_audio = audio_carried.to(torch.bfloat16) * carry
step_timesteps = None if model_timesteps is None else model_timesteps[previous_index]
hidden, times, segments, positions, video_segment, audio_segment = packer(
text,
video,
native_audio,
_model_sigma(sigma),
step_timesteps,
text_token_tags=text_token_tags,
cond_latents=cond_latents,
cond_frame_indices=cond_frame_indices,
frame_count=frame_count,
seed=seed,
)
hidden, times, segments, positions, video_segment, audio_segment = packer(text, video, native_audio, _model_sigma(sigma), step_timesteps)
raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
raw_video = raw_video.to(torch.bfloat16).float()
raw_audio = raw_audio.to(torch.bfloat16)
@ -189,16 +156,6 @@ def sample_video_res_multistep(
previous_sigma = sigmas[previous_index - 1] if previous_index else None
video = res_multistep_update(video, video_denoised, sigma, sigma_down, video_history, video_history_sigma, previous_sigma)
audio_carried = res_multistep_update(audio_carried, audio_denoised, sigma, sigma_down, audio_history, audio_history_sigma, previous_sigma)
if audio_step_trace is not None:
audio_step_trace.append({
"step": index,
"video_sigma": float(sigma),
"audio_sigma": float(_audio_sigma(sigma)),
"video_sigma_down": float(sigma_down),
"audio_before": audio_before,
"audio_denoised": audio_denoised.detach().cpu(),
"audio_after": audio_carried.detach().cpu(),
})
video_history, audio_history = video_denoised, audio_denoised
video_history_sigma = audio_history_sigma = sigma_down
if progress:
@ -212,51 +169,6 @@ def sample_video_res_multistep(
return (video, _decode_audio_latent(audio_carried)) if return_audio else video
@torch.inference_mode()
def sample_video_turbo(
model,
packer: H3PromptPacker,
text: torch.Tensor,
video: torch.Tensor,
audio: torch.Tensor,
*,
steps: int,
video_shift: float,
audio_shift: float = 3.0,
return_audio: bool = False,
seed: int = 0,
text_token_tags: torch.Tensor | None = None,
cond_latents: list[torch.Tensor] | None = None,
cond_frame_indices: list[int] | None = None,
frame_count: int | None = None,
) -> torch.Tensor:
"""Run the distilled H3 training-Euler schedule on independent AV clocks."""
video_schedule = turbo_sigmas(steps, video_shift, device=video.device)
audio_schedule = turbo_sigmas(steps, audio_shift, device=audio.device)
for index in range(steps):
video_sigma = video_schedule[index]
audio_sigma = audio_schedule[index]
model_timesteps = torch.stack((1.0 - video_sigma, 1.0 - audio_sigma))
hidden, times, segments, positions, video_segment, audio_segment = packer(
text,
video,
audio,
video_sigma,
model_timesteps,
text_token_tags=text_token_tags,
cond_latents=cond_latents,
cond_frame_indices=cond_frame_indices,
frame_count=frame_count,
seed=seed,
)
raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
video_output = unpatchify_video(raw_video.to(torch.bfloat16).float(), video.shape[2], video.shape[-2], video.shape[-1])
audio_output = _unpack_audio(raw_audio.to(torch.bfloat16)).float()
video = video.float() + (video_sigma - video_schedule[index + 1]) * video_output
audio = audio.float() + (audio_sigma - audio_schedule[index + 1]) * audio_output
return (video, audio) if return_audio else video
@torch.inference_mode()
def sample_video_euler(model, packer: H3PromptPacker, text: torch.Tensor, video: torch.Tensor, audio: torch.Tensor, *, steps: int = 2) -> torch.Tensor:
"""Use Euler updates to obtain a visual-only H3 preview, not parity sampling."""

101
src/h3_blackwell_runtime/tensor_parallel.py
View file

@ -1,101 +0,0 @@
"""True NVFP4 tensor-parallel sharding for the H3 denoiser."""
from __future__ import annotations
import torch
from .distributed import SequenceParallelContext, balanced_ranges
from .nvfp4 import Nvfp4Linear, Nvfp4LinearTensors
def aligned_balanced_ranges(total: int, parts: int, alignment: int) -> tuple[tuple[int, int], ...]:
"""Balance ranges in indivisible alignment-sized groups."""
if alignment < 1 or total % alignment:
raise ValueError(f"total {total} must be divisible by alignment {alignment}")
groups = balanced_ranges(total // alignment, parts)
return tuple((start * alignment, stop * alignment) for start, stop in groups)
def _reject_lora(linear: Nvfp4Linear) -> None:
if len(linear.lora_branches):
raise ValueError("tensor-parallel sharding must occur before loading Turbo LoRA branches")
def select_nvfp4_outputs(linear: Nvfp4Linear, ranges: tuple[tuple[int, int], ...]) -> Nvfp4Linear:
"""Create one column-parallel NVFP4 linear from selected output-row ranges."""
_reject_lora(linear)
indices = torch.cat([
torch.arange(start, stop, device=linear.weight.device)
for start, stop in ranges
])
bias = None if linear.bias is None else linear.bias.index_select(0, indices)
tensors = Nvfp4LinearTensors(
weight=linear.weight.index_select(0, indices).contiguous(),
weight_scale=linear.weight_scale.index_select(0, indices).contiguous(),
weight_scale_2=linear.weight_scale_2,
bias=bias.contiguous() if bias is not None else None,
pre_quant_scale=linear.pre_quant_scale,
full_precision_matrix_mult=linear.full_precision_matrix_mult,
in_features=linear.in_features,
out_features=indices.numel(),
)
return Nvfp4Linear(tensors, output_dtype=linear.output_dtype)
def slice_nvfp4_inputs(linear: Nvfp4Linear, start: int, stop: int) -> tuple[Nvfp4Linear, torch.Tensor | None]:
"""Create one row-parallel NVFP4 linear and return its once-only output bias."""
_reject_lora(linear)
if start < 0 or stop > linear.in_features or start >= stop:
raise ValueError(f"invalid input shard [{start}, {stop}) for width {linear.in_features}")
if start % 32 or stop % 32:
raise ValueError("NVFP4 input shards must align to 32 features")
bias = linear.bias
tensors = Nvfp4LinearTensors(
weight=linear.weight[:, start // 2 : stop // 2].contiguous(),
weight_scale=linear.weight_scale[:, start // 16 : stop // 16].contiguous(),
weight_scale_2=linear.weight_scale_2,
bias=None,
pre_quant_scale=(
None if linear.pre_quant_scale is None
else linear.pre_quant_scale[start:stop].contiguous()
),
full_precision_matrix_mult=linear.full_precision_matrix_mult,
in_features=stop - start,
out_features=linear.out_features,
)
return Nvfp4Linear(tensors, output_dtype=linear.output_dtype), bias
def configure_h3_tensor_parallel(model, context: SequenceParallelContext) -> None:
"""Shard all denoiser attention and MLP linears in place across ranks."""
for block in model.backbone.blocks:
attention = block.attention
global_heads = attention.heads
if global_heads != context.heads or attention.head_dim != context.head_dim:
raise ValueError("tensor-parallel context does not match H3 attention dimensions")
head_start, head_stop = context.local_head_range
inner = global_heads * attention.head_dim
local_start = head_start * attention.head_dim
local_stop = head_stop * attention.head_dim
attention.qkv_proj = select_nvfp4_outputs(attention.qkv_proj, (
(local_start, local_stop),
(inner + local_start, inner + local_stop),
(2 * inner + local_start, 2 * inner + local_stop),
))
attention.out_proj, output_bias = slice_nvfp4_inputs(
attention.out_proj, local_start, local_stop,
)
attention.heads = context.local_head_count
attention.register_buffer("tensor_parallel_output_bias", output_bias, persistent=False)
mlp = block.mlp
intermediate = mlp.fc2.in_features
mlp_ranges = aligned_balanced_ranges(intermediate, context.world_size, 32)
mlp_start, mlp_stop = mlp_ranges[context.rank]
mlp.fc1 = select_nvfp4_outputs(mlp.fc1, (
(mlp_start, mlp_stop),
(intermediate + mlp_start, intermediate + mlp_stop),
))
mlp.fc2, output_bias = slice_nvfp4_inputs(mlp.fc2, mlp_start, mlp_stop)
mlp.register_buffer("tensor_parallel_output_bias", output_bias, persistent=False)
mlp.tensor_parallel_intermediate_ranges = mlp_ranges

8
src/h3_blackwell_runtime/token_refiner.py
View file

@ -6,19 +6,15 @@ from torch import nn
from .attention import rms_norm, run_attention
from .checkpoint import H3Checkpoint
from .lora import DynamicLoraMixin
class _Linear(DynamicLoraMixin, nn.Module):
class _Linear(nn.Module):
def __init__(self, checkpoint: H3Checkpoint, prefix: str, dtype: torch.dtype):
super().__init__()
self.register_buffer("weight", checkpoint.tensor(f"{prefix}.weight", dtype=dtype), persistent=False)
self.in_features = self.weight.shape[1]
self.out_features = self.weight.shape[0]
self._init_dynamic_lora()
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self._apply_lora(x, functional.linear(x, self.weight))
return functional.linear(x, self.weight)
class _RefinerBlock(nn.Module):

28
src/h3_blackwell_runtime/vae_decoder.py
View file

@ -32,21 +32,12 @@ def dtype_from_name(name: str) -> torch.dtype:
def _rms_norm(x: torch.Tensor, weight: torch.Tensor | None, eps: float) -> torch.Tensor:
if os.getenv("H3_VAE_FAST_OPS", "").lower() in {"1", "true", "yes", "on"}:
if weight is None:
return F.rms_norm(x, (x.shape[-1],), eps=eps)
return F.rms_norm(x, weight.shape, weight=weight.to(device=x.device, dtype=x.dtype), eps=eps)
def _conv3d(x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor | None) -> torch.Tensor:
if x.is_cuda and weight.dtype in (torch.float16, torch.bfloat16):
output = torch.cudnn_convolution(
x, weight, (0, 0, 0), (1, 1, 1), (1, 1, 1), 1,
benchmark=False, deterministic=False, allow_tf32=True,
)
if bias is not None:
output += bias.reshape(1, -1, 1, 1, 1)
return output
return F.conv3d(x, weight, bias)
result = x * torch.rsqrt(x.float().square().mean(dim=-1, keepdim=True) + eps).to(x.dtype)
return result if weight is None else result * weight.to(dtype=x.dtype)
def create_token_ids(patch_dims: tuple[int, int, int], device: torch.device, dtype: torch.dtype) -> torch.Tensor:
@ -59,13 +50,16 @@ class RotaryEmbeddingND(nn.Module):
super().__init__()
self.rotary_base = rotary_base
self.step = 2 * n_dim / dim
inv_freq = 1 / rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32)
inv_freq = 1 / rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32, device=device)
self.register_buffer("inv_freq", inv_freq, persistent=False)
self.angle_scale = 2.0 * math.pi
def forward(self, img_ids: torch.Tensor) -> torch.Tensor:
inv_freq = self.inv_freq
inv_freq = inv_freq.to(device=img_ids.device, dtype=img_ids.dtype)
if inv_freq.device.type == "meta":
inv_freq = 1 / self.rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32, device=img_ids.device)
else:
inv_freq = inv_freq.to(img_ids.device)
angles = self.angle_scale * img_ids[:, :, :, None].float() * inv_freq[None, None, None, :]
angles = angles.flatten(2, 3)
cos, sin = torch.cos(angles), torch.sin(angles)
@ -95,6 +89,7 @@ class FeedForward(nn.Module):
def _apply_rope_split_half(x: torch.Tensor, table: torch.Tensor) -> torch.Tensor:
"""Apply the reference split-half RoPE layout to leading rotary channels."""
if os.getenv("H3_VAE_FAST_OPS", "").lower() in {"1", "true", "yes", "on"}:
try:
return torch.ops.comfy_kitchen.apply_rope_split_half1(x, table)
except Exception:
@ -122,6 +117,7 @@ class Attention(nn.Module):
qkv = self.to_qkv(x).view(batch, sequence, self.heads, 3 * self.dim_head)
query, key, value = qkv.chunk(3, dim=-1)
query, key = self.norm_q(query), self.norm_k(key)
if os.getenv("H3_VAE_FAST_OPS", "").lower() in {"1", "true", "yes", "on"}:
try:
rot = rotary_pos_emb.shape[-3] * 2
query_rot, key_rot = torch.ops.comfy_kitchen.apply_rope_split_half(query[..., :rot], key[..., :rot], rotary_pos_emb)
@ -129,6 +125,8 @@ class Attention(nn.Module):
key = torch.cat((key_rot, key[..., rot:]), dim=-1)
except Exception:
query, key = _apply_rope_split_half(query, rotary_pos_emb), _apply_rope_split_half(key, rotary_pos_emb)
else:
query, key = _apply_rope_split_half(query, rotary_pos_emb), _apply_rope_split_half(key, rotary_pos_emb)
query, key, value = query.transpose(1, 2), key.transpose(1, 2), value.transpose(1, 2)
try:
from comfy.ldm.modules.attention import optimized_attention
@ -252,7 +250,7 @@ class MiniMaxH3VideoVAE(nn.Module):
return model
def _decode_pixels(self, z: torch.Tensor) -> torch.Tensor:
return self.decoder(_conv3d(z, self.post_quant_conv.weight, self.post_quant_conv.bias))
return self.decoder(self.post_quant_conv(z))
def split_tiles(self, length: int) -> tuple[list[int], list[int], list[int]]:
if self.tile_size >= length:

327
src/h3_blackwell_runtime/vae_encoder.py
View file

@ -1,327 +0,0 @@
"""Direct, encoder-only MiniMax H3 video VAE implementation.
Mirrors the encoder half of ``upstream_vae.py`` so keyframe/reference images can
be encoded without ComfyUI. The encoder runs in FP16 like Comfy's keyframe VAE
path, and latent moments are upcast for mean/std normalization.
Causal-conv semantics: spatial padding is reflect; temporal padding is causal
(front-only zeros) with a stride grid that starts at the first input frame
(Comfy autopad "same" / ``causal``). For a single input frame the temporal taps
of the kernel are truncated (Comfy ``autopad="causal_zero"``) so a keyframe is
never convolved against zero frames.
Weights are read straight from the checkpoint's ``encoder.*`` / ``quant_conv.*``
keys (a direct name-for-name copy into plain tensors) and applied by the
stateless kernels below.
"""
from __future__ import annotations
import math
import os
from pathlib import Path
import torch
from safetensors import safe_open
from torch import nn
from torch.nn import functional as F
IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)
LATENTS_MEAN = (0.858090341091156, -0.9606591463088989, 1.0661640167236328, -0.5090325474739075, -0.2727581858634949, -1.3675414323806763, -0.2553254961967468, -0.26907554268836975, -0.5376840829849243, -0.0464097298681736, 0.6657370328903198, 0.19690127670764923, -0.5460608005523682, -0.4035342037677765, -0.23683024942874908, 0.25928452610969543, -0.30133944749832153, 0.211341992020607, -1.1206848621368408, 0.3581933379173279, -0.04225143790245056, 0.2604829967021942, 0.22864092886447906, 0.7056031823158264)
LATENTS_STD = (1.2223774194717407, 1.2767263650894165, 1.68317747116088865, 1.7549455165863037, 1.5636216402053833, 2.194143533706665, 0.96531379222869875, 1.05698859691619875, 0.841948926448822, 0.7729952931404114, 1.8955937623977661, 0.946841835975647, 0.7996809482574463, 0.44988900423049925, 0.7197399735450745, 0.69362932443618775, 2.961095094680786, 2.7694199085235595, 3.0496184825897215, 2.1088054180145265, 3.276226282119751, 3.1627357006073, 2.28168129920959475, 2.6127843856811525)
CH = 128
CH_MULT = (1, 2, 2, 4, 4, 8)
SPACE_DOWN = (2, 2, 2, 2, 1, 1)
TIME_DOWN = (1, 2, 2, 1, 1, 1)
NUM_RES_BLOCKS = 2
VAE_RATIO = 16
NIN_LEVELS = frozenset({1, 3, 5})
DOWNSAMPLE_LEVELS = frozenset({0, 1, 2, 3})
def _conv3d(x, weight, bias, stride=(1, 1, 1), padding=(0, 0, 0)):
if x.is_cuda and weight.dtype in (torch.float16, torch.bfloat16):
output = torch.cudnn_convolution(
x, weight, padding, stride, (1, 1, 1), 1,
benchmark=False, deterministic=False, allow_tf32=True,
)
if bias is not None:
output += bias.reshape(1, -1, 1, 1, 1)
return output
return F.conv3d(x, weight, bias, stride, padding)
def _causal_conv3d(x, weight, bias, *, kernel_size, stride, spatial_padding, temporal_causal):
"""Causal 3D conv (matches upstream_vae.CausalConv3d).
- ``spatial_padding > 0``: reflect H and W by ``spatial_padding`` on each side.
- ``temporal_causal``: front-zero T by ``kernel_size - 1`` (the reference's
``causal_padding[0] * 2`` for ``causal_padding[0]=1``, which is every
3D-causal conv in the H3 VAE). A single-frame input truncates the
temporal taps to the final input-aligned tap instead of convolving zero rows.
- If neither applies: no padding at all (reference early-return).
"""
if x.shape[2] == 1:
# Keyframe path (matches reference's `autopad="causal_zero"`):
# apply spatial-reflect pad (if any), then run an effective 2D conv
# by slicing the kernel to its final input-aligned temporal tap.
if spatial_padding > 0:
x = F.pad(x, (spatial_padding, spatial_padding, spatial_padding, spatial_padding, 0, 0), mode="reflect")
kernel_5d = weight[:, :, -1:, :, :]
return _conv3d(x, kernel_5d, bias, (1, stride[1], stride[2]))
if spatial_padding > 0:
x = F.pad(x, (spatial_padding, spatial_padding, spatial_padding, spatial_padding, 0, 0), mode="reflect")
if temporal_causal:
x = F.pad(x, (0, 0, 0, 0, kernel_size - 1, 0))
return _conv3d(x, weight, bias, stride)
def _group_norm_3d(x, weight, bias):
"""GroupNorm (32 groups, eps 1e-6) with per-frame statistics."""
b, c, t, h, w = x.shape
y = F.group_norm(x.permute(0, 2, 1, 3, 4).contiguous().view(b * t, c, 1, h, w), 32, weight, bias, 1e-6)
return y.view(b, t, c, h, w).permute(0, 2, 1, 3, 4).contiguous()
def _resnet(x, p):
# nin_shortcut uses CausalConv3d(k=1, padding=1) in the reference.
residual = x if p["nin"] is None else _conv3d(x, p["nin"][0], p["nin"][1])
h = _causal_conv3d(F.silu(_group_norm_3d(x, p["norm1_w"], p["norm1_b"])), p["conv1_w"], p["conv1_b"], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
h = _causal_conv3d(F.silu(_group_norm_3d(h, p["norm2_w"], p["norm2_b"])), p["conv2_w"], p["conv2_b"], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
return h.add_(residual)
def _downsample(x, p):
if p["space"] == 2:
# Reference Downsample3D pads only the right and bottom edges.
x = F.pad(x, (0, 1, 0, 1, 0, 0), mode="reflect")
# Conv uses padding=(1,0,0) -> causal_padding=(1,0,0), so spatial pad=0,
# temporal front-zero is applied.
return _causal_conv3d(x, p["w"], p["b"], kernel_size=3, stride=(p["time"], p["space"], p["space"]), spatial_padding=0, temporal_causal=True)
def _encoder_run(x, E):
h = _causal_conv3d(x, E["conv_in"][0], E["conv_in"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
for level in E["down"]:
for blk in level["blocks"]:
h = _resnet(h, blk)
if level["down"] is not None:
h = _downsample(h, level["down"])
h = F.silu(_group_norm_3d(h, E["norm_out_w"], E["norm_out_b"]))
return _causal_conv3d(h, E["conv_out"][0], E["conv_out"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
class MiniMaxH3VideoVAEEncoder(nn.Module):
"""Encoder-only H3 VAE. ``encode`` matches the public contract of upstream_vae.py.
Weights are plain tensors loaded from the checkpoint by canonical name into
``self.W`` (a dict), so no ``nn.Module`` sub-hierarchy is needed.
"""
def __init__(self, *, tiling: bool = True, compute_dtype: torch.dtype = torch.float16):
super().__init__()
self.compute_dtype = compute_dtype
self.vae_ratio, self.vae_ratio_t = VAE_RATIO, 4
self.clip_length, self.token_drop = 17, 3
self.frame_pre_padding = (-self.clip_length) % self.vae_ratio_t
self.tiling, self.tile_size, self.tile_overlap_min = tiling, 256, 64
self.quant_conv = nn.Conv3d(48, 48, 1)
self.register_buffer("latents_mean", torch.tensor(LATENTS_MEAN), persistent=False)
self.register_buffer("latents_std", torch.tensor(LATENTS_STD), persistent=False)
self.register_buffer("pixel_mean", torch.tensor(IMAGENET_MEAN).view(1, 3, 1, 1, 1), persistent=False)
self.register_buffer("pixel_std", torch.tensor(IMAGENET_STD).view(1, 3, 1, 1, 1), persistent=False)
def _required_encoder_names(self) -> list[str]:
names = [
"encoder.conv_in.weight", "encoder.conv_in.bias",
"encoder.norm_out.weight", "encoder.norm_out.bias",
"encoder.conv_out.weight", "encoder.conv_out.bias",
"quant_conv.weight", "quant_conv.bias",
"latents_mean", "latents_std",
]
for i in range(len(CH_MULT)):
for b in range(NUM_RES_BLOCKS):
base = f"encoder.down.{i}.block.{b}."
names += [
base + "conv1.weight", base + "conv1.bias",
base + "conv2.weight", base + "conv2.bias",
base + "norm1.weight", base + "norm1.bias",
base + "norm2.weight", base + "norm2.bias",
]
if b == 0 and i in NIN_LEVELS:
names += [base + "nin_shortcut.weight", base + "nin_shortcut.bias"]
if i in DOWNSAMPLE_LEVELS:
names += [f"encoder.down.{i}.downsample.conv.weight", f"encoder.down.{i}.downsample.conv.bias"]
return names
@classmethod
def from_safetensors(cls, path: str | Path, *, device: str | torch.device = "cuda", tiling: bool = True, dtype: torch.dtype = torch.float16) -> "MiniMaxH3VideoVAEEncoder":
model = cls(tiling=tiling, compute_dtype=dtype)
names = model._required_encoder_names()
if os.getenv("H3_FAST_SAFETENSORS", "").lower() in {"1", "true", "yes", "on"}:
from fastsafetensors import fastsafe_open
fast_device = "cuda:0" if str(device) == "cuda" else str(device)
with fastsafe_open(filenames=[str(path)], nogds=True, device=fast_device) as ck:
available = set(ck.keys())
missing = [n for n in names if n not in available]
if missing:
raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
W = {n: ck.get_tensor(n).clone().detach().to(dtype=dtype) for n in names}
elif os.getenv("H3_DISABLE_MMAP", "").lower() in {"1", "true", "yes", "on"}:
from safetensors.torch import load
with open(path, "rb") as file:
available_weights = load(file.read())
available = set(available_weights)
missing = [n for n in names if n not in available]
if missing:
raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
W = {n: available_weights[n].to(device=device, dtype=dtype) for n in names}
del available_weights
else:
with safe_open(str(path), framework="pt", device=str(device)) as ck:
available = set(ck.keys())
missing = [n for n in names if n not in available]
if missing:
raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
W = {n: ck.get_tensor(n).to(dtype=dtype).to(device) for n in names}
# Build the structured params dict.
down = []
for i in range(len(CH_MULT)):
mid = CH * CH_MULT[i]
blocks = []
for b in range(NUM_RES_BLOCKS):
base = f"encoder.down.{i}.block.{b}."
blk = {
"conv1_w": W[base + "conv1.weight"], "conv1_b": W[base + "conv1.bias"],
"conv2_w": W[base + "conv2.weight"], "conv2_b": W[base + "conv2.bias"],
"norm1_w": W[base + "norm1.weight"], "norm1_b": W[base + "norm1.bias"],
"norm2_w": W[base + "norm2.weight"], "norm2_b": W[base + "norm2.bias"],
"nin": None,
}
if base + "nin_shortcut.weight" in W:
blk["nin"] = (W[base + "nin_shortcut.weight"], W[base + "nin_shortcut.bias"])
blocks.append(blk)
level_down = None
if i in DOWNSAMPLE_LEVELS:
ds = f"encoder.down.{i}.downsample.conv."
level_down = {"w": W[ds + "weight"], "b": W[ds + "bias"], "time": TIME_DOWN[i], "space": SPACE_DOWN[i]}
down.append({"blocks": blocks, "down": level_down})
E = {
"conv_in": (W["encoder.conv_in.weight"], W["encoder.conv_in.bias"]),
"down": down,
"norm_out_w": W["encoder.norm_out.weight"], "norm_out_b": W["encoder.norm_out.bias"],
"conv_out": (W["encoder.conv_out.weight"], W["encoder.conv_out.bias"]),
}
model.W = E
model.quant_conv.to(device, dtype)
with torch.no_grad():
model.quant_conv.weight.copy_(W["quant_conv.weight"])
model.quant_conv.bias.copy_(W["quant_conv.bias"])
model.latents_mean.copy_(W["latents_mean"].float().cpu())
model.latents_std.copy_(W["latents_std"].float().cpu())
for b in ("latents_mean", "latents_std", "pixel_mean", "pixel_std"):
getattr(model, b).to(device)
return model
@torch.inference_mode()
def _encode_moments(self, x: torch.Tensor) -> torch.Tensor:
return _conv3d(_encoder_run(x.to(self.compute_dtype), self.W), self.quant_conv.weight, self.quant_conv.bias)
def _adaptive_encode(self, x: torch.Tensor) -> torch.Tensor:
if self.tiling:
return self.tiled_encode(x)
return self._encode_moments(x)
def split_tiles(self, length: int) -> tuple[list[int], list[int], list[int]]:
if self.tile_size >= length:
return [0], [length], []
count = math.ceil(length / self.tile_size)
while self.tile_size * count - self.tile_overlap_min * (count - 1) < length:
count += 1
overlaps = [self.tile_overlap_min] * (count - 1)
for index in range((self.tile_size * count - sum(overlaps) - length) // self.vae_ratio):
overlaps[index % len(overlaps)] += self.vae_ratio
starts = [0]
for overlap in overlaps:
starts.append(starts[-1] + self.tile_size - overlap)
return starts, [self.tile_size] * count, overlaps
@staticmethod
def blend(a: torch.Tensor, b: torch.Tensor, extent: int, dim: int) -> torch.Tensor:
extent = min(a.shape[dim], b.shape[dim], extent)
positions = torch.arange(extent, device=b.device, dtype=b.dtype)
weight_a = 1 - positions / extent
weight_b = positions / extent
shape = [1] * a.ndim
shape[dim] = extent
weight_a = weight_a.view(shape)
weight_b = weight_b.view(shape)
slice_a = [slice(None)] * a.ndim
slice_a[dim] = slice(-extent, None)
slice_b = [slice(None)] * a.ndim
slice_b[dim] = slice(0, extent)
blended = a[tuple(slice_a)] * weight_a + b[tuple(slice_b)] * weight_b
if extent < b.shape[dim]:
slice_b_rest = [slice(None)] * b.ndim
slice_b_rest[dim] = slice(extent, None)
return torch.cat((blended, b[tuple(slice_b_rest)]), dim=dim)
return blended
def tiled_encode(self, x: torch.Tensor) -> torch.Tensor:
height, width = x.shape[-2], x.shape[-1]
y_idx, y_len, y_overlap = self.split_tiles(height)
x_idx, x_len, x_overlap = self.split_tiles(width)
rows = [[self._encode_moments(x[..., i_pos:i_pos + i_len, j_pos:j_pos + j_len]) for j_pos, j_len in zip(x_idx, x_len)] for i_pos, i_len in zip(y_idx, y_len)]
latent_y_overlap = [o // self.vae_ratio for o in y_overlap]
latent_x_overlap = [o // self.vae_ratio for o in x_overlap]
result_rows = []
for i, row in enumerate(rows):
result_row = []
for j, tile in enumerate(row):
if i > 0:
tile = self.blend(rows[i - 1][j], tile, latent_y_overlap[i - 1], dim=-2)
if j > 0:
tile = self.blend(row[j - 1], tile, latent_x_overlap[j - 1], dim=-1)
if i < len(rows) - 1:
tile = tile[..., :-latent_y_overlap[i], :]
if j < len(row) - 1:
tile = tile[..., :, :-latent_x_overlap[j]]
result_row.append(tile)
result_rows.append(torch.cat(result_row, dim=-1))
return torch.cat(result_rows, dim=-2)
def encode_temporal(self, x: torch.Tensor) -> torch.Tensor:
if x.shape[2] % self.clip_length != 0:
pad_size = (-x.shape[2]) % self.clip_length
x = torch.cat([x, x[:, :, -1:].repeat(1, 1, pad_size, 1, 1)], dim=2)
num_chunks = x.shape[2] // self.clip_length
z_list = [self._adaptive_encode(x[:, :, i * self.clip_length : (i + 1) * self.clip_length, :, :]) for i in range(num_chunks)]
z = torch.cat(z_list, dim=2)
if self.token_drop > 0:
z = z[:, :, :-self.token_drop]
return z
def encode(self, x: torch.Tensor) -> torch.Tensor:
"""``[B,3,H,W]`` or ``[B,3,T,H,W]`` pixels in ``[-1, 1]`` -> normalized latents ``[B,24,T_lat,H//16,W//16]``."""
if x.ndim == 4:
x = x.unsqueeze(2)
# Comfy's VAE wrapper casts before entering the model, so image
# normalization rounds in the VAE compute dtype as well.
x = x.to(self.compute_dtype)
x = x.add(1.0).mul_(0.5).sub_(self.pixel_mean.to(x)).div_(self.pixel_std.to(x))
if x.shape[2] == 1:
moments = self._adaptive_encode(x)
moments = moments[:, :, -1:, :, :]
else:
moments = self.encode_temporal(x)
mean = torch.chunk(moments.float(), 2, dim=1)[0]
latents_mean = self.latents_mean.view(1, -1, 1, 1, 1).to(mean)
latents_std = self.latents_std.view(1, -1, 1, 1, 1).to(mean)
return (mean - latents_mean) / latents_std

262
tests/test_distributed.py
View file

@ -1,262 +0,0 @@
import tempfile
from pathlib import Path
import unittest
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn import functional as F
from h3_blackwell_runtime.distributed import (
SequenceParallelContext,
balanced_ranges,
localize_segments,
range_lengths,
)
from h3_blackwell_runtime.final import H3FinalLayer
from h3_blackwell_runtime.nvfp4 import Nvfp4Linear, Nvfp4LinearTensors
from h3_blackwell_runtime.tensor_parallel import (
aligned_balanced_ranges,
select_nvfp4_outputs,
slice_nvfp4_inputs,
)
def _init_gloo(rank: int, world_size: int, init_file: str) -> None:
dist.init_process_group(
"gloo",
init_method=f"file://{init_file}",
rank=rank,
world_size=world_size,
)
def _transport_identity_worker(rank: int, world_size: int, init_file: str) -> None:
_init_gloo(rank, world_size, init_file)
try:
sequence, heads, head_dim = 17, 56, 3
context = SequenceParallelContext.create(sequence, heads, head_dim)
start, stop = context.local_token_range
full = torch.arange(sequence * heads * head_dim, dtype=torch.float32).reshape(1, sequence, heads, head_dim)
q = full[:, start:stop].contiguous()
k = q + 1_000_000
v = q + 2_000_000
full_q, full_k, full_v = context.seq_to_heads(q, k, v)
torch.testing.assert_close(context.heads_to_seq(full_q), q[0], rtol=0, atol=0)
torch.testing.assert_close(context.heads_to_seq(full_k), k[0], rtol=0, atol=0)
torch.testing.assert_close(context.heads_to_seq(full_v), v[0], rtol=0, atol=0)
finally:
dist.destroy_process_group()
def _attention_parity_worker(rank: int, world_size: int, init_file: str) -> None:
_init_gloo(rank, world_size, init_file)
try:
torch.manual_seed(440420)
sequence, heads, head_dim = 19, 56, 8
context = SequenceParallelContext.create(sequence, heads, head_dim)
q = torch.randn(1, sequence, heads, head_dim)
k = torch.randn_like(q)
v = torch.randn_like(q)
start, stop = context.local_token_range
local_q, local_k, local_v = context.seq_to_heads(
q[:, start:stop].contiguous(),
k[:, start:stop].contiguous(),
v[:, start:stop].contiguous(),
)
local_heads = F.scaled_dot_product_attention(
local_q.transpose(1, 2),
local_k.transpose(1, 2),
local_v.transpose(1, 2),
).transpose(1, 2)
actual = context.heads_to_seq(local_heads)
expected = F.scaled_dot_product_attention(
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
).transpose(1, 2)[0, start:stop]
torch.testing.assert_close(actual, expected, rtol=1e-5, atol=1e-6)
finally:
dist.destroy_process_group()
def _final_projection_worker(rank: int, world_size: int, init_file: str) -> None:
_init_gloo(rank, world_size, init_file)
try:
sequence, hidden_size = 9, 4
context = SequenceParallelContext.create(sequence, heads=4, head_dim=2)
hidden = torch.arange(sequence * hidden_size, dtype=torch.float32).reshape(sequence, hidden_size) / 10
timesteps = torch.tensor([0.25, 0.75])
layer = H3FinalLayer(
torch.zeros(1025, 2),
torch.ones(hidden_size),
torch.zeros(2 * hidden_size, 2),
torch.zeros(2 * hidden_size),
torch.arange(3 * hidden_size, dtype=torch.float32).reshape(3, hidden_size) / 10,
torch.tensor([0.1, 0.2, 0.3]),
torch.arange(2 * hidden_size, dtype=torch.float32).reshape(2, hidden_size) / 20,
torch.tensor([-0.1, 0.1]),
hidden_size=hidden_size,
)
video_segment = (3, 9, 0)
audio_segment = (0, 3, 1)
expected_video, expected_audio = layer(hidden, timesteps, video_segment, audio_segment)
start, stop = context.local_token_range
actual_video, actual_audio = layer.forward_sequence_parallel(
hidden[start:stop], timesteps, video_segment, audio_segment, context,
)
torch.testing.assert_close(actual_video, expected_video)
torch.testing.assert_close(actual_audio, expected_audio)
finally:
dist.destroy_process_group()
def _ragged_row_collectives_worker(rank: int, world_size: int, init_file: str) -> None:
_init_gloo(rank, world_size, init_file)
try:
sequence, features = 17, 5
context = SequenceParallelContext.create(sequence, heads=56, head_dim=2)
full = torch.arange(sequence * features, dtype=torch.float32).reshape(sequence, features)
start, stop = context.local_token_range
gathered = context.all_gather_rows(full[start:stop].contiguous())
torch.testing.assert_close(gathered, full, rtol=0, atol=0)
partial = full * float(rank + 1)
reduced = context.reduce_scatter_rows(partial)
expected = full[start:stop] * sum(range(1, world_size + 1))
torch.testing.assert_close(reduced, expected, rtol=0, atol=0)
finally:
dist.destroy_process_group()
def _tensor_parallel_math_worker(rank: int, world_size: int, init_file: str) -> None:
_init_gloo(rank, world_size, init_file)
try:
torch.manual_seed(440421)
sequence, hidden, heads, head_dim, intermediate = 17, 16, 14, 4, 224
context = SequenceParallelContext.create(sequence, heads=heads, head_dim=head_dim)
full_x = torch.randn(sequence, hidden)
start, stop = context.local_token_range
gathered_x = context.all_gather_rows(full_x[start:stop].contiguous())
inner = heads * head_dim
qkv_weight = torch.randn(3 * inner, hidden)
output_weight = torch.randn(hidden, inner)
output_bias = torch.randn(hidden)
head_start, head_stop = context.local_head_range
feature_start, feature_stop = head_start * head_dim, head_stop * head_dim
indices = torch.cat((
torch.arange(feature_start, feature_stop),
torch.arange(inner + feature_start, inner + feature_stop),
torch.arange(2 * inner + feature_start, 2 * inner + feature_stop),
))
local_qkv = F.linear(gathered_x, qkv_weight.index_select(0, indices))
local_inner = context.local_head_count * head_dim
local_q, local_k, local_v = local_qkv.split(local_inner, dim=-1)
local_q = local_q.view(1, sequence, context.local_head_count, head_dim)
local_k = local_k.view_as(local_q)
local_v = local_v.view_as(local_q)
local_attention = F.scaled_dot_product_attention(
local_q.transpose(1, 2), local_k.transpose(1, 2), local_v.transpose(1, 2),
).transpose(1, 2).reshape(sequence, local_inner)
partial_attention = F.linear(
local_attention, output_weight[:, feature_start:feature_stop],
)
actual_attention = context.reduce_scatter_rows(partial_attention) + output_bias
full_q, full_k, full_v = F.linear(full_x, qkv_weight).split(inner, dim=-1)
full_q = full_q.view(1, sequence, heads, head_dim)
full_k = full_k.view_as(full_q)
full_v = full_v.view_as(full_q)
expected_attention = F.linear(
F.scaled_dot_product_attention(
full_q.transpose(1, 2), full_k.transpose(1, 2), full_v.transpose(1, 2),
).transpose(1, 2).reshape(sequence, inner),
output_weight,
output_bias,
)[start:stop]
torch.testing.assert_close(actual_attention, expected_attention, rtol=2e-5, atol=2e-5)
fc1_weight = torch.randn(2 * intermediate, hidden)
fc2_weight = torch.randn(hidden, intermediate)
fc2_bias = torch.randn(hidden)
mlp_ranges = aligned_balanced_ranges(intermediate, world_size, 32)
mlp_start, mlp_stop = mlp_ranges[rank]
local_fc1_weight = torch.cat((
fc1_weight[mlp_start:mlp_stop],
fc1_weight[intermediate + mlp_start:intermediate + mlp_stop],
))
gate, up = F.linear(gathered_x, local_fc1_weight).chunk(2, dim=-1)
partial_mlp = F.linear(F.silu(gate) * up, fc2_weight[:, mlp_start:mlp_stop])
actual_mlp = context.reduce_scatter_rows(partial_mlp) + fc2_bias
full_gate, full_up = F.linear(full_x, fc1_weight).chunk(2, dim=-1)
expected_mlp = F.linear(F.silu(full_gate) * full_up, fc2_weight, fc2_bias)[start:stop]
# TP reduction changes FP32 accumulation order across rank partials.
torch.testing.assert_close(actual_mlp, expected_mlp, rtol=1e-4, atol=2e-4)
finally:
dist.destroy_process_group()
def _run_distributed(worker, world_size: int) -> None:
with tempfile.TemporaryDirectory() as directory:
init_file = str(Path(directory) / "process-group")
mp.spawn(worker, args=(world_size, init_file), nprocs=world_size, join=True)
class DistributedPartitionContracts(unittest.TestCase):
def test_balanced_ragged_ranges(self):
ranges = balanced_ranges(56, 6)
self.assertEqual(range_lengths(ranges), (10, 10, 9, 9, 9, 9))
self.assertEqual(ranges[0], (0, 10))
self.assertEqual(ranges[-1], (47, 56))
def test_ranges_reject_empty_partitions(self):
with self.assertRaisesRegex(ValueError, "non-empty"):
balanced_ranges(3, 4)
def test_segments_are_clipped_and_rebased(self):
segments = [(0, 4, 1), (4, 10, 2), (10, 15, 3)]
self.assertEqual(localize_segments(segments, 3, 12), [(0, 1, 1), (1, 7, 2), (7, 9, 3)])
def test_transport_identity_for_planned_world_sizes(self):
for world_size in (2, 4, 6, 8):
with self.subTest(world_size=world_size):
_run_distributed(_transport_identity_worker, world_size)
def test_two_rank_sdpa_matches_single_process(self):
_run_distributed(_attention_parity_worker, 2)
def test_distributed_final_projection_matches_single_process(self):
_run_distributed(_final_projection_worker, 2)
def test_ragged_all_gather_and_reduce_scatter(self):
_run_distributed(_ragged_row_collectives_worker, 6)
def test_tensor_parallel_attention_and_mlp_match_dense_math(self):
for world_size in (2, 6):
with self.subTest(world_size=world_size):
_run_distributed(_tensor_parallel_math_worker, world_size)
def test_nvfp4_column_and_row_shards_preserve_layout(self):
tensors = Nvfp4LinearTensors(
weight=torch.arange(96 * 32, dtype=torch.int32).to(torch.uint8).reshape(96, 32),
weight_scale=torch.arange(96 * 4, dtype=torch.float32).to(torch.float8_e4m3fn).reshape(96, 4),
weight_scale_2=torch.tensor(0.5),
bias=torch.arange(96, dtype=torch.bfloat16),
pre_quant_scale=torch.arange(64, dtype=torch.bfloat16),
full_precision_matrix_mult=False,
in_features=64,
out_features=96,
)
linear = Nvfp4Linear(tensors)
column = select_nvfp4_outputs(linear, ((0, 16), (32, 48)))
self.assertEqual(tuple(column.weight.shape), (32, 32))
self.assertEqual(tuple(column.weight_scale.shape), (32, 4))
self.assertEqual(column.out_features, 32)
row, bias = slice_nvfp4_inputs(linear, 32, 64)
self.assertEqual(tuple(row.weight.shape), (96, 16))
self.assertEqual(tuple(row.weight_scale.shape), (96, 2))
self.assertEqual(row.in_features, 32)
self.assertIsNone(row.bias)
torch.testing.assert_close(bias, tensors.bias)
if __name__ == "__main__":
unittest.main()

270
tests/test_fl2va_contracts.py
View file

@ -1,270 +0,0 @@
import math
import sys
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import torch
from torch import nn
from torch.nn import functional as F
from h3_blackwell_runtime.packing import FRAME_RESCALE, H3PromptPacker, _video_t_spans
from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, run_attention
from h3_blackwell_runtime.qwen3vl_vision import (
TEXT_HEAD_DIM,
TEXT_ROPE_DIMS,
TEXT_ROPE_THETA,
VISION_HIDDEN,
Qwen3VL32BVision,
_VisionAttention,
_VisionPatchMerger,
_apply_rope_vision,
_text_run_ids,
mrope_freqs_cis,
)
from h3_blackwell_runtime.runtime import H3HotRuntime
from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder, _downsample
class Fl2vaVAEContracts(unittest.TestCase):
def test_quant_conv_is_a_required_checkpoint_weight(self):
names = MiniMaxH3VideoVAEEncoder()._required_encoder_names()
self.assertIn("quant_conv.weight", names)
self.assertIn("quant_conv.bias", names)
def test_single_frame_is_encoded_without_temporal_prepad(self):
encoder = MiniMaxH3VideoVAEEncoder(tiling=False)
seen = []
def fake_encode(x):
seen.append(tuple(x.shape))
return torch.zeros((x.shape[0], 48, x.shape[2], 1, 1), device=x.device)
encoder._adaptive_encode = fake_encode
result = encoder.encode(torch.zeros(1, 3, 8, 8))
self.assertEqual(seen, [(1, 3, 1, 8, 8)])
self.assertEqual(tuple(result.shape), (1, 24, 1, 1, 1))
def test_downsample_pads_only_right_and_bottom(self):
x = torch.arange(16, dtype=torch.float32).reshape(1, 1, 1, 4, 4)
weight = torch.ones(1, 1, 3, 3, 3)
params = {"w": weight, "b": torch.zeros(1), "time": 1, "space": 2}
actual = _downsample(x, params)
padded = F.pad(x, (0, 1, 0, 1, 0, 0), mode="reflect")
expected = F.conv3d(padded, weight[:, :, -1:], params["b"], stride=(1, 2, 2))
torch.testing.assert_close(actual, expected)
def test_hot_runtime_preserves_zero_to_one_images(self):
runtime = H3HotRuntime.__new__(H3HotRuntime)
image = torch.tensor([[[[0.0, 0.5, 1.0]]]])
converted = runtime._image_to_uint8_nhwc(image)
self.assertEqual(converted.flatten().tolist(), [0, 127, 255])
class AttentionBackendContracts(unittest.TestCase):
def test_hot_backends_include_benchmark_candidates(self):
self.assertTrue({"sage2", "cudnn_sdpa", "ck_int8", "flash4"}.issubset(AVAILABLE_BACKENDS))
def test_flash4_dispatches_bshd_and_restores_hnd(self):
q = torch.randn(1, 2, 3, 128, dtype=torch.bfloat16)
expected_bshd = torch.randn(1, 3, 2, 128, dtype=torch.bfloat16)
flash = unittest.mock.MagicMock(return_value=(expected_bshd, torch.empty(0)))
package = SimpleNamespace(cute=SimpleNamespace(flash_attn_func=flash))
with (
patch.dict(sys.modules, {"flash_attn": package, "flash_attn.cute": package.cute}),
patch.object(torch.Tensor, "is_cuda", new_callable=unittest.mock.PropertyMock, return_value=True),
):
actual = run_attention(q, q, q, backend="flash4", is_causal=False)
torch.testing.assert_close(actual, expected_bshd.transpose(1, 2))
flash.assert_called_once()
called_q, called_k, called_v = flash.call_args.args
self.assertEqual(called_q.shape, (1, 3, 2, 128))
self.assertTrue(called_q.is_contiguous())
self.assertTrue(called_k.is_contiguous())
self.assertTrue(called_v.is_contiguous())
self.assertFalse(flash.call_args.kwargs["causal"])
def test_cudnn_backend_is_forced_without_fallback(self):
q = torch.randn(1, 2, 3, 4)
expected = torch.randn_like(q)
context = unittest.mock.MagicMock()
with (
patch("torch.nn.attention.sdpa_kernel", return_value=context) as kernel,
patch("h3_blackwell_runtime.attention.functional.scaled_dot_product_attention", return_value=expected) as sdpa,
):
actual = run_attention(q, q, q, backend="cudnn_sdpa", is_causal=False)
self.assertIs(actual, expected)
self.assertEqual(kernel.call_args.args[0], [torch.nn.attention.SDPBackend.CUDNN_ATTENTION])
sdpa.assert_called_once_with(q, q, q, is_causal=False)
def test_comfy_kitchen_int8_backend_dispatches_hnd_tensors(self):
q = torch.randn(1, 2, 3, 4)
expected = torch.randn_like(q)
kitchen = SimpleNamespace(int8_attention=unittest.mock.MagicMock(return_value=expected))
with patch.dict(sys.modules, {"comfy_kitchen": kitchen}):
actual = run_attention(q, q, q, backend="ck_int8", is_causal=False)
self.assertIs(actual, expected)
kitchen.int8_attention.assert_called_once_with(q, q, q)
def test_comfy_kitchen_int8_rejects_causal_attention(self):
q = torch.randn(1, 2, 3, 4)
with self.assertRaisesRegex(ValueError, "does not support causal"):
run_attention(q, q, q, backend="ck_int8", is_causal=True)
class Fl2vaVisionContracts(unittest.TestCase):
def test_visual_rotary_coordinates_are_block_major(self):
class CoordinateTable(nn.Module):
def forward(self, length):
return torch.arange(length, dtype=torch.float32).unsqueeze(1)
vision = Qwen3VL32BVision.__new__(Qwen3VL32BVision)
nn.Module.__init__(vision)
vision.spatial_merge_size = 2
vision.rotary_pos_emb = CoordinateTable()
coordinates = vision.rot_pos_emb(torch.tensor([[1, 4, 4]])).tolist()
self.assertEqual(coordinates[:8], [
[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0],
[0.0, 2.0], [0.0, 3.0], [1.0, 2.0], [1.0, 3.0],
])
def test_sdpa_output_is_restored_to_token_major_layout(self):
torch.manual_seed(7)
sequence, heads, head_dim = 3, 2, 2
hidden = heads * head_dim
qkv_weight = torch.randn(hidden * 3, hidden)
qkv_bias = torch.randn(hidden * 3)
proj_weight = torch.randn(hidden, hidden)
proj_bias = torch.randn(hidden)
module = _VisionAttention(qkv_weight, qkv_bias, proj_weight, proj_bias, num_heads=heads, head_dim=head_dim)
x = torch.randn(sequence, hidden)
cos = torch.ones(sequence, 1, head_dim)
sin = torch.zeros(sequence, 1, head_dim // 2)
actual = module(x, torch.tensor([0, sequence], dtype=torch.int32), (cos, sin, sin))
qkv = F.linear(x, qkv_weight, qkv_bias)
query, key, value = qkv.reshape(sequence, 3, heads, head_dim).permute(1, 0, 2, 3).unbind(0)
output = F.scaled_dot_product_attention(
query.transpose(0, 1).unsqueeze(0),
key.transpose(0, 1).unsqueeze(0),
value.transpose(0, 1).unsqueeze(0),
)
expected = F.linear(output.transpose(1, 2).reshape(sequence, hidden), proj_weight, proj_bias)
torch.testing.assert_close(actual, expected)
def test_vision_rope_uses_original_halves(self):
q = torch.tensor([[[1.0, 2.0, 3.0, 4.0]]])
k = q + 4
cos = torch.full_like(q, 0.5)
sin = torch.full_like(q[..., :2], 0.25)
neg_sin = -sin
actual_q, actual_k = _apply_rope_vision(q, k, (cos, sin, neg_sin))
def expected(x):
return torch.cat((x[..., :2] * 0.5 + x[..., 2:] * -0.25,
x[..., 2:] * 0.5 + x[..., :2] * 0.25), dim=-1)
torch.testing.assert_close(actual_q, expected(q))
torch.testing.assert_close(actual_k, expected(k))
def test_mrope_uses_reference_section_boundaries(self):
positions = torch.stack((torch.arange(8), torch.arange(8) + 10, torch.arange(8) + 20))
actual = mrope_freqs_cis(positions)[0]
inv_freq = 1.0 / (
TEXT_ROPE_THETA ** (torch.arange(0, TEXT_HEAD_DIM, 2, dtype=torch.float32) / TEXT_HEAD_DIM)
)
freqs = (inv_freq[None, :, None].expand(3, -1, 1) @ positions[:, None, :].float()).transpose(1, 2)
interleaved = freqs[0].clone()
for axis, offset in ((1, 1), (2, 2)):
index = slice(offset, TEXT_ROPE_DIMS[axis] * 3, 3)
interleaved[..., index] = freqs[axis, ..., index]
expected = torch.cat((interleaved, interleaved), dim=-1).cos().unsqueeze(0)
torch.testing.assert_close(actual, expected)
def test_mergers_preserve_existing_block_major_order(self):
x = torch.arange(4 * VISION_HIDDEN, dtype=torch.float32).reshape(4, VISION_HIDDEN)
passthrough = lambda value, *args, **kwargs: value
with (
patch("h3_blackwell_runtime.qwen3vl_vision.F.layer_norm", side_effect=passthrough),
patch("h3_blackwell_runtime.qwen3vl_vision.F.linear", side_effect=passthrough),
patch("h3_blackwell_runtime.qwen3vl_vision.F.gelu", side_effect=passthrough),
):
main = _VisionPatchMerger(*(torch.empty(1) for _ in range(6)), merge_size=2,
out_hidden_size=1, norm_dim=VISION_HIDDEN)
deepstack = _VisionPatchMerger(*(torch.empty(1) for _ in range(6)), merge_size=2,
out_hidden_size=1)
torch.testing.assert_close(main(x), x.reshape(1, -1))
torch.testing.assert_close(deepstack(x), x.reshape(1, -1))
def test_configured_tokenizer_preserves_empty_prompt(self):
calls = []
class RawTokenizer:
def __call__(self, text, **kwargs):
calls.append((text, kwargs))
return SimpleNamespace(input_ids=torch.empty((1, 0), dtype=torch.long))
self.assertEqual(_text_run_ids(SimpleNamespace(tokenizer=RawTokenizer()), ""), [])
self.assertEqual(calls[0][0], "")
self.assertFalse(calls[0][1]["add_special_tokens"])
class _FakeCheckpoint:
def tensor(self, name, dtype=None):
if name == "video_patch_proj.weight":
value = torch.zeros(5376, 96)
elif name == "video_patch_proj.bias":
value = torch.zeros(5376)
elif name == "audio_patch_proj.weight":
value = torch.zeros(5376, 32)
elif name == "audio_patch_proj.bias":
value = torch.zeros(5376)
else:
value = torch.empty(0)
return value.to(dtype=dtype) if dtype is not None else value
class Fl2vaPackingContracts(unittest.TestCase):
def test_each_keyframe_keeps_its_own_condition_segment(self):
packer = H3PromptPacker(_FakeCheckpoint())
text = torch.zeros(1, 3, 5376)
video = torch.zeros(1, 24, 2, 2, 2)
audio = torch.zeros(1, 32, 2, 2)
keyframes = [torch.zeros(1, 24, 1, 2, 2) for _ in range(2)]
_, _, segments, _, _, _ = packer(
text,
video,
audio,
0.5,
cond_latents=keyframes,
cond_frame_indices=[0, 21],
frame_count=22,
)
self.assertEqual(segments[1][:2], (3, 4))
self.assertEqual(segments[2][:2], (4, 5))
self.assertEqual(segments[1][2], segments[2][2])
def test_last_only_anchor_and_targets_share_reference_cursor(self):
packer = H3PromptPacker(_FakeCheckpoint())
text = torch.zeros(1, 3, 5376)
video = torch.zeros(1, 24, 2, 2, 2)
audio = torch.zeros(1, 32, 2, 2)
last = torch.zeros(1, 24, 1, 2, 2)
_, _, _, positions, _, _ = packer(
text,
video,
audio,
0.5,
cond_latents=[last],
cond_frame_indices=[21],
frame_count=22,
seed=1,
)
expected_last_t = 3.0 + sum(_video_t_spans(2)) - FRAME_RESCALE
self.assertTrue(math.isclose(float(positions[3, 0]), expected_last_t))
self.assertEqual(float(positions[4, 0]), 3.0) # target audio
self.assertEqual(float(positions[8, 0]), 3.0) # target video
if __name__ == "__main__":
unittest.main()

52
tests/test_latent_upscaler.py
View file

@ -1,52 +0,0 @@
import unittest
import torch
from h3_blackwell_runtime.latent_upscaler import H3LatentResizer3D, _checkpoint_config, upscale_h3_latent
from h3_blackwell_runtime.runtime import normalize_upscale
class LatentUpscalerContracts(unittest.TestCase):
def test_request_upscale_normalization(self):
self.assertIsNone(normalize_upscale(None))
self.assertIsNone(normalize_upscale("none"))
self.assertIsNone(normalize_upscale(1))
self.assertEqual(normalize_upscale("2"), 2.0)
with self.assertRaisesRegex(ValueError, "greater than 1.0"):
normalize_upscale(0.5)
with self.assertRaisesRegex(ValueError, "at most 4.0"):
normalize_upscale(4.1)
with self.assertRaisesRegex(ValueError, "must be a number"):
normalize_upscale({"scale": 2})
def test_3d_model_preserves_time_and_scales_space(self):
model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, dropout=0, temporal_every=0).eval()
latent = torch.randn(1, 24, 3, 2, 4)
output = upscale_h3_latent(model, latent, scale=2)
self.assertEqual(output.shape, (1, 24, 3, 4, 8))
def test_invalid_latent_shape_is_rejected(self):
model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, temporal_every=0).eval()
with self.assertRaisesRegex(ValueError, r"\[B,24,T,H,W\]"):
upscale_h3_latent(model, torch.randn(1, 16, 3, 2, 4))
def test_scale_one_preserves_latent_exactly(self):
model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, temporal_every=0).eval()
latent = torch.randn(1, 24, 3, 2, 4)
self.assertTrue(torch.equal(upscale_h3_latent(model, latent, scale=1), latent))
def test_checkpoint_architecture_detection_matches_module_layout(self):
model = H3LatentResizer3D(in_blocks=3, out_blocks=2, channels=32, temporal_every=2, temporal_kernel=3)
config = _checkpoint_config(model.state_dict())
self.assertEqual(config, {
"in_channels": 24,
"in_blocks": 3,
"out_blocks": 2,
"channels": 32,
"temporal_every": 2,
"temporal_kernel": 3,
})
if __name__ == "__main__":
unittest.main()

120
tests/test_turbo.py
View file

@ -1,120 +0,0 @@
import unittest
from unittest.mock import patch
import torch
from torch import nn
from h3_blackwell_runtime.lora import DynamicLoraMixin
from h3_blackwell_runtime.sampler import sample_video_res_multistep, sample_video_turbo, turbo_sigmas
class _Linear(DynamicLoraMixin, nn.Module):
def __init__(self, weight):
super().__init__()
self.register_buffer("weight", weight)
self.in_features = weight.shape[1]
self.out_features = weight.shape[0]
self._init_dynamic_lora()
def forward(self, x):
return self._apply_lora(x, torch.nn.functional.linear(x, self.weight))
class TurboLoraContracts(unittest.TestCase):
def test_dynamic_lora_matches_unmerged_formula(self):
base_weight = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
down = torch.tensor([[2.0, -1.0]])
up = torch.tensor([[3.0], [-2.0]])
x = torch.tensor([[0.5, -1.0]])
linear = _Linear(base_weight)
linear.add_lora("turbo", down, up, alpha=0.5)
linear.set_lora("turbo", strength=0.75)
expected = torch.nn.functional.linear(x, base_weight)
expected += 0.75 * 0.5 * torch.nn.functional.linear(torch.nn.functional.linear(x, down), up)
torch.testing.assert_close(linear(x), expected)
def test_disabled_lora_returns_exact_base_result(self):
weight = torch.randn(3, 2)
x = torch.randn(4, 2)
linear = _Linear(weight)
linear.add_lora("turbo", torch.randn(1, 2), torch.randn(3, 1), alpha=1.0)
expected = torch.nn.functional.linear(x, weight)
self.assertTrue(torch.equal(linear(x), expected))
linear.set_lora("turbo", strength=0.0)
self.assertTrue(torch.equal(linear(x), expected))
def test_unknown_lora_is_rejected(self):
linear = _Linear(torch.randn(3, 2))
with self.assertRaisesRegex(ValueError, "not attached"):
linear.set_lora("missing")
def test_four_step_shift_six_schedule(self):
actual = turbo_sigmas(4, 6.0, device="cpu")
expected = torch.tensor([1.0, 0.9473684211, 0.8571428571, 0.6666666667, 0.0])
torch.testing.assert_close(actual, expected)
def test_eight_step_shift_twelve_schedule_has_nine_grid_points(self):
actual = turbo_sigmas(8, 12.0, device="cpu")
self.assertEqual(actual.shape, (9,))
self.assertEqual(actual[0].item(), 1.0)
self.assertEqual(actual[-1].item(), 0.0)
self.assertTrue(bool(torch.all(actual[:-1] > actual[1:])))
def test_turbo_sampler_uses_positive_dataward_updates_and_independent_clocks(self):
video = torch.zeros(1, 1, 1, 1, 1)
audio = torch.zeros(1, 1, 1, 1)
calls = []
def packer(text, current_video, current_audio, sigma, model_timesteps, **kwargs):
calls.append((current_video.clone(), current_audio.clone(), model_timesteps.clone()))
return (None, None, None, None, None, None)
def model(*args):
return torch.ones(1), torch.ones(1)
with (
patch("h3_blackwell_runtime.sampler.unpatchify_video", return_value=torch.full_like(video, 2.0)),
patch("h3_blackwell_runtime.sampler._unpack_audio", return_value=torch.full_like(audio, 3.0)),
):
result_video, result_audio = sample_video_turbo(
model, packer, torch.empty(0), video, audio,
steps=2, video_shift=6.0, audio_shift=3.0, return_audio=True,
)
torch.testing.assert_close(result_video, torch.full_like(video, 2.0))
torch.testing.assert_close(result_audio, torch.full_like(audio, 3.0))
torch.testing.assert_close(calls[1][0], torch.full_like(video, 2.0 / 7.0))
torch.testing.assert_close(calls[1][1], torch.full_like(audio, 0.75))
torch.testing.assert_close(calls[1][2], torch.tensor([1.0 / 7.0, 0.25]))
def test_base_sampler_records_opt_in_audio_step_trace(self):
video = torch.zeros(1, 1, 1, 1, 1)
audio = torch.zeros(1, 32, 2, 1)
trace = []
def packer(*args, **kwargs):
return (None, None, None, None, None, None)
def model(*args):
return torch.ones(1), torch.ones(1)
with (
patch("h3_blackwell_runtime.sampler.beta_sigmas", return_value=torch.tensor([1.0, 0.0])),
patch("h3_blackwell_runtime.sampler.unpatchify_video", return_value=torch.zeros_like(video)),
patch("h3_blackwell_runtime.sampler._unpack_audio", return_value=torch.ones_like(audio)),
):
_video, final_audio = sample_video_res_multistep(
model, packer, torch.empty(0), video, audio,
steps=1, return_audio=True, audio_step_trace=trace,
)
self.assertEqual(len(trace), 1)
self.assertEqual(trace[0]["step"], 1)
torch.testing.assert_close(trace[0]["audio_before"], torch.zeros_like(audio))
torch.testing.assert_close(trace[0]["audio_denoised"], torch.full_like(audio, 4.0))
torch.testing.assert_close(trace[0]["audio_after"], torch.full_like(audio, 4.0))
torch.testing.assert_close(final_audio, torch.ones_like(audio))
if __name__ == "__main__":
unittest.main()

123
tools/analyze_audio_boundary.py
View file

@ -1,123 +0,0 @@
"""Compare H3 audio-latent and lossless-waveform boundaries."""
import argparse
import json
import math
import wave
from pathlib import Path
import numpy as np
import torch
def dbfs(value: float) -> float:
return 20.0 * math.log10(max(value, 1e-20))
def load_audio_latent(path: Path) -> torch.Tensor:
state = torch.load(path, map_location="cpu", weights_only=False)
latent = state.get("audio_latent") if isinstance(state, dict) else state
if latent is None or latent.ndim != 4:
raise ValueError(f"{path} does not contain a [B,C,S,T] audio latent")
return latent.float()
def latent_metrics(latent: torch.Tensor) -> dict:
frames = latent.movedim(-1, 0).flatten(1)
frame_rms = frames.square().mean(1).sqrt()
frame_mean = frames.mean(1)
frame_max = frames.abs().amax(1)
deltas = frames[1:] - frames[:-1]
delta_rms = deltas.square().mean(1).sqrt()
adjacent_cosine = torch.nn.functional.cosine_similarity(frames[:-1], frames[1:], dim=1)
block_frames = min(4, frames.shape[0] // 2)
first = frames[:block_frames].flatten()
last = frames[-block_frames:].flatten()
first_last_cosine = torch.nn.functional.cosine_similarity(first, last, dim=0)
first_count = min(20, frames.shape[0])
return {
"shape": list(latent.shape),
"dtype": str(latent.dtype),
"first_20_frame_rms": frame_rms[:first_count].tolist(),
"first_20_frame_mean": frame_mean[:first_count].tolist(),
"first_20_frame_max_abs": frame_max[:first_count].tolist(),
"first_19_delta_rms": delta_rms[: max(0, first_count - 1)].tolist(),
"first_19_adjacent_cosine": adjacent_cosine[: max(0, first_count - 1)].tolist(),
"first_4_rms": float(frames[:block_frames].square().mean().sqrt()),
"frames_4_20_rms": float(frames[block_frames:first_count].square().mean().sqrt()),
"remaining_rms": float(frames[first_count:].square().mean().sqrt()),
"first_4_vs_last_4_cosine": float(first_last_cosine),
"largest_delta_frame": int(delta_rms.argmax().item() + 1),
"largest_delta_rms": float(delta_rms.max()),
}
def load_wav(path: Path) -> tuple[np.ndarray, int]:
with wave.open(str(path), "rb") as source:
if source.getsampwidth() != 2:
raise ValueError(f"{path} must be PCM S16")
channels = source.getnchannels()
sample_rate = source.getframerate()
samples = np.frombuffer(source.readframes(source.getnframes()), dtype="<i2")
return samples.reshape(-1, channels).astype(np.float32) / 32768.0, sample_rate
def waveform_metrics(samples: np.ndarray, sample_rate: int) -> dict:
first_half_second = samples[: sample_rate // 2]
mono = first_half_second.mean(1)
window_samples = sample_rate // 100
windows = []
for start in range(0, len(first_half_second), window_samples):
block = first_half_second[start : start + window_samples]
if len(block) == 0:
continue
windows.append({
"start_ms": start * 1000.0 / sample_rate,
"peak_dbfs": dbfs(float(np.max(np.abs(block)))),
"rms_dbfs": dbfs(float(np.sqrt(np.mean(block * block)))),
"mean": float(block.mean()),
})
spectrum_samples = min(sample_rate // 10, len(mono))
windowed = mono[:spectrum_samples] * np.hanning(spectrum_samples)
magnitudes = np.abs(np.fft.rfft(windowed))
frequencies = np.fft.rfftfreq(spectrum_samples, 1.0 / sample_rate)
dominant = np.argsort(magnitudes[1:])[-8:][::-1] + 1
derivatives = np.max(np.abs(np.diff(first_half_second, axis=0)), axis=1)
return {
"sample_rate": sample_rate,
"samples": len(samples),
"first_sample": samples[0].tolist(),
"first_sample_dbfs": [dbfs(float(abs(value))) for value in samples[0]],
"first_500ms_peak_dbfs": dbfs(float(np.max(np.abs(first_half_second)))),
"first_500ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_half_second**2)))),
"largest_derivative": float(derivatives.max()),
"largest_derivative_ms": float((derivatives.argmax() + 1) * 1000.0 / sample_rate),
"dominant_first_100ms_hz": [float(frequencies[index]) for index in dominant],
"windows_10ms": windows,
}
def analyze(latent_path: Path, wav_path: Path) -> dict:
samples, sample_rate = load_wav(wav_path)
return {
"latent_path": str(latent_path),
"wav_path": str(wav_path),
"latent": latent_metrics(load_audio_latent(latent_path)),
"waveform": waveform_metrics(samples, sample_rate),
}
parser = argparse.ArgumentParser()
parser.add_argument("--affected-latent", type=Path, required=True)
parser.add_argument("--affected-wav", type=Path, required=True)
parser.add_argument("--clean-latent", type=Path, required=True)
parser.add_argument("--clean-wav", type=Path, required=True)
args = parser.parse_args()
print(json.dumps({
"affected": analyze(args.affected_latent, args.affected_wav),
"clean": analyze(args.clean_latent, args.clean_wav),
}, indent=2))

143
tools/benchmark_ulysses.py
View file

@ -1,143 +0,0 @@
"""Benchmark ragged H3 Ulysses transport and attention under torchrun."""
import argparse
import json
import os
import statistics
import time
from pathlib import Path
import torch
import torch.distributed as dist
from h3_blackwell_runtime.attention import run_attention, run_flash4_attention_bshd, run_sol_attention_bshd
from h3_blackwell_runtime.distributed import SequenceParallelContext
def synchronize(device: torch.device) -> None:
if device.type == "cuda":
torch.cuda.synchronize(device)
def run_backend(backend: str, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
if backend == "flash4":
return run_flash4_attention_bshd(q, k, v, is_causal=False)
if backend == "sol_attn":
return run_sol_attention_bshd(q, k, v, is_causal=False)
return run_attention(
q.transpose(1, 2).contiguous(),
k.transpose(1, 2).contiguous(),
v.transpose(1, 2).contiguous(),
backend=backend,
is_causal=False,
).transpose(1, 2)
parser = argparse.ArgumentParser()
parser.add_argument("--sequence", type=int, default=20000)
parser.add_argument("--heads", type=int, default=56)
parser.add_argument("--head-dim", type=int, default=128)
parser.add_argument("--backend", default="sdpa")
parser.add_argument("--warmup", type=int, default=3)
parser.add_argument("--iterations", type=int, default=10)
parser.add_argument("--output", type=Path)
args = parser.parse_args()
local_rank = int(os.environ.get("LOCAL_RANK", "0"))
use_cuda = torch.cuda.is_available()
device = torch.device(f"cuda:{local_rank}" if use_cuda else "cpu")
if use_cuda:
torch.cuda.set_device(device)
dist.init_process_group(backend="nccl" if use_cuda else "gloo", device_id=device if use_cuda else None)
rank = dist.get_rank()
world_size = dist.get_world_size()
context = SequenceParallelContext.create(args.sequence, args.heads, args.head_dim)
generator = torch.Generator(device=device).manual_seed(440420 + rank)
shape = (1, context.local_token_length, args.heads, args.head_dim)
dtype = torch.bfloat16 if use_cuda else torch.float32
q = torch.randn(shape, generator=generator, dtype=dtype, device=device)
k = torch.randn(shape, generator=generator, dtype=dtype, device=device)
v = torch.randn(shape, generator=generator, dtype=dtype, device=device)
def iteration() -> tuple[float, float, float, torch.Tensor]:
dist.barrier()
synchronize(device)
started = time.perf_counter()
full_q, full_k, full_v = context.seq_to_heads(q, k, v)
synchronize(device)
after_forward = time.perf_counter()
head_output = run_backend(args.backend, full_q, full_k, full_v)
synchronize(device)
after_attention = time.perf_counter()
local_output = context.heads_to_seq(head_output)
synchronize(device)
finished = time.perf_counter()
return (
after_forward - started,
after_attention - after_forward,
finished - after_attention,
local_output,
)
for _ in range(args.warmup):
*_timings, output = iteration()
del output
rank_timings = []
for _ in range(args.iterations):
forward, attention, inverse, output = iteration()
rank_timings.append((forward, attention, inverse, forward + attention + inverse))
del output
timings = torch.tensor(rank_timings, dtype=torch.float64, device=device)
gathered = [torch.empty_like(timings) for _ in range(world_size)]
dist.all_gather(gathered, timings)
if use_cuda:
peak_memory = torch.tensor([torch.cuda.max_memory_allocated(device)], dtype=torch.int64, device=device)
else:
peak_memory = torch.tensor([0], dtype=torch.int64, device=device)
memory_by_rank = [torch.empty_like(peak_memory) for _ in range(world_size)]
dist.all_gather(memory_by_rank, peak_memory)
if rank == 0:
stacked = torch.stack(gathered).cpu()
stage_names = ("forward_all_to_all", "attention", "inverse_all_to_all", "total")
stages = {}
for index, name in enumerate(stage_names):
maximum_rank = stacked[:, :, index].amax(dim=0).tolist()
stages[name] = {
"median_seconds": statistics.median(maximum_rank),
"minimum_seconds": min(maximum_rank),
"maximum_seconds": max(maximum_rank),
}
element_size = q.element_size()
report = {
"world_size": world_size,
"backend": args.backend,
"device": torch.cuda.get_device_name(device) if use_cuda else "cpu",
"torch": torch.__version__,
"sequence": args.sequence,
"heads": args.heads,
"head_dim": args.head_dim,
"token_lengths": list(context.token_lengths),
"head_lengths": list(context.head_lengths),
"dtype": str(dtype),
"iterations": args.iterations,
"aggregate_transport_bytes_per_iteration": {
"forward_qkv": 3 * args.sequence * args.heads * args.head_dim * element_size,
"inverse_output": args.sequence * args.heads * args.head_dim * element_size,
},
"stages": stages,
"peak_allocated_bytes_by_rank": [int(value.item()) for value in memory_by_rank],
}
serialized = json.dumps(report, indent=2)
if args.output is not None:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(serialized + "\n", encoding="utf-8")
print(serialized)
dist.destroy_process_group()

64
tools/compare_audio_pcm.py
View file

@ -1,64 +0,0 @@
"""Compare decoded audio streams using aligned float PCM arrays."""
import argparse
import json
import subprocess
from pathlib import Path
import numpy as np
def decode(path: Path) -> np.ndarray:
raw = subprocess.check_output([
"ffmpeg", "-hide_banner", "-loglevel", "error", "-i", str(path),
"-map", "0:a:0", "-f", "f32le", "-acodec", "pcm_f32le", "-",
])
return np.frombuffer(raw, dtype="<f4").reshape(-1, 2)
def metrics(reference: np.ndarray, candidate: np.ndarray) -> dict:
count = min(len(reference), len(candidate))
reference = reference[:count]
candidate = candidate[:count]
error = candidate - reference
signal_power = np.maximum(np.mean(reference**2, axis=0), 1e-30)
noise_power = np.maximum(np.mean(error**2, axis=0), 1e-30)
first_250ms = error[:8000]
first_signal_power = np.maximum(np.mean(reference[:8000] ** 2, axis=0), 1e-30)
first_noise_power = np.maximum(np.mean(first_250ms**2, axis=0), 1e-30)
return {
"reference_samples": len(reference),
"candidate_samples": len(candidate),
"compared_samples": count,
"snr_db_by_channel": (10.0 * np.log10(signal_power / noise_power)).tolist(),
"rmse": float(np.sqrt(np.mean(error**2))),
"first_250ms_snr_db_by_channel": (
10.0 * np.log10(first_signal_power / first_noise_power)
).tolist(),
"first_250ms_rmse": float(np.sqrt(np.mean(first_250ms**2))),
"max_abs_error": float(np.max(np.abs(error))),
}
parser = argparse.ArgumentParser()
parser.add_argument("--reference", type=Path, required=True)
parser.add_argument("--candidate", action="append", default=[], metavar="NAME=PATH")
parser.add_argument("--output", type=Path)
args = parser.parse_args()
reference = decode(args.reference)
report = {"reference": str(args.reference), "candidates": {}}
for value in args.candidate:
if "=" not in value:
raise ValueError(f"candidate must be NAME=PATH, got {value!r}")
name, raw_path = value.split("=", 1)
report["candidates"][name] = {
"path": raw_path,
**metrics(reference, decode(Path(raw_path))),
}
serialized = json.dumps(report, indent=2)
if args.output is not None:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(serialized + "\n", encoding="utf-8")
print(serialized)

128
tools/compare_fl2va_key_sampler.py
View file

@ -1,128 +0,0 @@
"""Replay a matched Comfy keyframe sampler capture with exact static conditioning."""
import argparse
from pathlib import Path
import torch
from h3_blackwell_runtime.checkpoint import H3Checkpoint
from h3_blackwell_runtime.denoiser import H3PackedDenoiser
from h3_blackwell_runtime.packing import H3PromptPacker, unpatchify_video
from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, _unpack_audio, beta_sigmas, res_multistep_update
from h3_blackwell_runtime.t2v import random_av_latents
parser = argparse.ArgumentParser()
parser.add_argument("--sampler", type=Path, required=True)
parser.add_argument("--dit", type=Path, required=True)
parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
parser.add_argument("--result-latent", type=Path)
parser.add_argument("--production-trace", type=Path)
args = parser.parse_args()
checkpoint = H3Checkpoint(args.model)
model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend="sage2").eval()
packer = H3PromptPacker(checkpoint)
captured_input = torch.load(args.dit / "input.pt", map_location="cuda", weights_only=False)
text_length = next(start for start, _, code in captured_input["segments"] if code == 3)
prefix_stop = next(start for start, _, code in captured_input["segments"] if code % 3 == 2)
text = captured_input["hidden"][:text_length].unsqueeze(0).cuda()
prefix = captured_input["hidden"][:prefix_stop].cuda()
text_tags = torch.ones(text_length, dtype=torch.long, device="cuda")
for start, stop, code in captured_input["segments"]:
if stop <= text_length:
text_tags[start:stop] = code % 3
video, audio, frame_count = random_av_latents(384, 384, 22, 440207)
video_shape, audio_shape = video.shape, audio.shape
video_count, audio_count = video.numel(), audio.numel()
zero_cond = [torch.zeros(1, 24, 1, 24, 24, device="cuda") for _ in range(2)]
sigmas = torch.load(args.sampler / "initial.pt", map_location="cuda", weights_only=False)["sigmas"].cuda()
direct_sigmas = beta_sigmas(12, device="cuda")
sigma_delta = (direct_sigmas.float() - sigmas.float()).abs()
print({"stage": "sigmas", "direct": direct_sigmas.tolist(), "comfy": sigmas.tolist(), "mean_delta": float(sigma_delta.mean()), "max_delta": float(sigma_delta.max())}, flush=True)
video_history = audio_history = history_sigma = None
for index, sigma in enumerate(sigmas[:-1]):
reference = torch.load(args.sampler / f"step_{index:02d}.pt", map_location="cuda", weights_only=False)
reference_x = reference["x"].cuda().reshape(-1)
reference_video = reference_x[:video_count].reshape(video_shape)
reference_audio = reference_x[video_count:video_count + audio_count].reshape(audio_shape)
pre_video = (video.float() - reference_video.float()).abs()
pre_audio = (audio.float() - reference_audio.float()).abs()
sigma_audio = _audio_sigma(sigma)
carry = sigma_audio / sigma
hidden, times, segments, positions, video_segment, audio_segment = packer(
text,
video,
audio.to(torch.bfloat16) * carry,
_model_sigma(sigma),
text_token_tags=text_tags,
cond_latents=zero_cond,
cond_frame_indices=[0, frame_count - 1],
frame_count=frame_count,
seed=440207,
)
hidden[:prefix_stop] = prefix.to(hidden)
input_hidden = hidden.detach().clone()
with torch.inference_mode():
raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
raw_video = raw_video.to(torch.bfloat16).float()
raw_audio = raw_audio.to(torch.bfloat16)
video_denoised = video + sigma * unpatchify_video(raw_video, video.shape[2], video.shape[-2], video.shape[-1])
audio_model_output = (
(1.0 - 4.0) * (audio.to(torch.bfloat16) * carry.to(torch.bfloat16))
+ (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(raw_audio))
).float()
audio_denoised = audio - sigma * audio_model_output
reference_denoised = reference["denoised"].cuda().reshape(-1)
reference_video_denoised = reference_denoised[:video_count].reshape(video_shape)
reference_audio_denoised = reference_denoised[video_count:video_count + audio_count].reshape(audio_shape)
denoised_video = (video_denoised.float() - reference_video_denoised.float()).abs()
denoised_audio = (audio_denoised.float() - reference_audio_denoised.float()).abs()
production = torch.load(args.production_trace / f"step_{index:02d}.pt", map_location="cuda", weights_only=False) if args.production_trace else None
production_video = (production["video"].float() - reference_video.float()).abs() if production else None
production_audio = (production["audio"].float() - reference_audio.float()).abs() if production else None
production_denoised = (production["video_denoised"].float() - reference_video_denoised.float()).abs() if production else None
production_hidden = (production["hidden"].float() - input_hidden.float()).abs() if production and "hidden" in production else None
production_raw = (production["raw_video"].float() - raw_video.float()).abs() if production and "raw_video" in production else None
production_segments = [
(start, stop, code, float((production["hidden"][start:stop].float() - input_hidden[start:stop].float()).abs().mean()))
for start, stop, code in segments
] if production and "hidden" in production else None
print({
"step": index,
"pre_video_mean": float(pre_video.mean()),
"pre_video_max": float(pre_video.max()),
"pre_audio_mean": float(pre_audio.mean()),
"pre_audio_max": float(pre_audio.max()),
"denoised_video_mean": float(denoised_video.mean()),
"denoised_video_max": float(denoised_video.max()),
"denoised_audio_mean": float(denoised_audio.mean()),
"denoised_audio_max": float(denoised_audio.max()),
"production_video_mean": float(production_video.mean()) if production_video is not None else None,
"production_video_max": float(production_video.max()) if production_video is not None else None,
"production_audio_mean": float(production_audio.mean()) if production_audio is not None else None,
"production_audio_max": float(production_audio.max()) if production_audio is not None else None,
"production_denoised_mean": float(production_denoised.mean()) if production_denoised is not None else None,
"production_denoised_max": float(production_denoised.max()) if production_denoised is not None else None,
"production_hidden_mean": float(production_hidden.mean()) if production_hidden is not None else None,
"production_hidden_max": float(production_hidden.max()) if production_hidden is not None else None,
"production_raw_mean": float(production_raw.mean()) if production_raw is not None else None,
"production_raw_max": float(production_raw.max()) if production_raw is not None else None,
"production_segments": production_segments,
}, flush=True)
previous_sigma = sigmas[index - 1] if index else None
sigma_down = sigmas[index + 1]
video = res_multistep_update(video, video_denoised, sigma, sigma_down, video_history, history_sigma, previous_sigma)
audio = res_multistep_update(audio, audio_denoised, sigma, sigma_down, audio_history, history_sigma, previous_sigma)
video_history, audio_history, history_sigma = video_denoised, audio_denoised, sigma_down
if args.result_latent:
result = torch.load(args.result_latent, map_location="cuda", weights_only=False)
result_video = result["latent"] if isinstance(result, dict) else result
delta = (video.float() - result_video.cuda().float()).abs()
print({"stage": "final_video", "mean_delta": float(delta.mean()), "max_delta": float(delta.max())}, flush=True)

193
tools/compare_fl2va_keyframe_input.py
View file

@ -1,193 +0,0 @@
"""Compare direct keyframe FL2VA packing with a matched Comfy DiT capture."""
import argparse
from pathlib import Path
import numpy as np
from PIL import Image
import torch
from h3_blackwell_runtime.checkpoint import H3Checkpoint
from h3_blackwell_runtime.denoiser import H3PackedDenoiser
from h3_blackwell_runtime.packing import H3PromptPacker
from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
from h3_blackwell_runtime.qwen3vl_vision import Qwen3VL32BVision, build_fl2va_presentation, resize_keyframe
from h3_blackwell_runtime.qwen3vl_vision import mrope_freqs_cis, mrope_position_ids
from h3_blackwell_runtime.sampler import _model_sigma, beta_sigmas
from h3_blackwell_runtime.t2v import random_av_latents
from h3_blackwell_runtime.token_refiner import H3TokenRefiner
from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder
parser = argparse.ArgumentParser()
parser.add_argument("--capture", required=True)
parser.add_argument("--first", required=True)
parser.add_argument("--last", required=True)
parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
parser.add_argument("--qwen", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
parser.add_argument("--vae", default="/vae/minimax_h3_video_vae_fp16.safetensors")
parser.add_argument("--tokenizer", default="/opt/h3-blackwell-runtime/src/h3_blackwell_runtime/qwen25_tokenizer")
parser.add_argument("--reference-vision-first")
parser.add_argument("--reference-vision-last")
parser.add_argument("--qwen-capture-dir")
parser.add_argument("--sampler-capture-dir")
parser.add_argument("--oracle-qwen-input", action="store_true")
parser.add_argument("--vae-dtype", choices=("float16", "bfloat16", "float32"), default="float32")
parser.add_argument("--vae-no-tiling", action="store_true")
parser.add_argument("--load-dit-first", action="store_true")
parser.add_argument("--preview-order", action="store_true")
args = parser.parse_args()
prompt = "A studio time-lapse of the same pink peony bud opening into the same fully bloomed pink peony, fixed camera, cream background."
width = height = 384
requested_frames = 22
seed = 440207
def load_image(path):
image = Image.open(path).convert("RGB")
return torch.from_numpy(np.asarray(image).copy()).unsqueeze(0).cuda().float().div(255.0)
def report(name, actual, expected):
expected = expected.to(actual.device)
delta = (actual.float() - expected.float()).abs()
print({
"stage": name,
"shape": tuple(actual.shape),
"mean_delta": float(delta.mean()),
"max_delta": float(delta.max()),
}, flush=True)
capture = torch.load(args.capture, map_location="cuda", weights_only=False)
checkpoint = H3Checkpoint(args.model)
dit_probe = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend="sage2").eval() if args.load_dit_first else None
early_refiner = H3TokenRefiner(checkpoint, attention_backend="sage2") if args.preview_order else None
early_packer = H3PromptPacker(checkpoint) if args.preview_order else None
conditioner = Qwen3VLPromptConditioner(args.qwen, args.tokenizer)
vision = Qwen3VL32BVision(args.qwen, device="cuda", dtype=torch.float32)
if args.reference_vision_first and args.reference_vision_last:
class CapturedVision:
def __init__(self, paths):
self.outputs = [torch.load(path, map_location="cuda", weights_only=False) for path in paths]
def __call__(self, flatten, grid):
output = self.outputs.pop(0)
return output["merged"].cuda(), [value.cuda() for value in output["deepstack"]]
vision = CapturedVision([args.reference_vision_first, args.reference_vision_last])
video, audio, frame_count = random_av_latents(width, height, requested_frames, seed)
if args.sampler_capture_dir:
initial = torch.load(Path(args.sampler_capture_dir) / "initial.pt", map_location="cuda", weights_only=False)
direct_initial = torch.cat((video.reshape(-1), audio.reshape(-1)))
report("sampler_initial", direct_initial, initial["initial_x"].reshape(-1))
presentation = build_fl2va_presentation(
prompt,
load_image(args.first),
load_image(args.last),
width=width,
height=height,
frame_count=frame_count,
tokenizer=conditioner.tokenizer,
vision=vision,
text_encoder=conditioner.encoder,
device="cuda",
)
if args.qwen_capture_dir:
qwen_capture = Path(args.qwen_capture_dir)
expected_ids = torch.load(qwen_capture / "qwen_input_ids.pt", map_location="cuda", weights_only=False)
print({"stage": "qwen_input_ids", "equal": torch.equal(presentation.input_ids, expected_ids), "direct_shape": tuple(presentation.input_ids.shape), "comfy_shape": tuple(expected_ids.shape)}, flush=True)
direct_embeds = conditioner.encoder._embed_rows(presentation.input_ids)
visual_mask = torch.zeros((1, direct_embeds.shape[1]), dtype=torch.bool, device="cuda")
deepstack_by_index = {}
for embed in presentation.embeds_info:
start = embed["index"]
end = start + embed["size"]
direct_embeds[0, start:end] = embed["extra"]["merged"].to(direct_embeds)
visual_mask[0, start:end] = True
for index, value in enumerate(embed["extra"]["deepstack"]):
deepstack_by_index.setdefault(index, []).append(value)
compact_ids_path = qwen_capture / "qwen_compact_token_ids.pt"
if compact_ids_path.exists():
compact_ids = torch.load(compact_ids_path, map_location="cuda", weights_only=False)
print({"stage": "qwen_compact_token_ids", "equal": torch.equal(presentation.input_ids[~visual_mask], compact_ids.reshape(-1)), "direct": presentation.input_ids[~visual_mask].tolist(), "comfy": compact_ids.reshape(-1).tolist()}, flush=True)
direct_deepstack = [torch.cat(values, dim=0) for _, values in sorted(deepstack_by_index.items())]
expected_embeds = torch.load(qwen_capture / "qwen_input_embeds.pt", map_location="cuda", weights_only=False)
raw_rows = torch.nn.functional.embedding(presentation.input_ids, conditioner.encoder.embed_tokens)
raw_scales = torch.nn.functional.embedding(presentation.input_ids, conditioner.encoder.embed_scale)
fp16_embeds = (raw_rows.to(torch.float16) * raw_scales.to(torch.float16)).float()
bf16_embeds = (raw_rows.to(torch.bfloat16) * raw_scales.to(torch.bfloat16)).float()
fp32_to_fp16_embeds = (raw_rows.float() * raw_scales.float()).half().float()
fp32_to_bf16_embeds = (raw_rows.float() * raw_scales.float()).bfloat16().float()
report("qwen_input_text_rows_fp16", fp16_embeds[~visual_mask], expected_embeds[~visual_mask])
report("qwen_input_text_rows_bf16", bf16_embeds[~visual_mask], expected_embeds[~visual_mask])
report("qwen_input_text_rows_fp32_to_fp16", fp32_to_fp16_embeds[~visual_mask], expected_embeds[~visual_mask])
report("qwen_input_text_rows_fp32_to_bf16", fp32_to_bf16_embeds[~visual_mask], expected_embeds[~visual_mask])
report("qwen_input_embeds", direct_embeds, expected_embeds)
report("qwen_input_text_rows", direct_embeds[~visual_mask], expected_embeds.to(direct_embeds.device)[~visual_mask])
report("qwen_input_visual_rows", direct_embeds[visual_mask], expected_embeds.to(direct_embeds.device)[visual_mask])
position_ids = mrope_position_ids(presentation.embeds_info, direct_embeds.shape[1], "cuda")
freqs = mrope_freqs_cis(position_ids)
hidden = (expected_embeds if args.oracle_qwen_input else direct_embeds).to(conditioner.encoder.dtype)
for index, layer in enumerate(conditioner.encoder.layers):
hidden = layer(hidden, freqs)
expected_layer = torch.load(qwen_capture / "qwen_layers" / f"{index:02d}.pt", map_location="cuda", weights_only=False)
report(f"qwen_layer_{index:02d}", hidden, expected_layer)
if index < len(direct_deepstack):
hidden[visual_mask] = hidden[visual_mask] + direct_deepstack[index].to(hidden)
expected_layer50 = torch.load(qwen_capture / "qwen_layer50.pt", map_location="cuda", weights_only=False)
report("qwen_layer50", presentation.text_states, expected_layer50)
vae_dtype = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[args.vae_dtype]
vae = MiniMaxH3VideoVAEEncoder.from_safetensors(args.vae, device="cuda", dtype=vae_dtype, tiling=not args.vae_no_tiling).eval()
cond_latents = []
cond_images = []
for keyframe in presentation.keyframes:
resized = resize_keyframe(
keyframe["image"],
width,
height,
crop="disabled" if keyframe["resolved_frame_index"] == 0 else "center",
)
cond_images.append(resized)
pixels = resized.movedim(-1, 1).cuda().float().mul(2.0).sub(1.0)
cond_latents.append(vae.encode(pixels))
if args.qwen_capture_dir:
for index, latent in enumerate(cond_latents):
captured_vae = torch.load(Path(args.qwen_capture_dir) / f"vae_keyframe_{index}.pt", map_location="cuda", weights_only=False)
print({"stage": f"vae_meta_{index}", **captured_vae.get("meta", {})}, flush=True)
report(f"vae_image_{index}", cond_images[index], captured_vae["image"])
report(f"vae_keyframe_{index}", latent, captured_vae["latent"])
text = (early_refiner or H3TokenRefiner(checkpoint))(presentation.text_states)
packer = early_packer or H3PromptPacker(checkpoint)
sigma = beta_sigmas(12, device=video.device)[0]
hidden, times, segments, positions, _, _ = packer(
text,
video,
audio,
_model_sigma(sigma),
text_token_tags=presentation.text_token_tags,
cond_latents=cond_latents,
cond_frame_indices=[keyframe["resolved_frame_index"] for keyframe in presentation.keyframes],
frame_count=frame_count,
seed=seed,
)
expected_hidden = capture["hidden"]
text_length = text.shape[1]
frame_rows = (video.shape[-2] // 2) * (video.shape[-1] // 2)
direct_first = hidden[text_length:text_length + frame_rows]
direct_last = hidden[text_length + frame_rows:text_length + 2 * frame_rows]
comfy_first = expected_hidden[text_length:text_length + frame_rows]
comfy_last = expected_hidden[text_length + frame_rows:text_length + 2 * frame_rows]
print({"stage": "lengths", "text": text_length, "cond_each": frame_rows, "direct_total": hidden.shape[0], "comfy_total": expected_hidden.shape[0]}, flush=True)
report("text_rows", hidden[:text_length], expected_hidden[:text_length])
report("cond_first_to_first", direct_first, comfy_first)
report("cond_first_to_last", direct_first, comfy_last)
report("cond_last_to_last", direct_last, comfy_last)
report("cond_last_to_first", direct_last, comfy_first)
report("timesteps", times, capture["timesteps"])
report("positions", positions, capture["position_ids"])
print({"stage": "segments", "direct": segments, "comfy": capture["segments"]}, flush=True)

32
tools/compare_generation_latents.py
View file

@ -1,32 +0,0 @@
"""Compare saved hot-runtime video and audio latents against one reference run."""
import argparse
import json
import torch
parser = argparse.ArgumentParser()
parser.add_argument("reference")
parser.add_argument("candidates", nargs="+")
args = parser.parse_args()
reference = torch.load(args.reference, map_location="cpu", weights_only=False)
results = {}
for path in args.candidates:
candidate = torch.load(path, map_location="cpu", weights_only=False)
metrics = {}
for name in ("latent", "audio_latent"):
expected = reference[name].float()
actual = candidate[name].float()
delta = actual - expected
metrics[name] = {
"max_abs": delta.abs().max().item(),
"mean_abs": delta.abs().mean().item(),
"rmse": delta.square().mean().sqrt().item(),
"relative_rmse": (delta.square().mean().sqrt() / expected.square().mean().sqrt()).item(),
"cosine": torch.nn.functional.cosine_similarity(actual.flatten(), expected.flatten(), dim=0).item(),
}
results[path] = metrics
print(json.dumps({"reference": args.reference, "results": results}, indent=2))

55
tools/compare_h3_keyframe_capture.py
View file

@ -1,55 +0,0 @@
"""Replay a matched Comfy keyframe DiT capture through the direct H3 model."""
import argparse
from pathlib import Path
import torch
from h3_blackwell_runtime.checkpoint import H3Checkpoint
from h3_blackwell_runtime.denoiser import H3PackedDenoiser
from h3_blackwell_runtime.packing import unpatchify_video
from h3_blackwell_runtime.rope import h3_rope_rotation
from h3_blackwell_runtime.sampler import _unpack_audio
parser = argparse.ArgumentParser()
parser.add_argument("--capture", type=Path, required=True)
parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
parser.add_argument("--attention", default="sage2")
args = parser.parse_args()
def report(name, actual, expected):
expected = expected.to(actual.device)
delta = (actual.float() - expected.float()).abs()
print({"stage": name, "shape": tuple(actual.shape), "mean_delta": float(delta.mean()), "max_delta": float(delta.max())}, flush=True)
captured_input = torch.load(args.capture / "input.pt", map_location="cuda", weights_only=False)
captured_output = torch.load(args.capture / "output.pt", map_location="cuda", weights_only=False)
checkpoint = H3Checkpoint(args.model)
model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend=args.attention).eval()
hidden = captured_input["hidden"].cuda()
timesteps = captured_input["timesteps"].cuda()
positions = captured_input["position_ids"].cuda()
segments = captured_input["segments"]
rotation = h3_rope_rotation(positions, model.backbone.inv_freq, hidden.dtype)
with torch.inference_mode():
for index, (block, adaln) in enumerate(zip(model.backbone.blocks, model.backbone.adaln, strict=True)):
hidden = block(hidden, rotation, *adaln(timesteps), segments)
expected = torch.load(args.capture / "blocks" / f"{index:02d}.pt", map_location="cuda", weights_only=False)
report(f"block_{index:02d}", hidden, expected)
video_rows, audio_rows = model.final_layer(
hidden,
timesteps,
tuple(captured_output["video_segment"]),
tuple(captured_output["audio_segment"]),
)
expected_video = captured_output["video"].cuda()
expected_audio = captured_output["audio"].cuda()
video = unpatchify_video(video_rows, expected_video.shape[2], expected_video.shape[3], expected_video.shape[4])
audio = _unpack_audio(audio_rows)
report("video_output", video, expected_video)
report("audio_output", audio, expected_audio)

212
tools/compare_qwen_vision.py
View file

@ -1,212 +0,0 @@
"""Compare direct and Comfy Qwen3-VL vision outputs on one keyframe."""
import argparse
from pathlib import Path
import sys
import numpy as np
from PIL import Image
from safetensors import safe_open
import torch
from torch.nn import functional as F
from h3_blackwell_runtime.qwen3vl_vision import (
Qwen3VL32BVision,
_apply_rope_vision,
process_image,
resize_keyframe,
)
parser = argparse.ArgumentParser()
parser.add_argument("--image", required=True)
parser.add_argument("--checkpoint", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
parser.add_argument("--width", type=int, default=384)
parser.add_argument("--height", type=int, default=384)
parser.add_argument("--comfy-path", default="/opt/ComfyUI")
parser.add_argument("--save-reference")
parser.add_argument("--dtype", choices=("float16", "bfloat16", "float32"), default="bfloat16")
parser.add_argument("--captured-reference")
args = parser.parse_args()
sys.path.insert(0, args.comfy_path)
import comfy.ops # noqa: E402
from comfy.ldm.modules.attention import optimized_attention_for_device # noqa: E402
from comfy.text_encoders.qwen3vl import ( # noqa: E402
QWEN3VL_VISION,
QWEN3VL_VISION_COMMON,
Qwen3VLVisionModel,
)
from comfy.text_encoders.qwen_vl import process_qwen2vl_images # noqa: E402
from comfy.text_encoders.llama import apply_rope # noqa: E402
def report(name, actual, expected):
actual = actual.detach()
expected = expected.detach().to(actual.device)
delta = (actual.float() - expected.float()).abs()
print({
"stage": name,
"shape": tuple(actual.shape),
"actual_dtype": str(actual.dtype),
"expected_dtype": str(expected.dtype),
"mean_delta": float(delta.mean()),
"max_delta": float(delta.max()),
}, flush=True)
image = Image.open(args.image).convert("RGB")
pixels = torch.from_numpy(np.asarray(image).copy()).unsqueeze(0).cuda().float().div(255.0)
pixels = resize_keyframe(pixels, args.width, args.height)
direct_flatten, direct_grid = process_image(pixels)
reference_flatten, reference_grid = process_qwen2vl_images(
pixels,
patch_size=16,
image_mean=[0.5, 0.5, 0.5],
image_std=[0.5, 0.5, 0.5],
)
report("flatten_patches", direct_flatten, reference_flatten)
print({"stage": "grid", "direct": direct_grid.tolist(), "reference": reference_grid.tolist()}, flush=True)
dtype = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[args.dtype]
config = {
**QWEN3VL_VISION_COMMON,
**QWEN3VL_VISION["qwen3vl_32b"],
"out_hidden_size": 5120,
}
reference = Qwen3VLVisionModel(
config,
device="cuda",
dtype=dtype,
ops=comfy.ops.disable_weight_init,
).to("cuda").eval()
with safe_open(args.checkpoint, framework="pt", device="cuda") as checkpoint:
print({
"stage": "checkpoint_dtypes",
"text_norm": str(checkpoint.get_tensor("model.layers.0.input_layernorm.weight").dtype),
"vision_norm": str(checkpoint.get_tensor("visual.blocks.0.norm1.weight").dtype),
"vision_patch": str(checkpoint.get_tensor("visual.patch_embed.proj.weight").dtype),
}, flush=True)
visual_state = {
name.removeprefix("visual."): checkpoint.get_tensor(name).to(dtype)
for name in checkpoint.keys()
if name.startswith("visual.")
}
reference.load_state_dict(visual_state, strict=True)
del visual_state
direct = Qwen3VL32BVision(args.checkpoint, device="cuda", dtype=dtype).eval()
with torch.inference_mode():
direct_x = direct.patch_embed(direct_flatten.cuda().to(dtype))
direct_patch_embed = direct_x
reference_x = reference.patch_embed(reference_flatten.cuda().to(dtype))
report("patch_embed", direct_x, reference_x)
direct_pos = direct.fast_pos_embed_interpolate(direct_grid).to(direct_x.device)
reference_pos = reference.fast_pos_embed_interpolate(reference_grid).to(reference_x.device)
report("position_embed", direct_pos, reference_pos)
direct_x = direct_x + direct_pos
direct_vision_input = direct_x
reference_x = reference_x + reference_pos
report("vision_input", direct_x, reference_x)
direct_rotary = direct.rot_pos_emb(direct_grid.to(direct_x.device)).reshape(direct_x.shape[0], -1)
reference_rotary = reference.rot_pos_emb(reference_grid).to(reference_x.device).reshape(reference_x.shape[0], -1)
report("rotary", direct_rotary, reference_rotary)
def position_tuple(rotary):
embedding = torch.cat((rotary, rotary), dim=-1)
cosine = embedding.cos().unsqueeze(-2)
sine = embedding.sin().unsqueeze(-2)
split = sine.shape[-1] // 2
return cosine, sine[..., :split], -sine[..., split:]
direct_position = position_tuple(direct_rotary)
reference_position = position_tuple(reference_rotary)
cu_seqlens = F.pad(
torch.repeat_interleave(direct_grid[:, 1] * direct_grid[:, 2], direct_grid[:, 0]).cumsum(0, dtype=torch.int32),
(1, 0),
value=0,
)
optimized_attention = optimized_attention_for_device(reference_x.device, mask=False, small_input=True)
direct_block0 = direct.blocks[0]
reference_block0 = reference.blocks[0]
direct_norm = F.layer_norm(
direct_x,
(direct_x.shape[-1],),
weight=direct_block0.norm1_weight,
bias=direct_block0.norm1_bias,
eps=1e-6,
)
reference_norm = reference_block0.norm1(reference_x)
report("block0_norm1", direct_norm, reference_norm)
direct_qkv = F.linear(direct_norm, direct_block0.attn.qkv_weight, direct_block0.attn.qkv_bias)
reference_qkv = reference_block0.attn.qkv(reference_norm)
report("block0_qkv", direct_qkv, reference_qkv)
direct_q, direct_k, direct_v = direct_qkv.reshape(direct_x.shape[0], 3, 16, 72).permute(1, 0, 2, 3).unbind(0)
reference_q, reference_k, reference_v = reference_qkv.reshape(reference_x.shape[0], 3, 16, 72).permute(1, 0, 2, 3).unbind(0)
direct_q, direct_k = _apply_rope_vision(direct_q.float(), direct_k.float(), direct_position)
direct_q, direct_k = direct_q.to(dtype), direct_k.to(dtype)
reference_q, reference_k = apply_rope(reference_q, reference_k, reference_position)
report("block0_rope_q", direct_q, reference_q)
report("block0_rope_k", direct_k, reference_k)
direct_attention_heads = F.scaled_dot_product_attention(
direct_q.transpose(0, 1).unsqueeze(0),
direct_k.transpose(0, 1).unsqueeze(0),
direct_v.transpose(0, 1).unsqueeze(0),
)
direct_attention = direct_attention_heads.transpose(1, 2).reshape(1, direct_x.shape[0], -1)
reference_attention = optimized_attention(
reference_q.transpose(0, 1).unsqueeze(0),
reference_k.transpose(0, 1).unsqueeze(0),
reference_v.transpose(0, 1).unsqueeze(0),
16,
skip_reshape=True,
)
report("block0_attention", direct_attention, reference_attention)
direct_projected = F.linear(direct_attention[0], direct_block0.attn.proj_weight, direct_block0.attn.proj_bias)
reference_projected = reference_block0.attn.proj(reference_attention)[0]
report("block0_projected", direct_projected, reference_projected)
direct_deepstack = []
direct_blocks = []
reference_deepstack = []
for index, (direct_block, reference_block) in enumerate(zip(direct.blocks, reference.blocks)):
direct_x = direct_block(direct_x, cu_seqlens, direct_position)
direct_blocks.append(direct_x)
reference_x = reference_block(
reference_x,
cu_seqlens,
reference_position,
optimized_attention=optimized_attention,
)
report(f"block_{index:02d}", direct_x, reference_x)
if index in direct.deepstack_visual_indexes:
merger_index = direct.deepstack_visual_indexes.index(index)
direct_deepstack.append(direct.deepstack_merger_list[merger_index](direct_x))
reference_deepstack.append(reference.deepstack_merger_list[merger_index](reference_x))
direct_merged = direct.merger(direct_x)
reference_merged = reference.merger(reference_x)
report("merged", direct_merged, reference_merged)
for index, (actual, expected) in enumerate(zip(direct_deepstack, reference_deepstack)):
report(f"deepstack_{index}", actual, expected)
if args.captured_reference:
captured = torch.load(args.captured_reference, map_location="cuda", weights_only=False)
report("loaded_comfy_pixel_values", direct_flatten, captured["pixel_values"])
print({"stage": "loaded_comfy_grid", "direct": direct_grid.tolist(), "expected": captured["grid"].tolist()})
trace_path = Path(args.captured_reference).with_name(Path(args.captured_reference).name.replace("qwen_vision_", "qwen_vision_trace_"))
trace = torch.load(trace_path, map_location="cuda", weights_only=False)
report("loaded_comfy_patch_embed", direct_patch_embed, trace["patch_embed"])
report("loaded_comfy_position_embed", direct_pos, trace["position_embed"])
report("loaded_comfy_vision_input", direct_vision_input, trace["vision_input"])
for index, block_output in enumerate(direct_blocks):
report(f"loaded_comfy_block_{index:02d}", block_output, trace[f"block_{index:02d}"])
report("loaded_comfy_merged", direct_merged, captured["merged"])
for index, (actual, expected) in enumerate(zip(direct_deepstack, captured["deepstack"])):
report(f"loaded_comfy_deepstack_{index}", actual, expected)
if args.save_reference:
torch.save({
"merged": reference_merged.detach().cpu(),
"deepstack": [value.detach().cpu() for value in reference_deepstack],
}, args.save_reference)

6
tools/compare_vae_decoder_clip.py
View file

@ -7,7 +7,7 @@ from pathlib import Path
import torch
from safetensors.torch import load_file
from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE as DirectVAE, _conv3d
from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE as DirectVAE
parser = argparse.ArgumentParser()
@ -30,7 +30,7 @@ state = torch.load(args.latent, map_location="cuda", weights_only=False)
latent = state["latent"].to("cuda") if isinstance(state, dict) else state.to("cuda")
direct = DirectVAE.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda", tiling=False).eval()
upstream = UpstreamVAE(tiling=False).to("cuda", dtype=torch.float16).eval()
upstream = UpstreamVAE(tiling=False).to("cuda").eval()
upstream.load_state_dict(load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda"), strict=True)
with torch.inference_mode():
@ -38,7 +38,7 @@ with torch.inference_mode():
z_u = z_d.clone().to(next(upstream.parameters()).dtype)
z_d = z_d * direct.latents_std.view(1, -1, 1, 1, 1).to(z_d) + direct.latents_mean.view(1, -1, 1, 1, 1).to(z_d)
z_u = z_u * upstream.latents_std.view(1, -1, 1, 1, 1).to(z_u) + upstream.latents_mean.view(1, -1, 1, 1, 1).to(z_u)
z_d = _conv3d(z_d, direct.post_quant_conv.weight, direct.post_quant_conv.bias)
z_d = direct.post_quant_conv(z_d)
z_u = upstream.post_quant_conv(z_u)
stats("post_quant_conv", z_d, z_u)

110
tools/compare_vae_encoder.py
View file

@ -1,110 +0,0 @@
"""Compare direct and upstream MiniMax H3 VAE encoding on one keyframe."""
import argparse
import os
import sys
import numpy as np
from PIL import Image
import torch
from h3_blackwell_runtime.vae_encoder import (
MiniMaxH3VideoVAEEncoder as DirectVAEEncoder,
_causal_conv3d,
_downsample,
_group_norm_3d,
_resnet,
)
parser = argparse.ArgumentParser()
parser.add_argument("--image", required=True)
parser.add_argument("--vae", default="/vae/minimax_h3_video_vae_fp16.safetensors")
parser.add_argument("--comfy-path", default="/opt/ComfyUI")
parser.add_argument("--tiling", action="store_true")
args = parser.parse_args()
sys.path.insert(0, args.comfy_path)
from h3_blackwell_runtime.upstream_vae import MiniMaxH3VideoVAE as UpstreamVAE # noqa: E402
def load_checkpoint(path):
if os.getenv("H3_DISABLE_MMAP", "").lower() in {"1", "true", "yes", "on"}:
from safetensors.torch import load
with open(path, "rb") as file:
return load(file.read())
from safetensors.torch import load_file
return load_file(path, device="cuda")
def report(name, actual, expected):
actual = actual.detach()
expected = expected.detach().to(actual.device)
delta = (actual.float() - expected.float()).abs()
print({
"stage": name,
"shape": tuple(actual.shape),
"actual_min": float(actual.min()),
"actual_max": float(actual.max()),
"expected_min": float(expected.min()),
"expected_max": float(expected.max()),
"mean_delta": float(delta.mean()),
"max_delta": float(delta.max()),
}, flush=True)
image = Image.open(args.image).convert("RGB")
pixels = torch.from_numpy(np.asarray(image).copy()).permute(2, 0, 1).unsqueeze(0).cuda().float()
pixels = pixels.div(127.5).sub(1.0)
direct = DirectVAEEncoder.from_safetensors(args.vae, device="cuda", tiling=args.tiling).eval()
upstream = UpstreamVAE(tiling=args.tiling).to("cuda").eval()
upstream = upstream.to(dtype=direct.compute_dtype)
state = load_checkpoint(args.vae)
upstream.load_state_dict(state, strict=True)
del state
trace = {}
trace_names = {"encoder.conv_in", "encoder.norm_out", "encoder.conv_out", "quant_conv"}
for level in range(6):
trace_names.update(f"encoder.down.{level}.block.{block}" for block in range(2))
if level < 4:
trace_names.add(f"encoder.down.{level}.downsample")
for name, module in upstream.named_modules():
if name in trace_names:
module.register_forward_hook(lambda _module, _inputs, output, name=name: trace.__setitem__(name, output.detach().cpu()))
report("quant_conv_weight", direct.quant_conv.weight, upstream.quant_conv.weight)
report("quant_conv_bias", direct.quant_conv.bias, upstream.quant_conv.bias)
with torch.inference_mode():
direct_latent = direct.encode(pixels.clone())
upstream_latent = upstream.encode(pixels.to(direct.compute_dtype))
report("normalized_latent", direct_latent, upstream_latent)
with torch.inference_mode():
x = pixels.unsqueeze(2)
x = (x + 1.0) * 0.5
x = (x - direct.pixel_mean.to(x)) / direct.pixel_std.to(x)
params = direct.W
x = _causal_conv3d(x, params["conv_in"][0], params["conv_in"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
report("encoder.conv_in", x, trace.pop("encoder.conv_in"))
for level_index, level in enumerate(params["down"]):
for block_index, block in enumerate(level["blocks"]):
x = _resnet(x, block)
name = f"encoder.down.{level_index}.block.{block_index}"
report(name, x, trace.pop(name))
if level["down"] is not None:
x = _downsample(x, level["down"])
name = f"encoder.down.{level_index}.downsample"
report(name, x, trace.pop(name))
x = _group_norm_3d(x, params["norm_out_w"], params["norm_out_b"])
report("encoder.norm_out", x, trace.pop("encoder.norm_out"))
x = torch.nn.functional.silu(x)
x = _causal_conv3d(x, params["conv_out"][0], params["conv_out"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
report("encoder.conv_out", x, trace.pop("encoder.conv_out"))
x = torch.nn.functional.conv3d(x, direct.quant_conv.weight, direct.quant_conv.bias)
report("quant_conv", x, trace.pop("quant_conv"))
report("latents_mean", direct.latents_mean, upstream.latents_mean)
report("latents_std", direct.latents_std, upstream.latents_std)

2
tools/compare_vae_full_decode.py
View file

@ -51,7 +51,7 @@ with torch.inference_mode():
torch.cuda.empty_cache()
gc.collect()
upstream = UpstreamVAE().to("cuda", dtype=torch.float16).eval()
upstream = UpstreamVAE().to("cuda").eval()
upstream.load_state_dict(load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda"), strict=True)
upstream_pixels = pixelize(upstream.decode(latent.to(next(upstream.parameters()).dtype)), frames)

4
tools/decode_audio_latent.py
View file

@ -18,9 +18,9 @@ args = parser.parse_args()
state = torch.load(args.latent, map_location="cuda", weights_only=False)
if isinstance(state, dict):
latent = state.get("audio_latent", state.get("final_audio", state.get("latent")))
latent = state.get("audio_latent", state.get("latent"))
if latent is None:
raise ValueError("saved state does not contain 'audio_latent', 'final_audio', or 'latent'")
raise ValueError("saved state does not contain 'audio_latent' or 'latent'")
else:
latent = state
latent = latent.to("cuda")

80
tools/direct_t2v_preview.py
View file

@ -7,7 +7,6 @@ import subprocess
import time
from datetime import datetime, timezone
import warnings
import numpy as np
warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)
@ -22,10 +21,8 @@ from h3_blackwell_runtime.packing import H3PromptPacker
from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
from h3_blackwell_runtime.sampler import sample_video_res_multistep
from h3_blackwell_runtime.t2v import random_av_latents
from h3_blackwell_runtime.qwen3vl_vision import build_fl2va_presentation, resize_keyframe
from h3_blackwell_runtime.token_refiner import H3TokenRefiner
from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE, dtype_from_name
from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder
parser = argparse.ArgumentParser()
@ -36,8 +33,6 @@ parser.add_argument("--height", type=int, default=192)
parser.add_argument("--frames", type=int, default=22)
parser.add_argument("--steps", type=int, default=12)
parser.add_argument("--seed", type=int, default=440204)
parser.add_argument("--first-frame", type=Path, help="First keyframe image (fl2va), PNG/JPG path.")
parser.add_argument("--last-frame", type=Path, help="Last keyframe image (fl2va), PNG/JPG path.")
parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default=DEFAULT_ATTENTION_BACKEND)
parser.add_argument("--model-timesteps-capture", type=Path, help="Directory containing captured input_XX.pt H3 timesteps for strict parity checks.")
parser.add_argument("--progress", action="store_true", help="Print per-step sampler progress.")
@ -102,6 +97,8 @@ video, audio, frames = random_av_latents(args.width, args.height, args.frames, a
model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend=args.attention).eval()
configure_mlp_chunking(model, args.mlp_chunks, args.mlp_chunk_threshold)
report_memory("h3_loaded")
text = H3TokenRefiner(checkpoint, attention_backend=args.attention)(conditioner(args.prompt))
report_memory("text_conditioned")
model_timesteps = None
if args.model_timesteps_capture is not None:
model_timesteps = [
@ -110,50 +107,9 @@ if args.model_timesteps_capture is not None:
]
want_audio = args.save_audio_latent is not None or args.audio_output is not None or args.mux_audio
cache_stats = {}
refiner = H3TokenRefiner(checkpoint, attention_backend=args.attention)
packer = H3PromptPacker(checkpoint)
if args.first_frame is not None or args.last_frame is not None:
from PIL import Image
def load_image(path: Path) -> torch.Tensor:
img = Image.open(path).convert("RGB")
return torch.from_numpy(np.array(img)).unsqueeze(0).float() / 255.0
first = load_image(args.first_frame) if args.first_frame is not None else None
last = load_image(args.last_frame) if args.last_frame is not None else None
vae_encoder = MiniMaxH3VideoVAEEncoder.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda").eval()
report_memory("vae_encoder_loaded")
from h3_blackwell_runtime.qwen3vl_vision import Qwen3VL32BVision
vision_tower = Qwen3VL32BVision(
"/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", device="cuda", dtype=torch.float32
)
report_memory("vision_tower_loaded")
presentation = build_fl2va_presentation(
args.prompt,
first,
last,
width=args.width,
height=args.height,
frame_count=frames,
tokenizer=conditioner.tokenizer,
vision=vision_tower,
text_encoder=conditioner.encoder,
device="cuda",
)
cond_latents = []
for kf in presentation.keyframes:
resized = resize_keyframe(kf["image"].cuda(), args.width, args.height, crop="disabled" if kf["resolved_frame_index"] == 0 else "center")
pixels = resized.movedim(-1, 1).cuda().float().mul(2.0).sub(1.0)
cond_latents.append(vae_encoder.encode(pixels))
report_memory("fl2va_conditioned")
text = refiner(presentation.text_states)
report_memory("text_conditioned")
seed = args.seed
sampled = sample_video_res_multistep(
sampled = sample_video_res_multistep(
model,
packer,
H3PromptPacker(checkpoint),
text,
video,
audio,
@ -161,39 +117,13 @@ if args.first_frame is not None or args.last_frame is not None:
model_timesteps=model_timesteps,
return_audio=want_audio,
progress=args.progress,
seed=seed,
text_token_tags=presentation.text_token_tags,
cond_latents=cond_latents,
cond_frame_indices=[kf["resolved_frame_index"] for kf in presentation.keyframes],
frame_count=frames,
cache_mode=args.cache_mode,
cache_threshold=args.cache_threshold,
cache_start_percent=args.cache_start_percent,
cache_end_percent=args.cache_end_percent,
cache_subsample_factor=args.cache_subsample_factor,
cache_stats=cache_stats,
)
else:
text = refiner(conditioner(args.prompt))
report_memory("text_conditioned")
sampled = sample_video_res_multistep(
model,
packer,
text,
video,
audio,
steps=args.steps,
model_timesteps=model_timesteps,
return_audio=want_audio,
progress=args.progress,
seed=args.seed,
cache_mode=args.cache_mode,
cache_threshold=args.cache_threshold,
cache_start_percent=args.cache_start_percent,
cache_end_percent=args.cache_end_percent,
cache_subsample_factor=args.cache_subsample_factor,
cache_stats=cache_stats,
)
)
if cache_stats:
report({"cache": cache_stats})
if want_audio:

185
tools/distributed_t2va.py
View file

@ -1,185 +0,0 @@
"""Run prompt-only H3 T2VA with Ulysses or TP+sequence parallelism."""
import argparse
import json
import os
import time
from pathlib import Path
import torch
import torch.distributed as dist
from h3_blackwell_runtime.checkpoint import H3Checkpoint
from h3_blackwell_runtime.denoiser import H3PackedDenoiser
from h3_blackwell_runtime.distributed import SequenceParallelContext
from h3_blackwell_runtime.packing import H3PromptPacker
from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
from h3_blackwell_runtime.sampler import sample_video_res_multistep
from h3_blackwell_runtime.t2v import random_av_latents
from h3_blackwell_runtime.tensor_parallel import configure_h3_tensor_parallel
from h3_blackwell_runtime.token_refiner import H3TokenRefiner
class ParallelDenoiser:
def __init__(self, model: H3PackedDenoiser, mode: str):
self.model = model
self.mode = mode
def __call__(self, hidden, timesteps, positions, segments, video_segment, audio_segment):
context = SequenceParallelContext.create(hidden.shape[0], heads=56, head_dim=128)
if self.mode == "ulysses":
return self.model.forward_sequence_parallel(
hidden, timesteps, positions, segments, video_segment, audio_segment, context,
)
return self.model.forward_tensor_parallel(
hidden, timesteps, positions, segments, video_segment, audio_segment, context,
)
def synchronize(device: torch.device) -> None:
torch.cuda.synchronize(device)
parser = argparse.ArgumentParser()
parser.add_argument("--benchmark", type=Path, required=True)
parser.add_argument("--mode", choices=("ulysses", "tensor"), required=True)
parser.add_argument("--attention", default="sdpa")
parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
parser.add_argument("--text-encoder", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
parser.add_argument("--save-latent", type=Path)
parser.add_argument("--report", type=Path)
args = parser.parse_args()
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
device = torch.device(f"cuda:{local_rank}")
dist.init_process_group("nccl", device_id=device)
rank = dist.get_rank()
world_size = dist.get_world_size()
benchmark = json.loads(args.benchmark.read_text(encoding="utf-8"))
load_started = time.perf_counter()
checkpoint = H3Checkpoint(args.model, device=device)
model = H3PackedDenoiser.from_checkpoint(
checkpoint, output_dtype=torch.bfloat16, attention_backend=args.attention,
).eval()
packer = H3PromptPacker(checkpoint)
if args.mode == "tensor":
partition_context = SequenceParallelContext.create(world_size, heads=56, head_dim=128)
configure_h3_tensor_parallel(model, partition_context)
synchronize(device)
model_load_seconds = time.perf_counter() - load_started
model_load_peak = torch.cuda.max_memory_allocated(device)
torch.cuda.reset_peak_memory_stats(device)
conditioning_started = time.perf_counter()
if rank == 0:
conditioner = Qwen3VLPromptConditioner(args.text_encoder, device=device, dtype=torch.float32)
refiner = H3TokenRefiner(checkpoint, attention_backend="sdpa").eval()
text = refiner(conditioner(benchmark["prompt"])).to(torch.bfloat16)
text_length = torch.tensor([text.shape[1]], dtype=torch.int64, device=device)
del conditioner, refiner
else:
text = None
text_length = torch.zeros(1, dtype=torch.int64, device=device)
checkpoint.release_cache()
torch.cuda.empty_cache()
dist.broadcast(text_length, src=0)
if rank != 0:
text = torch.empty(1, int(text_length.item()), 5376, dtype=torch.bfloat16, device=device)
dist.broadcast(text, src=0)
synchronize(device)
conditioning_seconds = time.perf_counter() - conditioning_started
conditioning_peak = torch.cuda.max_memory_allocated(device)
torch.cuda.reset_peak_memory_stats(device)
width, height = benchmark["resolution"]
video, audio, aligned_frames = random_av_latents(
width, height, benchmark["frames"], benchmark["seed"], device=device,
)
dist.barrier()
synchronize(device)
sampling_started = time.perf_counter()
video, audio = sample_video_res_multistep(
ParallelDenoiser(model, args.mode),
packer,
text,
video,
audio,
steps=benchmark["steps"],
seed=benchmark["seed"],
return_audio=True,
progress=rank == 0,
)
synchronize(device)
sampling_seconds = time.perf_counter() - sampling_started
timing = torch.tensor(
[model_load_seconds, conditioning_seconds, sampling_seconds],
dtype=torch.float64,
device=device,
)
timings = [torch.empty_like(timing) for _ in range(world_size)]
dist.all_gather(timings, timing)
checksums = torch.stack((video.float().sum(), audio.float().sum())).to(torch.float64)
all_checksums = [torch.empty_like(checksums) for _ in range(world_size)]
dist.all_gather(all_checksums, checksums)
checksum_stack = torch.stack(all_checksums)
if not torch.allclose(checksum_stack, checksum_stack[0].expand_as(checksum_stack), rtol=0, atol=1e-5):
raise RuntimeError(f"rank outputs diverged: {checksum_stack.cpu().tolist()}")
peak_memory = torch.tensor(
[model_load_peak, conditioning_peak, torch.cuda.max_memory_allocated(device)],
dtype=torch.int64,
device=device,
)
memory = [torch.empty_like(peak_memory) for _ in range(world_size)]
dist.all_gather(memory, peak_memory)
if rank == 0:
if args.save_latent is not None:
args.save_latent.parent.mkdir(parents=True, exist_ok=True)
torch.save({
"latent": video.cpu(),
"audio_latent": audio.cpu(),
"frames": aligned_frames,
"width": width,
"height": height,
"prompt": benchmark["prompt"],
"seed": benchmark["seed"],
"distributed_mode": args.mode,
"world_size": world_size,
"attention": args.attention,
}, args.save_latent)
timing_stack = torch.stack(timings).cpu()
report = {
"mode": args.mode,
"world_size": world_size,
"attention": args.attention,
"device": torch.cuda.get_device_name(device),
"torch": torch.__version__,
"benchmark": str(args.benchmark),
"resolution": [width, height],
"frames": aligned_frames,
"steps": benchmark["steps"],
"seed": benchmark["seed"],
"timings_max_rank_seconds": {
"model_load": float(timing_stack[:, 0].max()),
"conditioning": float(timing_stack[:, 1].max()),
"sampling": float(timing_stack[:, 2].max()),
},
"peak_allocated_bytes_by_rank": {
"model_load": [int(value[0].item()) for value in memory],
"conditioning": [int(value[1].item()) for value in memory],
"sampling": [int(value[2].item()) for value in memory],
},
"checksums": checksum_stack[0].cpu().tolist(),
"latent": str(args.save_latent) if args.save_latent is not None else None,
}
serialized = json.dumps(report, indent=2)
if args.report is not None:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(serialized + "\n", encoding="utf-8")
print(serialized)
dist.destroy_process_group()

16
tools/inspect_checkpoint_tensors.py
View file

@ -1,18 +1,10 @@
"""List selected H3 checkpoint tensor shapes."""
import argparse
from safetensors import safe_open
parser = argparse.ArgumentParser()
parser.add_argument("--checkpoint", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
parser.add_argument("--prefix", action="append", default=[])
args = parser.parse_args()
prefixes = tuple(args.prefix) or ("adaln_t_table", "blocks.0.adaln_proj", "final_layer.adaln_proj")
with safe_open(args.checkpoint, framework="pt", device="cpu") as checkpoint:
path = "/models/minimax_h3_ref2va_pruned_nvfp4.safetensors"
with safe_open(path, framework="pt", device="cpu") as checkpoint:
for name in checkpoint.keys():
if any(name == prefix or name.startswith(prefix) for prefix in prefixes):
tensor = checkpoint.get_tensor(name)
print(name, tuple(tensor.shape), tensor.dtype)
if name == "adaln_t_table" or name.startswith("blocks.0.adaln_proj") or name.startswith("final_layer.adaln_proj"):
print(name, tuple(checkpoint.get_tensor(name).shape))

74
tools/patch_comfy_qwen_vision_capture.py
View file

@ -1,74 +0,0 @@
"""Capture actual loaded-Comfy Qwen merged and DeepStack vision tensors."""
from pathlib import Path
path = Path("/opt/ComfyUI/comfy/text_encoders/qwen3vl.py")
source = path.read_text(encoding="utf-8")
if "import os\n" not in source:
source = source.replace("import os\n", "import os\n", 1) if "import os\n" in source else "import os\n" + source
if "import traceback\n" not in source:
source = "import traceback\n" + source
old = (
" merged, deepstack = self.visual(image.to(device, dtype=torch.float32), grid)\n"
" return merged, {\"grid\": grid, \"deepstack\": deepstack}\n"
)
new = (
" merged, deepstack = self.visual(image.to(device, dtype=torch.float32), grid)\n"
" capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n"
" if capture_dir:\n"
" os.makedirs(capture_dir, exist_ok=True)\n"
" capture_index = getattr(self, \"_h3_vision_capture_index\", 0)\n"
" torch.save({\"merged\": merged.detach().cpu(), \"deepstack\": [value.detach().cpu() for value in deepstack], \"pixel_values\": image.detach().cpu(), \"grid\": grid.detach().cpu(), \"stack\": traceback.format_stack()}, os.path.join(capture_dir, f\"qwen_vision_{capture_index}.pt\"))\n"
" open(os.path.join(capture_dir, \"qwen_vision_stack.txt\"), \"w\", encoding=\"utf-8\").writelines(traceback.format_stack())\n"
" self._h3_vision_capture_index = capture_index + 1\n"
" return merged, {\"grid\": grid, \"deepstack\": deepstack}\n"
)
if source.count(old) != 1:
raise RuntimeError("Unable to locate Qwen3-VL vision preprocess block.")
path.write_text(source.replace(old, new), encoding="utf-8")
vision_path = Path("/opt/ComfyUI/comfy/text_encoders/qwen35.py")
vision_source = vision_path.read_text(encoding="utf-8")
vision_source = vision_source.replace(
" x = self.patch_embed(x)\n pos_embeds = self.fast_pos_embed_interpolate(grid_thw).to(x.device)\n x = x + pos_embeds\n",
" x = self.patch_embed(x)\n capture_trace = {\"patch_embed\": x.detach().cpu()}\n pos_embeds = self.fast_pos_embed_interpolate(grid_thw).to(x.device)\n capture_trace[\"position_embed\"] = pos_embeds.detach().cpu()\n x = x + pos_embeds\n capture_trace[\"vision_input\"] = x.detach().cpu()\n",
1,
)
vision_source = vision_source.replace(
" x = blk(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings, optimized_attention=optimized_attention)\n",
" x = blk(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings, optimized_attention=optimized_attention)\n capture_trace[f\"block_{layer_num:02d}\"] = x.detach().cpu()\n",
1,
)
vision_source = vision_source.replace(
" merged = self.merger(x)\n if self.deepstack_merger_list is not None:\n",
" merged = self.merger(x)\n capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n if capture_dir:\n trace_index = getattr(self, \"_h3_vision_trace_index\", 0)\n torch.save(capture_trace, os.path.join(capture_dir, f\"qwen_vision_trace_{trace_index}.pt\"))\n self._h3_vision_trace_index = trace_index + 1\n if self.deepstack_merger_list is not None:\n",
1,
)
vision_path.write_text(vision_source, encoding="utf-8")
clip_path = Path("/opt/ComfyUI/comfy/sd1_clip.py")
clip_source = clip_path.read_text(encoding="utf-8")
if "import os\n" not in clip_source:
clip_source = "import os\n" + clip_source
clip_source = clip_source.replace(
" tokens_embed = torch.tensor([tokens_temp], device=device, dtype=torch.long)\n tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)\n",
" tokens_embed = torch.tensor([tokens_temp], device=device, dtype=torch.long)\n capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n if capture_dir:\n torch.save(tokens_embed.detach().cpu(), os.path.join(capture_dir, \"qwen_compact_token_ids.pt\"))\n tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)\n",
1,
)
clip_path.write_text(clip_source, encoding="utf-8")
nodes_path = Path("/opt/ComfyUI/comfy_extras/nodes_minimax_h3.py")
nodes_source = nodes_path.read_text(encoding="utf-8")
if "import os\n" not in nodes_source:
nodes_source = "import os\n" + nodes_source
vae_encode_line = "kf[\"latent\"] = vae.encode(kf.pop(\"image\"))"
if nodes_source.count(vae_encode_line) != 1:
raise RuntimeError("Unable to locate MiniMax H3 keyframe VAE encode call.")
nodes_source = nodes_source.replace(
vae_encode_line,
"keyframe_image = kf.pop(\"image\")\n kf[\"latent\"] = vae.encode(keyframe_image)\n capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n if capture_dir:\n capture_index = getattr(vae, \"_h3_vae_capture_index\", 0)\n vae_model = vae.first_stage_model\n vae_meta = {\"parameter_dtype\": str(next(vae_model.parameters()).dtype), \"tiling\": vae_model.tiling, \"tile_size\": vae_model.tile_size, \"tile_overlap_min\": vae_model.tile_overlap_min}\n torch.save({\"image\": keyframe_image.detach().cpu(), \"latent\": kf[\"latent\"].detach().cpu(), \"meta\": vae_meta}, os.path.join(capture_dir, f\"vae_keyframe_{capture_index}.pt\"))\n vae._h3_vae_capture_index = capture_index + 1",
1,
)
nodes_path.write_text(nodes_source, encoding="utf-8")
print("Applied Qwen3-VL vision capture patch.")

27
tools/poll_qwen38_health_spark.sh
View file

@ -1,27 +0,0 @@
#!/usr/bin/env bash
set -u
for i in $(seq 0 39); do
status=$(docker ps -a --filter name=qwen38-vllm --format '{{.Status}}')
listening=$(ss -ltnp | grep -c ':8002 ' || true)
if curl -fsS --max-time 5 http://127.0.0.1:8002/health >/tmp/qwen38-health.out 2>/tmp/qwen38-health.err; then
echo "HEALTHY poll=$i status=$status listening=$listening"
docker logs --tail 60 qwen38-vllm
exit 0
fi
if [[ $status != Up* ]]; then
echo "NOT_RUNNING poll=$i status=$status listening=$listening"
docker logs --tail 160 qwen38-vllm
exit 2
fi
echo "poll=$i status=$status listening=$listening"
docker logs --tail 10 qwen38-vllm
sleep 30
done
echo "TIMEOUT"
docker logs --tail 200 qwen38-vllm
exit 1

182
tools/probe_audio_vae_boundary.py
View file

@ -1,182 +0,0 @@
"""Decode controlled H3 audio-latent boundary variants for diagnosis."""
import argparse
import json
import math
import subprocess
from pathlib import Path
import numpy as np
import torch
from h3_blackwell_runtime.audio_vae_decoder import MiniMaxH3AudioVAE
SAMPLE_RATE = 32000
SAMPLES_PER_LATENT = 800
def load_latent(path: Path) -> torch.Tensor:
state = torch.load(path, map_location="cpu", weights_only=False)
latent = state.get("audio_latent") if isinstance(state, dict) else state
if latent is None or latent.ndim != 4:
raise ValueError(f"{path} does not contain a [B,C,S,T] audio latent")
return latent
def dbfs(value: float) -> float:
return 20.0 * math.log10(max(value, 1e-20))
def waveform_metrics(waveform: torch.Tensor) -> dict:
samples = waveform.float().numpy().T
first_100ms = samples[: SAMPLE_RATE // 10]
first_500ms = samples[: SAMPLE_RATE // 2]
derivatives = np.max(np.abs(np.diff(first_500ms, axis=0)), axis=1)
return {
"samples": len(samples),
"first_sample": samples[0].tolist(),
"first_sample_dbfs": [dbfs(float(abs(value))) for value in samples[0]],
"first_100ms_peak_dbfs": dbfs(float(np.max(np.abs(first_100ms)))),
"first_100ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_100ms**2)))),
"first_500ms_peak_dbfs": dbfs(float(np.max(np.abs(first_500ms)))),
"first_500ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_500ms**2)))),
"largest_derivative": float(derivatives.max()),
"largest_derivative_ms": float((derivatives.argmax() + 1) * 1000.0 / SAMPLE_RATE),
}
def comparison(reference: torch.Tensor, candidate: torch.Tensor) -> dict:
count = min(reference.shape[-1], candidate.shape[-1])
reference = reference[..., :count].float()
candidate = candidate[..., :count].float()
def region_metrics(samples: int) -> dict:
ref = reference[..., :samples]
test = candidate[..., :samples]
error = test - ref
signal_power = ref.square().mean(dim=-1)
noise_power = error.square().mean(dim=-1)
psnr = 10.0 * torch.log10(signal_power.clamp_min(1e-30) / noise_power.clamp_min(1e-30))
return {
"rmse": float(error.square().mean().sqrt()),
"max_abs": float(error.abs().max()),
"psnr_db_by_channel": psnr.flatten().tolist(),
}
return {
"first_100ms": region_metrics(SAMPLE_RATE // 10),
"first_500ms": region_metrics(SAMPLE_RATE // 2),
"full": region_metrics(count),
}
def write_waveform(path: Path, waveform: torch.Tensor) -> None:
raw = path.with_suffix(".f32le")
waveform.transpose(0, 1).contiguous().numpy().tofile(raw)
subprocess.run([
"ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(SAMPLE_RATE), "-ac", "2", "-i", str(raw),
"-c:a", "pcm_f32le", str(path),
], check=True)
raw.unlink()
parser = argparse.ArgumentParser()
parser.add_argument("--affected-latent", type=Path, required=True)
parser.add_argument("--clean-latent", type=Path, required=True)
parser.add_argument("--vae", type=Path, default=Path("/vae/minimax_h3_audio_vae_fp32.safetensors"))
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--report", type=Path)
args = parser.parse_args()
affected = load_latent(args.affected_latent)
clean = load_latent(args.clean_latent)
if affected.shape != clean.shape:
raise ValueError(f"latent shapes differ: {tuple(affected.shape)} != {tuple(clean.shape)}")
boundary_frames = 4
variants = {
"affected-original": (affected, 0),
"clean-original": (clean, 0),
"zero-normalized-latent": (torch.zeros_like(affected), 0),
"affected-repeat-frame0": (affected[..., :1].expand_as(affected).clone(), 0),
"affected-repeat-frame4": (affected[..., 4:5].expand_as(affected).clone(), 0),
}
silent_carrier = affected[..., 4:5].expand_as(affected).clone()
carrier_start = silent_carrier.clone()
carrier_start[..., :boundary_frames] = affected[..., :boundary_frames]
variants["carrier-affected-first4-at-start"] = (carrier_start, 0)
interior_frame = 40
carrier_interior = silent_carrier.clone()
carrier_interior[..., interior_frame : interior_frame + boundary_frames] = affected[..., :boundary_frames]
variants["carrier-affected-first4-at-frame40"] = (carrier_interior, 0)
replaced_with_frame4 = affected.clone()
replaced_with_frame4[..., :boundary_frames] = affected[..., 4:5]
variants["affected-first4-repeat-frame4"] = (replaced_with_frame4, 0)
affected_with_clean = affected.clone()
affected_with_clean[..., :boundary_frames] = clean[..., :boundary_frames]
variants["affected-first4-from-clean"] = (affected_with_clean, 0)
clean_with_affected = clean.clone()
clean_with_affected[..., :boundary_frames] = affected[..., :boundary_frames]
variants["clean-first4-from-affected"] = (clean_with_affected, 0)
prefix_repeat = affected[..., :1].expand(*affected.shape[:-1], boundary_frames)
variants["affected-prefix-repeat-frame0"] = (
torch.cat((prefix_repeat, affected), dim=-1),
boundary_frames * SAMPLES_PER_LATENT,
)
variants["affected-prefix-own-first4"] = (
torch.cat((affected[..., :boundary_frames], affected), dim=-1),
boundary_frames * SAMPLES_PER_LATENT,
)
vae = MiniMaxH3AudioVAE.from_safetensors(args.vae, device="cuda").eval()
args.output_dir.mkdir(parents=True, exist_ok=True)
decoded = {}
report = {"boundary_frames": boundary_frames, "variants": {}}
with torch.inference_mode():
for name, (latent, crop_start) in variants.items():
waveform = vae.decode(latent.to("cuda", dtype=next(vae.parameters()).dtype)).cpu()[0]
waveform = waveform[:, crop_start : crop_start + affected.shape[-1] * SAMPLES_PER_LATENT]
decoded[name] = waveform
output = args.output_dir / f"{name}.wav"
write_waveform(output, waveform)
report["variants"][name] = {
"output": str(output),
"crop_start_samples": crop_start,
"metrics": waveform_metrics(waveform),
}
affected_reference = decoded["affected-original"]
for name, waveform in decoded.items():
if name != "affected-original":
report["variants"][name]["difference_from_affected_original"] = comparison(
affected_reference, waveform,
)
interior_start = interior_frame * SAMPLES_PER_LATENT
segment_samples = boundary_frames * SAMPLES_PER_LATENT
report["interior_placement"] = {
"frame": interior_frame,
"start_seconds": interior_start / SAMPLE_RATE,
"affected_onset_vs_carrier_interior_event": comparison(
affected_reference[..., :segment_samples],
decoded["carrier-affected-first4-at-frame40"][..., interior_start : interior_start + segment_samples],
),
"carrier_start_event_vs_carrier_interior_event": comparison(
decoded["carrier-affected-first4-at-start"][..., :segment_samples],
decoded["carrier-affected-first4-at-frame40"][..., interior_start : interior_start + segment_samples],
),
}
serialized = json.dumps(report, indent=2)
if args.report is not None:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(serialized + "\n", encoding="utf-8")
print(serialized)

20
tools/run_distributed_matrix.sh
View file

@ -1,20 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
available="$(python -c 'import torch; print(torch.cuda.device_count())')"
if [[ "$available" -lt 1 ]]; then
echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
exit 1
fi
read -ra counts <<< "${H3_GPU_COUNTS:-1 2 4 6 8}"
read -ra modes <<< "${H3_DISTRIBUTED_MODES:-ulysses tensor}"
for count in "${counts[@]}"; do
if [[ "$count" -gt "$available" ]]; then
echo "Skipping ${count} GPUs; only ${available} are visible." >&2
continue
fi
for mode in "${modes[@]}"; do
tools/run_distributed_t2va.sh "$mode" "$count" "${H3_ATTENTION:-sdpa}"
done
done

35
tools/run_distributed_t2va.sh
View file

@ -1,35 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
mode="${1:?usage: run_distributed_t2va.sh ulysses|tensor [WORLD_SIZE] [ATTENTION]}"
world_size="${2:-${H3_WORLD_SIZE:-}}"
if [[ -z "$world_size" ]]; then
world_size="$(python -c 'import torch; print(torch.cuda.device_count())')"
fi
if [[ "$world_size" -lt 1 ]]; then
echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
exit 1
fi
attention="${3:-sdpa}"
benchmark="${H3_DISTRIBUTED_BENCHMARK:-benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json}"
output_root="${H3_DISTRIBUTED_OUTPUT:-/output/h3-baselines}"
stem="distributed-${mode}-${world_size}gpu-${attention}"
model="${H3_MODEL_PATH:-/models/minimax_h3_fl2va_pruned_nvfp4.safetensors}"
text_encoder="${H3_TEXT_ENCODER_PATH:-/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors}"
extra_args=()
if [[ "${H3_SAVE_LATENTS:-1}" == "1" ]]; then
extra_args+=(--save-latent "$output_root/$stem.latent.pt")
fi
torchrun \
--standalone \
--nnodes=1 \
--nproc-per-node="$world_size" \
tools/distributed_t2va.py \
--benchmark "$benchmark" \
--mode "$mode" \
--attention "$attention" \
--model "$model" \
--text-encoder "$text_encoder" \
--report "$output_root/$stem.json" \
"${extra_args[@]}"

23
tools/run_ulysses_benchmark.sh
View file

@ -1,23 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
world_size="${1:-${H3_WORLD_SIZE:-}}"
if [[ -z "$world_size" ]]; then
world_size="$(python -c 'import torch; print(torch.cuda.device_count())')"
fi
if [[ "$world_size" -lt 1 ]]; then
echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
exit 1
fi
backend="${2:-sdpa}"
sequence="${3:-20000}"
output="${H3_BENCHMARK_OUTPUT:-/output/h3-baselines/ulysses-${world_size}gpu-${backend}-${sequence}t.json}"
torchrun \
--standalone \
--nnodes=1 \
--nproc-per-node="$world_size" \
tools/benchmark_ulysses.py \
--sequence "$sequence" \
--backend "$backend" \
--output "$output"

129
tools/runpod_api.py
View file

@ -1,129 +0,0 @@
"""Minimal RunPod API v2 client for the H3 single-node benchmark pod."""
import argparse
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
API = "https://api.runpod.io/v2"
BLACKWELL_GPUS = (
"NVIDIA RTX PRO 6000 Blackwell Server Edition",
"NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
"NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition",
)
DEFAULT_IMAGE = "runpod/pytorch:1.1.0-cu1300-torch291-ubuntu2404"
def request(method: str, path: str, body=None, query=None):
key = os.environ.get("RUNPOD_API_KEY")
if not key:
raise SystemExit("RUNPOD_API_KEY is required")
url = f"{API}{path}"
if query:
url += "?" + urllib.parse.urlencode(query)
data = None if body is None else json.dumps(body).encode("utf-8")
call = urllib.request.Request(url, data=data, method=method)
call.add_header("User-Agent", "h3-blackwell-runtime/0.1")
call.add_header("Authorization", f"Bearer {key}")
call.add_header("Accept", "application/json")
if data is not None:
call.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(call, timeout=60) as response:
payload = response.read()
return json.loads(payload) if payload else {"status": response.status}
except urllib.error.HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
raise SystemExit(f"RunPod API returned HTTP {error.code}: {detail}") from error
parser = argparse.ArgumentParser()
commands = parser.add_subparsers(dest="command", required=True)
catalog_parser = commands.add_parser("catalog")
catalog_parser.add_argument("--count", type=int, default=8)
catalog_parser.add_argument("--cloud", choices=("SECURE", "COMMUNITY"), default="SECURE")
create_parser = commands.add_parser("create")
create_parser.add_argument("--gpu", choices=BLACKWELL_GPUS, default=BLACKWELL_GPUS[0])
create_parser.add_argument("--count", type=int, default=8)
create_parser.add_argument("--cloud", choices=("SECURE", "COMMUNITY"), default="SECURE")
create_parser.add_argument("--datacenter")
create_parser.add_argument("--image", default=DEFAULT_IMAGE)
create_parser.add_argument("--disk", type=int, default=100)
create_parser.add_argument("--volume", type=int, default=100)
create_parser.add_argument("--network-volume")
create_parser.add_argument("--volume-mount-path", default="/runpod-volume")
create_parser.add_argument("--ssh-public-key", type=Path)
create_parser.add_argument("--yes", action="store_true")
get_parser = commands.add_parser("get")
get_parser.add_argument("pod_id")
terminate_parser = commands.add_parser("terminate")
terminate_parser.add_argument("pod_id")
terminate_parser.add_argument("--yes", action="store_true")
ssh_keys_parser = commands.add_parser("ssh-keys")
ssh_keys_parser.add_argument("--replace-with", type=Path)
ssh_keys_parser.add_argument("--yes", action="store_true")
args = parser.parse_args()
if args.command == "catalog":
response = request("GET", "/catalog/gpus", query={
"include": "AVAILABILITY",
"product": "POD",
"count": args.count,
"cloud": args.cloud,
"minCudaVersion": "12.8",
})
response["gpus"] = [gpu for gpu in response["gpus"] if gpu["id"] in BLACKWELL_GPUS]
elif args.command == "create":
if not args.yes:
raise SystemExit("create rents billable GPUs; repeat with --yes after checking catalog")
body = {
"name": "h3-blackwell-distributed",
"image": args.image,
"gpu": {"id": args.gpu, "count": args.count, "minCudaVersion": "12.8"},
"cloud": args.cloud,
"disk": args.disk,
"ports": ["22/tcp"],
"startSsh": True,
}
if args.network_volume:
body["mounts"] = {
"network": [{"volumeId": args.network_volume, "path": args.volume_mount_path}],
}
else:
body["mounts"] = {
"persistent": {"size": args.volume, "path": args.volume_mount_path},
}
if args.ssh_public_key:
body["env"] = {
"PUBLIC_KEY": args.ssh_public_key.read_text(encoding="utf-8").strip(),
}
if args.datacenter:
body["dataCenterIds"] = [args.datacenter]
response = request("POST", "/pods", body=body)
elif args.command == "get":
response = request("GET", f"/pods/{args.pod_id}")
elif args.command == "terminate":
if not args.yes:
raise SystemExit("termination is irreversible; repeat with --yes")
response = request("POST", f"/pods/{args.pod_id}/action", body={"action": "terminate"})
else:
if args.replace_with:
if not args.yes:
raise SystemExit("replacing account SSH keys requires --yes")
public_key = args.replace_with.read_text(encoding="utf-8").strip()
response = request("PUT", "/account/ssh-keys", body={"keys": [public_key]})
else:
response = request("GET", "/account/ssh-keys")
json.dump(response, sys.stdout, indent=2)
sys.stdout.write("\n")

80
tools/serve_hot_runtime.py
View file

@ -10,42 +10,8 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse
import torch
from PIL import Image
from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND, attention_backend_status
from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig, TURBO_VARIANTS, normalize_upscale
def _load_image(value) -> torch.Tensor | None:
"""Accept a keyframe as on-disk path or base64 JPEG/PNG -> ``[1,3,H,W]`` float ``[0,1]``."""
if value in (None, ""):
return None
if isinstance(value, (list, tuple)):
value = value[0]
if isinstance(value, dict):
value = value.get("url") or value.get("path") or value.get("b64")
if isinstance(value, str) and "\n" not in value and len(value) < 2048 and not value.startswith("data:"):
path = Path(value)
if path.exists():
image = Image.open(path).convert("RGB")
import numpy as np
tensor = torch.from_numpy(np.array(image))[None].permute(0, 3, 1, 2).float() / 255.0
return tensor
data = value
if isinstance(value, str) and value.startswith("data:"):
data = value.split(",", 1)[1]
if isinstance(data, str) and len(data) >= 1024:
import base64
import io
raw = base64.b64decode(data)
image = Image.open(io.BytesIO(raw)).convert("RGB")
import numpy as np
return torch.from_numpy(np.array(image))[None].permute(0, 3, 1, 2).float() / 255.0
raise ValueError("first_frame/last_frame must be a path or a base64/data-URL image")
from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND
from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig
parser = argparse.ArgumentParser()
@ -56,20 +22,11 @@ parser.add_argument("--vae-dtype", choices=("float32", "float16", "bfloat16"), d
parser.add_argument("--vae-tile-size", type=int, default=256)
parser.add_argument("--mlp-chunks", type=int, default=1)
parser.add_argument("--mlp-chunk-threshold", type=int, default=4096)
parser.add_argument("--turbo-lora", action="append", default=[], metavar="NAME=PATH", help="Load a resident 4step or 8step Turbo adapter.")
parser.add_argument("--latent-upscaler", help="Load the optional H3 3D latent upscaler for request-level spatial upscaling.")
parser.add_argument("--warmup", action="store_true", help="Run a tiny generation before accepting traffic.")
parser.add_argument("--warmup-output", type=Path, default=Path("/output/h3-blackwell-runtime/hot-runtime-warmup.mp4"))
args = parser.parse_args()
turbo_loras = []
for value in args.turbo_lora:
if "=" not in value:
parser.error("--turbo-lora must use NAME=PATH")
name, path = value.split("=", 1)
turbo_loras.append((name, path))
runtime = H3HotRuntime(RuntimeConfig(attention=args.attention, vae_dtype=args.vae_dtype, vae_tile_size=args.vae_tile_size, mlp_chunks=args.mlp_chunks, mlp_chunk_threshold=args.mlp_chunk_threshold, turbo_loras=tuple(turbo_loras), latent_upscaler_path=args.latent_upscaler))
runtime = H3HotRuntime(RuntimeConfig(attention=args.attention, vae_dtype=args.vae_dtype, vae_tile_size=args.vae_tile_size, mlp_chunks=args.mlp_chunks, mlp_chunk_threshold=args.mlp_chunk_threshold))
runtime_lock = threading.Lock()
warmup_result = None
if args.warmup:
@ -89,7 +46,6 @@ def service_status() -> dict:
return {
"ready": True,
"attention_backends": list(AVAILABLE_BACKENDS),
"attention_backend_status": attention_backend_status(),
"runtime": runtime.status(),
}
@ -127,39 +83,16 @@ class Handler(BaseHTTPRequestHandler):
width = int(payload.get("width", 960))
height = int(payload.get("height", 544))
frames = int(payload.get("frames", 124))
turbo = payload.get("turbo")
if turbo in {"", "none"}:
turbo = None
if turbo is not None and turbo not in runtime.turbo_loras:
write_json(self, 400, {"error": "unsupported turbo", "turbo": turbo, "available": sorted(runtime.turbo_loras)})
return
steps = int(payload.get("steps", TURBO_VARIANTS[turbo]["steps"] if turbo else 12))
if turbo is not None and steps != TURBO_VARIANTS[turbo]["steps"]:
write_json(self, 400, {"error": "invalid turbo steps", "turbo": turbo, "required_steps": TURBO_VARIANTS[turbo]["steps"]})
return
steps = int(payload.get("steps", 12))
seed = int(payload.get("seed", 440407))
attention = payload.get("attention")
if attention is not None and attention not in AVAILABLE_BACKENDS:
write_json(self, 400, {"error": "unsupported attention", "attention": attention, "available": list(AVAILABLE_BACKENDS)})
return
mux_audio = bool(payload.get("mux_audio", True))
keep_intermediates = bool(payload.get("keep_intermediates", False))
try:
upscale = normalize_upscale(payload.get("upscale"))
except ValueError as exc:
write_json(self, 400, {"error": "invalid upscale", "message": str(exc)})
return
if upscale is not None and runtime.latent_upscaler is None:
write_json(self, 400, {"error": "H3 latent upscaler is not loaded"})
return
ffmpeg_loglevel = payload.get("ffmpeg_loglevel", "error")
first_frame = _load_image(payload.get("first_frame"))
last_frame = _load_image(payload.get("last_frame"))
save_latent = payload.get("save_latent")
cache_mode = payload.get("cache_mode")
if turbo is not None and cache_mode not in {None, "", "disabled", "none"}:
write_json(self, 400, {"error": "turbo does not support denoiser caching", "turbo": turbo})
return
cache_threshold = float(payload.get("cache_threshold", 0.0))
cache_start_percent = float(payload.get("cache_start_percent", 0.0))
cache_end_percent = float(payload.get("cache_end_percent", 1.0))
@ -175,12 +108,7 @@ class Handler(BaseHTTPRequestHandler):
steps=steps,
seed=seed,
attention=attention,
turbo=turbo,
upscale=upscale,
first_frame=first_frame,
last_frame=last_frame,
mux_audio=mux_audio,
keep_intermediates=keep_intermediates,
ffmpeg_loglevel=ffmpeg_loglevel,
save_latent=save_latent,
cache_mode=cache_mode,

58
tools/smoke_attention_backends.py
View file

@ -1,58 +0,0 @@
"""Verify and time request-selectable H3 attention kernels on the active GPU."""
import argparse
import json
import time
import torch
from h3_blackwell_runtime.attention import run_attention
parser = argparse.ArgumentParser()
parser.add_argument("--backends", nargs="+", default=("sage2", "cudnn_sdpa", "ck_int8"))
parser.add_argument("--sequence", type=int, default=512)
parser.add_argument("--heads", type=int, default=56)
parser.add_argument("--head-dim", type=int, default=128)
parser.add_argument("--warmup", type=int, default=2)
parser.add_argument("--iterations", type=int, default=5)
parser.add_argument("--seed", type=int, default=440407)
args = parser.parse_args()
torch.manual_seed(args.seed)
q = torch.randn(1, args.heads, args.sequence, args.head_dim, device="cuda", dtype=torch.bfloat16)
results = {}
reference = None
with torch.inference_mode():
for backend in args.backends:
for _ in range(args.warmup):
output = run_attention(q, q, q, backend=backend, is_causal=False)
torch.cuda.synchronize()
elapsed = []
for _ in range(args.iterations):
started = time.perf_counter()
output = run_attention(q, q, q, backend=backend, is_causal=False)
torch.cuda.synchronize()
elapsed.append(time.perf_counter() - started)
if reference is None:
reference = output
delta = (output.float() - reference.float()).abs()
results[backend] = {
"mean_seconds": sum(elapsed) / len(elapsed),
"min_seconds": min(elapsed),
"finite": bool(torch.isfinite(output).all()),
"shape": list(output.shape),
"dtype": str(output.dtype),
"max_abs_vs_reference": delta.max().item(),
"mean_abs_vs_reference": delta.mean().item(),
}
print(json.dumps({
"gpu": torch.cuda.get_device_name(),
"torch": torch.__version__,
"cuda": torch.version.cuda,
"shape": list(q.shape),
"reference": args.backends[0],
"results": results,
}, indent=2))

50
tools/smoke_flash4.py
View file

@ -1,50 +0,0 @@
"""Compare FlashAttention-4 against PyTorch SDPA on an H3-shaped operation."""
import argparse
import json
import torch
parser = argparse.ArgumentParser()
parser.add_argument("--sequence", type=int, default=257)
parser.add_argument("--heads", type=int, default=8)
parser.add_argument("--head-dim", type=int, default=128)
parser.add_argument("--iterations", type=int, default=5)
args = parser.parse_args()
from flash_attn.cute import flash_attn_func
torch.manual_seed(440411)
shape = (1, args.sequence, args.heads, args.head_dim)
q = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
k = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
v = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
with torch.inference_mode():
result = flash_attn_func(q, k, v, causal=False)
actual = result[0] if isinstance(result, tuple) else result
expected = torch.nn.functional.scaled_dot_product_attention(
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=False,
).transpose(1, 2)
torch.cuda.synchronize()
started = torch.cuda.Event(enable_timing=True)
finished = torch.cuda.Event(enable_timing=True)
started.record()
for _ in range(args.iterations):
result = flash_attn_func(q, k, v, causal=False)
actual = result[0] if isinstance(result, tuple) else result
finished.record()
torch.cuda.synchronize()
error = (actual.float() - expected.float()).abs()
print(json.dumps({
"device": torch.cuda.get_device_name(),
"capability": torch.cuda.get_device_capability(),
"shape": tuple(actual.shape),
"dtype": str(actual.dtype),
"contiguous": actual.is_contiguous(),
"max_abs_error": error.max().item(),
"mean_abs_error": error.mean().item(),
"milliseconds": started.elapsed_time(finished) / args.iterations,
}, indent=2))

8
tools/smoke_qwen38_vllm_spark.sh
View file

@ -1,8 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
curl -fsS --max-time 120 http://127.0.0.1:8002/v1/chat/completions \
-H 'Content-Type: application/json' \
--data-binary @- <<'JSON'
{"model":"qwen38","messages":[{"role":"user","content":"Reply with exactly: ready"}],"max_tokens":64,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}
JSON

61
tools/spark_stack_spark.sh
View file

@ -1,61 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT=/home/daniel/aeon-spark-test/h3
STACK_DIR="$ROOT/spark-stack"
COMPOSE_FILE="$STACK_DIR/compose.spark-stack.yml"
usage() {
cat <<'EOF'
Usage:
spark_stack_spark.sh start h3|qwen|flux
spark_stack_spark.sh stop
spark_stack_spark.sh status
Ports:
h3 -> http://192.168.1.162:8001
qwen -> http://192.168.1.162:8002
flux -> http://192.168.1.162:8192
EOF
}
managed_containers=(
h3-hot-runtime
h3-blackwell-runtime-h3-hot-runtime-1
qwen38-vllm
flux2-klein-comfy
flux-nvfp4-test
)
stop_managed() {
for name in "${managed_containers[@]}"; do
if docker ps -a --format '{{.Names}}' | grep -qx "$name"; then
docker stop "$name" >/dev/null 2>&1 || true
docker rm "$name" >/dev/null 2>&1 || true
fi
done
}
case "${1:-}" in
start)
target="${2:-}"
case "$target" in
h3|qwen|flux) ;;
*) usage; exit 2 ;;
esac
mkdir -p "$ROOT/vllm-cache"
stop_managed
docker compose -f "$COMPOSE_FILE" --profile "$target" up -d
;;
stop)
stop_managed
;;
status)
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' | grep -E 'NAMES|h3-hot-runtime|qwen38-vllm|flux2-klein-comfy|flux-nvfp4-test' || true
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits || true
;;
*)
usage
exit 2
;;
esac

27
tools/start_qwen38_vllm_spark.sh
View file

@ -1,27 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT=/home/daniel/aeon-spark-test/h3
MODEL_DIR="$ROOT/models/Qwen3.8-27B-NVFP4"
COMPOSE_FILE="$ROOT/qwen38-vllm/compose.qwen38-vllm.yml"
if [[ ! -f "$MODEL_DIR/model.safetensors" ]]; then
echo "model.safetensors is not present yet: $MODEL_DIR/model.safetensors" >&2
echo "Wait for qwen38-27b-nvfp4-download to finish before starting vLLM." >&2
exit 1
fi
python3 - <<'PY'
import json
from pathlib import Path
tokenizer = Path('/home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4/tokenizer.json')
truncation = json.loads(tokenizer.read_text(encoding='utf-8')).get('truncation')
if truncation is not None:
raise SystemExit(f'tokenizer truncation must be null, got: {truncation!r}')
print('tokenizer truncation: null')
PY
mkdir -p "$ROOT/vllm-cache"
docker compose -f "$COMPOSE_FILE" up -d
docker logs -f qwen38-vllm

46
tools/submit_fl2va_keyframes.py
View file

@ -1,46 +0,0 @@
"""Submit a matched Comfy FL2VA first/last-frame reference workflow."""
import argparse
import json
from urllib.request import Request, urlopen
parser = argparse.ArgumentParser()
parser.add_argument("--url", default="http://localhost:8188")
parser.add_argument("--first", default="fl2va_key_first.png")
parser.add_argument("--last", default="fl2va_key_last.png")
parser.add_argument("--prefix", default="fl2va-comfy-keyframes-seed440207")
args = parser.parse_args()
prompt = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "minimax_h3_fl2va_pruned_nvfp4.safetensors", "weight_dtype": "default"}},
"3": {"class_type": "CLIPLoader", "inputs": {"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", "type": "minimax"}},
"4": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_video_vae_fp16.safetensors"}},
"5": {"class_type": "LoadImage", "inputs": {"image": args.first}},
"6": {"class_type": "LoadImage", "inputs": {"image": args.last}},
"8": {"class_type": "MiniMaxH3ImageToVideo", "inputs": {
"clip": ["3", 0],
"vae": ["4", 0],
"prompt": "A studio time-lapse of the same pink peony bud opening into the same fully bloomed pink peony, fixed camera, cream background.",
"width": 384,
"height": 384,
"length": 22,
"first_frame": ["5", 0],
"last_frame": ["6", 0],
}},
"9": {"class_type": "BasicGuider", "inputs": {"model": ["1", 0], "conditioning": ["8", 0]}},
"10": {"class_type": "RandomNoise", "inputs": {"noise_seed": 440207}},
"11": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "res_multistep"}},
"12": {"class_type": "BasicScheduler", "inputs": {"model": ["1", 0], "scheduler": "beta", "steps": 12, "denoise": 1.0}},
"13": {"class_type": "SamplerCustomAdvanced", "inputs": {"noise": ["10", 0], "guider": ["9", 0], "sampler": ["11", 0], "sigmas": ["12", 0], "latent_image": ["8", 1]}},
"14": {"class_type": "VAEDecode", "inputs": {"samples": ["13", 0], "vae": ["4", 0]}},
"15": {"class_type": "SaveImage", "inputs": {"images": ["14", 0], "filename_prefix": args.prefix}},
}
request = Request(
args.url.rstrip("/") + "/prompt",
data=json.dumps({"prompt": prompt}).encode(),
headers={"Content-Type": "application/json"},
)
with urlopen(request) as response:
print(response.read().decode())

91
tools/submit_h3_comfy_t2v.py
View file

@ -1,91 +0,0 @@
"""Submit a prompt-only MiniMax H3 T2V graph to ComfyUI."""
from __future__ import annotations
import argparse
import json
import time
from urllib.request import Request, urlopen
def post_json(url: str, payload: dict) -> dict:
request = Request(url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
with urlopen(request, timeout=30) as response:
return json.loads(response.read().decode())
def get_json(url: str) -> dict:
with urlopen(url, timeout=30) as response:
return json.loads(response.read().decode())
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--server", default="http://127.0.0.1:8188")
parser.add_argument("--prompt", required=True)
parser.add_argument("--filename-prefix", required=True)
parser.add_argument("--seed", type=int, default=440407)
parser.add_argument("--width", type=int, default=960)
parser.add_argument("--height", type=int, default=544)
parser.add_argument("--frames", type=int, default=124)
parser.add_argument("--steps", type=int, default=12)
parser.add_argument("--upscale", type=float)
args = parser.parse_args()
graph = {
"1": {"class_type": "UNETLoader", "inputs": {"unet_name": "minimax_h3_fl2va_pruned_nvfp4.safetensors", "weight_dtype": "default"}},
"3": {"class_type": "CLIPLoader", "inputs": {"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", "type": "minimax"}},
"4": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_video_vae_fp16.safetensors"}},
"5": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_audio_vae_fp32.safetensors"}},
"8": {"class_type": "MiniMaxH3ImageToVideo", "inputs": {"clip": ["3", 0], "vae": ["4", 0], "prompt": args.prompt, "width": args.width, "height": args.height, "length": args.frames}},
"9": {"class_type": "BasicGuider", "inputs": {"model": ["1", 0], "conditioning": ["8", 0]}},
"10": {"class_type": "RandomNoise", "inputs": {"noise_seed": args.seed}},
"11": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "res_multistep"}},
"12": {"class_type": "BasicScheduler", "inputs": {"model": ["1", 0], "scheduler": "beta", "steps": args.steps, "denoise": 1.0}},
"13": {"class_type": "SamplerCustomAdvanced", "inputs": {"noise": ["10", 0], "guider": ["9", 0], "sampler": ["11", 0], "sigmas": ["12", 0], "latent_image": ["8", 1]}},
}
decoded_latent = ["13", 0]
if args.upscale is not None:
graph.update({
"18": {"class_type": "LTXVSeparateAVLatent", "inputs": {"av_latent": ["13", 0]}},
"19": {"class_type": "MinimaxH3LatentUpscaler3D", "inputs": {
"latent": ["18", 0],
"model_name": "minimax_h3_latent_upscaler_3d_fp16.safetensors",
"mode": "scale by multiplier",
"mode.scale": args.upscale,
"align": 32,
"keep_proportion": True,
"device": "cuda",
"precision": "fp16",
}},
"20": {"class_type": "LTXVConcatAVLatent", "inputs": {"video_latent": ["19", 0], "audio_latent": ["18", 1]}},
})
decoded_latent = ["20", 0]
graph.update({
"14": {"class_type": "VAEDecode", "inputs": {"samples": decoded_latent, "vae": ["4", 0]}},
"15": {"class_type": "VAEDecodeAudio", "inputs": {"samples": decoded_latent, "vae": ["5", 0]}},
"16": {"class_type": "CreateVideo", "inputs": {"images": ["14", 0], "audio": ["15", 0], "bit_depth": 8, "fps": 24.0}},
"17": {"class_type": "SaveVideo", "inputs": {"video": ["16", 0], "filename_prefix": args.filename_prefix, "format": "mp4", "codec": "auto"}},
})
started = time.perf_counter()
response = post_json(f"{args.server}/prompt", {"prompt": graph})
prompt_id = response["prompt_id"]
while True:
time.sleep(1)
history = get_json(f"{args.server}/history/{prompt_id}").get(prompt_id)
if history is None:
continue
status = history.get("status", {})
if status.get("completed") or status.get("status_str") in {"success", "error"}:
break
print(json.dumps({
"prompt_id": prompt_id,
"wall_seconds": time.perf_counter() - started,
"status": history.get("status", {}),
"outputs": history.get("outputs", {}),
}, indent=2))
if __name__ == "__main__":
main()

186
tools/sweep_dialogue_audio_boundary.py
View file

@ -1,186 +0,0 @@
"""Run a paired tagged-versus-quoted H3 dialogue audio sweep."""
import argparse
import json
import math
import subprocess
import time
from pathlib import Path
import torch
from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig
from h3_blackwell_runtime.sampler import sample_video_res_multistep
from h3_blackwell_runtime.t2v import random_av_latents
SAMPLE_RATE = 32000
def dbfs(value: float) -> float:
return 20.0 * math.log10(max(value, 1e-20))
def waveform_metrics(waveform: torch.Tensor) -> dict:
waveform = waveform.float()
first_100ms = waveform[..., :3200]
next_400ms = waveform[..., 3200:16000]
first_500ms = waveform[..., :16000]
derivatives = (first_500ms[..., 1:] - first_500ms[..., :-1]).abs()
windows = waveform.unfold(-1, 320, 320)
window_rms = windows.square().mean(dim=(0, 2)).sqrt()
active = (20.0 * torch.log10(window_rms.clamp_min(1e-20)) > -40.0).nonzero()
first_active_ms = None if active.numel() == 0 else int(active[0, 0]) * 10
first_rms = dbfs(float(first_100ms.square().mean().sqrt()))
next_rms = dbfs(float(next_400ms.square().mean().sqrt()))
return {
"first_sample": waveform[..., 0].flatten().tolist(),
"first_100ms_peak_dbfs": dbfs(float(first_100ms.abs().max())),
"first_100ms_rms_dbfs": first_rms,
"next_400ms_peak_dbfs": dbfs(float(next_400ms.abs().max())),
"next_400ms_rms_dbfs": next_rms,
"boundary_decay_db": first_rms - next_rms,
"first_500ms_peak_dbfs": dbfs(float(first_500ms.abs().max())),
"first_500ms_rms_dbfs": dbfs(float(first_500ms.square().mean().sqrt())),
"full_peak_dbfs": dbfs(float(waveform.abs().max())),
"full_rms_dbfs": dbfs(float(waveform.square().mean().sqrt())),
"largest_first_500ms_derivative": float(derivatives.max()),
"first_10ms_window_above_minus_40_dbfs_ms": first_active_ms,
}
def latent_metrics(latent: torch.Tensor) -> dict:
frames = latent.float().movedim(-1, 0).flatten(1)
return {
"shape": list(latent.shape),
"first_4_rms": float(frames[:4].square().mean().sqrt()),
"frames_4_20_rms": float(frames[4:20].square().mean().sqrt()),
"first_frame_rms": float(frames[0].square().mean().sqrt()),
"frame_0_to_1_delta_rms": float((frames[1] - frames[0]).square().mean().sqrt()),
}
def write_waveform(path: Path, waveform: torch.Tensor) -> None:
raw = path.with_suffix(".f32le")
waveform.transpose(0, 1).contiguous().numpy().tofile(raw)
subprocess.run([
"ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(SAMPLE_RATE), "-ac", "2", "-i", str(raw),
"-c:a", "pcm_f32le", str(path),
], check=True)
raw.unlink()
def save_report(path: Path, report: dict) -> None:
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
temporary.replace(path)
parser = argparse.ArgumentParser()
parser.add_argument("--tagged-benchmark", type=Path, required=True)
parser.add_argument("--quoted-benchmark", type=Path, required=True)
parser.add_argument("--seed-start", type=int, default=440420)
parser.add_argument("--seed-count", type=int, default=10)
parser.add_argument("--output-dir", type=Path, required=True)
parser.add_argument("--report", type=Path, required=True)
parser.add_argument("--attention", default="sage2")
args = parser.parse_args()
tagged = json.loads(args.tagged_benchmark.read_text(encoding="utf-8"))
quoted = json.loads(args.quoted_benchmark.read_text(encoding="utf-8"))
for field in ("resolution", "frames", "steps"):
if tagged[field] != quoted[field]:
raise ValueError(f"benchmark {field} differs: {tagged[field]} != {quoted[field]}")
args.output_dir.mkdir(parents=True, exist_ok=True)
args.report.parent.mkdir(parents=True, exist_ok=True)
if args.report.exists():
report = json.loads(args.report.read_text(encoding="utf-8"))
else:
report = {
"tagged_benchmark": str(args.tagged_benchmark),
"quoted_benchmark": str(args.quoted_benchmark),
"attention": args.attention,
"seed_start": args.seed_start,
"seed_count": args.seed_count,
"cases": {},
"pairs": {},
}
runtime = H3HotRuntime(RuntimeConfig(attention=args.attention))
conditioned = {
"tagged": runtime.refiner(runtime.conditioner(tagged["prompt"])),
"quoted": runtime.refiner(runtime.conditioner(quoted["prompt"])),
}
width, height = tagged["resolution"]
for seed in range(args.seed_start, args.seed_start + args.seed_count):
for prompt_format, benchmark in (("tagged", tagged), ("quoted", quoted)):
key = f"{seed}:{prompt_format}"
if key in report["cases"]:
print(f"skip completed {key}", flush=True)
continue
started = time.perf_counter()
video, audio, aligned_frames = random_av_latents(
width, height, benchmark["frames"], seed, device=runtime.config.device,
)
sampled_video, audio_latent = sample_video_res_multistep(
runtime.model,
runtime.packer,
conditioned[prompt_format],
video,
audio,
steps=benchmark["steps"],
seed=seed,
return_audio=True,
)
with torch.inference_mode():
waveform = runtime.audio_vae.decode(
audio_latent.to("cuda", dtype=next(runtime.audio_vae.parameters()).dtype),
).cpu()[0]
stem = f"dialogue-{prompt_format}-base12-sage2-seed{seed}"
wav_path = args.output_dir / f"{stem}.wav"
latent_path = args.output_dir / f"{stem}.audio-latent.pt"
write_waveform(wav_path, waveform)
torch.save({
"audio_latent": audio_latent.detach().cpu(),
"prompt_format": prompt_format,
"prompt": benchmark["prompt"],
"seed": seed,
}, latent_path)
report["cases"][key] = {
"seed": seed,
"prompt_format": prompt_format,
"wav": str(wav_path),
"audio_latent": str(latent_path),
"frames": aligned_frames,
"seconds": time.perf_counter() - started,
"waveform": waveform_metrics(waveform),
"latent": latent_metrics(audio_latent.cpu()),
}
del sampled_video, audio_latent, waveform, video, audio
save_report(args.report, report)
print(json.dumps(report["cases"][key]), flush=True)
tagged_case = report["cases"][f"{seed}:tagged"]
quoted_case = report["cases"][f"{seed}:quoted"]
report["pairs"][str(seed)] = {
"quoted_peak_reduction_db": (
tagged_case["waveform"]["first_100ms_peak_dbfs"]
- quoted_case["waveform"]["first_100ms_peak_dbfs"]
),
"quoted_rms_reduction_db": (
tagged_case["waveform"]["first_100ms_rms_dbfs"]
- quoted_case["waveform"]["first_100ms_rms_dbfs"]
),
"tagged_boundary_decay_db": tagged_case["waveform"]["boundary_decay_db"],
"quoted_boundary_decay_db": quoted_case["waveform"]["boundary_decay_db"],
}
save_report(args.report, report)
print(json.dumps(report["pairs"], indent=2))

106
tools/trace_audio_denoising.py
View file

@ -1,106 +0,0 @@
"""Trace when an H3 audio-boundary artifact emerges during base sampling."""
import argparse
import json
import math
from pathlib import Path
import torch
from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig
from h3_blackwell_runtime.sampler import _decode_audio_latent, sample_video_res_multistep
from h3_blackwell_runtime.t2v import random_av_latents
def dbfs(value: float) -> float:
return 20.0 * math.log10(max(value, 1e-20))
def latent_metrics(latent: torch.Tensor) -> dict:
frames = latent.float().movedim(-1, 0).flatten(1)
return {
"first_4_rms": float(frames[:4].square().mean().sqrt()),
"frames_4_20_rms": float(frames[4:20].square().mean().sqrt()),
"first_frame_rms": float(frames[0].square().mean().sqrt()),
"frame_0_to_1_delta_rms": float((frames[1] - frames[0]).square().mean().sqrt()),
}
def waveform_metrics(waveform: torch.Tensor) -> dict:
waveform = waveform.float()
first_100ms = waveform[..., :3200]
first_500ms = waveform[..., :16000]
derivative = (first_500ms[..., 1:] - first_500ms[..., :-1]).abs()
return {
"first_sample": waveform[..., 0].flatten().tolist(),
"first_100ms_peak_dbfs": dbfs(float(first_100ms.abs().max())),
"first_100ms_rms_dbfs": dbfs(float(first_100ms.square().mean().sqrt())),
"first_500ms_peak_dbfs": dbfs(float(first_500ms.abs().max())),
"first_500ms_rms_dbfs": dbfs(float(first_500ms.square().mean().sqrt())),
"largest_derivative": float(derivative.max()),
}
parser = argparse.ArgumentParser()
parser.add_argument("--benchmark", type=Path, required=True)
parser.add_argument("--trace", type=Path, required=True)
parser.add_argument("--report", type=Path, required=True)
parser.add_argument("--attention", default="sage2")
args = parser.parse_args()
benchmark = json.loads(args.benchmark.read_text(encoding="utf-8"))
runtime = H3HotRuntime(RuntimeConfig(attention=args.attention))
video, initial_audio, aligned_frames = random_av_latents(
benchmark["resolution"][0],
benchmark["resolution"][1],
benchmark["frames"],
benchmark["seed"],
device=runtime.config.device,
)
text = runtime.refiner(runtime.conditioner(benchmark["prompt"]))
trace = []
video, final_audio = sample_video_res_multistep(
runtime.model,
runtime.packer,
text,
video,
initial_audio,
steps=benchmark["steps"],
seed=benchmark["seed"],
return_audio=True,
audio_step_trace=trace,
)
report = {
"benchmark": str(args.benchmark),
"attention": args.attention,
"seed": benchmark["seed"],
"frames": aligned_frames,
"steps": [],
}
with torch.inference_mode():
for entry in trace:
denoised = _decode_audio_latent(entry["audio_denoised"]).to(
"cuda", dtype=next(runtime.audio_vae.parameters()).dtype,
)
waveform = runtime.audio_vae.decode(denoised).cpu()[0]
report["steps"].append({
"step": entry["step"],
"video_sigma": entry["video_sigma"],
"audio_sigma": entry["audio_sigma"],
"latent": latent_metrics(denoised.cpu()),
"denoised_waveform": waveform_metrics(waveform),
})
args.trace.parent.mkdir(parents=True, exist_ok=True)
torch.save({
"initial_audio": initial_audio.detach().cpu(),
"final_audio": final_audio.detach().cpu(),
"steps": trace,
"prompt": benchmark["prompt"],
"seed": benchmark["seed"],
}, args.trace)
args.report.parent.mkdir(parents=True, exist_ok=True)
serialized = json.dumps(report, indent=2)
args.report.write_text(serialized + "\n", encoding="utf-8")
print(serialized)

55
tools/upscale_video_latent.py
View file

@ -1,55 +0,0 @@
"""Upscale a saved direct-runtime H3 latent with the learned 3D model."""
import argparse
import json
import time
from pathlib import Path
import torch
from h3_blackwell_runtime.latent_upscaler import load_h3_latent_upscaler, upscale_h3_latent
parser = argparse.ArgumentParser()
parser.add_argument("--latent", type=Path, required=True)
parser.add_argument("--model", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--scale", type=float, default=2.0)
parser.add_argument("--precision", choices=("float16", "bfloat16", "float32"), default="float16")
args = parser.parse_args()
dtype = getattr(torch, args.precision)
state = torch.load(args.latent, map_location="cpu", weights_only=False)
if not isinstance(state, dict) or "latent" not in state:
state = {"latent": state}
source = state["latent"]
torch.cuda.synchronize()
started = time.perf_counter()
model = load_h3_latent_upscaler(args.model, dtype=dtype)
torch.cuda.synchronize()
loaded = time.perf_counter()
upscaled = upscale_h3_latent(model, source, scale=args.scale).cpu()
torch.cuda.synchronize()
finished = time.perf_counter()
result = dict(state)
result["latent"] = upscaled
result["width"] = upscaled.shape[-1] * 16
result["height"] = upscaled.shape[-2] * 16
result["upscale"] = {
"model": args.model.name,
"scale": args.scale,
"precision": args.precision,
"source_shape": tuple(source.shape),
"output_shape": tuple(upscaled.shape),
}
args.output.parent.mkdir(parents=True, exist_ok=True)
torch.save(result, args.output)
print(json.dumps({
"output": str(args.output),
"source_shape": tuple(source.shape),
"output_shape": tuple(upscaled.shape),
"model_load_seconds": loaded - started,
"upscale_seconds": finished - loaded,
}, indent=2))