h3-blackwell-runtime/README.md

# H3 Blackwell Runtime

Direct MiniMax H3 audiovisual inference for NVIDIA Blackwell and Grace Blackwell.
The runtime consumes the current Comfy-format NVFP4 checkpoints while keeping
ComfyUI out of the inference critical path. ComfyUI remains the checkpoint and
correctness oracle.

## Current Status

Implemented and validated:

- Prompt-only T2VA with jointly generated video and stereo audio.
- First-frame I2VA, last-frame L2VA, and first/last-frame FL2VA conditioning.
- Qwen3-VL text and image conditioning, token refinement, H3 packed denoising,
  beta/RES sampling, video and audio VAE decode, H.264/AAC encoding, and muxing.
- Resident HTTP runtime with startup warmup, model reuse, request timing, and
  request-level attention selection.
- SageAttention2 correctness default plus SDPA, forced cuDNN SDPA,
  FlashAttention-4, Sage3, KJ Sage, Comfy Kitchen INT8, head-sliced, and Sol-Attn
  experimental backends.
- Official FL2VA Turbo 4-step and 8-step adapters.
- Optional H3-native 3D latent upscaling.
- Opt-in EasyCache and H3-Cache experiments.
- Ragged Ulysses sequence parallelism and true NVFP4 tensor parallelism with
  automatic 1/2/4/6/8-GPU launch tooling.

The main remaining feature gap is full arbitrary Ref2VA, especially reference
video/audio, identity, and voice conditioning. Multi-GPU code is CPU- and
one-GPU-validated; real 2/4/6/8-GPU NCCL scaling measurements are still pending.
See [`CURRENT_STATE.md`](CURRENT_STATE.md) for the canonical detailed status.

## Checkpoints

The default runtime paths are:

```text
/models/minimax_h3_fl2va_pruned_nvfp4.safetensors
/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors
/vae/ae.safetensors
/vae/mini_vae.safetensors
```

The denoiser and Qwen checkpoints use Comfy Kitchen NVFP4 layouts. Do not
convert or dequantize them during loading.

## DGX Spark

`Dockerfile.spark` and `compose.spark.yml` provide the ARM64 CUDA 13 runtime used
on GB10. Build and start the resident service from the Spark checkout:

```bash
docker compose -f compose.spark.yml build h3-hot-runtime
docker compose -f compose.spark.yml up -d h3-hot-runtime
curl http://127.0.0.1:8001/ready
```

The service listens on container port 8000 and Spark host port 8001. It keeps
Qwen, H3, both VAEs, Turbo adapters, and the optional latent upscaler resident.
SageAttention2 is the default because it matches the established Comfy quality
baseline.

## HTTP API

`GET /health` and `GET /ready` report readiness, loaded options, warmup results,
and attention backend status. `POST /generate` performs one serialized request:

```bash
curl -X POST http://127.0.0.1:8001/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "A quiet medium shot of a woman by a rain-streaked window. She says, \"We should leave before dawn.\"",
    "output": "/output/h3-blackwell-runtime/example.mp4",
    "width": 864,
    "height": 480,
    "frames": 141,
    "steps": 12,
    "seed": 440420,
    "attention": "sage2"
  }'
```

Optional request fields include `first_frame`, `last_frame`, `turbo`, `upscale`,
`mux_audio`, `save_latent`, `keep_intermediates`, `cache_mode`, and cache tuning
parameters. Keyframes accept an on-disk image path, base64 payload, or data URL.
The presence of first and last frames selects I2VA, L2VA, or FL2VA behavior; a
named task field is not yet exposed.

Set `turbo` to `"4step"` or `"8step"`; the service enforces the corresponding
step count and Turbo schedule. Set `upscale` to `2.0` to run the resident learned
latent upscaler before video decode. Turbo and denoiser caching cannot be
combined.

Quoted dialogue is the project prompt default. Tagged `<d>[English]...` dialogue
is a repeatable startup-audio trigger and should not be used as the default.
Never apply unconditional audio trimming or fading because valid sound can begin
at the first sample. See [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md) and
[`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md).

## Distributed Execution

Two batch-generation modes are available through `torchrun`:

- `ulysses`: token-sharded blocks with ragged sequence-to-head all-to-all around
  attention.
- `tensor`: sequence-sharded residuals plus NVFP4-sharded QKV, attention output,
  MLP FC1, and MLP FC2 projections.

Use all visible GPUs or pass an explicit world size:

```bash
tools/run_distributed_t2va.sh ulysses
tools/run_distributed_t2va.sh tensor 8 sdpa
H3_SAVE_LATENTS=0 tools/run_distributed_matrix.sh
```

The matrix runs feasible 1/2/4/6/8-GPU configurations and skips counts larger
than the visible device count. Six ranks use ragged head ownership
`[10, 10, 9, 9, 9, 9]`; no semantic token padding is introduced.

See [`DISTRIBUTED.md`](DISTRIBUTED.md) for the collective contracts, validation
evidence, environment variables, RunPod provisioning client, and x86 packaging
status.

## Validation

Run the contract suite with:

```bash
python -m unittest discover -s tests -v
```

The current suite has 38 passing tests. Distributed tests cover 2/4/6/8-rank
transport identity, ragged collectives, SDPA parity, final projection parity,
NVFP4 shard layout, and 2/6-rank TP attention/MLP math. On GB10, real-checkpoint
one-rank Ulysses and tensor paths produced exactly equal video and audio latents
at 864x480, 141 frames, 12 steps, seed 440420.

A matched one-GPU RunPod RTX PRO 6000 Blackwell Server SDPA test averaged
`28.50s` sampling over two runs versus `126.66s` for the same tensor runner on
GB10, a `4.44x` speedup. The two RTX runs were within `0.49%` and produced
identical checksums. See
`benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`.

Two-GPU Ulysses scaling improves as the packed sequence grows: `1.21x` at
864x480/141 frames, `1.55x` at 1344x768/124 frames, and `1.72x` at
1344x768/243 frames. These RTX PRO 6000 Server cards have no NVLink. CUDA P2P
read/write is available and NCCL selected `P2P/CUMEM`; topology varied between
`SYS` and `NODE` across RunPod allocations. See
`benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`.

Matched GB10 backend results and parity evidence are recorded in [`PLAN.md`](PLAN.md)
and [`PARITY.md`](PARITY.md). Standalone `tools/compare_*`, `tools/trace_*`,
`tools/inspect_*`, and `tools/patch_comfy_*` commands are diagnostic utilities,
not runtime startup requirements.

## Documentation

- [`CURRENT_STATE.md`](CURRENT_STATE.md): canonical implemented and missing scope.
- [`DISTRIBUTED.md`](DISTRIBUTED.md): Ulysses, TP, launchers, and RunPod workflow.
- [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md): audiovisual prompting contract.
- [`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md): startup
  audio localization and prompt-format evidence.
- [`TURBO.md`](TURBO.md): official Turbo adapters and schedules.
- [`H3_LATENT_UPSCALER.md`](H3_LATENT_UPSCALER.md): learned latent upscaler.
- [`FLASH4.md`](FLASH4.md): FlashAttention-4 integration and benchmark.
- [`PARITY.md`](PARITY.md): direct-versus-Comfy evidence ledger.
- [`PLAN.md`](PLAN.md): historical investigation and future kernel plan.

## Known Gaps

- Full arbitrary Ref2VA reference video/audio and identity/voice conditioning.
- Explicit task schemas and stricter production request validation.
- Real multi-GPU NCCL parity and 1/2/4/6/8 scaling results.
- x86 SageAttention2 packaging for the generic RunPod image; SDPA is the initial
  cloud validation backend.
- Production queueing, cancellation, authentication, TLS, metrics, and durable
  job state.
- Full quality sweeps for approximate attention, cache, Turbo, and upscaler paths.