# H3 Runtime Current State

Status date: 2026-08-22

This document is the canonical snapshot of implemented scope and remaining work.
Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.

## Implemented And Validated

- Single-GPU prompt-only T2VA with joint video/audio generation.
- First-frame I2VA, last-frame L2VA, and first/last FL2VA through the shared
  keyframe-conditioning path.
- Qwen text and vision conditioning, token refinement, video VAE encoding, H3
  packed denoising, beta/RES sampling, video/audio decoding, and final MP4 mux.
- Resident HTTP runtime with warmup, readiness reporting, request-level backend
  selection, timing stages, optional latent saving, and diagnostic intermediates.
- SageAttention2 as the default quality backend.
- SDPA, forced cuDNN SDPA, FlashAttention-4, Sage3, Comfy Kitchen INT8, KJ Sage,
  head-sliced, and Sol-Attn experimental backends.
- Official FL2VA Turbo 4-step and 8-step adapters.
- Optional resident H3-native latent upscaling.
- Experimental EasyCache and H3-Cache delta-reuse modes.
- Quoted dialogue as the project prompt default. In a matched 10-seed test,
  quoted dialogue eliminated immediate first-100ms activity in all ten cases and
  all ten quoted WAVs passed subjective review.
- Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
- Sequence-sharded 50-block execution and distributed final projection.
- True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
- Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
- Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames,
  and 12 steps, including exact video and audio latent equality.
- Matched one-GPU RTX PRO 6000 Blackwell Server SDPA sampling averaged `28.50s`
  over two runs versus `126.66s` for the same tensor runner on GB10 (`4.44x`).
  RTX repeat variance was `0.49%` and checksums were identical between repeats.
- Two-GPU Ulysses SDPA speedup grows with sequence size: `1.21x` at
  864x480/141 frames, `1.55x` at 1344x768/124 frames, and `1.72x` at
  1344x768/243 frames. The tested cards have no NVLink; P2P read/write is
  available and NCCL uses `P2P/CUMEM`.

## Primary Missing Scope

### Full Ref2VA

- Arbitrary reference image, video, and audio inputs.
- Reference-audio encoder and reference soundtrack conditioning.
- Reference identity/voice blocks in the standalone packer.
- Ref2VA position, modality, and scheduling contracts.
- Direct-versus-Comfy full Ref2VA per-step and final-output parity benchmark.

### Explicit Task API

- Named `task` selection for T2VA, I2VA, L2VA, FL2VA, and Ref2VA.
- Mode-specific request schemas and incompatible-input validation.
- Intermediate keyframe anchors beyond the current first/last restriction.

### Distributed Execution

- Output parity above one rank; real two-rank NCCL transport is validated.
- 2/4/6/8-GPU topology and performance sweeps on one Blackwell machine.
- Distributed resident-service orchestration; the current launcher is batch
  generation through `torchrun`.
- x86 SageAttention2 packaging; RunPod validation initially uses SDPA.

### Owned Performance Kernels

- H3-specific attention backend optimized for real GB10 tensor shapes.
- Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
- CUDA graph capture and shape buckets.
- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.

## Quality Work Remaining

- Generate full quoted-dialogue videos and validate wording, voice consistency,
  speech timing, and lip-sync before closing the startup-audio work.
- Complete strict per-step LightX2V parity for Turbo adapters.
- Add real-adapter Turbo end-to-end fixtures.
- Add the optional target-resolution refinement stage after latent upscaling.
- Resolve or formally bound upscaler ringing, texture, chromatic-edge, and
  identity changes.
- Run full-size cache threshold and quality sweeps before enabling caches for
  production output.
- Keep Sage3, Sol-Attn, INT8, and other approximate backends quality-gated.
- Fix the inactive fused Sol QKV-layout path, which currently references an
  undefined `qkv` value. The deployed native Sol layout does not use this path.

## Production Work Remaining

- Asynchronous jobs, queueing, progress, cancellation, and timeouts.
- Strict request validation, including Boolean fields and mode combinations.
- Input/output path sandboxing, request-size limits, authentication, and TLS.
- Configurable FPS, video codec, audio codec, sample rate, and media policy.
- Container healthcheck, restart policy, resource limits, durable structured
  request logs, and runtime metrics.
- Batch generation and an intentional worker/concurrency model.

## Validation And Packaging Gaps

- GPU end-to-end fixtures for T2VA, I2VA, L2VA, and FL2VA.
- Full Ref2VA, AudioVAE waveform, cache, HTTP API, real Turbo, real upscaler,
  attention-quality, CUDA-graph, and distributed tests.
- Reproducible local fixtures for parity evidence currently stored on Spark/SMB.
- Explicit package declarations/checks for NumPy, SciPy, Pillow, and FFmpeg.
- A standalone base image if removing the Comfy-derived image becomes a product
  requirement; the current denoising path still intentionally uses Comfy Kitchen
  kernels.
- Align Docker `H3_MODEL_PATH` and `RuntimeConfig`; the environment variable is
  currently not consumed by the runtime default.

## Recommended Execution Order

1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
2. Correct the inactive fused Sol path.
3. Add explicit task schemas and automated single-GPU mode tests.
4. Implement full Ref2VA, including reference-audio encoding.
5. Build the H3-specific attention backend and CUDA graph buckets.
6. Harden the service API and operational deployment.
7. Complete RunPod NCCL validation and distributed scaling benchmarks.

The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is
implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still
blocked on an eight-GPU host. The other largest gap is standalone Ref2VA.