h3-blackwell-runtime/CURRENT_STATE.md

H3 Runtime Current State

Status date: 2026-08-22

This document is the canonical snapshot of implemented scope and remaining work. Historical handoffs in PLAN.md and PARITY.md may describe older states.

Implemented And Validated

• Single-GPU prompt-only T2VA with joint video/audio generation.
• First-frame I2VA, last-frame L2VA, and first/last FL2VA through the shared keyframe-conditioning path.
• Qwen text and vision conditioning, token refinement, video VAE encoding, H3 packed denoising, beta/RES sampling, video/audio decoding, and final MP4 mux.
• Resident HTTP runtime with warmup, readiness reporting, request-level backend selection, timing stages, optional latent saving, and diagnostic intermediates.
• SageAttention2 as the default quality backend.
• SDPA, forced cuDNN SDPA, FlashAttention-4, Sage3, Comfy Kitchen INT8, KJ Sage, head-sliced, and Sol-Attn experimental backends.
• Official FL2VA Turbo 4-step and 8-step adapters.
• Optional resident H3-native latent upscaling.
• Experimental EasyCache and H3-Cache delta-reuse modes.
• Quoted dialogue as the project prompt default. In a matched 10-seed test, quoted dialogue eliminated immediate first-100ms activity in all ten cases and all ten quoted WAVs passed subjective review.
• Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
• Sequence-sharded 50-block execution and distributed final projection.
• True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
• Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
• Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames, and 12 steps, including exact video and audio latent equality.
• Matched one-GPU RTX PRO 6000 Blackwell Server SDPA sampling averaged 28.50s over two runs versus 126.66s for the same tensor runner on GB10 (4.44x). RTX repeat variance was 0.49% and checksums were identical between repeats.

Primary Missing Scope

Full Ref2VA

• Arbitrary reference image, video, and audio inputs.
• Reference-audio encoder and reference soundtrack conditioning.
• Reference identity/voice blocks in the standalone packer.
• Ref2VA position, modality, and scheduling contracts.
• Direct-versus-Comfy full Ref2VA per-step and final-output parity benchmark.

Explicit Task API

• Named task selection for T2VA, I2VA, L2VA, FL2VA, and Ref2VA.
• Mode-specific request schemas and incompatible-input validation.
• Intermediate keyframe anchors beyond the current first/last restriction.

Distributed Execution

• Real NCCL transport and output parity above one rank.
• 2/4/6/8-GPU topology and performance sweeps on one Blackwell machine.
• Distributed resident-service orchestration; the current launcher is batch generation through torchrun.
• x86 SageAttention2 packaging; RunPod validation initially uses SDPA.

Owned Performance Kernels

• H3-specific attention backend optimized for real GB10 tensor shapes.
• Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
• CUDA graph capture and shape buckets.
• Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.

Quality Work Remaining

• Generate full quoted-dialogue videos and validate wording, voice consistency, speech timing, and lip-sync before closing the startup-audio work.
• Complete strict per-step LightX2V parity for Turbo adapters.
• Add real-adapter Turbo end-to-end fixtures.
• Add the optional target-resolution refinement stage after latent upscaling.
• Resolve or formally bound upscaler ringing, texture, chromatic-edge, and identity changes.
• Run full-size cache threshold and quality sweeps before enabling caches for production output.
• Keep Sage3, Sol-Attn, INT8, and other approximate backends quality-gated.
• Fix the inactive fused Sol QKV-layout path, which currently references an undefined qkv value. The deployed native Sol layout does not use this path.

Production Work Remaining

• Asynchronous jobs, queueing, progress, cancellation, and timeouts.
• Strict request validation, including Boolean fields and mode combinations.
• Input/output path sandboxing, request-size limits, authentication, and TLS.
• Configurable FPS, video codec, audio codec, sample rate, and media policy.
• Container healthcheck, restart policy, resource limits, durable structured request logs, and runtime metrics.
• Batch generation and an intentional worker/concurrency model.

Validation And Packaging Gaps

• GPU end-to-end fixtures for T2VA, I2VA, L2VA, and FL2VA.
• Full Ref2VA, AudioVAE waveform, cache, HTTP API, real Turbo, real upscaler, attention-quality, CUDA-graph, and distributed tests.
• Reproducible local fixtures for parity evidence currently stored on Spark/SMB.
• Explicit package declarations/checks for NumPy, SciPy, Pillow, and FFmpeg.
• A standalone base image if removing the Comfy-derived image becomes a product requirement; the current denoising path still intentionally uses Comfy Kitchen kernels.
• Align Docker H3_MODEL_PATH and RuntimeConfig; the environment variable is currently not consumed by the runtime default.

Recommended Execution Order

1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
2. Correct the inactive fused Sol path.
3. Add explicit task schemas and automated single-GPU mode tests.
4. Implement full Ref2VA, including reference-audio encoding.
5. Build the H3-specific attention backend and CUDA graph buckets.
6. Harden the service API and operational deployment.
7. Complete RunPod NCCL validation and distributed scaling benchmarks.

The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still blocked on an eight-GPU host. The other largest gap is standalone Ref2VA.