121 lines
6 KiB
Markdown
121 lines
6 KiB
Markdown
# H3 Runtime Current State
|
|
|
|
Status date: 2026-08-22
|
|
|
|
This document is the canonical snapshot of implemented scope and remaining work.
|
|
Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
|
|
|
|
## Implemented And Validated
|
|
|
|
- Single-GPU prompt-only T2VA with joint video/audio generation.
|
|
- First-frame I2VA, last-frame L2VA, and first/last FL2VA through the shared
|
|
keyframe-conditioning path.
|
|
- Qwen text and vision conditioning, token refinement, video VAE encoding, H3
|
|
packed denoising, beta/RES sampling, video/audio decoding, and final MP4 mux.
|
|
- Resident HTTP runtime with warmup, readiness reporting, request-level backend
|
|
selection, timing stages, optional latent saving, and diagnostic intermediates.
|
|
- SageAttention2 as the default quality backend.
|
|
- SDPA, forced cuDNN SDPA, FlashAttention-4, Sage3, Comfy Kitchen INT8, KJ Sage,
|
|
head-sliced, and Sol-Attn experimental backends.
|
|
- Official FL2VA Turbo 4-step and 8-step adapters.
|
|
- Optional resident H3-native latent upscaling.
|
|
- Experimental EasyCache and H3-Cache delta-reuse modes.
|
|
- Quoted dialogue as the project prompt default. In a matched 10-seed test,
|
|
quoted dialogue eliminated immediate first-100ms activity in all ten cases and
|
|
all ten quoted WAVs passed subjective review.
|
|
- Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
|
|
- Sequence-sharded 50-block execution and distributed final projection.
|
|
- True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
|
|
- Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
|
|
- Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames,
|
|
and 12 steps, including exact video and audio latent equality.
|
|
- Matched one-GPU RTX PRO 6000 Blackwell Server SDPA sampling averaged `28.50s`
|
|
over two runs versus `126.66s` for the same tensor runner on GB10 (`4.44x`).
|
|
RTX repeat variance was `0.49%` and checksums were identical between repeats.
|
|
- Two-GPU Ulysses SDPA speedup grows with sequence size: `1.21x` at
|
|
864x480/141 frames, `1.55x` at 1344x768/124 frames, and `1.72x` at
|
|
1344x768/243 frames. The tested cards have no NVLink; P2P read/write is
|
|
available and NCCL uses `P2P/CUMEM`.
|
|
|
|
## Primary Missing Scope
|
|
|
|
### Full Ref2VA
|
|
|
|
- Arbitrary reference image, video, and audio inputs.
|
|
- Reference-audio encoder and reference soundtrack conditioning.
|
|
- Reference identity/voice blocks in the standalone packer.
|
|
- Ref2VA position, modality, and scheduling contracts.
|
|
- Direct-versus-Comfy full Ref2VA per-step and final-output parity benchmark.
|
|
|
|
### Explicit Task API
|
|
|
|
- Named `task` selection for T2VA, I2VA, L2VA, FL2VA, and Ref2VA.
|
|
- Mode-specific request schemas and incompatible-input validation.
|
|
- Intermediate keyframe anchors beyond the current first/last restriction.
|
|
|
|
### Distributed Execution
|
|
|
|
- Output parity above one rank; real two-rank NCCL transport is validated.
|
|
- 2/4/6/8-GPU topology and performance sweeps on one Blackwell machine.
|
|
- Distributed resident-service orchestration; the current launcher is batch
|
|
generation through `torchrun`.
|
|
- x86 SageAttention2 packaging; RunPod validation initially uses SDPA.
|
|
|
|
### Owned Performance Kernels
|
|
|
|
- H3-specific attention backend optimized for real GB10 tensor shapes.
|
|
- Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
|
|
- CUDA graph capture and shape buckets.
|
|
- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.
|
|
|
|
## Quality Work Remaining
|
|
|
|
- Generate full quoted-dialogue videos and validate wording, voice consistency,
|
|
speech timing, and lip-sync before closing the startup-audio work.
|
|
- Complete strict per-step LightX2V parity for Turbo adapters.
|
|
- Add real-adapter Turbo end-to-end fixtures.
|
|
- Add the optional target-resolution refinement stage after latent upscaling.
|
|
- Resolve or formally bound upscaler ringing, texture, chromatic-edge, and
|
|
identity changes.
|
|
- Run full-size cache threshold and quality sweeps before enabling caches for
|
|
production output.
|
|
- Keep Sage3, Sol-Attn, INT8, and other approximate backends quality-gated.
|
|
- Fix the inactive fused Sol QKV-layout path, which currently references an
|
|
undefined `qkv` value. The deployed native Sol layout does not use this path.
|
|
|
|
## Production Work Remaining
|
|
|
|
- Asynchronous jobs, queueing, progress, cancellation, and timeouts.
|
|
- Strict request validation, including Boolean fields and mode combinations.
|
|
- Input/output path sandboxing, request-size limits, authentication, and TLS.
|
|
- Configurable FPS, video codec, audio codec, sample rate, and media policy.
|
|
- Container healthcheck, restart policy, resource limits, durable structured
|
|
request logs, and runtime metrics.
|
|
- Batch generation and an intentional worker/concurrency model.
|
|
|
|
## Validation And Packaging Gaps
|
|
|
|
- GPU end-to-end fixtures for T2VA, I2VA, L2VA, and FL2VA.
|
|
- Full Ref2VA, AudioVAE waveform, cache, HTTP API, real Turbo, real upscaler,
|
|
attention-quality, CUDA-graph, and distributed tests.
|
|
- Reproducible local fixtures for parity evidence currently stored on Spark/SMB.
|
|
- Explicit package declarations/checks for NumPy, SciPy, Pillow, and FFmpeg.
|
|
- A standalone base image if removing the Comfy-derived image becomes a product
|
|
requirement; the current denoising path still intentionally uses Comfy Kitchen
|
|
kernels.
|
|
- Align Docker `H3_MODEL_PATH` and `RuntimeConfig`; the environment variable is
|
|
currently not consumed by the runtime default.
|
|
|
|
## Recommended Execution Order
|
|
|
|
1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
|
|
2. Correct the inactive fused Sol path.
|
|
3. Add explicit task schemas and automated single-GPU mode tests.
|
|
4. Implement full Ref2VA, including reference-audio encoding.
|
|
5. Build the H3-specific attention backend and CUDA graph buckets.
|
|
6. Harden the service API and operational deployment.
|
|
7. Complete RunPod NCCL validation and distributed scaling benchmarks.
|
|
|
|
The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is
|
|
implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still
|
|
blocked on an eight-GPU host. The other largest gap is standalone Ref2VA.
|