694 lines
30 KiB
Markdown
694 lines
30 KiB
Markdown
|
# Buyer Questions: Current Evidence Status
|
||
|
|
|
||
|
|
**Internal - do not share without review**
|
||
|
|
**Evidence date:** 2026-08-25
|
||
|
|
**Purpose:** Track what Vortex can currently answer, what is only partially evidenced, and what remains open.
|
||
|
|
|
||
|
|
## Status Summary
|
||
|
|
|
||
|
|
| Status | Count | Meaning |
|
||
|
|
|---|---:|---|
|
||
|
|
| Answerable | 1 | The current product state supports a direct, evidence-backed answer. The answer may be negative. |
|
||
|
|
| Partially answerable | 13 | Useful measurements or implementation facts exist, but they do not completely answer the buyer's exact question. |
|
||
|
|
| Open | 11 | The required comparison, integration, evaluation, or operating evidence does not exist. |
|
||
|
|
| **Total** | **25** | 14 performance questions and 11 quality questions. |
|
||
|
|
|
||
|
|
These classifications measure whether the buyer's full question can be answered, not whether Vortex has done related technical work. Component-level checksums and kernel timings are useful evidence, but they are not substitutes for end-to-end performance, reliability, or perceptual-quality evidence.
|
||
|
|
|
||
|
|
## Evidence Rules
|
||
|
|
|
||
|
|
- Same-workload cross-hardware measurements must not be presented as Vortex-versus-official-runtime comparisons.
|
||
|
|
- Scalar checksums do not prove complete tensor equality.
|
||
|
|
- Kernel and block parity do not prove final video or audio parity.
|
||
|
|
- Native sampling resolution must be distinguished from latent-upscaled output.
|
||
|
|
- A demonstrated workload is not a proven maximum-safe workload.
|
||
|
|
- Projected performance is not measured performance.
|
||
|
|
- Historical Markdown can describe superseded states. `CURRENT_STATE.md` is the canonical status source where documents conflict.
|
||
|
|
- Much of the newest evidence is currently preserved in the working tree rather than an earlier Git commit. Release claims should cite the eventual preservation commit.
|
||
|
|
|
||
|
|
# Performance Questions
|
||
|
|
|
||
|
|
## 1. What models are supported?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
The implemented runtime currently supports the MiniMax H3 stack centered on the pruned FL2VA NVFP4 denoiser and Qwen3-VL 32B NVFP4-AWQ conditioner. Implemented H3 modes include prompt-only T2VA, first-frame I2VA, last-frame L2VA, first/last-frame FL2VA, joint video and stereo audio, base beta/RES sampling, official Turbo 4-step v1.1 and 8-step v1.0 adapters, and an optional learned latent upscaler.
|
||
|
|
|
||
|
|
Full arbitrary Ref2VA video, audio, identity, and voice conditioning is not implemented.
|
||
|
|
|
||
|
|
This does not meet the required buyer-facing model scope. The support matrix must also include:
|
||
|
|
|
||
|
|
- Wan, with exact model families, variants, revisions, and task modes still to be selected.
|
||
|
|
- LTX 2.3.
|
||
|
|
- LTX 2.5.
|
||
|
|
- Any additional strategically selected model family.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `README.md`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
- `TURBO.md`
|
||
|
|
- `H3_LATENT_UPSCALER.md`
|
||
|
|
- `src/h3_blackwell_runtime/runtime.py`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no implemented or validated Vortex runtime for the required Wan and LTX versions. The phrase "supports Wan" is also too broad until exact checkpoints and modes are pinned.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Define a versioned model-support contract. For every selected model, record checkpoint revision and SHA-256, T2V/I2V capabilities, audio support, precision modes, valid step counts, native resolutions, duration limits, supported GPUs, parity status, and measured performance. Then integrate and validate Wan and LTX 2.3/2.5 against that contract.
|
||
|
|
|
||
|
|
## 2. What is Vortex versus the official H3 runtime on the same RTX PRO 6000?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No same-RTX-PRO-6000 comparison against the official H3 runtime exists. The measured `4.44x` result compares the same direct SDPA workload on RTX PRO 6000 and GB10. It is a cross-hardware result, not a Vortex-versus-official-runtime result.
|
||
|
|
|
||
|
|
The closest older same-hardware evidence is a GB10 direct-versus-Comfy warm comparison around `149.304s` versus `150.26s` at 960x544x124. It is not an isolated RTX comparison and does not establish a Vortex advantage.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`
|
||
|
|
- `PARITY.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no licensed or pinned official runtime running beside Vortex on the same RTX GPU, checkpoint, prompt, seed, precision, attention backend, and output settings.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run alternated cold and warm trials of official H3 and Vortex on one RTX PRO 6000. Retain full stage timings, final video/audio latents, decoded media, environment versions, and checkpoint hashes.
|
||
|
|
|
||
|
|
## 3. What is Vortex versus reference on the same B200 or GB300 GPU?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
B200 has been tested only as a hardware comparison against RTX PRO 6000 using the same direct SDPA runner. At 1344x768x124, B200 sampling was `79.7213s` versus RTX `97.9439s`, a `1.2286x` speedup. At 1344x768x243, B200 was `257.8620s` versus RTX `311.0163s`, a `1.2061x` speedup. Under the recorded provider prices, B200 sampling cost was approximately `2.64-2.69x` the RTX cost.
|
||
|
|
|
||
|
|
These results do not compare Vortex with the official runtime on B200. No GB300 evidence exists.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no same-B200 official-runtime baseline, no architecture-matched attribution explaining the modest B200 speedup, and no GB300 run.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run official H3 and Vortex on the same B200, then repeat the unchanged protocol on GB300. Record clocks, power, kernels, precision, complete timings, latents, and media.
|
||
|
|
|
||
|
|
## 4. How much gain comes from hardware, NVFP4, attention, fusion, runtime residency, and distributed execution?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
GB10 block composition and several isolated software gains are measured. In a fully fused block-24 profile, Sage2 represented approximately `57.60%` of block time, NVFP4 GEMMs `26.87%`, NVFP4 scale and packing `8.92%`, norm/RoPE `4.27%`, and gates `2.35%`.
|
||
|
|
|
||
|
|
Measured controlled improvements include:
|
||
|
|
|
||
|
|
- Fused elementwise path: approximately `3.0%` over the 12-step run and about 390 MiB lower peak allocation.
|
||
|
|
- Strided-NHD QKV copy removal: approximately `3.6%` over its baseline.
|
||
|
|
- NVFP4 modulation fusion: approximately `0.56%`.
|
||
|
|
- NVFP4 SwiGLU producer fusion: approximately `4.07%`.
|
||
|
|
- Two-GPU RTX Ulysses: `1.21x`, `1.55x`, and `1.72x` as workload size increased.
|
||
|
|
|
||
|
|
These numbers do not form a complete additive decomposition. NVFP4's share of runtime is not the gain caused by NVFP4, and independently measured percentages cannot safely be summed.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/gb10-fully-fused-fresh-nsight-summary.json`
|
||
|
|
- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`
|
||
|
|
- `benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json`
|
||
|
|
- `benchmarks/gb10-nvfp4-modulate-fusion-summary.json`
|
||
|
|
- `benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`
|
||
|
|
- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no controlled BF16-versus-NVFP4 model comparison, no complete residency ablation, and no factorial matrix across hardware, attention, fusion, precision, and distribution.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Add explicit component toggles and run a fixed hardware x precision x attention x fusion x residency x GPU-count matrix with parity gates and uninstrumented end-to-end timing.
|
||
|
|
|
||
|
|
## 5. What are cold-start and warm-start times?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
A historical GB10 deployment recorded approximately `30.73s` for startup warmup and `2.322s` for a later resident 320x192x22 smoke request. RTX H3 model loading was approximately `4.33-4.82s`; B200 model loading was approximately `10.71-11.83s`.
|
||
|
|
|
||
|
|
These values measure different scopes and workloads. They do not provide controlled process-launch-to-ready, first-request, and warmed-request timings for one configuration.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/gb10-post-lora-revert-deployment-smoke.json`
|
||
|
|
- `benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json`
|
||
|
|
- `benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json`
|
||
|
|
- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
Container startup, model loading, CUDA compilation, warmup, conditioning, sampling, decoding, encoding, and muxing have not been separated under one controlled protocol.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Restart the service and issue one fixed cold request followed by at least ten identical warm requests. Record every stage separately.
|
||
|
|
|
||
|
|
## 6. What are p50, p95, and p99 latencies?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Percentiles exist for selected blocks and kernels. For example, fully fused GB10 block-24 p50 was approximately `458.775ms`. Some three-iteration component artifacts report interpolated p95 and p99 values, and a 50-sample Sage2 study reports stable isolated medians.
|
||
|
|
|
||
|
|
There are no buyer-grade service-request or end-to-end p50, p95, and p99 measurements. Microkernel percentiles must not be presented as request latency.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/gb10-fully-fused-fresh-nsight-summary.json`
|
||
|
|
- `benchmarks/gb10-cute-qkv-block0-baseline.json`
|
||
|
|
- `benchmarks/gb10-sage2-p3-temporal-pair-analysis.json`
|
||
|
|
- `PERFORMANCE_ROADMAP.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no sufficiently large resident-request distribution, queue timing, or full-media-completion distribution.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run at least 100 serialized resident requests per supported operating point and report p50/p95/p99 for queue time, request wall time, sampling, and full media completion.
|
||
|
|
|
||
|
|
## 7. How does performance change across 4, 8, 12, and higher step counts?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Official Turbo 4-step and 8-step modes and base 12-step mode are operational. One 960x544x124 first-frame case measured `60.4s` sampling for Turbo-4, `121.1s` for Turbo-8, and `148.1s` for the stated base 8-step comparison. Other canonical base results use 12 steps.
|
||
|
|
|
||
|
|
Turbo and base runs use different adapters and schedules. They are not a controlled step-count-only sweep, and no buyer-grade result above 12 steps exists.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `TURBO.md`
|
||
|
|
- `tests/test_turbo.py`
|
||
|
|
- `tools/serve_hot_runtime.py`
|
||
|
|
- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
No matched 4/8/12/16+ latency and quality sweep exists with adapter and sampler held or explicitly accounted for.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run a declared 4/8/12/16-step matrix and score both performance and final quality. Separate the effect of fewer steps from the effect of the Turbo adapter.
|
||
|
|
|
||
|
|
## 8. How does it scale with 5-, 10-, 20-, and 30-second videos?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Approximately five-second and ten-second workloads have been measured. At 24 fps, 124 frames is approximately `5.17s` and 243 frames is approximately `10.13s`. Source guidance describes roughly 124-362 frames, or about 5-15 seconds, as the trained range, with longer output untested.
|
||
|
|
|
||
|
|
No 20- or 30-second performance, memory, completion, audio-continuity, or quality evidence exists.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`
|
||
|
|
- `src/h3_blackwell_runtime/upstream_nodes.py`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
Longer requests may exceed memory or the model's trained range. That failure boundary is not known.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run native 5/10/20/30-second sweeps with peak memory, latency, completion rate, latent health, audio continuity, and temporal-quality review.
|
||
|
|
|
||
|
|
## 9. What happens at native 720p, 1080p, 2K, and 4K?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Demonstrated native sampling resolutions include 864x480, 960x544, and 1344x768. A 1920x1088 result exists after 2x latent upscaling from 960x544. It is not native 1080p generation. No native 2K or 4K evidence exists.
|
||
|
|
|
||
|
|
The term "2K delivery" must therefore be described as an upscaled output workflow, not native 2K sampling.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
- `H3_LATENT_UPSCALER.md`
|
||
|
|
- `TURBO.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
Exact native 1280x720 and 1920x1080 have not been validated, and memory-safe native 2K/4K limits are unknown.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Test exact native 720p and 1080p first. Attempt native 2K and 4K only with explicit memory, duration, quality, and failure criteria. Keep upscaled controls separately labeled.
|
||
|
|
|
||
|
|
## 10. What are maximum safe frame and token counts?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
The largest demonstrated complete sampling case is 1344x768x243 frames. The canonical 1344x768x124 workload contains 37,810 packed tokens. On B200, the 243-frame sampling case recorded approximately 22.96 GB peak sampling allocation.
|
||
|
|
|
||
|
|
These are demonstrated points, not maximum-safe limits. The direct API does not currently enforce a validated maximum frame or token count.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`
|
||
|
|
- `tools/serve_hot_runtime.py`
|
||
|
|
- `src/h3_blackwell_runtime/upstream_nodes.py`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
No push-to-failure matrix defines safe limits by GPU, model, resolution, duration, conditioning load, or concurrency.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Increase frames and conditioning tokens until a predeclared memory, latency, quality, or error boundary is reached. Repeatedly validate the preceding safe point and enforce it in the API.
|
||
|
|
|
||
|
|
## 11. Can one rack run many independent jobs efficiently, or is it optimized only for one distributed job?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
The resident HTTP service accepts threaded connections but serializes generation through a single runtime lock. It therefore executes one independent generation at a time per process. Distributed Ulysses or tensor execution instead shards one generation across GPUs.
|
||
|
|
|
||
|
|
On two RTX PRO 6000 GPUs, one Ulysses job achieved speedups of `1.21x`, `1.55x`, and `1.72x` as workload size increased. This does not answer whether two independent one-GPU workers would provide better rack throughput or economics.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `tools/serve_hot_runtime.py`
|
||
|
|
- `DISTRIBUTED.md`
|
||
|
|
- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no rack scheduler, intentional concurrency contract, or equal-work comparison between independent workers and distributed jobs.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
On identical GPUs, compare concurrent one-GPU workers with one multi-GPU Ulysses job. Report aggregate throughput, p95 latency, VRAM, queue behavior, and cost.
|
||
|
|
|
||
|
|
## 12. Does throughput remain stable after hours or days of continuous operation?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No claim can currently be made. The repository contains repeatability studies, isolated 50-sample kernel tests, short benchmark matrices, and deployment smoke tests. It does not contain an hours- or days-long resident soak.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`
|
||
|
|
- `benchmarks/gb10-sage2-p3-temporal-pair-analysis.json`
|
||
|
|
- `compose.spark-stack.yml`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
No sustained throughput, memory-leak, thermal, clock, checksum, restart, or failure trend exists.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run a 24-hour soak followed by a 72-hour mixed-workload soak. Capture throughput, p50/p95/p99, host/GPU memory, allocator state, temperature, clocks, power, checksums, failures, and restarts.
|
||
|
|
|
||
|
|
## 13. What happens when GPU memory becomes fragmented?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Selected experiments report peak allocated and reserved memory, including an elementwise-fusion reduction of approximately 408.8 MB allocated and 130.0 MB reserved. These are single-run peaks and do not measure fragmentation.
|
||
|
|
|
||
|
|
A duplicate-model FC1 experiment caused a pressure-related container stop, but its timing was invalid and it does not establish allocator behavior.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`
|
||
|
|
- `PERFORMANCE_ROADMAP.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no mixed-size workload, allocator fragmentation telemetry, largest-free-block tracking, or recovery test.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run hundreds of alternating small and large requests while recording allocated, reserved, inactive split blocks, largest free allocation, failures, and post-request recovery.
|
||
|
|
|
||
|
|
## 14. What are failure and retry rates?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
The service returns HTTP 400 for selected validation failures and HTTP 500 for other exceptions. There is no automatic generation retry logic. Individual records mention a pre-sampling dependency failure and a container stop under experimental memory pressure, but no rate can be calculated.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `tools/serve_hot_runtime.py`
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
- `PERFORMANCE_ROADMAP.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
Failure frequency, timeout behavior, cancellation, idempotency, cleanup, retry success, and durable job recovery are unmeasured.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Add structured request outcome logging and run a fault-injection matrix covering invalid input, OOM, missing checkpoints, CUDA errors, FFmpeg failure, client disconnect, timeout, and process restart.
|
||
|
|
|
||
|
|
# Quality Questions
|
||
|
|
|
||
|
|
## 15. Are complete per-step latents equal to the official reference?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
One small fixed prompt-only FL2VA fixture at 320x192x22, seed 440204, and 12 steps has extensive ComfyUI-oracle replay evidence. The ledger reports exact H3 calls and RES updates at selected boundaries, with some wrapper and state-boundary qualifications.
|
||
|
|
|
||
|
|
This does not prove complete per-step equality for the current production runtime across T2VA, I2VA, L2VA, FL2VA, Turbo, different prompts, or different hardware. Exact QDATA/SFA, projection, block, or encoder hashes prove bounded seams rather than complete end-to-end equality.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `PARITY.md`
|
||
|
|
- `tools/compare_fl2va_steps.py`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
The complete official initial state, conditioning, every scheduler pre/post state, and final video/audio latent set are not preserved in one self-contained assertion-based fixture.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Capture and assert `torch.equal` at every step for at least base T2VA, FL2VA, and Turbo-4 using pinned official-reference artifacts.
|
||
|
|
|
||
|
|
## 16. Are final video and audio latents equal across hardware?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Cross-hardware final-latent equality has not been tested. RTX repeat runs recorded identical scalar checksums, but complete latents were not retained. B200 records timing and scalar values without matching RTX or GB10 tensors. One-rank Ulysses and tensor modes were equal on one GB10, but that tests execution paths on one hardware class rather than cross-hardware equality.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json`
|
||
|
|
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
|
||
|
|
- `DISTRIBUTED.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
Complete matching video/audio tensors are absent across GB10, RTX PRO 6000, B200, and GB300.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Save full final video and audio latents for identical requests on each available GPU. Compare raw-byte SHA-256, `torch.equal`, maximum and mean error, RMSE, and cosine similarity.
|
||
|
|
|
||
|
|
## 17. Does NVFP4 introduce perceptual degradation relative to BF16?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No defensible perceptual answer exists. The extensive NVFP4 parity work compares optimized implementations with the existing NVFP4/Comfy Kitchen path. It does not compare the pruned NVFP4 model's output quality with the official BF16 base.
|
||
|
|
|
||
|
|
Turbo adapters were released for a BF16 base; use over the pruned NVFP4 base is operationally validated but not upstream-certified.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `PLAN.md`
|
||
|
|
- `PERFORMANCE_ROADMAP.md`
|
||
|
|
- `TURBO.md`
|
||
|
|
- `research/EXPERIMENT_REGISTRY.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no matched BF16/NVFP4 prompt-and-seed corpus, latent comparison, video/audio metric set, or blinded preference study.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run official BF16 and pruned NVFP4 models at equal prompts and seeds. Retain all latents and decoded media, score objective quality, and conduct blinded human comparison.
|
||
|
|
|
||
|
|
## 18. Are results stable across hundreds of prompts and seeds?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No hundreds-of-prompts-and-seeds campaign exists. The strongest related evidence is a narrow ten-seed audio-format investigation and a ten-prompt text-conditioning repeatability tool. Neither is a broad end-to-end generation study.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json`
|
||
|
|
- `tools/compare_text_repeatability.py`
|
||
|
|
- `tests/`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no representative prompt taxonomy, broad seed matrix, retained media set, failure count, collapse rate, or quality distribution.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Run at least 100 prompts across at least three seeds, spanning dialogue, music, action, static scenes, keyframes, identities, text, faces, hands, and difficult camera motion.
|
||
|
|
|
||
|
|
## 19. What are temporal consistency, motion quality, and identity-preservation results?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No systematic results exist. One Turbo comparison contains informal observations about composition, motion, and exposure drift. There are no standard temporal or identity metrics and no multi-seed evaluation.
|
||
|
|
|
||
|
|
Full arbitrary identity/reference conditioning is also outside the currently implemented H3 direct-runtime scope.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `TURBO.md`
|
||
|
|
- `VORTEX_RESEARCH_AGENDA.md`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no subject-tracking suite, motion-adherence test, temporal perceptual score, optical-flow consistency score, or identity-embedding drift measurement.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Build a fixed multi-seed suite with prescribed subjects and motion. Add temporal perceptual, flow-consistency, subject/face embedding, motion-adherence, and human-review scoring.
|
||
|
|
|
||
|
|
## 20. How well does it handle text, hands, faces, and complex camera motion?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
The repository contains prompt guidance and isolated diagnostic prompts, not category-level quality measurements. No hand/finger benchmark, OCR suite, face-quality suite, or camera-trajectory evaluation exists.
|
||
|
|
|
||
|
|
The latent upscaler is explicitly described as unsafe for text, QR codes, and identity-critical content, and it can alter faces.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `H3_PROMPT_GUIDE.md`
|
||
|
|
- `H3_LATENT_UPSCALER.md`
|
||
|
|
- `benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no curated hard-case corpus, automated task scoring, defect taxonomy, or blinded review.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Create four explicit test buckets: readable text/OCR, hand-object interaction, close and multi-angle faces, and prescribed static/pan/dolly/handheld camera motion.
|
||
|
|
|
||
|
|
## 21. What are speech quality, voice consistency, and lip-sync scores?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Stereo 32 kHz audio generation, decoding, and muxing are operational. A ten-seed quoted-dialogue diagnostic found useful prompt-format behavior and received subjective WAV review. Quoted prompts removed immediate startup activity in all ten recorded cases.
|
||
|
|
|
||
|
|
Speech accuracy, voice consistency, and full-video lip-sync remain unscored. Signal levels and successful muxing are not speech-quality evidence.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json`
|
||
|
|
- `AUDIO_BOUNDARY_INVESTIGATION.md`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
- `PARITY.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There are no ASR/WER results, speaker-embedding measurements, onset/offset accuracy, lip-sync metrics, or structured human scores.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Generate full dialogue videos with known scripts and visible speakers. Retain lossless PCM and score WER, speaker consistency, speech timing, automated lip-sync, and blinded human lip-sync.
|
||
|
|
|
||
|
|
## 22. Does the latent upscaler introduce ringing, texture, or chromatic artifacts?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Yes, known examples show harsher illustrated texture, ringing or chromatic edges, and facial changes. One principal source-to-downscaled comparison recorded SSIM `0.924986`. The upscaler is not considered fidelity-safe for text, QR codes, or identity-critical content.
|
||
|
|
|
||
|
|
What is unknown is how often these defects occur, their severity distribution, and whether they are preferable to conventional scaling.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `H3_LATENT_UPSCALER.md`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
- `tests/test_latent_upscaler.py`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no representative multi-clip artifact study, edge-overshoot metric, color-fringe metric, LPIPS distribution, or blind crop review.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Evaluate at least 20 clips containing text, high-contrast edges, faces, skin, hair, and fine texture against bicubic/Lanczos and native-resolution controls.
|
||
|
|
|
||
|
|
## 23. Has there been a blind human A/B evaluation?
|
||
|
|
|
||
|
|
**Status: Open**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
No randomized, blinded, multi-rater A/B evaluation is documented. Existing audio, Turbo, and upscaler observations are informal and do not record blindness, randomization, rater count, inter-rater agreement, or confidence intervals.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json`
|
||
|
|
- `TURBO.md`
|
||
|
|
- `H3_LATENT_UPSCALER.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There is no evaluation protocol, recruited panel, anonymized media set, scoring rubric, or statistical analysis.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Randomize and anonymize at least 30 matched clip pairs, recruit at least three independent raters, collect preference and defect labels, and report confidence intervals and agreement.
|
||
|
|
|
||
|
|
## 24. Does the four-step Turbo model materially reduce quality?
|
||
|
|
|
||
|
|
**Status: Partially answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Turbo-4 is operational and materially faster, but its general quality delta is unknown. On one matched 960x544x124 case, Turbo-4 sampled in `60.4s`, Turbo-8 in `121.1s`, and the stated base comparison in `148.1s`. Informal review found Turbo-4 somewhat more compositionally stable in that example, Turbo-8 showed more motion, and both showed exposure drift.
|
||
|
|
|
||
|
|
One example cannot establish population-level quality. There is no scored comparison with base 12-step output or official LightX2V BF16 Turbo.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `TURBO.md`
|
||
|
|
- `benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`
|
||
|
|
- `tests/test_turbo.py`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
No matched prompt suite, latent-quality analysis, video/audio metrics, or blind preference result exists.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Compare base 12-step, Turbo-8, and Turbo-4 over at least 20 prompts and three seeds, then separately compare NVFP4 Turbo-4 with the official BF16 path.
|
||
|
|
|
||
|
|
## 25. Can customers choose between reference-quality and maximum-throughput modes?
|
||
|
|
|
||
|
|
**Status: Answerable**
|
||
|
|
|
||
|
|
### Current answer
|
||
|
|
|
||
|
|
Not as coherent, named, validated product modes today.
|
||
|
|
|
||
|
|
The runtime exposes low-level controls for attention backend, Turbo adapter, upscaling, cache behavior, VAE dtype, and distributed mode. An expert can approximate a conservative path using base 12-step, Sage2, and no cache/upscaler, or a faster path using Turbo-4 and selected acceleration controls. Production fusions and layouts are also selected through startup environment variables.
|
||
|
|
|
||
|
|
These combinations are not immutable buyer-facing `reference_quality` and `maximum_throughput` contracts. "Reference quality" cannot currently mean proven official-BF16 equivalence because that quality comparison remains open.
|
||
|
|
|
||
|
|
### Existing evidence
|
||
|
|
|
||
|
|
- `src/h3_blackwell_runtime/runtime.py`
|
||
|
|
- `src/h3_blackwell_runtime/attention.py`
|
||
|
|
- `tools/serve_hot_runtime.py`
|
||
|
|
- `compose.spark.yml`
|
||
|
|
- `CURRENT_STATE.md`
|
||
|
|
|
||
|
|
### Remaining gap
|
||
|
|
|
||
|
|
There are no named profiles, frozen configuration contracts, per-profile regression tests, quality floors, or supported-model matrix.
|
||
|
|
|
||
|
|
### Next action
|
||
|
|
|
||
|
|
Define immutable `reference_quality` and `maximum_throughput` profiles per supported model. Validate the reference profile against the official oracle and the throughput profile against explicit latency, reliability, and perceptual-quality thresholds.
|
||
|
|
|
||
|
|
# Recommended Closure Order
|
||
|
|
|
||
|
|
## Immediate evidence work on available hardware
|
||
|
|
|
||
|
|
1. End-to-end p50/p95/p99 resident-request harness.
|
||
|
|
2. Controlled cold-start and warm-start instrumentation.
|
||
|
|
3. Full per-step latent capture and assertion fixture.
|
||
|
|
4. Five-, ten-, twenty-, and thirty-second duration sweep.
|
||
|
|
5. Maximum safe frame/token push-to-failure matrix.
|
||
|
|
6. Structured failure logging and fault injection.
|
||
|
|
7. Fixed 100-prompt, three-seed stability suite.
|
||
|
|
8. Upscaler artifact benchmark.
|
||
|
|
|
||
|
|
## Highest-leverage engineering work
|
||
|
|
|
||
|
|
1. Pin and implement the Wan and LTX 2.3/2.5 support matrix.
|
||
|
|
2. Build a shared video/audio quality-scoring pipeline.
|
||
|
|
3. Add controlled precision, attention, fusion, residency, and distribution ablations.
|
||
|
|
4. Define named reference-quality and maximum-throughput profiles.
|
||
|
|
5. Add allocator telemetry and mixed-workload soak support.
|
||
|
|
|
||
|
|
## External or additional-hardware work
|
||
|
|
|
||
|
|
1. Official H3 versus Vortex on the same RTX PRO 6000.
|
||
|
|
2. Official H3 versus Vortex on the same B200 and GB300.
|
||
|
|
3. Independent-job versus distributed-job rack-scale testing.
|
||
|
|
4. Four-, eight-, and higher-GPU scaling.
|
||
|
|
5. Blind human A/B evaluation.
|
||
|
|
|
||
|
|
# Primary Evidence Index
|
||
|
|
|
||
|
|
- `CURRENT_STATE.md`: canonical implementation and open-work status.
|
||
|
|
- `README.md`: runtime scope and operating modes.
|
||
|
|
- `PARITY.md`: historical correctness ledger; interpret contradictory rows cautiously.
|
||
|
|
- `PERFORMANCE_ROADMAP.md`: profiles, measured opportunities, and projected targets.
|
||
|
|
- `DISTRIBUTED.md`: distributed implementation, topology, timing, and parity gaps.
|
||
|
|
- `TURBO.md`: Turbo revisions, schedules, timing, and limited subjective observations.
|
||
|
|
- `H3_LATENT_UPSCALER.md`: upscaler implementation, timing, and known quality defects.
|
||
|
|
- `AUDIO_BOUNDARY_INVESTIGATION.md`: audio diagnostics and unresolved quality work.
|
||
|
|
- `research/EXPERIMENT_REGISTRY.md`: permanent performance-experiment registry.
|
||
|
|
- `research/REPRODUCIBILITY.md`: environment, command, and reproducibility gaps.
|
||
|
|
- `benchmarks/`: raw and summarized timing, parity, memory, profiler, and smoke evidence.
|