Add distributed H3 execution and validation

Expand H3 runtime acceleration and validation
Add resident FL2VA Turbo variants
2026-08-22 14:09:45 +07:00 · 2026-08-20 22:08:52 +07:00 · 2026-08-20 19:13:23 +07:00 · 2026-08-20 17:39:44 +07:00 · 2026-08-20 16:43:22 +07:00 · 2026-08-19 23:03:58 +07:00
92 changed files with 10262 additions and 179 deletions
--- a/.dockerignore
+++ b/.dockerignore
@ -0,0 +1,5 @@
+.git
+.pytest_cache
+artifacts
+**/__pycache__
+**/*.pyc
--- a/AUDIO_BOUNDARY_INVESTIGATION.md
+++ b/AUDIO_BOUNDARY_INVESTIGATION.md
@ -0,0 +1,109 @@
+# H3 Startup Audio Investigation
+
+Investigation date: 2026-08-21
+
+## Reproduction Cases
+
+- Affected dialogue: base 12-step, Sage2, seed `440420`, tagged dialogue.
+- Clean control: base 12-step, Sage2, seed `440421`, immediate nightclub music.
+- Prompt-format control: affected dialogue prompt and seed with only the two
+  `<d>[English]...</d>` spans replaced by quoted speech.
+
+All diagnostic assets are under `/home/daniel/StoryStudioAssets/H3-output/h3-baselines`.
+
+## Findings
+
+1. The affected transient is already present in the retained lossless WAV. AAC
+   encoding and MP4 muxing are not the source.
+2. The AudioVAE is not the primary source. Moving the affected first four audio
+   latent frames to frame 40 in an otherwise near-silent latent carrier creates
+   a similarly strong event at `1.0s`: `-18.44 dBFS` peak versus `-19.12 dBFS`
+   when the same frames are placed at the start.
+3. AudioVAE boundary context changes the exact waveform, but the event remains.
+   This makes the decoder a secondary shaper rather than the origin.
+4. Zero normalized latents are not silence. The official AudioVAE decodes them
+   to approximately `-26.17 dBFS` RMS in the first 100ms, so zero replacement is
+   not a valid repair.
+5. Repeating affected latent frame 4 produces near-silence (`-56.02 dBFS` RMS),
+   but unconditional replacement is unsafe because valid music begins in the
+   same first four frames in the clean control.
+6. There is no evidence for end-to-start wraparound in this sample. First/last
+   four-frame latent cosine is `-0.122`; first/last 100ms PCM correlation is
+   `0.006`.
+7. In the tagged-dialogue denoising trace, the unwanted onset is near silence
+   through step 6 and begins growing materially at step 7 (`audio sigma 0.751`).
+   It reaches `-18.94 dBFS` peak and `-34.23 dBFS` RMS in the final first 100ms.
+8. Clean immediate music is strongly predicted from step 1. This distinguishes
+   legitimate onset generation from the late-forming dialogue artifact.
+9. Replacing only tagged dialogue with quoted speech suppresses the final first
+   100ms by about `21.3 dB` peak and `15.9 dB` RMS. The quoted result remains
+   below `-40.28 dBFS` peak and retains two later non-silent speech regions.
+
+## Current Diagnosis
+
+The startup artifact is encoded into the first four generated audio latent
+frames during late denoising. Dialogue markup is a strong trigger for the
+reproduced seed. The AudioVAE renders and contextually shapes the event but does
+not create it independently.
+
+This does not prove that markup is the only trigger or that quoted dialogue is
+universally clean. It does establish markup as a repeatable trigger for this
+prompt family. Automatic prompt transformation still requires subjective speech,
+word-accuracy, and lip-sync review.
+
+## Ten-Seed Prompt-Format Sweep
+
+A matched sweep used seeds `440420` through `440429`, Sage2, base 12-step
+beta/RES sampling, and identical dialogue semantics. Each seed generated one
+tagged and one quoted audio latent and lossless waveform.
+
+- Quoted speech reduced first-100ms peak level for all 10 seeds.
+- Median peak reduction was `21.55 dB`; mean was `20.11 dB`.
+- Median RMS reduction was `17.50 dB`; mean was `17.93 dB`.
+- Tagged speech exceeded `-40 dBFS` peak in the first 100ms for 9/10 seeds.
+- Quoted speech exceeded that threshold for 0/10 seeds.
+- Tagged speech became active within 100ms for 9/10 seeds; quoted speech did so
+  for 0/10 seeds.
+- A greater-than-10dB boundary decay occurred for 8/10 tagged cases and 0/10
+  quoted cases.
+- Simple first-frame and first-four-frame latent magnitude/delta features overlap
+  between groups and cannot safely detect the artifact by themselves.
+
+The complete report and paired WAV/latent files are in
+`h3-baselines/audio-dialogue-format-sweep`. A subjective listening pass on
+2026-08-21 judged all ten quoted WAVs good. Quoted dialogue is therefore the
+project default; full-video lip-sync validation remains pending.
+
+## Diagnostic Assets
+
+- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.latent.pt`
+- `audio-diagnostic-affected-dialogue-864x480-141f-base12-sage2-seed440420.wav`
+- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt`
+- `audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav`
+- `audio-diagnostic-dialogue-quoted-base12-sage2-seed440420.wav`
+- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.pt`
+- `audio-diagnostic-affected-dialogue-sage2-denoise-trace.json`
+- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.pt`
+- `audio-diagnostic-clean-nightclub-sage2-denoise-trace.json`
+- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.pt`
+- `audio-diagnostic-dialogue-quoted-sage2-denoise-trace.json`
+- `audio-vae-boundary-probes/report.json`
+
+## Next Experiments
+
+1. Generate selected full videos to compare lip-sync and prompt adherence with
+   the new quoted-dialogue default.
+2. Add one ambience-only prompt and one immediate-impact sound prompt to prevent
+   a detector from equating quiet starts with correctness.
+3. Compare tagged-versus-quoted Qwen conditioning and per-step first-four-frame
+   denoiser outputs to localize the conditioning pathway.
+4. Prototype a selective late-step boundary re-denoise only after a reliable
+   latent classifier exists.
+5. Reject unconditional trimming, fading, zeroing, or fixed-frame replacement.
+
+## Measurement Correction
+
+FFmpeg's `apsnr` results previously recorded for attention-backend audio were
+inconsistent with direct decoded-PCM array comparisons and must not be used.
+Future audio comparisons must decode each stream to aligned float PCM and compute
+error metrics directly.
--- a/CURRENT_STATE.md
+++ b/CURRENT_STATE.md
@ -0,0 +1,114 @@
+# H3 Runtime Current State
+
+Status date: 2026-08-22
+
+This document is the canonical snapshot of implemented scope and remaining work.
+Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
+
+## Implemented And Validated
+
+- Single-GPU prompt-only T2VA with joint video/audio generation.
+- First-frame I2VA, last-frame L2VA, and first/last FL2VA through the shared
+  keyframe-conditioning path.
+- Qwen text and vision conditioning, token refinement, video VAE encoding, H3
+  packed denoising, beta/RES sampling, video/audio decoding, and final MP4 mux.
+- Resident HTTP runtime with warmup, readiness reporting, request-level backend
+  selection, timing stages, optional latent saving, and diagnostic intermediates.
+- SageAttention2 as the default quality backend.
+- SDPA, forced cuDNN SDPA, FlashAttention-4, Sage3, Comfy Kitchen INT8, KJ Sage,
+  head-sliced, and Sol-Attn experimental backends.
+- Official FL2VA Turbo 4-step and 8-step adapters.
+- Optional resident H3-native latent upscaling.
+- Experimental EasyCache and H3-Cache delta-reuse modes.
+- Quoted dialogue as the project prompt default. In a matched 10-seed test,
+  quoted dialogue eliminated immediate first-100ms activity in all ten cases and
+  all ten quoted WAVs passed subjective review.
+- Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
+- Sequence-sharded 50-block execution and distributed final projection.
+- True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
+- Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
+- Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames,
+  and 12 steps, including exact video and audio latent equality.
+
+## Primary Missing Scope
+
+### Full Ref2VA
+
+- Arbitrary reference image, video, and audio inputs.
+- Reference-audio encoder and reference soundtrack conditioning.
+- Reference identity/voice blocks in the standalone packer.
+- Ref2VA position, modality, and scheduling contracts.
+- Direct-versus-Comfy full Ref2VA per-step and final-output parity benchmark.
+
+### Explicit Task API
+
+- Named `task` selection for T2VA, I2VA, L2VA, FL2VA, and Ref2VA.
+- Mode-specific request schemas and incompatible-input validation.
+- Intermediate keyframe anchors beyond the current first/last restriction.
+
+### Distributed Execution
+
+- Real NCCL transport and output parity above one rank.
+- 2/4/6/8-GPU topology and performance sweeps on one Blackwell machine.
+- Distributed resident-service orchestration; the current launcher is batch
+  generation through `torchrun`.
+- x86 SageAttention2 packaging; RunPod validation initially uses SDPA.
+
+### Owned Performance Kernels
+
+- H3-specific attention backend optimized for real GB10 tensor shapes.
+- Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
+- CUDA graph capture and shape buckets.
+- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.
+
+## Quality Work Remaining
+
+- Generate full quoted-dialogue videos and validate wording, voice consistency,
+  speech timing, and lip-sync before closing the startup-audio work.
+- Complete strict per-step LightX2V parity for Turbo adapters.
+- Add real-adapter Turbo end-to-end fixtures.
+- Add the optional target-resolution refinement stage after latent upscaling.
+- Resolve or formally bound upscaler ringing, texture, chromatic-edge, and
+  identity changes.
+- Run full-size cache threshold and quality sweeps before enabling caches for
+  production output.
+- Keep Sage3, Sol-Attn, INT8, and other approximate backends quality-gated.
+- Fix the inactive fused Sol QKV-layout path, which currently references an
+  undefined `qkv` value. The deployed native Sol layout does not use this path.
+
+## Production Work Remaining
+
+- Asynchronous jobs, queueing, progress, cancellation, and timeouts.
+- Strict request validation, including Boolean fields and mode combinations.
+- Input/output path sandboxing, request-size limits, authentication, and TLS.
+- Configurable FPS, video codec, audio codec, sample rate, and media policy.
+- Container healthcheck, restart policy, resource limits, durable structured
+  request logs, and runtime metrics.
+- Batch generation and an intentional worker/concurrency model.
+
+## Validation And Packaging Gaps
+
+- GPU end-to-end fixtures for T2VA, I2VA, L2VA, and FL2VA.
+- Full Ref2VA, AudioVAE waveform, cache, HTTP API, real Turbo, real upscaler,
+  attention-quality, CUDA-graph, and distributed tests.
+- Reproducible local fixtures for parity evidence currently stored on Spark/SMB.
+- Explicit package declarations/checks for NumPy, SciPy, Pillow, and FFmpeg.
+- A standalone base image if removing the Comfy-derived image becomes a product
+  requirement; the current denoising path still intentionally uses Comfy Kitchen
+  kernels.
+- Align Docker `H3_MODEL_PATH` and `RuntimeConfig`; the environment variable is
+  currently not consumed by the runtime default.
+
+## Recommended Execution Order
+
+1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
+2. Correct the inactive fused Sol path.
+3. Add explicit task schemas and automated single-GPU mode tests.
+4. Implement full Ref2VA, including reference-audio encoding.
+5. Build the H3-specific attention backend and CUDA graph buckets.
+6. Harden the service API and operational deployment.
+7. Complete RunPod NCCL validation and distributed scaling benchmarks.
+
+The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is
+implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still
+blocked on an eight-GPU host. The other largest gap is standalone Ref2VA.
--- a/DISTRIBUTED.md
+++ b/DISTRIBUTED.md
@ -0,0 +1,134 @@
+# Distributed H3 Execution
+
+The runtime supports two single-node distributed denoiser modes:
+
+- `ulysses`: token-sharded blocks with QKV sequence-to-head all-to-all and the
+  inverse head-to-sequence all-to-all around attention.
+- `tensor`: token-sharded blocks plus true NVFP4 tensor parallelism across QKV,
+  attention output, MLP FC1, and MLP FC2.
+
+Both modes preserve the native packed H3 sequence and reject empty token or head
+partitions. They do not add semantic padding. Temporary collective padding used
+by ragged row gathering is removed before model operations.
+
+## Partition Contract
+
+H3 has 56 attention heads of width 128. Head ownership is balanced and can be
+ragged:
+
+| GPUs | Heads per rank |
+| --- | --- |
+| 2 | 28, 28 |
+| 4 | 14, 14, 14, 14 |
+| 6 | 10, 10, 9, 9, 9, 9 |
+| 8 | 7, 7, 7, 7, 7, 7, 7, 7 |
+
+Token ranges use the same quotient/remainder partitioning. Segment boundaries
+are clipped to each rank's token interval and rebased before AdaLN modulation
+and residual gating.
+
+In tensor mode, QKV output rows are selected by local head ownership. Attention
+output and MLP FC2 input columns are sliced on NVFP4 alignment boundaries; each
+rank computes a partial output and a ragged reduce-scatter sums and assigns token
+rows. MLP FC1 selects corresponding local ranges from both the gate and value
+halves. Projection bias is omitted from rank partials and added exactly once
+after reduction.
+
+## Launch
+
+The launcher uses every visible GPU when the world size is omitted:
+
+```bash
+tools/run_distributed_t2va.sh ulysses
+tools/run_distributed_t2va.sh tensor
+```
+
+Pass an explicit count and attention backend when needed:
+
+```bash
+tools/run_distributed_t2va.sh tensor 8 sdpa
+```
+
+Relevant environment variables:
+
+- `H3_WORLD_SIZE`: fallback world size when no positional count is supplied.
+- `H3_DISTRIBUTED_BENCHMARK`: benchmark JSON path.
+- `H3_DISTRIBUTED_OUTPUT`: report and optional latent output directory.
+- `H3_MODEL_PATH` and `H3_TEXT_ENCODER_PATH`: checkpoint paths.
+- `H3_SAVE_LATENTS=0`: write reports without retaining large latent files.
+
+Run every feasible target count on the current machine with:
+
+```bash
+H3_SAVE_LATENTS=0 tools/run_distributed_matrix.sh
+```
+
+The default matrix runs Ulysses and tensor modes at 1, 2, 4, 6, and 8 GPUs and
+skips counts larger than the visible device count. Override the lists with
+`H3_GPU_COUNTS` and `H3_DISTRIBUTED_MODES`.
+
+The lower-level transport benchmark does not load H3 weights:
+
+```bash
+tools/run_ulysses_benchmark.sh
+tools/run_ulysses_benchmark.sh 6 sdpa
+```
+
+## Validation
+
+Automated Gloo tests cover:
+
+- Ulysses transport identity at 2, 4, 6, and 8 ranks.
+- Two-rank distributed SDPA parity.
+- Ragged gather and reduce-scatter behavior.
+- Distributed final-projection parity.
+- TP attention and MLP parity at 2 and 6 ranks.
+- Packed NVFP4 column- and row-shard layout preservation.
+
+On GB10, the real 50-block NVFP4 checkpoint completed both one-rank distributed
+paths at 864x480, 141 frames, 12 steps, seed 440420. Ulysses and tensor modes
+produced identical video and audio tensors with zero maximum absolute error.
+This validates integration and the world-size-one identity path, but it does not
+replace multi-GPU NCCL parity testing.
+
+## RunPod
+
+The target is one eight-GPU RTX PRO 6000 Blackwell machine. Query current stock
+using the guarded API v2 client:
+
+```powershell
+$env:RUNPOD_API_KEY = "..."
+python .\tools\runpod_api.py catalog --count 8
+```
+
+After choosing an available data center, create the pod explicitly:
+
+```powershell
+python .\tools\runpod_api.py create --count 8 --datacenter US-XX-N --yes
+```
+
+The client defaults to the server-edition RTX PRO 6000 Blackwell and RunPod's
+x86_64 CUDA 13.0, Torch 2.9.1 image. It creates persistent workspace storage and
+enables SSH. Creation and termination require `--yes` to avoid accidental spend
+or data loss. Use `get` to poll status and `terminate POD_ID --yes` when finished.
+
+Transfer this checkout plus these two existing checkpoints to the pod workspace:
+
+- `minimax_h3_fl2va_pruned_nvfp4.safetensors` (12,528,636,800 bytes)
+- `qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` (15,687,142,551 bytes)
+
+Install the project dependencies in the RunPod image, set the checkpoint path
+variables, run `test_distributed.py`, then run the transport and generation
+matrices. `Dockerfile.runpod` provides the equivalent reproducible custom image
+once an x86_64 image builder and registry are available.
+
+SageAttention2 is not currently packaged in the generic x86 image, so cloud
+correctness and scaling start with SDPA. Sage2 can be measured after an x86
+wheel is added without changing the distributed layout.
+
+## Remaining GPU Gates
+
+- Real NCCL identity at 2, 4, 6, and 8 GPUs.
+- Distributed-versus-single latent parity above one rank.
+- Full 1/2/4/6/8 timing, transport, and memory reports.
+- Quality comparison after selecting an x86 attention backend.
--- a/Dockerfile.runpod
+++ b/Dockerfile.runpod
@ -0,0 +1,24 @@
+# x86_64 CUDA 13 image for RunPod Blackwell distributed validation.
+FROM pytorch/pytorch:2.9.1-cuda13.0-cudnn9-devel
+
+ARG COMFY_KITCHEN_VERSION=0.2.31
+
+RUN apt-get update \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ffmpeg git \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/h3-blackwell-runtime
+COPY . .
+
+RUN python -m pip install --no-cache-dir \
+    "comfy-kitchen==${COMFY_KITCHEN_VERSION}" \
+    "fastsafetensors>=0.1.10" \
+    "numpy>=2.0" \
+    "pillow>=11" \
+    "safetensors>=0.5.0" \
+    "scipy>=1.14" \
+    "transformers>=4.51,<5"
+
+ENV PYTHONPATH=/opt/h3-blackwell-runtime/src
+ENTRYPOINT []
+CMD ["bash"]
--- a/Dockerfile.spark
+++ b/Dockerfile.spark
@ -1,6 +1,8 @@
 # GB10/Grace Blackwell development image. It does not start inference by default.
 FROM ghcr.io/aeon-7/comfyui-aeon-spark:slim

+ARG SOL_ATTN_COMMIT=930a4d6e432ff8b8ed5e30ff2f72519b92d69bdf
+
 WORKDIR /opt/h3-blackwell-runtime
 COPY . .

@ -9,14 +11,28 @@ COPY wheels/sageattn3-*.whl /tmp/wheels/
 RUN python -m pip install --no-cache-dir --no-deps /tmp/wheels/sageattn3-*.whl \
    && rm -rf /tmp/wheels

-RUN python -m pip install --no-cache-dir --no-deps comfy-kitchen==0.2.28
+RUN python -m pip install --no-cache-dir --no-deps comfy-kitchen==0.2.31

 RUN python -m pip install --no-cache-dir "fastsafetensors>=0.1.10"

+# Official CuTeDSL FlashAttention-4 beta with CUDA 13 Blackwell support.
+RUN python -m pip install --no-cache-dir --pre \
+    "flash-attn-4[cu13]==4.0.0b27" \
+    "nvidia-cutlass-dsl[cu13]==4.6.2" \
+    "quack-kernels==0.6.4"
+
+RUN python -m pip uninstall -y pynvml \
+    && python -m pip install --no-cache-dir nvidia-ml-py
+
+RUN git clone https://github.com/Saganaki22/ComfyUI-sol-attn.git /opt/ComfyUI-sol-attn \
+    && cd /opt/ComfyUI-sol-attn \
+    && git checkout ${SOL_ATTN_COMMIT}
+
 RUN python -m pip install --no-cache-dir --no-deps -e . \
-    && python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); print(torch.__version__, torch.version.cuda)"
+    && python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); assert hasattr(comfy_kitchen, 'int8_attention'); assert hasattr(comfy_kitchen, 'int8_attention_is_available'); print(torch.__version__, torch.version.cuda)"

 ENV H3_MODEL_PATH=/models/minimax_h3_ref2va_pruned_nvfp4.safetensors
+ENV PYTHONPATH=/opt/ComfyUI-sol-attn
 ENV TORCH_COMPILE_DISABLE=0 TORCHDYNAMO_DISABLE=0
 ENTRYPOINT []
 CMD ["bash"]
--- a/FLASH4.md
+++ b/FLASH4.md
@ -0,0 +1,71 @@
+# FlashAttention-4
+
+The hot runtime exposes the official FlashAttention-4 CuTeDSL Blackwell kernel
+as the strict `flash4` attention backend. It does not silently fall back to
+SageAttention or SDPA.
+
+## Versions
+
+- `flash-attn-4==4.0.0b27`
+- `nvidia-cutlass-dsl==4.6.2` with the CUDA 13 extra
+- `quack-kernels==0.6.4`
+- Upstream: <https://github.com/Dao-AILab/flash-attention>
+- License: BSD-3-Clause
+
+The package is a beta release. The Spark image pins these versions rather than
+tracking the latest prerelease.
+
+## API
+
+Select Flash4 per request:
+
+```json
+{
+  "prompt": "A simple cinematic scene.",
+  "output": "/output/flash4-example.mp4",
+  "width": 960,
+  "height": 544,
+  "frames": 124,
+  "steps": 12,
+  "attention": "flash4"
+}
+```
+
+H3's main attention blocks call Flash4 in native
+`[batch, sequence, heads, head_dim]` layout. The adapter requires CUDA, matching
+FP16/BF16 Q/K/V tensors, matching shapes, and H3's head dimension of 128. Any
+violation raises an error instead of changing kernels.
+
+## GB10 Validation
+
+Flash4 dispatches its SM120 forward implementation on the GB10, which PyTorch
+reports as compute capability 12.1.
+
+| Validation | Result |
+| --- | --- |
+| BF16 `[1,257,8,128]` vs SDPA | max abs `0.001953125`, mean abs `5.04e-05` |
+| BF16 `[1,20480,56,128]` vs SDPA | max abs `0.000244141`, mean abs `1.96e-08` |
+| Tiny joint AV H3 smoke | passed, no fallback |
+| 28 runtime contracts | passed |
+
+Matched base 12-step talking benchmark at `960x544x124`, seed `440410`:
+
+| Backend | Sampling | Relative to Flash4 |
+| --- | ---: | ---: |
+| Sage2 | `114.67s` | `-19.4%` |
+| Flash4 | `142.23s` | baseline |
+| PyTorch SDPA | `144.40s` | `+1.5%` |
+
+Flash4 is therefore close to SDPA and materially slower than Sage2 for this H3
+workload on GB10. Its value is as another exact-style attention and audio-quality
+comparison, not as the current speed default.
+
+The matched Flash4 proof contains H.264 video and stereo 32 kHz AAC audio:
+
+```text
+/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-flash4-960x544.mp4
+```
+
+The current CuTeDSL package emits a one-time `AuxData` JIT argument warning.
+Compilation and inference still complete successfully, but the warning should
+be rechecked when upgrading FlashAttention-4 or CUTLASS DSL.
--- a/H3_LATENT_UPSCALER.md
+++ b/H3_LATENT_UPSCALER.md
@ -0,0 +1,149 @@
+# H3 Native Latent Upscaler
+
+The direct runtime can apply LBH-123-AI's learned 3D upscaler to a saved,
+normalized MiniMax H3 video latent without a ComfyUI dependency. This is an
+experimental spatial upscale. It preserves latent time and does not interpolate
+frames.
+
+Upstream implementation and checkpoint:
+
+- <https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler>
+- <https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler>
+- Revision: `13ccf95d85d120bdbc92c05b1247a6e147bf54bf`
+- FP16 SHA-256: `043e5a48e161610ef6c3ea974645220354d06fa618abca15f76d084812eb55c2`
+- License: Apache-2.0
+
+## Usage
+
+The hot service supports the complete generation, latent upscale, video/audio
+decode, and mux process in one request:
+
+```json
+{
+  "prompt": "A simple cinematic scene.",
+  "output": "/output/h3-native-upscale/example.mp4",
+  "width": 960,
+  "height": 544,
+  "frames": 124,
+  "turbo": "4step",
+  "upscale": 2.0,
+  "mux_audio": true,
+  "keep_intermediates": false
+}
+```
+
+The response reports `source_width`, `source_height`, final `width` and
+`height`, and a `latent_upscaled` timing stage. H3 audio bypasses the spatial
+upscaler and is decoded and muxed normally.
+
+Only the final muxed MP4 is retained by default. Set `keep_intermediates` to
+`true` only when a separate WAV and video-only MP4 are needed for diagnostics;
+`save_latent` remains a separate opt-in field.
+
+For offline diagnostics, save the normalized latent during generation by
+setting `save_latent`, then run:
+
+```bash
+python tools/upscale_video_latent.py \
+  --latent /output/source.pt \
+  --model /upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors \
+  --output /output/source-h3latent2x.pt \
+  --scale 2 \
+  --precision float16
+
+python tools/decode_video_latent.py \
+  --latent /output/source-h3latent2x.pt \
+  --output /output/source-h3latent2x.mp4 \
+  --vae-dtype float16 \
+  --vae-tile-size 256
+```
+
+The upscaled state retains other source metadata and any audio latent. Its
+`width` and `height` fields are updated from the resulting H3 latent dimensions.
+
+## GB10 Validation
+
+Validated on 2026-08-20 with the Posly hero image, Turbo 4-step v1.1, Sage2,
+seed `440408`, and 124 frames:
+
+| Stage | Result |
+| --- | --- |
+| Source | `960x544`, latent `[1,24,37,34,60]` |
+| Learned 3D upscale | `10.94s`, latent `[1,24,37,68,120]` |
+| Checkpoint load | `1.36s` |
+| Tiled FP16 decode command | `137.67s`, `1920x1088` H.264 |
+| Source vs downscaled result | SSIM `0.924986` |
+
+The learned stage adds visible edge, face, and food detail. It also produces a
+harsher illustrated texture, ringing/chromatic edges in some high-contrast
+areas, and small facial changes. It is not fidelity-safe for protected UI,
+text, QR codes, or identity-critical material without visual review.
+
+The upstream example follows latent upscaling with a second target-resolution
+H3 sampling pass using sigmas `0.9035, 0.6316, 0.3158, 0.0`. That refinement
+was deliberately excluded from this first test so the upscaler itself could be
+evaluated. It adds generation cost and another opportunity to alter content.
+
+## Proofs
+
+Spark output directory:
+
+```text
+/home/daniel/StoryStudioAssets/H3-output/posly/upscale-h3-native/
+```
+
+Key files:
+
+- `01-posly-hero-turbo4-source-960x544.mp4`
+- `01-posly-hero-turbo4-h3latent2x-1920x1088.mp4`
+- `source-frame60-right-crop-lanczos.png`
+- `h3latent2x-frame60-right-crop.png`
+
+This result is promising for cinematic footage but does not replace the proven
+LTX 2.3 chain yet. Run a matched LTX 2.3 comparison and a carefully bounded H3
+refinement test before selecting a default production path.
+
+## Integrated T2V Validation
+
+The single-request path was validated with a matched 5.17-second talking-head
+T2V pair at seed `440410`. Both requests sampled at `960x544`, applied the
+resident 2x upscaler, decoded at `1920x1088`, and muxed native H3 audio into the
+only retained artifact.
+
+| Variant | Sampling | Latent upscale | VAE decode | Wall time |
+| --- | ---: | ---: | ---: | ---: |
+| Turbo 4-step v1.1 | `56.54s` | `7.52s` | `82.74s` | `163.02s` |
+| Turbo 8-step v1.0 | `113.41s` | `7.28s` | `83.69s` | `220.06s` |
+| Base 12-step beta/RES | `114.67s` | `7.26s` | `83.34s` | `221.66s` |
+
+Both outputs contain H.264 video at 24 fps and stereo AAC audio at 32 kHz. The
+4-step audio measured `-31.4 dB` mean / `-10.9 dB` peak; the 8-step audio
+measured `-23.0 dB` mean / `-5.0 dB` peak; and the base 12-step audio measured
+`-22.4 dB` mean / `-5.2 dB` peak.
+
+```text
+/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-h3native2x-1920x1088.mp4
+/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-turbo8-h3native2x-1920x1088.mp4
+/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-h3native2x-1920x1088.mp4
+```
+
+### Attention Audio Isolation
+
+A matched base 12-step run used direct PyTorch SDPA instead of Sage2. Upscaling
+was disabled, but that does not affect the audio comparison because the learned
+upscaler runs after joint AV sampling and only receives the separated video
+latent.
+
+| Attention | Sampling | Final size | Audio level | Wall time |
+| --- | ---: | ---: | ---: | ---: |
+| Sage2 | `114.67s` | `1920x1088` after post-sample 2x | `-22.4 dB` mean / `-5.2 dB` peak | `221.66s` |
+| PyTorch SDPA | `144.40s` | `960x544`, no upscale | `-22.5 dB` mean / `-2.3 dB` peak | `171.21s` |
+
+The SDPA comparison file is:
+
+```text
+/home/daniel/StoryStudioAssets/H3-output/h3-native-upscale-t2v/talking-woman-base12-sdpa-960x544.mp4
+```
+
+Signal levels alone cannot establish dialogue quality or lip sync; the matched
+clips require listening review to determine whether Sage2 is the cause.
--- a/H3_PROMPT_GUIDE.md
+++ b/H3_PROMPT_GUIDE.md
@ -0,0 +1,167 @@
+# H3 Prompt Guide
+
+Project guidance for MiniMax H3 audiovisual prompts. This document combines
+MiniMax's official prompt format with observations from our direct-runtime and
+ComfyUI tests.
+
+## Core Principle
+
+Treat an H3 prompt as an audiovisual timeline, not as a loose prose image
+description. For T2VA and keyframe modes, use these exact fields in this order:
+
+```text
+integrated_multimodal_description: ...
+
+overall_soundscape: ...
+
+non_diegetic_music: ...
+```
+
+- `integrated_multimodal_description` contains shots, visible actions,
+  speakers, exact dialogue, singing, and synchronized diegetic events.
+- `overall_soundscape` contains ambience, Foley, and non-verbal human sounds
+  such as breathing, laughter, and panting.
+- `non_diegetic_music` contains only music heard by the audience rather than
+  the characters. Use `N/A` when no score is wanted.
+
+T2VA begins directly with these fields. I2VA, FL2VA, and L2VA add the official
+picture-alignment instruction before them. Ref2VA uses a different six-section
+format and should follow the official reference guide.
+
+## Dialogue Syntax
+
+Give every speaking character a stable speaker ID such as `(S1)` or `(S2)`.
+Describe the speaker, voice, action, and delivery outside the dialogue block.
+Use quoted speech as the project default and state the language outside it:
+
+```text
+The woman with a low, clear voice and measured pace (S1) says in English in a
+flat, matter-of-fact delivery: "The meeting starts at three."
+```
+
+MiniMax's official guide recommends `<d>[Language]...</d>`. Do not use that
+markup by default in this project. In a matched 10-seed Sage2 sweep, tagged
+dialogue produced immediate first-100ms activity in 9/10 cases, while quoted
+dialogue produced it in 0/10. All ten quoted WAVs passed subjective listening.
+See [`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md).
+
+Rules:
+
+- Preserve the dialogue wording and punctuation exactly.
+- Specify the language explicitly in prose, such as `says in English`.
+- Describe pitch, timbre, pace, volume, accent, and emotional restraint only
+  when useful.
+- Keep the same speaker ID across shots.
+- Use a compound ID such as `(S1,S2)` only when speakers vocalize together.
+- Reserve `<d>[Language]...</d>` for controlled compatibility experiments until
+  the upstream startup-audio defect is resolved.
+- Describe visible text explicitly without reusing dialogue syntax, for example
+  `a sign visibly reads MEETING ROOM`.
+- For voiceover, use `says in an off-screen voiceover` and state that the
+  visible character's lips remain completely closed.
+
+## Lip Sync And End State
+
+Describe speech as a physical action and specify what happens when it ends:
+
+```text
+Her mouth movements naturally synchronize with each spoken word. Immediately
+after the final word, her lips meet and her jaw ceases speaking motion. She
+remains silent through the final frame.
+```
+
+This is especially important in short clips. Without an explicit end state, H3
+may continue mouth motion or add a non-verbal vocal reaction after the line.
+
+For multiple speakers, state whose lips move and whose remain still during each
+line and pause.
+
+## Avoiding Unwanted Laughter
+
+Laughter is a non-verbal human sound, so control it in `overall_soundscape`, not
+inside the dialogue block:
+
+```text
+overall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum.
+No laughter, chuckling, giggling, sighing, gasping, audible breathing, filler
+sounds, audience reaction, or other voices.
+```
+
+Also reinforce the visible end state in the shot description: closed lips,
+stopped jaw motion, neutral expression, and silence through the final frame.
+
+These exclusions are project guidance, not a guaranteed negative-prompt
+mechanism. H3 jointly generates video and audio, so unwanted sounds remain
+probabilistic.
+
+Avoid positive cues that can conflict with the exclusion. In our first dialogue
+test, `She smiles slightly` repeatedly became laughter across Sage2, PyTorch
+SDPA, and FlashAttention-4. Since the behavior survived attention changes, the
+prompt was the stronger common cause. Use observable neutral behavior instead:
+
+```text
+She maintains a neutral, composed, closed-mouth expression.
+```
+
+## Short Dialogue Tests
+
+For a five-second diagnostic clip:
+
+- Use one visible speaker and one short sentence.
+- Use one static shot with no cuts.
+- Choose emotionally neutral wording.
+- Request a flat or matter-of-fact delivery.
+- Specify the post-dialogue closed-mouth state.
+- Permit only a simple, positive acoustic bed such as dry room tone.
+- Set `non_diegetic_music: N/A`.
+
+Do not use `overall_soundscape: N/A` for a speaking test. The official guide
+reserves it for complete silence throughout the video.
+
+## Reusable Neutral T2VA Prompt
+
+```text
+integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static
+medium close-up frames exactly one adult woman seated alone in an otherwise
+empty, quiet cafe. Warm natural window light falls evenly across her face. She
+maintains a neutral, composed, closed-mouth expression and looks steadily
+toward the camera. The woman with a low, clear voice and slow, even speaking
+pace (S1) physically speaks once in a flat, matter-of-fact delivery:
+"The meeting starts at three." Her mouth movements naturally
+synchronize with each spoken word. Immediately after the final word, her lips
+meet and her jaw ceases speaking motion. She remains silent and maintains the
+same neutral expression through the final frame. The camera remains completely
+static with no cuts.
+
+overall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum.
+No laughter, chuckling, giggling, smiling vocalization, sighing, gasping,
+audible breathing, filler sounds, audience reaction, or other voices.
+
+non_diegetic_music: N/A
+```
+
+## Prompt Checklist
+
+- Correct task mode selected.
+- Exact field names and order preserved.
+- Every shot has observable visual and audible events.
+- Every speaker has a stable ID.
+- Dialogue uses quoted speech.
+- Spoken language, delivery, and voice are stated outside the quotation.
+- `<d>[Language]...</d>` is avoided unless explicitly testing official syntax.
+- Lip motion and post-speech closure are explicit.
+- Ambience and non-verbal sounds are in `overall_soundscape`.
+- Music is isolated in `non_diegetic_music`.
+- Script length fits the requested duration.
+- Positive emotional cues do not conflict with audio exclusions.
+
+## Official Sources
+
+- Base prompt guide:
+  <https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md>
+- Reference prompt guide:
+  <https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md>
+- Official prompt-writing skill:
+  <https://github.com/MiniMax-AI/MiniMax-H3/tree/main/.agents/skills/h3-prompt-writing>
+- Official model repository:
+  <https://huggingface.co/MiniMaxAI/MiniMax-H3>
--- a/PARITY.md
+++ b/PARITY.md
@ -180,31 +180,27 @@ gate and is exact.

 | Component | Implemented | Known limitation |
 | --- | --- | --- |
-| Text-only Qwen | Yes | No vision encoder, MRoPE, image/video expansion, reference labels, or modality tags |
+| Qwen text and vision conditioning | Yes | Text-only prompt parity is established; first/last keyframe vision conditioning is implemented, while arbitrary reference video/audio remains missing |
 | Token refiner | Yes | Bit-exact from captured 5376-wide refiner input through both blocks and final RMSNorm; Qwen-to-refiner projection boundary is still not separately captured |
-| Prompt-only FL2VA packer | Yes | Bit-exact for the coherent captured text-only FL2VA DiT input; no keyframe/reference condition rows |
+| T2VA/keyframe packer | Yes | Bit-exact for the coherent captured prompt-only DiT input; first-frame, last-frame, and first/last keyframe condition rows are implemented |
 | H3 DiT backbone | Yes | Bit-exact through all 50 blocks from the coherent assembled FL2VA input; requires the standalone Comfy Kitchen fused Q/K RMSNorm + split-half RoPE operator |
 | H3 final layer | Yes | Bit-exact final AdaLN, target-row modulation, and video/audio patch rows; Comfy materializes the AdaLN and output-head biases through BF16 |
 | H3 DiT | Yes | Strict all-block numeric parity not achieved |
 | Beta/RES sampler | Yes | H3 replay is bit-exact for all 12 captured sampler calls and RES update is bit-exact from captured denoised; direct raw-output to model-wrapper denoised conversion retains small BF16/state boundary deltas |
-| Video VAE decoder | Yes | Same final sampled latent decoded through direct VAE versus Comfy PNG frames: mean abs pixel delta `0.0833898`, max `6` |
-| Audio VAE/decode/mux | No | Preview intentionally produces video only |
-| End-to-end prompt-only FL2VA preview | Yes | Direct 320x192x22 dragon smoke videos generated with Sage2. Comfy-equivalent CPU joint AV seed initialization matches captured `initial.pt` exactly. Video-side wrapper denoised conversion is exact after BF16 raw-output materialization; remaining seed-exact final-latent delta is localized to regenerated text conditioning plus audio-side wrapper conversion |
-| Full Ref2VA | No | References, vision conditioning, VAE encode, audio, and muxing are unimplemented |
+| Video VAE decoder | Yes | Direct VAE temporal assembly matches upstream after overlap fix. FP16 is the default Comfy-equivalent runtime path; cat benchmark VAE decode is `25.085s`. Use FP32 only for exact direct diagnostics. |
+| Audio VAE/decode/mux | Yes | Direct decoder-only MiniMax H3 audio VAE returns stereo `32000 Hz` waveform and muxes with generated video. Native audio latent scaling is fixed. |
+| End-to-end prompt-only FL2VA preview | Yes | Apples-to-apples warm cat benchmark is at Comfy parity: Comfy warm `150.26s`; direct warm after text conditioning `149.304s`; direct warm including text conditioning `151.465s`. |
+| Full Ref2VA | No | Image keyframes, vision conditioning, VAE encode, generated audio, and muxing are implemented; arbitrary reference video/audio and identity/voice conditioning remain missing |

 ## Remaining Gates, In Dependency Order

 Only these are outstanding. Do not recapture or revisit rows marked complete
 unless the checkpoint, Comfy version, prompt, or backend changes.

-1. **Final video latent layout.** Compare direct `unpatchify_video` output
-   with Comfy before entering the sampler.
-2. **Text/audio end-to-end closure.** Direct H3, video wrapper conversion,
-   RES update, VAE decode, and initial AV noise now match their captures.
-   Close regenerated text conditioning and audio-side wrapper conversion before
-   claiming identical seed output.
-3. **Feature/performance work.** Only then add audio, Ref2VA/reference paths,
-    Sage3, CUDA graphs, and multi-GPU execution.
+1. **Distributed validation.** Run real NCCL parity and performance sweeps at
+   2/4/6/8 GPUs; CPU transport and one-GPU real-checkpoint identity are complete.
+2. **Full Ref2VA support.** Add arbitrary reference video/audio, identity and
+   voice conditioning, and reference-path validation gates.

 ## Existing Tools And Their Intended Gate

--- a/PLAN.md
+++ b/PLAN.md
@ -1,5 +1,8 @@
 # H3 Blackwell Runtime Plan

+Current implementation status is tracked in [`CURRENT_STATE.md`](CURRENT_STATE.md).
+The dated handoffs below are retained as historical investigation records.
+
 ## Goal

 Build a direct MiniMax H3 Ref2VA runtime for Blackwell and Grace Blackwell that consumes the current Comfy safetensors checkpoints while removing ComfyUI and Raylight from the denoising critical path.
@ -48,6 +51,61 @@ The direct runner must first match the model contract and output quality. Beatin
 5. Implement ragged Ulysses Sage3 with transport-identity and distributed-versus-single-Sage3 tests.
 6. Sweep Ulysses/tensor-parallel layouts on 2/4/6/8 GPUs in an NVLink/NVSwitch domain.

+## Performance Backend Plan
+
+Prompt-only FL2VA is now at warm Comfy parity with the direct Sage2 baseline. Feature and performance work should proceed in this order:
+
+1. Validate and benchmark the existing `sage3` backend against the same cat prompt, seed, dimensions, and FP16 VAE runtime path used for Sage2 parity.
+   - First cat benchmark result: Sage3 runs successfully but is slower than Sage2 in this direct path. Sampling was `123.675s` versus Sage2 `114.414s`; warm after text conditioning was `158.111s` versus Sage2 `149.304s`. Same-seed MP4 frame diff versus Sage2 was mean `46.563`, max `255`, so keep Sage3 experimental pending human visual review and stricter tensor gates.
+2. Build a persistent hot runtime service instead of measuring only process-per-run CLIs. A warm container must preload and retain Qwen, H3, video VAE, and audio VAE in GPU memory, then accept video jobs without model-load latency. Add explicit startup warmup, readiness reporting, request-level attention selection (`sage2`, `sdpa`, `sage3` initially), and timing fields that separate resident-model request latency from cold startup.
+3. Add exact memory/lifetime optimizations next: `kj_head_sliced` and `kj_chunked_ffn`. These must preserve the validated direct outputs before being kept.
+4. Evaluate prior H3-tested attention candidates as standalone adapters: `sol_attn` and `kj_sage`.
+   - `kj_sage` is implemented as explicit SageAttention mode backends: `kj_sage_cuda`, `kj_sage_triton`, `kj_sage_fp8`, and `kj_sage_fp8pp`; all passed hot-runtime smoke tests.
+   - `sol_attn` is implemented through the pinned ComfyUI Triton source vendored
+     into the Spark image. The native QKV layout is deployed; the inactive fused
+     layout remains a known repair item.
+5. Evaluate approximate denoiser caches only after exact baselines are recorded: `easycache` and `h3_cache`.
+   - Initial direct cache modes are implemented as opt-in approximate sampler modes. They reuse cached denoised deltas and report skipped-step stats; full-size quality/threshold sweeps are still required before using them for production output.
+6. Keep every backend explicit per run, with separate quality and timing records for sampling, VAE, audio, and end-to-end output.
+
+## Next Performance Plan: Blackwell NVFP4 GEMMs
+
+The latest attention/cache sweeps show that attention backend swaps are not the main remaining speed lever. Full request times for `sage2`, `kj_sage_fp8`, `kj_sage_fp8pp`, `kj_head_sliced`, and `sol_attn` are close, while cache gains come from skipping denoiser calls and must remain quality-gated. The next exact performance target is therefore the dense transformer linear stack.
+
+Target mixed-precision policy:
+
+- Keep residual stream, RMSNorm, RoPE, modulation/AdaLN, residual adds, timestep embeddings, softmax, final layer, audio VAE, and video VAE in BF16/FP16.
+- Keep attention accumulation/output in the best measured BF16/FP16 backend until a separate quality gate proves otherwise.
+- Replace only the large transformer GEMMs first: QKV projection, attention output projection, MLP gate/up projection, and MLP down projection.
+- Prioritize Blackwell-native NVFP4 activation x NVFP4 weight GEMMs with BF16 accumulation/output, avoiding unnecessary dequantize/contiguous boundaries.
+
+Kernel candidates to test, in order:
+
+1. Current `Nvfp4Linear`/Comfy Kitchen path as the measured baseline.
+2. Transformer Engine NVFP4BlockScaling prototype to establish expected Blackwell FP4 behavior with a higher-level NVIDIA stack.
+3. CUTLASS/CuTe block-scaled NVFP4 GEMM prototype for owned hot-path kernels, exact layouts, fused epilogues, and persistent-kernel experiments.
+4. cuBLASLt NVFP4/grouped GEMM where CUDA exposes a supported operation on the current Blackwell target.
+
+Current NVFP4 activation quantization status:
+
+- `H3_NVFP4_SCALE_BACKEND=vortex` is parity-safe for BF16 H3 activations and is the useful optimization seam. The tuned scale kernel uses 128 threads / 256 blocks and reaches about 88-90% of DGX Spark's advertised LPDDR5X bandwidth on FC1/FC2 scale discovery.
+- Standalone scale discovery is effectively done for Spark; remaining theoretical headroom is too small to justify more absmax-only tuning.
+- `vortex_native` BF16->NVFP4 pack is parity-safe for real H3 FC1/FC2 activation shapes, including qdata, cuBLAS tiled block-scale layout, signed zero, tie-to-even FP4 rounding, and downstream `scaled_mm_nvfp4` linear output.
+- Keep Comfy Kitchen as the default production packer. Pack-only sweep `8d90cec` shows native 512-thread pack is essentially tied on FC1 (`1.090 ms` vs CK `1.098 ms`) but still slower on FC2 (`2.977 ms` vs CK `2.886 ms`). Use `vortex_native` only as an experimental benchmark path until the wide-FC2 pack kernel is redesigned.
+
+Validation and profiling sequence:
+
+1. Use `tools/profile_h3_block.py` to measure one representative H3 block before writing kernels. Record QKV, RoPE/RMS, attention kernel, output projection, MLP fc1, activation, MLP fc2, modulation/gating, and total block time.
+2. Repeat profiling for representative block indices, at least block `0`, `24`, and `49`, because token statistics and cache behavior can differ through depth.
+3. Confirm which component dominates before implementing a kernel replacement.
+4. Add candidate GEMM backends behind the existing `Nvfp4Linear` API so model code and correctness tests remain stable.
+5. Gate each candidate by layer-level max/mean error, one-block output error, one denoiser-step tensor error, and finally full video quality.
+
+DGX Spark caveat:
+
+- GB10 reports SM121, not B200/GB200 SM100. CUTLASS, Transformer Engine, and cuBLASLt FP4 coverage must be probed independently on Spark before assuming B200 examples work unchanged.
+- Keep separate kernel policy notes for GB10/Spark and B200/GB200-class Blackwell.
+
 ## Non-Negotiable Validation

 - Never silently pad semantic H3 tokens for unmasked attention.
@ -56,8 +114,40 @@ The direct runner must first match the model contract and output quality. Beatin
 - Record attention, GEMM, communication, VAE, and end-to-end timings separately.
 - Treat SageAttention3 as an experimental quality-gated kernel for H3.

+## 2026-08-15 Handoff
+
+Current quality baseline:
+
+- `sage2` is the correct visual path for the 960x544, 124-frame, 12-step cat prompt at seed `440407`.
+- Fresh Spark hot-runtime Sage2/Vortex A/B asset: `\\192.168.1.162\StoryStudioAssets\H3-output\h3-blackwell-runtime\cat-sage2-vortex-5s-440407-ab.mp4`.
+- `sol_attn` is faster but visually wrong for the same prompt/seed. Keep it experimental until localized and quality-gated.
+- Sol-native BSHD layout is exact versus the old Sol path, so the visual issue is likely Sol's approximation/configuration, not the layout removal.
+
+Tomorrow priority:
+
+1. Diagnose `sol_attn` quality failure.
+   - Save per-step latents for `sage2` and `sol_attn` with identical prompt, seed, dimensions, and sampler settings.
+   - Measure per-step max/mean/RMSE/cosine divergence.
+   - Try hybrid schedules: early steps `sage2`, later steps `sol_attn`; also try Sage2 every Nth step.
+   - Sweep Sol params only after the divergence point is known: `H3_SOL_TAU`, `H3_SOL_THRESH_TYPE`, `H3_SOL_INT8_QK`, `H3_SOL_INT8_PV`.
+2. Optimize the quality path (`sage2`).
+   - Use `sage2` as visual/numerical reference.
+   - Attack Q/K/V HND layout and Q/K RMS+RoPE first; a fused Q/K RMS+RoPE + Sage HND layout kernel is the most plausible exact win.
+   - Keep Sage3 experimental; previous measurements were slower and visually/numerically different.
+3. Add durable hot-runtime request logging.
+   - Log `generate_start`, `generate_complete`, and `generate_failed` to stdout with output path, seed, attention, dimensions, and timings.
+   - This prevents losing timing data when a client disconnects before reading the synchronous response.
+4. Check LTX 2.5 downloads.
+   - Detached Spark container: `ltx25-nvfp4-download`.
+   - Expected files under `/home/daniel/aeon-spark-test/h3/comfy-models`:
+     - `diffusion_models/ltx-2.5-22b-distilled-transformer-nvfp4.safetensors`
+     - `text_encoders/gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors`
+   - Official `Lightricks/LTX-2.5` does not list a matching NVFP4 text encoder; the int8 ConvRot Gemma4 encoder is the matching low-VRAM official text encoder.
+
 ## 2026-08-13 VAE Debug Handoff

+Resolved on `vae-decode-optimization`: direct VAE temporal overlap constants now match upstream, audio decode/mux is implemented, and Comfy-equivalent FP16 video VAE is the default runtime path. The 960x544x124 cat benchmark now matches warm Comfy performance: Comfy `150.26s`, direct `149.304s` after text conditioning, direct VAE decode `25.085s`.
+
 Current saved latent and comparison assets live under:

 - Spark: `/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime`
--- a/README.md
+++ b/README.md
@ -1,49 +1,165 @@
 # H3 Blackwell Runtime

-Direct MiniMax H3 Ref2VA runtime research project. ComfyUI is the checkpoint and correctness oracle, not the target runtime.
+Direct MiniMax H3 audiovisual inference for NVIDIA Blackwell and Grace Blackwell.
+The runtime consumes the current Comfy-format NVFP4 checkpoints while keeping
+ComfyUI out of the inference critical path. ComfyUI remains the checkpoint and
+correctness oracle.

-## First Gate
+## Current Status

-Inspect the mounted H3 NVFP4 safetensors headers before designing an importer:
+Implemented and validated:

-```powershell
-python .\tools\inspect_safetensors.py /runpod-volume/ComfyUI/models/diffusion_models/minimax_h3_ref2va_pruned_nvfp4.safetensors
+- Prompt-only T2VA with jointly generated video and stereo audio.
+- First-frame I2VA, last-frame L2VA, and first/last-frame FL2VA conditioning.
+- Qwen3-VL text and image conditioning, token refinement, H3 packed denoising,
+  beta/RES sampling, video and audio VAE decode, H.264/AAC encoding, and muxing.
+- Resident HTTP runtime with startup warmup, model reuse, request timing, and
+  request-level attention selection.
+- SageAttention2 correctness default plus SDPA, forced cuDNN SDPA,
+  FlashAttention-4, Sage3, KJ Sage, Comfy Kitchen INT8, head-sliced, and Sol-Attn
+  experimental backends.
+- Official FL2VA Turbo 4-step and 8-step adapters.
+- Optional H3-native 3D latent upscaling.
+- Opt-in EasyCache and H3-Cache experiments.
+- Ragged Ulysses sequence parallelism and true NVFP4 tensor parallelism with
+  automatic 1/2/4/6/8-GPU launch tooling.
+
+The main remaining feature gap is full arbitrary Ref2VA, especially reference
+video/audio, identity, and voice conditioning. Multi-GPU code is CPU- and
+one-GPU-validated; real 2/4/6/8-GPU NCCL scaling measurements are still pending.
+See [`CURRENT_STATE.md`](CURRENT_STATE.md) for the canonical detailed status.
+
+## Checkpoints
+
+The default runtime paths are:
+
+```text
+/models/minimax_h3_fl2va_pruned_nvfp4.safetensors
+/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors
+/vae/ae.safetensors
+/vae/mini_vae.safetensors
 ```

-Write the output to `artifacts/checkpoints/` on the mounted volume. The result must identify packed weights, scales, and tensor naming before any kernel conversion work begins.
-
-## Benchmark Contract
-
-`benchmarks/ref2va-960x544-124f.json` is the single-GPU performance contract. Record direct-runner results as JSON and compare them with:
-
-```powershell
-python .\tools\compare_benchmark.py --result direct-result.json
-```
+The denoiser and Qwen checkpoints use Comfy Kitchen NVFP4 layouts. Do not
+convert or dequantize them during loading.

 ## DGX Spark

-`Dockerfile.spark` and `compose.spark.yml` prepare an ARM64 GB10 development image using the existing AEON CUDA 13/SageAttention3 base. The compose target opens a shell only; it does not start inference.
-
-### Forgejo Pulls From Spark
-
-The Spark checkout uses Forgejo through the host's published local SSH port and a dedicated key:
+`Dockerfile.spark` and `compose.spark.yml` provide the ARM64 CUDA 13 runtime used
+on GB10. Build and start the resident service from the Spark checkout:

 ```bash
-cd /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime
-git config core.sshCommand 'ssh -i ~/.ssh/id_ed25519_forgejo_h3 -o IdentitiesOnly=yes'
-git remote set-url origin ssh://git@127.0.0.1:2222/daniel/h3-blackwell-runtime.git
-git pull --ff-only origin master
+docker compose -f compose.spark.yml build h3-hot-runtime
+docker compose -f compose.spark.yml up -d h3-hot-runtime
+curl http://127.0.0.1:8001/ready
 ```

-The private key remains on Spark at `~/.ssh/id_ed25519_forgejo_h3`; only its public key is registered in Forgejo.
+The service listens on container port 8000 and Spark host port 8001. It keeps
+Qwen, H3, both VAEs, Turbo adapters, and the optional latent upscaler resident.
+SageAttention2 is the default because it matches the established Comfy quality
+baseline.

-## Runtime Output
+## HTTP API

-Generation and latent-decode tools are quiet by default: they suppress ffmpeg banners and only print compact JSON summaries. Use these flags when debugging:
+`GET /health` and `GET /ready` report readiness, loaded options, warmup results,
+and attention backend status. `POST /generate` performs one serialized request:

- `--progress`: print per-step sampler timing in `tools/direct_t2v_preview.py`.
- `--profile-memory`: print memory checkpoints in `tools/direct_t2v_preview.py`.
- `--ffmpeg-loglevel info`: show ffmpeg details instead of the default `error` level.
- `--quiet`: suppress JSON summary lines.
+```bash
+curl -X POST http://127.0.0.1:8001/generate \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "prompt": "A quiet medium shot of a woman by a rain-streaked window. She says, \"We should leave before dawn.\"",
+    "output": "/output/h3-blackwell-runtime/example.mp4",
+    "width": 864,
+    "height": 480,
+    "frames": 141,
+    "steps": 12,
+    "seed": 440420,
+    "attention": "sage2"
+  }'
+```

-Standalone `tools/compare_*`, `tools/trace_*`, `tools/inspect_*`, and `tools/patch_comfy_*` scripts are debugging utilities and remain opt-in by being separate commands.
+Optional request fields include `first_frame`, `last_frame`, `turbo`, `upscale`,
+`mux_audio`, `save_latent`, `keep_intermediates`, `cache_mode`, and cache tuning
+parameters. Keyframes accept an on-disk image path, base64 payload, or data URL.
+The presence of first and last frames selects I2VA, L2VA, or FL2VA behavior; a
+named task field is not yet exposed.
+
+Set `turbo` to `"4step"` or `"8step"`; the service enforces the corresponding
+step count and Turbo schedule. Set `upscale` to `2.0` to run the resident learned
+latent upscaler before video decode. Turbo and denoiser caching cannot be
+combined.
+
+Quoted dialogue is the project prompt default. Tagged `<d>[English]...` dialogue
+is a repeatable startup-audio trigger and should not be used as the default.
+Never apply unconditional audio trimming or fading because valid sound can begin
+at the first sample. See [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md) and
+[`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md).
+
+## Distributed Execution
+
+Two batch-generation modes are available through `torchrun`:
+
+- `ulysses`: token-sharded blocks with ragged sequence-to-head all-to-all around
+  attention.
+- `tensor`: sequence-sharded residuals plus NVFP4-sharded QKV, attention output,
+  MLP FC1, and MLP FC2 projections.
+
+Use all visible GPUs or pass an explicit world size:
+
+```bash
+tools/run_distributed_t2va.sh ulysses
+tools/run_distributed_t2va.sh tensor 8 sdpa
+H3_SAVE_LATENTS=0 tools/run_distributed_matrix.sh
+```
+
+The matrix runs feasible 1/2/4/6/8-GPU configurations and skips counts larger
+than the visible device count. Six ranks use ragged head ownership
+`[10, 10, 9, 9, 9, 9]`; no semantic token padding is introduced.
+
+See [`DISTRIBUTED.md`](DISTRIBUTED.md) for the collective contracts, validation
+evidence, environment variables, RunPod provisioning client, and x86 packaging
+status.
+
+## Validation
+
+Run the contract suite with:
+
+```bash
+python -m unittest discover -s tests -v
+```
+
+The current suite has 38 passing tests. Distributed tests cover 2/4/6/8-rank
+transport identity, ragged collectives, SDPA parity, final projection parity,
+NVFP4 shard layout, and 2/6-rank TP attention/MLP math. On GB10, real-checkpoint
+one-rank Ulysses and tensor paths produced exactly equal video and audio latents
+at 864x480, 141 frames, 12 steps, seed 440420.
+
+Matched GB10 backend results and parity evidence are recorded in [`PLAN.md`](PLAN.md)
+and [`PARITY.md`](PARITY.md). Standalone `tools/compare_*`, `tools/trace_*`,
+`tools/inspect_*`, and `tools/patch_comfy_*` commands are diagnostic utilities,
+not runtime startup requirements.
+
+## Documentation
+
+- [`CURRENT_STATE.md`](CURRENT_STATE.md): canonical implemented and missing scope.
+- [`DISTRIBUTED.md`](DISTRIBUTED.md): Ulysses, TP, launchers, and RunPod workflow.
+- [`H3_PROMPT_GUIDE.md`](H3_PROMPT_GUIDE.md): audiovisual prompting contract.
+- [`AUDIO_BOUNDARY_INVESTIGATION.md`](AUDIO_BOUNDARY_INVESTIGATION.md): startup
+  audio localization and prompt-format evidence.
+- [`TURBO.md`](TURBO.md): official Turbo adapters and schedules.
+- [`H3_LATENT_UPSCALER.md`](H3_LATENT_UPSCALER.md): learned latent upscaler.
+- [`FLASH4.md`](FLASH4.md): FlashAttention-4 integration and benchmark.
+- [`PARITY.md`](PARITY.md): direct-versus-Comfy evidence ledger.
+- [`PLAN.md`](PLAN.md): historical investigation and future kernel plan.
+
+## Known Gaps
+
+- Full arbitrary Ref2VA reference video/audio and identity/voice conditioning.
+- Explicit task schemas and stricter production request validation.
+- Real multi-GPU NCCL parity and 1/2/4/6/8 scaling results.
+- x86 SageAttention2 packaging for the generic RunPod image; SDPA is the initial
+  cloud validation backend.
+- Production queueing, cancellation, authentication, TLS, metrics, and durable
+  job state.
+- Full quality sweeps for approximate attention, cache, Turbo, and upscaler paths.
--- a/TURBO.md
+++ b/TURBO.md
@ -0,0 +1,94 @@
+# FL2VA Turbo
+
+The hot runtime can keep both official LightX2V FL2VA Turbo adapters resident and switch them per request without rebuilding the packed NVFP4 base model.
+
+## Variants
+
+| API value | Official artifact | NFE | Video/audio shift | Schedule |
+| --- | --- | ---: | --- | --- |
+| `4step` | `minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors` | 4 | 6/3 | Uniform shifted training-Euler |
+| `8step` | `minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors` | 8 | 12/3 | Uniform shifted training-Euler |
+
+Downloaded from Hugging Face revision `ec01fa4c86263832faa0bd1d6d8f36a281eaabb2`:
+
+| Artifact | SHA-256 |
+| --- | --- |
+| 4-step v1.1 Comfy BF16 | `449d80f301ac571622c72e28b8fd72a4b3681b7a8df8a92f17c8f6ec43f56558` |
+| 8-step v1.0 Comfy BF16 | `2339acdf19bfe123f46b971ea35d367a84adb85de43627e1eceafa5a5b2b111e` |
+
+The older unversioned 744 MB adapters remain on Spark but are not loaded by the hot service.
+
+## Implementation
+
+Each official file contains 624 tensors covering 208 linear targets: four projections in each of 50 denoiser blocks and two token-refiner blocks. The runtime validates the complete key set, dimensions, rank, and alpha while loading.
+
+The adapters are applied dynamically:
+
+```text
+output = NVFP4_base(x) + strength * (alpha / rank) * B(A(x))
+```
+
+The BF16 branch receives the original activation. It does not receive the base checkpoint's `pre_quant_scale`-modified activation, and the LoRA delta is never merged into packed NVFP4 weights. Fused QKV uses rank/alpha 384; the other released projections use rank/alpha 128.
+
+Turbo sampling maintains independent native video and audio states. The 4-step and 8-step variants use their own shifted sigma grids and positive data-ward training-Euler updates. The original beta/RES sampler remains unchanged when Turbo is disabled.
+
+## Hot API
+
+The compose service registers both adapters with repeated startup arguments:
+
+```text
+--turbo-lora 4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors
+--turbo-lora 8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors
+```
+
+Select one per request:
+
+```json
+{
+  "prompt": "A calm restaurant at dusk.",
+  "output": "/output/example.mp4",
+  "width": 960,
+  "height": 544,
+  "frames": 124,
+  "seed": 440408,
+  "turbo": "4step"
+}
+```
+
+The server chooses and enforces four or eight steps. Use `null` or `"none"` for the base path. Turbo and denoiser caching cannot be combined. `/ready` reports `available_turbos` and `current_turbo`.
+
+## Matched GB10 Benchmark
+
+All runs used the same Posly first frame, prompt, seed `440408`, 960x544 canvas, 124 frames, Sage2, FP16 tiled VAE, and no audio output.
+
+| Mode | Steps | Sampling | Total wall | Speedup vs base 8-step | Sampling reduction |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Base beta/RES | 8 | 148.1 s | 179.3 s | 1.00x | baseline |
+| Turbo v1.0 | 8 | 121.1 s | 152.3 s | 1.18x | 18.2% |
+| Turbo v1.1 768p | 4 | 60.4 s | 91.2 s | 1.97x | 59.2% |
+
+The 4-step run was 1.67x faster end-to-end than the 8-step Turbo run. These are observed hot-service request times, not isolated steady-state kernel timings; shape-specific compilation can affect the first request.
+
+## Validation
+
+- 22 unit/contract tests cover disabled exactness, dynamic branch arithmetic, adapter selection, both official sigma grids, and independent AV training-Euler updates.
+- Both official files loaded with all 624 expected keys and all 208 target dimensions validated.
+- Both variants generated matched 960x544x124 FL2VA proofs.
+- Both variants completed joint AV smoke runs and produced stereo 32 kHz AAC muxed outputs.
+- Base warmup remains bit-preserving while no adapter is selected.
+
+Comparison outputs:
+
+```text
+\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo4-v1.1-seed440408-5s.mp4
+\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo8-v1.0-seed440408-5s.mp4
+\\192.168.1.162\StoryStudioAssets\H3-output\posly\proofs-960x540\01-posly-hero-turbo4-left-turbo8-right-seed440408.mp4
+```
+
+## Quality Notes
+
+For the Posly restaurant source, both Turbo variants avoided the severe central lens flare produced by the base-model trials. The 4-step result appeared slightly more compositionally stable; the 8-step result showed more subject motion. Both still brightened the exterior over time, so Turbo does not by itself satisfy the fixed-exposure requirement.
+
+The adapters were released against the BF16 H3 base. Dynamic use over the pruned NVFP4 checkpoint is operationally validated here but is not an upstream-certified pairing. A future strict parity gate should compare per-step denoiser outputs and final latents against the latest LightX2V BF16 reference.
+
+SLA Turbo is intentionally excluded. It requires LightX2V's separate 85% dynamic sparse-attention path and should not be represented as equivalent to these dense variants.
--- a/benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json
+++ b/benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json
@ -0,0 +1,50 @@
+{
+  "name": "audio-dialogue-format-sweep-sage2-seeds440420-440429",
+  "measured_at": "2026-08-21",
+  "hardware": "NVIDIA GB10",
+  "model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
+  "attention": "sage2",
+  "scheduler": "beta",
+  "sampler": "res_multistep",
+  "steps": 12,
+  "resolution": [864, 480],
+  "frames": 141,
+  "seeds": [440420, 440421, 440422, 440423, 440424, 440425, 440426, 440427, 440428, 440429],
+  "comparison": "Identical dialogue semantics with <d>[English]...</d> markup versus quoted speech",
+  "cases": 20,
+  "aggregate": {
+    "quoted_first_100ms_peak_reduction_db": {
+      "mean": 20.112007323193758,
+      "median": 21.550717420448272,
+      "minimum": 3.800458970207643,
+      "maximum": 27.93839234881301
+    },
+    "quoted_first_100ms_rms_reduction_db": {
+      "mean": 17.932329146978255,
+      "median": 17.504237701162523,
+      "minimum": 3.147146068931228,
+      "maximum": 27.10315527948587
+    },
+    "first_100ms_peak_above_minus_40_dbfs": {
+      "tagged": 9,
+      "quoted": 0
+    },
+    "first_activity_under_100ms": {
+      "tagged": 9,
+      "quoted": 0
+    },
+    "boundary_decay_above_10db": {
+      "tagged": 8,
+      "quoted": 0
+    }
+  },
+  "latent_feature_result": "Simple first-frame and first-block RMS/delta ranges overlap between tagged and quoted cases; no scalar latent threshold is justified.",
+  "subjective_audio_review": {
+    "reviewed_at": "2026-08-21",
+    "quoted_cases_reviewed": 10,
+    "result": "All ten quoted WAVs judged good"
+  },
+  "full_video_lip_sync_review_required": true,
+  "output_directory": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/audio-dialogue-format-sweep",
+  "full_report": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/audio-dialogue-format-sweep/report.json"
+}
--- a/benchmarks/ref2va-960x544-124f.json
+++ b/benchmarks/ref2va-960x544-124f.json
@ -18,5 +18,35 @@
  "sampler": "res_multistep",
  "seed": 440202,
  "reference_comfy_sage3_seconds": 49.893,
-  "measurement": "ComfyUI prompt execution time after warm-up"
+  "measurement": "ComfyUI prompt execution time after warm-up",
+  "prompt_only_fl2va_cat_benchmark": {
+    "seed": 440407,
+    "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
+    "comfy_warm_seconds": 150.26,
+    "direct_warm_after_text_conditioned_seconds": 149.304,
+    "direct_warm_including_text_conditioning_seconds": 151.465,
+    "direct_cold_through_audio_decode_seconds": 180.494,
+    "direct_video_vae_dtype": "float16",
+    "direct_video_vae_tile_size": 256,
+    "direct_video_vae_decode_seconds": 25.085,
+    "direct_output": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\direct-cat-benchmark-960x544-124f-seed440407-fp16-vae.mp4",
+    "direct_log": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\benchmarks\\direct-cat-benchmark-960x544-124f-seed440407-fp16-vae.log",
+    "comfy_output": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\comfy-cat-benchmark-960x544-124f-seed440407_00001_.mp4",
+    "comfy_metrics": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\benchmarks\\comfy-cat-benchmark-960x544-124f-seed440407.json"
+  },
+  "prompt_only_fl2va_cat_sage3_benchmark": {
+    "seed": 440407,
+    "attention_backend": "sage3",
+    "direct_cold_through_audio_decode_seconds": 184.646,
+    "direct_warm_after_text_conditioned_seconds": 158.111,
+    "direct_warm_including_text_conditioning_seconds": 160.38,
+    "direct_sample_seconds": 123.675,
+    "direct_video_vae_dtype": "float16",
+    "direct_video_vae_tile_size": 256,
+    "direct_video_vae_decode_seconds": 25.024,
+    "sage2_mp4_frame_diff_max": 255,
+    "sage2_mp4_frame_diff_mean": 46.56300230273561,
+    "direct_output": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\direct-cat-benchmark-960x544-124f-seed440407-sage3-fp16-vae.mp4",
+    "direct_log": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-blackwell-runtime\\benchmarks\\direct-cat-benchmark-960x544-124f-seed440407-sage3-fp16-vae.log"
+  }
 }
--- a/benchmarks/t2va-dialogue-864x480-141f-base12-sdpa-seed440420.json
+++ b/benchmarks/t2va-dialogue-864x480-141f-base12-sdpa-seed440420.json
@ -0,0 +1,349 @@
+{
+  "name": "t2va-dialogue-864x480-141f-base12-sdpa-seed440420",
+  "measured_at": "2026-08-20",
+  "measurement": "Warm resident direct-runtime request including conditioning, sampling, video/audio decode, encoding, and mux",
+  "hardware": "NVIDIA GB10",
+  "runtime_image": "sha256:d842225c9c5036647ee5cab1fae2e093357cc8041bd8617e33a82c81d0bcc874",
+  "torch": "2.9.1+cu130",
+  "model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
+  "task": "t2va",
+  "resolution": [864, 480],
+  "frames": 141,
+  "fps": 24,
+  "duration_seconds": 5.875,
+  "steps": 12,
+  "scheduler": "beta",
+  "sampler": "res_multistep",
+  "seed": 440420,
+  "attention": "sdpa",
+  "turbo": null,
+  "upscale": null,
+  "mux_audio": true,
+  "keep_intermediates": false,
+  "vae_dtype": "float16",
+  "vae_tile_size": 256,
+  "mlp_chunks": 1,
+  "prompt": "integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static medium two-shot frames exactly two adults seated across from each other at a small table in a quiet, otherwise empty meeting room. A composed adult man sits on the left and a composed adult woman sits on the right. Both maintain neutral, attentive expressions. The man with a low, clear baritone voice and measured speaking pace (S1) looks toward the woman and says in a calm, matter-of-fact delivery: <d>[English] The north entrance closes at six.</d> During S1's line, only his lips and jaw move; the woman's lips remain completely closed. Immediately after his final word, his lips meet and his jaw ceases speaking motion. After a brief silent pause, the woman with a clear alto voice and measured speaking pace (S2) looks toward the man and replies in a calm, matter-of-fact delivery: <d>[English] Then we should leave by five thirty.</d> During S2's line, only her lips and jaw move; the man's lips remain completely closed. Immediately after her final word, her lips meet and her jaw ceases speaking motion. Both remain silent with neutral, closed-mouth expressions through the final frame. There is no overlapping speech. The camera remains completely static with no cuts.\n\noverall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum. Only S1 and S2 are audible, one at a time. No laughter, chuckling, giggling, smiling vocalization, sighing, gasping, audible breathing, filler sounds, audience reaction, narration, or other voices.\n\nnon_diegetic_music: N/A",
+  "timings_seconds": {
+    "latents_initialized": 0.018280818010680377,
+    "text_conditioned": 3.531974215002265,
+    "sampled": 125.30079188900709,
+    "vae_decoded": 23.84665890400356,
+    "pixels_cpu": 0.06607734999852255,
+    "raw_write": 2.75871228199685,
+    "video_encode": 0.39305945999512915,
+    "audio_decoded": 0.3400787889986532,
+    "audio_raw_write": 0.0008342489891219884,
+    "audio_encode": 0.034553833000245504,
+    "mux": 0.19705491400964092,
+    "request": 156.48807670301176,
+    "wall": 156.56338787100685
+  },
+  "throughput": {
+    "sampling_frames_per_second": 1.1252921699401506,
+    "wall_frames_per_second": 0.9005936950992043,
+    "sampling_realtime_factor": 21.327794364086314,
+    "wall_realtime_factor": 26.649087297192655
+  },
+  "cache": {
+    "mode": null,
+    "threshold": 0.0,
+    "skipped_steps": 0,
+    "rates": []
+  },
+  "output": {
+    "linux_path": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
+    "unc_path": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-baselines\\dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
+    "sha256": "2eb61334978b42bcec00ec68449f3f39139d9e24ef5ba70ef2240dc8bf7116e0",
+    "size_bytes": 429062,
+    "container_bit_rate": 584254,
+    "duration_seconds": 5.875,
+    "video": {
+      "codec": "h264",
+      "profile": "High",
+      "pixel_format": "yuv420p",
+      "width": 864,
+      "height": 480,
+      "fps": 24,
+      "frames": 141,
+      "bit_rate": 439001
+    },
+    "audio": {
+      "codec": "aac",
+      "profile": "LC",
+      "sample_rate": 32000,
+      "channels": 2,
+      "channel_layout": "stereo",
+      "bit_rate": 136335,
+      "mean_volume_db": -25.8,
+      "max_volume_db": -4.3,
+      "integrated_loudness_lufs": -21.18,
+      "true_peak_dbtp": -4.34,
+      "loudness_range_lu": 4.10,
+      "loudness_threshold_lufs": -33.94
+    }
+  },
+  "attention_backend_matrix": {
+    "reference_backend": "sdpa",
+    "comparison_note": "Video PSNR and decoded-PCM audio SNR measure output drift from SDPA, not subjective quality. Audio SNR is computed directly from aligned float PCM arrays; earlier FFmpeg apsnr results above 170 dB were invalid and have been replaced.",
+    "runs": {
+      "sdpa": {
+        "sampled_seconds": 125.30079188900709,
+        "request_seconds": 156.48807670301176,
+        "wall_seconds": 156.56338787100685,
+        "sampling_speedup_percent_vs_sdpa": 0.0,
+        "output_file": "dialogue-two-character-864x480-141f-base12-sdpa-seed440420.mp4",
+        "sha256": "2eb61334978b42bcec00ec68449f3f39139d9e24ef5ba70ef2240dc8bf7116e0",
+        "size_bytes": 429062
+      },
+      "sage2": {
+        "sampled_seconds": 101.26120180900034,
+        "request_seconds": 132.59474182801205,
+        "wall_seconds": 132.66185540499282,
+        "sampling_speedup_percent_vs_sdpa": 19.185505,
+        "video_psnr_average_db_vs_sdpa": 22.4225,
+        "audio_pcm_snr_db_vs_sdpa": [2.515815, 2.492644],
+        "output_file": "dialogue-two-character-864x480-141f-base12-sage2-seed440420.mp4",
+        "sha256": "c3e0691586c4101987f240c35a79637965af75088731108037cd084be945ba9d",
+        "size_bytes": 433450
+      },
+      "cudnn_sdpa": {
+        "sampled_seconds": 125.72722270998929,
+        "request_seconds": 157.14237468996726,
+        "wall_seconds": 157.2081264879962,
+        "sampling_speedup_percent_vs_sdpa": -0.340326,
+        "video_psnr_average_db_vs_sdpa": 22.093143,
+        "audio_pcm_snr_db_vs_sdpa": [2.056909, 2.162249],
+        "output_file": "dialogue-two-character-864x480-141f-base12-cudnn_sdpa-seed440420.mp4",
+        "sha256": "cd17833c12a99cd71d1c0a9137a1dd9feba03943980d6eb8fff296cd9f9529a6",
+        "size_bytes": 429636
+      },
+      "ck_int8": {
+        "sampled_seconds": 104.05008868798905,
+        "request_seconds": 135.57199566195777,
+        "wall_seconds": 135.62453711099806,
+        "sampling_speedup_percent_vs_sdpa": 16.959752,
+        "video_psnr_average_db_vs_sdpa": 22.989999,
+        "audio_pcm_snr_db_vs_sdpa": [2.308146, 2.398058],
+        "output_file": "dialogue-two-character-864x480-141f-base12-ck_int8-seed440420.mp4",
+        "sha256": "a54f9f3f2ba351d31b0b807646aea2740aae7c13c03d4eb7d44182aab3cf5618",
+        "size_bytes": 423190
+      },
+      "sage3": {
+        "sampled_seconds": 109.8837421490025,
+        "request_seconds": 142.34848491402227,
+        "wall_seconds": 142.41395058500348,
+        "sampling_speedup_percent_vs_sdpa": 12.304032,
+        "video_psnr_average_db_vs_sdpa": 19.63725,
+        "audio_pcm_snr_db_vs_sdpa": [-3.047541, -2.395828],
+        "output_file": "dialogue-two-character-864x480-141f-base12-sage3-seed440420.mp4",
+        "sha256": "3763ca4b79399037d997d5a37c795fdaf233aca88fdb90e122bf25450c75dddd",
+        "size_bytes": 596651
+      },
+      "sage3_mean": {
+        "sampled_seconds": 109.88214365398744,
+        "request_seconds": 141.45796999098093,
+        "wall_seconds": 141.51303354099218,
+        "sampling_speedup_percent_vs_sdpa": 12.305308,
+        "video_psnr_average_db_vs_sdpa": 19.63725,
+        "audio_pcm_snr_db_vs_sdpa": [-3.047541, -2.395828],
+        "output_file": "dialogue-two-character-864x480-141f-base12-sage3_mean-seed440420.mp4",
+        "sha256": "3763ca4b79399037d997d5a37c795fdaf233aca88fdb90e122bf25450c75dddd",
+        "size_bytes": 596651,
+        "identical_output_to": "sage3"
+      },
+      "kj_sage_cuda": {
+        "sampled_seconds": 113.68463072601298,
+        "request_seconds": 145.39383504101716,
+        "wall_seconds": 145.45757150900317,
+        "sampling_speedup_percent_vs_sdpa": 9.270621,
+        "video_psnr_average_db_vs_sdpa": 22.83642,
+        "audio_pcm_snr_db_vs_sdpa": [2.039297, 2.098551],
+        "output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_cuda-seed440420.mp4",
+        "sha256": "d85aaca677d2c678604b0aa2aba9bbf4921934bfc2970c20d0498b1762373ab5",
+        "size_bytes": 427568
+      },
+      "kj_sage_triton": {
+        "sampled_seconds": 118.28390433100867,
+        "request_seconds": 150.37281119299587,
+        "wall_seconds": 150.43365036998875,
+        "sampling_speedup_percent_vs_sdpa": 5.600034,
+        "video_psnr_average_db_vs_sdpa": 22.086796,
+        "audio_pcm_snr_db_vs_sdpa": [5.920034, 6.184134],
+        "output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_triton-seed440420.mp4",
+        "sha256": "9fe63aafa51aae8ecc0940ba75fbc13a978a50091fd8f0c336c2bb04e3658a54",
+        "size_bytes": 430038
+      },
+      "kj_sage_fp8": {
+        "sampled_seconds": 102.6086639950081,
+        "request_seconds": 134.1615118059999,
+        "wall_seconds": 134.2236167689989,
+        "sampling_speedup_percent_vs_sdpa": 18.110123,
+        "video_psnr_average_db_vs_sdpa": 22.814833,
+        "audio_pcm_snr_db_vs_sdpa": [3.442309, 3.542901],
+        "output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_fp8-seed440420.mp4",
+        "sha256": "8c5bc2a8bf7e496c2108194479fe60e67883dea0ab50e95f3ab4bac916274816",
+        "size_bytes": 420189
+      },
+      "kj_sage_fp8pp": {
+        "sampled_seconds": 103.24360375599645,
+        "request_seconds": 134.77945204998832,
+        "wall_seconds": 134.84802630099875,
+        "sampling_speedup_percent_vs_sdpa": 17.603391,
+        "video_psnr_average_db_vs_sdpa": 22.296461,
+        "audio_pcm_snr_db_vs_sdpa": [3.893655, 4.120454],
+        "output_file": "dialogue-two-character-864x480-141f-base12-kj_sage_fp8pp-seed440420.mp4",
+        "sha256": "71643c7163e5dc16d64ac0c5d567bdd846f7944471eb0a97fd54f9a1990c24e9",
+        "size_bytes": 433672
+      },
+      "kj_head_sliced": {
+        "sampled_seconds": 103.97128305501246,
+        "request_seconds": 135.67128342803335,
+        "wall_seconds": 135.71258915099315,
+        "sampling_speedup_percent_vs_sdpa": 17.022645,
+        "video_psnr_average_db_vs_sdpa": 21.679726,
+        "audio_pcm_snr_db_vs_sdpa": [1.20378, 1.17592],
+        "output_file": "dialogue-two-character-864x480-141f-base12-kj_head_sliced-seed440420.mp4",
+        "sha256": "48ee4da6c0a91f18714973ca23e26f1844b302024c8111946772cbc9a4a4dd83",
+        "size_bytes": 421728
+      },
+      "sol_attn": {
+        "sampled_seconds": 96.45773334198748,
+        "request_seconds": 128.10418028896675,
+        "wall_seconds": 128.17140150099294,
+        "sampling_speedup_percent_vs_sdpa": 23.019055,
+        "video_psnr_average_db_vs_sdpa": 16.653003,
+        "audio_pcm_snr_db_vs_sdpa": [1.186243, 1.042278],
+        "output_file": "dialogue-two-character-864x480-141f-base12-sol_attn-seed440420.mp4",
+        "sha256": "8b999630210692301549d11d20d9319a6af5873a60d4cd735035e2ed79c469d6",
+        "size_bytes": 570667,
+        "note": "Experimental sparse Triton dispatch; eligible calls use sol_attn and shorter calls fall back."
+      },
+      "flash4": {
+        "sampled_seconds": 128.5334454880067,
+        "request_seconds": 160.91555801399227,
+        "wall_seconds": 160.96679471699463,
+        "sampling_speedup_percent_vs_sdpa": -2.579915,
+        "video_psnr_average_db_vs_sdpa": 21.238297,
+        "audio_pcm_snr_db_vs_sdpa": [3.230108, 3.821894],
+        "output_file": "dialogue-two-character-864x480-141f-base12-flash4-seed440420.mp4",
+        "sha256": "cfb8a046cb79f8448988e7392ca748c0a08a2a034a43f6ab55a9058faa1a72ef",
+        "size_bytes": 449687
+      }
+    }
+  },
+  "turbo_sage2_matrix": {
+    "reference": "Base 12-step Sage2 from attention_backend_matrix",
+    "runs": {
+      "4step": {
+        "adapter": "Official FL2VA Turbo 4-step v1.1 768p",
+        "steps": 4,
+        "attention": "sage2",
+        "sampled_seconds": 51.661242722009774,
+        "request_seconds": 83.01122188399313,
+        "wall_seconds": 83.06598260099418,
+        "sampling_speedup_percent_vs_base_sage2": 48.982195,
+        "wall_speedup_percent_vs_base_sage2": 37.38518,
+        "output_file": "dialogue-two-character-864x480-141f-turbo4step-sage2-seed440420.mp4",
+        "sha256": "b02c29a8732d88c7dcf8e85462afcbe3aa83328541ce1647995e7d0d4188744a",
+        "size_bytes": 455325,
+        "container_bit_rate": 620017,
+        "audio": {
+          "integrated_loudness_lufs": -31.78,
+          "true_peak_dbtp": -14.55,
+          "first_0_25_seconds_peak_dbfs": -39.394894,
+          "first_0_25_seconds_rms_dbfs": -57.733654,
+          "first_sample_dbfs": [-54.110104, -50.553467]
+        }
+      },
+      "8step": {
+        "adapter": "Official FL2VA Turbo 8-step v1.0",
+        "steps": 8,
+        "attention": "sage2",
+        "sampled_seconds": 102.70967868898879,
+        "request_seconds": 134.31820170898573,
+        "wall_seconds": 134.36813215899747,
+        "sampling_speedup_percent_vs_base_sage2": -1.430436,
+        "wall_speedup_percent_vs_base_sage2": -1.286185,
+        "output_file": "dialogue-two-character-864x480-141f-turbo8step-sage2-seed440420.mp4",
+        "sha256": "21f450848c2ded8c02054cc241c8080d55a17979022eadfd892f3ce5205833e6",
+        "size_bytes": 585250,
+        "container_bit_rate": 796936,
+        "audio": {
+          "integrated_loudness_lufs": -24.39,
+          "true_peak_dbtp": -7.18,
+          "first_0_25_seconds_peak_dbfs": -21.305752,
+          "first_0_25_seconds_rms_dbfs": -41.479687,
+          "first_sample_dbfs": [-35.859436, -39.793118]
+        }
+      }
+    }
+  },
+  "comfy_sdpa_comparison": {
+    "comfyui_version": "0.31.0",
+    "attention": "PyTorch attention without --use-sage-attention",
+    "prompt_id": "29c45a77-efdf-4cc8-8ae9-9b8a40823b41",
+    "execution_seconds": 152.767,
+    "wall_seconds": 153.03360149999207,
+    "wall_difference_from_direct_percent": -2.25454138353405,
+    "output": {
+      "linux_path": "/home/daniel/StoryStudioAssets/H3-output/h3-baselines/comfy-sdpa-dialogue-two-character-864x480-141f-base12-seed440420_00001_.mp4",
+      "unc_path": "\\\\192.168.1.162\\StoryStudioAssets\\H3-output\\h3-baselines\\comfy-sdpa-dialogue-two-character-864x480-141f-base12-seed440420_00001_.mp4",
+      "sha256": "f4510d6b8ac06b8f2cb22172454c6977a214e061dcc5539ace68e1cb70abc2ee",
+      "size_bytes": 443723,
+      "container_bit_rate": 604218,
+      "duration_seconds": 5.875,
+      "video": {
+        "codec": "h264",
+        "profile": "High",
+        "pixel_format": "yuv420p",
+        "width": 864,
+        "height": 480,
+        "fps": 24,
+        "frames": 141,
+        "bit_rate": 457452
+      },
+      "audio": {
+        "codec": "aac",
+        "profile": "LC",
+        "sample_rate": 32000,
+        "channels": 2,
+        "channel_layout": "stereo",
+        "bit_rate": 133407,
+        "mean_volume_db": -25.7,
+        "max_volume_db": -4.3,
+        "integrated_loudness_lufs": -21.12,
+        "true_peak_dbtp": -4.25,
+        "loudness_range_lu": 4.10,
+        "loudness_threshold_lufs": -33.88
+      }
+    },
+    "first_0_25_seconds": {
+      "direct": {
+        "peak_level_db": -19.429245,
+        "rms_level_db": -33.601728,
+        "max_sample_difference": 0.022571,
+        "peak_count": 2,
+        "nan_count": 0,
+        "inf_count": 0
+      },
+      "comfy": {
+        "peak_level_db": -19.261140,
+        "rms_level_db": -33.545443,
+        "max_sample_difference": 0.023026,
+        "peak_count": 2,
+        "nan_count": 0,
+        "inf_count": 0
+      },
+      "decoded_pcm_snr_db": {
+        "channel_1": 31.498066,
+        "channel_2": 29.897251
+      },
+      "conclusion": "No amplitude clipping. Direct and Comfy closely match during the first 250ms and both exhibit the startup artifact, but they are not numerically identical. The shared defect originates before runtime-specific audio encoding and mux."
+    },
+    "full_decoded_pcm_snr_db": {
+      "channel_1": 21.414368,
+      "channel_2": 26.592979
+    }
+  }
+}
--- a/benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json
+++ b/benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json
@ -0,0 +1,19 @@
+{
+  "name": "t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420",
+  "measured_at": "2026-08-21",
+  "hardware": "NVIDIA GB10",
+  "model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
+  "task": "t2va",
+  "resolution": [864, 480],
+  "frames": 141,
+  "fps": 24,
+  "duration_seconds": 5.875,
+  "steps": 12,
+  "scheduler": "beta",
+  "sampler": "res_multistep",
+  "seed": 440420,
+  "attention": "sage2",
+  "turbo": null,
+  "prompt_change": "Only the two <d>[English]...</d> spans were replaced with quoted dialogue.",
+  "prompt": "integrated_multimodal_description: [Shot 1] Live-action, cinematic, a static medium two-shot frames exactly two adults seated across from each other at a small table in a quiet, otherwise empty meeting room. A composed adult man sits on the left and a composed adult woman sits on the right. Both maintain neutral, attentive expressions. The man with a low, clear baritone voice and measured speaking pace (S1) looks toward the woman and says in a calm, matter-of-fact delivery: \"The north entrance closes at six.\" During S1's line, only his lips and jaw move; the woman's lips remain completely closed. Immediately after his final word, his lips meet and his jaw ceases speaking motion. After a brief silent pause, the woman with a clear alto voice and measured speaking pace (S2) looks toward the man and replies in a calm, matter-of-fact delivery: \"Then we should leave by five thirty.\" During S2's line, only her lips and jaw move; the man's lips remain completely closed. Immediately after her final word, her lips meet and her jaw ceases speaking motion. Both remain silent with neutral, closed-mouth expressions through the final frame. There is no overlapping speech. The camera remains completely static with no cuts.\n\noverall_soundscape: Quiet, dry indoor room tone with a faint ventilation hum. Only S1 and S2 are audible, one at a time. No laughter, chuckling, giggling, smiling vocalization, sighing, gasping, audible breathing, filler sounds, audience reaction, narration, or other voices.\n\nnon_diegetic_music: N/A"
+}
--- a/benchmarks/t2va-distributed-smoke-256x256-9f-1step.json
+++ b/benchmarks/t2va-distributed-smoke-256x256-9f-1step.json
@ -0,0 +1,12 @@
+{
+  "name": "t2va-distributed-smoke-256x256-9f-1step",
+  "mode": "t2va",
+  "prompt": "A paper windmill turns steadily on a plain table. Quiet room tone.",
+  "resolution": [
+    256,
+    256
+  ],
+  "frames": 9,
+  "steps": 1,
+  "seed": 440420
+}
--- a/benchmarks/t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421.json
+++ b/benchmarks/t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421.json
@ -0,0 +1,25 @@
+{
+  "name": "t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421",
+  "measured_at": "2026-08-21",
+  "hardware": "NVIDIA GB10",
+  "model": "minimax_h3_fl2va_pruned_nvfp4.safetensors",
+  "task": "t2va",
+  "resolution": [864, 480],
+  "frames": 141,
+  "fps": 24,
+  "duration_seconds": 5.875,
+  "steps": 12,
+  "scheduler": "beta",
+  "sampler": "res_multistep",
+  "seed": 440421,
+  "attention": "sage2",
+  "turbo": null,
+  "prompt": "integrated_multimodal_description: 0.0-2.0s: Inside a packed underground nightclub, the camera glides low across a crowded dance floor toward a raised DJ booth. A clean four-on-the-floor kick and deep bassline begin immediately at 0.0s. Cyan and magenta strobes strike precisely on the beat while dancers move in synchronized rhythm. 2.0-4.0s: The camera sweeps around the DJ as she works the mixer, one hand adjusting a filter while the other raises toward the crowd. The house groove remains continuous and coherent; crisp hi-hats enter over the kick and bass. The crowd cheers naturally beneath the music. 4.0-5.875s: The camera pushes close to the mixer and then tilts up as the room erupts under a bright white strobe hit. The beat continues without interruption, ending on an energetic club moment. No dialogue. overall_soundscape: Loud but clean diegetic nightclub house music coming from the venue sound system, beginning exactly at the first frame, with a steady kick, deep controlled bass, crisp hi-hats, room reflections, dancing footsteps, and a lively crowd. No clipping, crackling, popping, startup noise, gibberish, or speech. non_diegetic_music: N/A",
+  "diagnostic_capture": {
+    "output_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.mp4",
+    "latent_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.latent.pt",
+    "wav_file": "audio-diagnostic-clean-nightclub-864x480-141f-base12-sage2-seed440421.wav",
+    "sampled_seconds": 104.5049012459931,
+    "wall_seconds": 135.91074104901054
+  }
+}
--- a/compose.qwen38-vllm.yml
+++ b/compose.qwen38-vllm.yml
@ -0,0 +1,46 @@
+services:
+  qwen38-vllm:
+    image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest
+    container_name: qwen38-vllm
+    restart: unless-stopped
+    gpus: all
+    ipc: host
+    shm_size: 16g
+    network_mode: host
+    volumes:
+      - /home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4:/model:ro
+      - /home/daniel/aeon-spark-test/h3/vllm-cache:/root/.cache/vllm
+    entrypoint: ["vllm"]
+    command:
+      - serve
+      - /model
+      - --served-model-name
+      - qwen38
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8002"
+      - --tensor-parallel-size
+      - "1"
+      - --gpu-memory-utilization
+      - "0.45"
+      - --max-model-len
+      - "262144"
+      - --max-num-seqs
+      - "4"
+      - --max-num-batched-tokens
+      - "8192"
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      - --kv-cache-dtype
+      - fp8_e4m3
+      - --reasoning-parser
+      - qwen3
+      - --tool-call-parser
+      - qwen3_xml
+      - --enable-auto-tool-choice
+      - --distributed-executor-backend
+      - mp
+      - --speculative-config
+      - '{"method":"mtp","num_speculative_tokens":5}'
+      - --trust-remote-code
--- a/compose.spark-stack.yml
+++ b/compose.spark-stack.yml
@ -0,0 +1,84 @@
+services:
+  h3-hot-runtime:
+    profiles: ["h3"]
+    image: h3-blackwell-runtime:dev
+    container_name: h3-hot-runtime
+    restart: unless-stopped
+    gpus: all
+    volumes:
+      - /home/daniel/aeon-spark-test/h3/comfy-models/diffusion_models:/models:ro
+      - /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
+      - /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
+      - /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
+      - /home/daniel/StoryStudioAssets/H3-output:/output
+    ports:
+      - "8001:8000"
+    environment:
+      H3_DISABLE_MMAP: "1"
+      H3_NVFP4_SCALE_BACKEND: "vortex"
+      H3_NVFP4_SCALE_VERSION: "1"
+      H3_SOL_QKV_LAYOUT: "native"
+    command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--warmup"]
+
+  qwen38-vllm:
+    profiles: ["qwen"]
+    image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest
+    container_name: qwen38-vllm
+    restart: unless-stopped
+    gpus: all
+    ipc: host
+    shm_size: 16g
+    network_mode: host
+    volumes:
+      - /home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4:/model:ro
+      - /home/daniel/aeon-spark-test/h3/vllm-cache:/root/.cache/vllm
+    entrypoint: ["vllm"]
+    command:
+      - serve
+      - /model
+      - --served-model-name
+      - qwen38
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8002"
+      - --tensor-parallel-size
+      - "1"
+      - --gpu-memory-utilization
+      - "0.45"
+      - --max-model-len
+      - "262144"
+      - --max-num-seqs
+      - "4"
+      - --max-num-batched-tokens
+      - "8192"
+      - --enable-chunked-prefill
+      - --enable-prefix-caching
+      - --kv-cache-dtype
+      - fp8_e4m3
+      - --reasoning-parser
+      - qwen3
+      - --tool-call-parser
+      - qwen3_xml
+      - --enable-auto-tool-choice
+      - --distributed-executor-backend
+      - mp
+      - --speculative-config
+      - '{"method":"mtp","num_speculative_tokens":5}'
+      - --trust-remote-code
+
+  flux2-klein-comfy:
+    profiles: ["flux"]
+    image: ghcr.io/aeon-7/comfyui-aeon-spark:slim
+    container_name: flux2-klein-comfy
+    restart: unless-stopped
+    gpus: all
+    ipc: host
+    shm_size: 16g
+    volumes:
+      - /home/daniel/aeon-spark-test/workspace:/workspace/ComfyUI:rw
+      - /home/daniel/StoryStudioAssets/H3-output:/workspace/ComfyUI/output:rw
+      - /home/daniel/aeon-spark-test/extra_model_paths.yaml:/opt/ComfyUI/extra_model_paths.yaml:ro
+      - /home/daniel/comfy-data/models:/shared-models:ro
+    ports:
+      - "8192:8188"
--- a/compose.spark.yml
+++ b/compose.spark.yml
@ -10,5 +10,29 @@ services:
      - /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
      - /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
      - /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
+      - /home/daniel/aeon-spark-test/h3/models/MiniMax-H3-Turbo-FL2VA/latest:/turbo:ro
+      - /home/daniel/aeon-spark-test/h3/models/Minimax-H3-Latent-Upscaler:/latent-upscaler:ro
      - /home/daniel/StoryStudioAssets/H3-output:/output
    command: ["sleep", "infinity"]
+  h3-hot-runtime:
+    build:
+      context: .
+      dockerfile: Dockerfile.spark
+    image: h3-blackwell-runtime:dev
+    gpus: all
+    volumes:
+      - /home/daniel/aeon-spark-test/h3/comfy-models/diffusion_models:/models:ro
+      - /home/daniel/aeon-spark-test/h3/comfy-models/text_encoders:/text-encoders:ro
+      - /home/daniel/aeon-spark-test/h3/comfy-models/vae:/vae:ro
+      - /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts:/artifacts:ro
+      - /home/daniel/aeon-spark-test/h3/models/MiniMax-H3-Turbo-FL2VA/latest:/turbo:ro
+      - /home/daniel/aeon-spark-test/h3/models/Minimax-H3-Latent-Upscaler:/latent-upscaler:ro
+      - /home/daniel/StoryStudioAssets/H3-output:/output
+    ports:
+      - "8001:8000"
+    environment:
+      H3_DISABLE_MMAP: "1"
+      H3_NVFP4_SCALE_BACKEND: "vortex"
+      H3_NVFP4_SCALE_VERSION: "1"
+      H3_SOL_QKV_LAYOUT: "native"
+    command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--turbo-lora", "4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors", "--turbo-lora", "8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors", "--latent-upscaler", "/latent-upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors", "--warmup"]
--- a/pyproject.toml
+++ b/pyproject.toml
@ -4,12 +4,15 @@ version = "0.1.0"
 description = "Direct MiniMax H3 Blackwell inference research runtime"
 requires-python = ">=3.12"
 dependencies = [
-  "comfy-kitchen==0.2.28",
+  "comfy-kitchen==0.2.31",
  "fastsafetensors>=0.1.10",
  "safetensors>=0.5.0",
  "torch==2.9.1+cu130",
  "transformers>=4.51,<5"
 ]

+[project.optional-dependencies]
+flash4 = ["flash-attn-4[cu13]==4.0.0b27"]
+
 [tool.setuptools.packages.find]
 where = ["src"]
--- a/src/h3_blackwell_runtime/attention.py
+++ b/src/h3_blackwell_runtime/attention.py
@ -1,5 +1,8 @@
 """Direct H3 self-attention using packed NVFP4 linears and SageAttention3."""

+import os
+from typing import TYPE_CHECKING
+
 import torch
 import torch.nn.functional as functional
 from torch import nn
@ -7,21 +10,25 @@ from torch import nn
 from .checkpoint import H3Checkpoint
 from .nvfp4 import Nvfp4Linear

+if TYPE_CHECKING:
+    from .distributed import SequenceParallelContext

-AVAILABLE_BACKENDS = ("sage2", "sdpa", "sage3")
-PLANNED_BACKENDS = ("flash4", "easycache", "h3_cache", "sol_attn", "kj_sage", "kj_chunked_ffn", "kj_head_sliced")
+
+AVAILABLE_BACKENDS = ("sage2", "cudnn_sdpa", "ck_int8", "sdpa", "flash4", "sage3", "sage3_mean", "kj_sage_cuda", "kj_sage_triton", "kj_sage_fp8", "kj_sage_fp8pp", "kj_head_sliced", "sol_attn")
+PLANNED_BACKENDS = ("easycache", "h3_cache", "kj_chunked_ffn")
+DEFAULT_ATTENTION_BACKEND = os.getenv("H3_DEFAULT_ATTENTION", "sage2")


 def attention_backend_status() -> dict[str, str]:
    """Report direct-runtime attention choices without importing ComfyUI nodes."""
    status = {name: "available" for name in AVAILABLE_BACKENDS}
-    status.update({"flash4": "planned: exact Blackwell kernel adapter"})
+    status.update({"cudnn_sdpa": "available: forced cuDNN SDPA with no backend fallback"})
+    status.update({"ck_int8": "available: approximate Comfy Kitchen INT8 Q/K/V attention"})
+    status.update({"sol_attn": "experimental: sparse Triton attention for eligible non-causal H3 attention calls; falls back below H3_SOL_MIN_TOKENS unless H3_SOL_STRICT=1"})
+    status.update({"flash4": "available: official FlashAttention-4 CuTeDSL Blackwell kernel (strict, no fallback)"})
    status.update({"easycache": "planned: approximate denoiser cache"})
    status.update({"h3_cache": "planned: approximate H3-specific cache"})
-    status.update({"sol_attn": "experimental: prior H3-tested sparse Triton attention; standalone adapter pending"})
-    status.update({"kj_sage": "experimental: prior H3-tested Sage patch; standalone adapter pending"})
-    status.update({"kj_chunked_ffn": "planned: exact memory-lifetime adapter"})
-    status.update({"kj_head_sliced": "planned: exact memory-lifetime adapter"})
+    status.update({"kj_chunked_ffn": "available: exact H3 MLP row chunking via H3_MLP_CHUNKS or runtime args"})
    return status


@ -29,16 +36,134 @@ def rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float) -> torch.Tensor:
    return torch.nn.functional.rms_norm(x, (x.shape[-1],), weight.to(x.dtype), eps)


+def run_sol_attention_bshd(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, is_causal: bool) -> torch.Tensor:
+    """Run Sol-Attn on `[batch, sequence, heads, dim]` tensors and return the same layout."""
+    try:
+        from sol_kernel import sol_attn
+
+        tau = float(os.getenv("H3_SOL_TAU", "1.3"))
+        min_tokens = int(os.getenv("H3_SOL_MIN_TOKENS", "4096"))
+        thresh_type = os.getenv("H3_SOL_THRESH_TYPE", "diag")
+        int8_qk = os.getenv("H3_SOL_INT8_QK", "").lower() in {"1", "true", "yes", "on"}
+        int8_pv = os.getenv("H3_SOL_INT8_PV", "").lower() in {"1", "true", "yes", "on"}
+        if is_causal:
+            raise ValueError("Sol-Attn backend only supports non-causal H3 attention")
+        if q.shape[-1] != 128:
+            raise ValueError(f"Sol-Attn requires head dim 128, got {q.shape[-1]}")
+        if q.shape[1] < min_tokens:
+            raise ValueError(f"{q.shape[1]} tokens < H3_SOL_MIN_TOKENS={min_tokens}")
+        return sol_attn(
+            q.contiguous(),
+            k.contiguous(),
+            v.contiguous(),
+            tau=tau,
+            thresh_type=thresh_type,
+            int8_qk=int8_qk,
+            int8_pv=int8_pv,
+        )
+    except Exception:
+        if os.getenv("H3_SOL_STRICT", "").lower() in {"1", "true", "yes", "on"}:
+            raise
+        fallback = os.getenv("H3_SOL_FALLBACK", "sage2")
+        hnd = run_attention(q.transpose(1, 2).contiguous(), k.transpose(1, 2).contiguous(), v.transpose(1, 2).contiguous(), backend=fallback, is_causal=is_causal)
+        return hnd.transpose(1, 2)
+
+
+def qkv_to_bshd(qkv: torch.Tensor, heads: int, head_dim: int) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+    """Split `[S, 3*H*D]` QKV into contiguous Sol-native `[1,S,H,D]` tensors."""
+    if os.getenv("H3_SOL_QKV_LAYOUT", "native").lower() != "fused":
+        raise RuntimeError("fused QKV layout disabled")
+    try:
+        from .nvfp4_quant import _vortex_scale_extension
+
+        return tuple(_vortex_scale_extension().qkv_to_bshd(qkv, heads, head_dim))
+    except Exception:
+        if os.getenv("H3_SOL_QKV_LAYOUT_STRICT", "").lower() in {"1", "true", "yes", "on"}:
+            raise
+        inner = heads * head_dim
+        sequence = qkv.shape[0]
+        q, k, v = qkv.split(inner, dim=-1)
+        return (
+            q.view(1, sequence, heads, head_dim).contiguous(),
+            k.view(1, sequence, heads, head_dim).contiguous(),
+            v.view(1, sequence, heads, head_dim).contiguous(),
+        )
+
+
+def run_flash4_attention_bshd(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, is_causal: bool) -> torch.Tensor:
+    """Run official FlashAttention-4 on `[batch, sequence, heads, dim]` tensors."""
+    if not q.is_cuda or not k.is_cuda or not v.is_cuda:
+        raise ValueError("FlashAttention-4 requires CUDA tensors")
+    if q.dtype not in {torch.float16, torch.bfloat16} or k.dtype != q.dtype or v.dtype != q.dtype:
+        raise ValueError("FlashAttention-4 requires matching FP16 or BF16 Q/K/V tensors")
+    if q.shape != k.shape or q.shape != v.shape:
+        raise ValueError("FlashAttention-4 requires matching Q/K/V shapes")
+    if q.shape[-1] != 128:
+        raise ValueError(f"H3 FlashAttention-4 requires head dim 128, got {q.shape[-1]}")
+
+    from flash_attn.cute import flash_attn_func
+
+    result = flash_attn_func(q.contiguous(), k.contiguous(), v.contiguous(), causal=is_causal)
+    return result[0] if isinstance(result, tuple) else result
+
+
 def run_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, backend: str, is_causal: bool) -> torch.Tensor:
    """Run one `[batch, heads, sequence, dim]` attention operation."""
+    if backend == "sol_attn":
+        return run_sol_attention_bshd(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=is_causal).transpose(1, 2)
+    if backend == "kj_head_sliced":
+        head_slice_size = int(os.getenv("H3_HEAD_SLICE_SIZE", "8"))
+        base_backend = os.getenv("H3_HEAD_SLICE_BACKEND", "sage2")
+        if head_slice_size <= 0:
+            raise ValueError("H3_HEAD_SLICE_SIZE must be positive")
+        outputs = [
+            run_attention(q[:, start:start + head_slice_size], k[:, start:start + head_slice_size], v[:, start:start + head_slice_size], backend=base_backend, is_causal=is_causal)
+            for start in range(0, q.shape[1], head_slice_size)
+        ]
+        return torch.cat(outputs, dim=1)
    if backend == "sage2":
        from sageattention import sageattn

        return sageattn(q, k, v, is_causal=is_causal, tensor_layout="HND", smooth_k=False)
+    if backend == "flash4":
+        return run_flash4_attention_bshd(
+            q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=is_causal,
+        ).transpose(1, 2)
+    if backend == "cudnn_sdpa":
+        from torch.nn.attention import SDPBackend, sdpa_kernel
+
+        with sdpa_kernel([SDPBackend.CUDNN_ATTENTION]):
+            return functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal)
+    if backend == "ck_int8":
+        if is_causal:
+            raise ValueError("Comfy Kitchen INT8 attention does not support causal H3 attention")
+        import comfy_kitchen
+
+        return comfy_kitchen.int8_attention(q, k, v)
    if backend == "sage3":
        from sageattn3 import sageattn3_blackwell

        return sageattn3_blackwell(q, k, v, is_causal=is_causal)
+    if backend == "sage3_mean":
+        from sageattn3 import sageattn3_blackwell
+
+        return sageattn3_blackwell(q, k, v, is_causal=is_causal, per_block_mean=True)
+    if backend == "kj_sage_cuda":
+        from sageattention import sageattn_qk_int8_pv_fp16_cuda
+
+        return sageattn_qk_int8_pv_fp16_cuda(q, k, v, is_causal=is_causal, pv_accum_dtype="fp32", tensor_layout="HND")
+    if backend == "kj_sage_triton":
+        from sageattention import sageattn_qk_int8_pv_fp16_triton
+
+        return sageattn_qk_int8_pv_fp16_triton(q, k, v, is_causal=is_causal, tensor_layout="HND")
+    if backend == "kj_sage_fp8":
+        from sageattention import sageattn_qk_int8_pv_fp8_cuda
+
+        return sageattn_qk_int8_pv_fp8_cuda(q, k, v, is_causal=is_causal, pv_accum_dtype="fp32+fp32", tensor_layout="HND")
+    if backend == "kj_sage_fp8pp":
+        from sageattention import sageattn_qk_int8_pv_fp8_cuda
+
+        return sageattn_qk_int8_pv_fp8_cuda(q, k, v, is_causal=is_causal, pv_accum_dtype="fp32+fp16", tensor_layout="HND")
    if backend == "sdpa":
        return functional.scaled_dot_product_attention(q, k, v, is_causal=is_causal)
    raise ValueError(f"Unsupported H3 attention backend: {backend}")
@ -87,7 +212,7 @@ class H3SageAttention(nn.Module):
        heads: int = 56,
        head_dim: int = 128,
        eps: float = 1e-5,
-        backend: str = "sage2",
+        backend: str = DEFAULT_ATTENTION_BACKEND,
    ):
        super().__init__()
        self.qkv_proj = qkv_proj
@ -104,7 +229,7 @@ class H3SageAttention(nn.Module):
        self.register_buffer("k_norm_weight", k_norm_weight, persistent=False)

    @classmethod
-    def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16, backend: str = "sage2"):
+    def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16, backend: str = DEFAULT_ATTENTION_BACKEND):
        return cls(
            checkpoint.nvfp4_linear(f"{prefix}.qkv_proj", output_dtype=output_dtype),
            checkpoint.nvfp4_linear(f"{prefix}.out_proj", output_dtype=output_dtype),
@ -113,20 +238,97 @@ class H3SageAttention(nn.Module):
            backend=backend,
        )

-    def forward(self, x: torch.Tensor, rope_rotation: torch.Tensor) -> torch.Tensor:
+    def forward(
+        self,
+        x: torch.Tensor,
+        rope_rotation: torch.Tensor,
+        sequence_parallel: "SequenceParallelContext | None" = None,
+        tensor_parallel: "SequenceParallelContext | None" = None,
+    ) -> torch.Tensor:
        if x.ndim != 2:
            raise ValueError("H3 attention expects `[sequence, hidden]` input.")
+        if sequence_parallel is not None and tensor_parallel is not None:
+            raise ValueError("choose Ulysses sequence parallelism or tensor parallelism, not both")
+        if tensor_parallel is not None:
+            return self._forward_tensor_parallel(x, rope_rotation, tensor_parallel)
        sequence = x.shape[0]
        inner = self.heads * self.head_dim
-        q, k, v = self.qkv_proj(x).split(inner, dim=-1)
+        qkv = self.qkv_proj(x)
+        q, k, v = qkv.split(inner, dim=-1)
        q = q.view(1, sequence, self.heads, self.head_dim)
        k = k.view(1, sequence, self.heads, self.head_dim)
        v = v.view(1, sequence, self.heads, self.head_dim)

        q, k = rms_rope_split_half_(q, k, rope_rotation, self.q_norm_weight, self.k_norm_weight, self.eps)
+        if sequence_parallel is not None:
+            q, k, v = sequence_parallel.seq_to_heads(q, k, v)
+            if self.backend == "sol_attn":
+                out = run_sol_attention_bshd(q, k, v, is_causal=False)
+            elif self.backend == "flash4":
+                out = run_flash4_attention_bshd(q, k, v, is_causal=False)
+            else:
+                out = run_attention(
+                    q.transpose(1, 2).contiguous(),
+                    k.transpose(1, 2).contiguous(),
+                    v.transpose(1, 2).contiguous(),
+                    backend=self.backend,
+                    is_causal=False,
+                ).transpose(1, 2)
+            local_out = sequence_parallel.heads_to_seq(out)
+            return self.out_proj(local_out.reshape(sequence, inner))
+        if self.backend == "sol_attn":
+            if os.getenv("H3_SOL_QKV_LAYOUT", "native").lower() == "fused":
+                q, k, v = qkv_to_bshd(qkv, self.heads, self.head_dim)
+            else:
+                q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
+            out = run_sol_attention_bshd(q, k, v, is_causal=False)
+            return self.out_proj(out.reshape(sequence, inner).contiguous())
+        if self.backend == "flash4":
+            out = run_flash4_attention_bshd(q, k, v, is_causal=False)
+            return self.out_proj(out.reshape(sequence, inner).contiguous())
+
        q = q.transpose(1, 2).contiguous()
        k = k.transpose(1, 2).contiguous()
        v = v.transpose(1, 2).contiguous()

        out = run_attention(q, k, v, backend=self.backend, is_causal=False)
        return self.out_proj(out.transpose(1, 2).reshape(sequence, inner).contiguous())
+
+    def _forward_tensor_parallel(
+        self,
+        local_x: torch.Tensor,
+        local_rotation: torch.Tensor,
+        context: "SequenceParallelContext",
+    ) -> torch.Tensor:
+        """Run local-head attention with column/row-parallel NVFP4 projections."""
+        local_sequence = local_x.shape[0]
+        full_x = context.all_gather_rows(local_x)
+        full_rotation = context.all_gather_rows(local_rotation[0]).unsqueeze(0)
+        inner = self.heads * self.head_dim
+        q, k, v = self.qkv_proj(full_x).split(inner, dim=-1)
+        q = q.view(1, context.sequence_length, self.heads, self.head_dim)
+        k = k.view(1, context.sequence_length, self.heads, self.head_dim)
+        v = v.view(1, context.sequence_length, self.heads, self.head_dim)
+        q, k = rms_rope_split_half_(
+            q, k, full_rotation, self.q_norm_weight, self.k_norm_weight, self.eps,
+        )
+        if self.backend == "sol_attn":
+            out = run_sol_attention_bshd(q, k, v, is_causal=False)
+        elif self.backend == "flash4":
+            out = run_flash4_attention_bshd(q, k, v, is_causal=False)
+        else:
+            out = run_attention(
+                q.transpose(1, 2).contiguous(),
+                k.transpose(1, 2).contiguous(),
+                v.transpose(1, 2).contiguous(),
+                backend=self.backend,
+                is_causal=False,
+            ).transpose(1, 2)
+        partial = self.out_proj(out.reshape(context.sequence_length, inner).contiguous())
+        local_output = context.reduce_scatter_rows(partial)
+        bias = getattr(self, "tensor_parallel_output_bias", None)
+        if bias is not None:
+            local_output = local_output + bias.to(local_output)
+        if local_output.shape[0] != local_sequence:
+            raise RuntimeError("tensor-parallel attention returned the wrong local token count")
+        return local_output
--- a/src/h3_blackwell_runtime/backbone.py
+++ b/src/h3_blackwell_runtime/backbone.py
@ -2,12 +2,17 @@

 import torch
 from torch import nn
+from typing import TYPE_CHECKING

 from .adaln import H3CurveAdaLN
+from .attention import DEFAULT_ATTENTION_BACKEND
 from .block import H3DiTBlock
 from .checkpoint import H3Checkpoint
 from .rope import h3_rope_rotation

+if TYPE_CHECKING:
+    from .distributed import SequenceParallelContext
+

 class H3DenoiserBackbone(nn.Module):
    """Execute H3 transformer blocks over an already packed Ref2VA hidden sequence."""
@ -21,7 +26,7 @@ class H3DenoiserBackbone(nn.Module):
        self.register_buffer("inv_freq", inv_freq.to(torch.float32), persistent=False)

    @classmethod
-    def from_checkpoint(cls, checkpoint: H3Checkpoint, *, output_dtype=torch.bfloat16, attention_backend: str = "sage2"):
+    def from_checkpoint(cls, checkpoint: H3Checkpoint, *, output_dtype=torch.bfloat16, attention_backend: str = DEFAULT_ATTENTION_BACKEND):
        return cls(
            [H3DiTBlock.from_checkpoint(checkpoint, index, output_dtype=output_dtype, attention_backend=attention_backend) for index in range(50)],
            [H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{index}.adaln_proj") for index in range(50)],
@ -34,8 +39,29 @@ class H3DenoiserBackbone(nn.Module):
        timesteps: torch.Tensor,
        position_ids: torch.Tensor,
        segments: list[tuple[int, int, int]],
+        sequence_parallel: "SequenceParallelContext | None" = None,
+        tensor_parallel: "SequenceParallelContext | None" = None,
    ) -> torch.Tensor:
+        if sequence_parallel is not None and tensor_parallel is not None:
+            raise ValueError("choose Ulysses sequence parallelism or tensor parallelism, not both")
+        parallel = sequence_parallel or tensor_parallel
+        if parallel is not None:
+            if hidden.shape[0] != parallel.local_token_length:
+                raise ValueError(
+                    f"local hidden length {hidden.shape[0]} does not match sequence-parallel "
+                    f"partition {parallel.local_token_length}"
+                )
+            if position_ids.shape[0] != hidden.shape[0]:
+                raise ValueError("local position IDs must match local hidden rows")
+            segments = parallel.localize_segments(segments)
        rotation = h3_rope_rotation(position_ids.to(hidden.device), self.inv_freq, hidden.dtype)
        for block, adaln in zip(self.blocks, self.adaln, strict=True):
-            hidden = block(hidden, rotation, *adaln(timesteps), segments)
+            hidden = block(
+                hidden,
+                rotation,
+                *adaln(timesteps),
+                segments,
+                sequence_parallel,
+                tensor_parallel,
+            )
        return hidden
--- a/src/h3_blackwell_runtime/block.py
+++ b/src/h3_blackwell_runtime/block.py
@ -2,11 +2,15 @@

 import torch
 from torch import nn
+from typing import TYPE_CHECKING

-from .attention import H3SageAttention, rms_norm
+from .attention import DEFAULT_ATTENTION_BACKEND, H3SageAttention, rms_norm
 from .checkpoint import H3Checkpoint
 from .nvfp4 import Nvfp4Linear

+if TYPE_CHECKING:
+    from .distributed import SequenceParallelContext
+

 def modulate_segments(
    x: torch.Tensor,
@ -39,6 +43,8 @@ class H3SwiGLU(nn.Module):
        super().__init__()
        self.fc1 = fc1
        self.fc2 = fc2
+        self.chunks = 1
+        self.chunk_threshold = 4096

    @classmethod
    def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16):
@ -47,11 +53,38 @@ class H3SwiGLU(nn.Module):
            checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype),
        )

-    def forward(self, x: torch.Tensor) -> torch.Tensor:
+    def forward(
+        self,
+        x: torch.Tensor,
+        tensor_parallel: "SequenceParallelContext | None" = None,
+    ) -> torch.Tensor:
+        if tensor_parallel is not None:
+            full_x = tensor_parallel.all_gather_rows(x)
+            partial = self._forward_chunk(full_x)
+            local_output = tensor_parallel.reduce_scatter_rows(partial)
+            bias = getattr(self, "tensor_parallel_output_bias", None)
+            if bias is not None:
+                local_output = local_output + bias.to(local_output)
+            return local_output
+        if self.chunks > 1 and x.shape[0] >= self.chunk_threshold:
+            return torch.cat([self._forward_chunk(chunk) for chunk in x.chunk(self.chunks, dim=0)], dim=0)
+        return self._forward_chunk(x)
+
+    def _forward_chunk(self, x: torch.Tensor) -> torch.Tensor:
        gate, up = self.fc1(x).chunk(2, dim=-1)
        return self.fc2(torch.nn.functional.silu(gate).mul_(up))


+def configure_mlp_chunking(model: nn.Module, chunks: int, threshold: int = 4096) -> None:
+    """Configure exact row-chunked H3 SwiGLU execution to reduce peak activation memory."""
+    if chunks < 1:
+        raise ValueError("MLP chunks must be >= 1")
+    for module in model.modules():
+        if isinstance(module, H3SwiGLU):
+            module.chunks = chunks
+            module.chunk_threshold = threshold
+
+
 class H3DiTBlock(nn.Module):
    """One H3 transformer block with externally supplied AdaLN tensors."""

@ -72,7 +105,7 @@ class H3DiTBlock(nn.Module):
        self.register_buffer("norm2_weight", norm2_weight, persistent=False)

    @classmethod
-    def from_checkpoint(cls, checkpoint: H3Checkpoint, index: int, *, output_dtype=torch.bfloat16, attention_backend: str = "sage2"):
+    def from_checkpoint(cls, checkpoint: H3Checkpoint, index: int, *, output_dtype=torch.bfloat16, attention_backend: str = DEFAULT_ATTENTION_BACKEND):
        prefix = f"blocks.{index}"
        return cls(
            checkpoint.tensor(f"{prefix}.norm1.weight", dtype=output_dtype),
@ -92,8 +125,15 @@ class H3DiTBlock(nn.Module):
        scale_mlp: torch.Tensor,
        gate_mlp: torch.Tensor,
        segments: list[tuple[int, int, int]],
+        sequence_parallel: "SequenceParallelContext | None" = None,
+        tensor_parallel: "SequenceParallelContext | None" = None,
    ) -> torch.Tensor:
        h = modulate_segments(rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, segments)
-        x = gate_segments(x, self.attention(h, rope_rotation), gate_msa, segments)
+        x = gate_segments(
+            x,
+            self.attention(h, rope_rotation, sequence_parallel, tensor_parallel),
+            gate_msa,
+            segments,
+        )
        h = modulate_segments(rms_norm(x, self.norm2_weight, self.norm_eps), shift_mlp, scale_mlp, segments)
-        return gate_segments(x, self.mlp(h), gate_mlp, segments)
+        return gate_segments(x, self.mlp(h, tensor_parallel), gate_mlp, segments)
--- a/src/h3_blackwell_runtime/checkpoint.py
+++ b/src/h3_blackwell_runtime/checkpoint.py
@ -59,6 +59,10 @@ class H3Checkpoint:
            value = checkpoint.get_tensor(name)
        return value.to(dtype=dtype) if dtype is not None else value

+    def release_cache(self) -> None:
+        """Release tensors retained by whole-file loading after modules are built."""
+        self._no_mmap_tensors = None
+
    def nvfp4_linear(self, prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear:
        names = ("comfy_quant", "weight", "weight_scale", "weight_scale_2", "bias", "pre_quant_scale")
        tensors = {}
--- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp
+++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp
@ -0,0 +1,59 @@
+#include <torch/extension.h>
+
+#include <vector>
+
+torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor);
+torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divisor, torch::Tensor partials, torch::Tensor output, int64_t blocks, int64_t threads);
+std::vector<torch::Tensor> quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads);
+std::vector<torch::Tensor> qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim);
+
+torch::Tensor nvfp4_activation_scale(torch::Tensor input, double divisor) {
+  TORCH_CHECK(input.is_cuda(), "nvfp4_activation_scale expects a CUDA tensor");
+  TORCH_CHECK(input.is_contiguous(), "nvfp4_activation_scale expects contiguous input");
+  TORCH_CHECK(input.dim() == 2, "nvfp4_activation_scale expects a 2D tensor");
+  return nvfp4_activation_scale_cuda(input, divisor);
+}
+
+torch::Tensor nvfp4_activation_scale_into(torch::Tensor input, double divisor, torch::Tensor partials, torch::Tensor output, int64_t blocks, int64_t threads) {
+  TORCH_CHECK(input.is_cuda(), "nvfp4_activation_scale_into expects a CUDA tensor");
+  TORCH_CHECK(input.is_contiguous(), "nvfp4_activation_scale_into expects contiguous input");
+  TORCH_CHECK(input.dim() == 2, "nvfp4_activation_scale_into expects a 2D tensor");
+  TORCH_CHECK(partials.is_cuda() && output.is_cuda(), "nvfp4_activation_scale_into workspace must be CUDA tensors");
+  TORCH_CHECK(partials.device() == input.device() && output.device() == input.device(), "nvfp4_activation_scale_into workspace must be on the input device");
+  TORCH_CHECK(partials.is_contiguous() && output.is_contiguous(), "nvfp4_activation_scale_into workspace must be contiguous");
+  TORCH_CHECK(partials.scalar_type() == torch::kFloat32 && output.scalar_type() == torch::kFloat32, "nvfp4_activation_scale_into workspace must be float32");
+  TORCH_CHECK(blocks > 0, "nvfp4_activation_scale_into blocks must be positive");
+  TORCH_CHECK(threads > 0 && threads <= 1024, "nvfp4_activation_scale_into threads must be between 1 and 1024");
+  TORCH_CHECK((threads & (threads - 1)) == 0, "nvfp4_activation_scale_into threads must be a power of two");
+  TORCH_CHECK(partials.numel() >= blocks, "nvfp4_activation_scale_into partial workspace is too small");
+  TORCH_CHECK(output.numel() >= 1, "nvfp4_activation_scale_into output workspace is too small");
+  return nvfp4_activation_scale_into_cuda(input, divisor, partials, output, blocks, threads);
+}
+
+std::vector<torch::Tensor> quantize_nvfp4_bf16(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads) {
+  TORCH_CHECK(input.is_cuda(), "quantize_nvfp4_bf16 expects a CUDA tensor");
+  TORCH_CHECK(input.is_contiguous(), "quantize_nvfp4_bf16 expects contiguous input");
+  TORCH_CHECK(input.dim() == 2, "quantize_nvfp4_bf16 expects a 2D tensor");
+  TORCH_CHECK(input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_bf16 expects bfloat16 input");
+  TORCH_CHECK(scale.is_cuda(), "quantize_nvfp4_bf16 expects a CUDA scale tensor");
+  TORCH_CHECK(scale.device() == input.device(), "quantize_nvfp4_bf16 scale must be on the input device");
+  TORCH_CHECK(scale.numel() == 1, "quantize_nvfp4_bf16 scale must be scalar");
+  TORCH_CHECK(threads == 64 || threads == 128 || threads == 256 || threads == 512, "quantize_nvfp4_bf16 threads must be 64, 128, 256, or 512");
+  return quantize_nvfp4_bf16_cuda(input, scale, pad_16x, threads);
+}
+
+std::vector<torch::Tensor> qkv_to_bshd(torch::Tensor qkv, int64_t heads, int64_t head_dim) {
+  TORCH_CHECK(qkv.is_cuda(), "qkv_to_bshd expects a CUDA tensor");
+  TORCH_CHECK(qkv.is_contiguous(), "qkv_to_bshd expects contiguous input");
+  TORCH_CHECK(qkv.dim() == 2, "qkv_to_bshd expects a 2D [sequence, 3 * heads * head_dim] tensor");
+  TORCH_CHECK(heads > 0 && head_dim > 0, "qkv_to_bshd heads and head_dim must be positive");
+  TORCH_CHECK(qkv.size(1) == 3 * heads * head_dim, "qkv_to_bshd input feature dimension does not match 3 * heads * head_dim");
+  return qkv_to_bshd_cuda(qkv, heads, head_dim);
+}
+
+PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
+  m.def("nvfp4_activation_scale", &nvfp4_activation_scale, "Vortex NVFP4 activation scale");
+  m.def("nvfp4_activation_scale_into", &nvfp4_activation_scale_into, "Vortex NVFP4 activation scale with caller workspace");
+  m.def("quantize_nvfp4_bf16", &quantize_nvfp4_bf16, "Vortex BF16 to TensorCore NVFP4 quantizer");
+  m.def("qkv_to_bshd", &qkv_to_bshd, "Fused H3 QKV split to BSHD tensors");
+}
--- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu
+++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu
@ -0,0 +1,398 @@
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAGuard.h>
+#include <torch/extension.h>
+
+#include <cuda_fp8.h>
+
+#include <cmath>
+#include <cstdint>
+#include <limits>
+#include <vector>
+
+namespace {
+
+constexpr int kThreads = 256;
+
+int64_t roundup(int64_t value, int64_t multiple) {
+  return ((value + multiple - 1) / multiple) * multiple;
+}
+
+__inline__ __device__ float warp_reduce_max(float value) {
+  for (int offset = 16; offset > 0; offset >>= 1) {
+    value = fmaxf(value, __shfl_down_sync(0xffffffff, value, offset));
+  }
+  return value;
+}
+
+__inline__ __device__ float block_reduce_max(float value) {
+  __shared__ float warp_values[32];
+  const int lane = threadIdx.x & 31;
+  const int warp = threadIdx.x >> 5;
+  value = warp_reduce_max(value);
+  if (lane == 0) {
+    warp_values[warp] = value;
+  }
+  __syncthreads();
+  value = threadIdx.x < ((blockDim.x + 31) >> 5) ? warp_values[lane] : 0.0f;
+  if (warp == 0) {
+    value = warp_reduce_max(value);
+  }
+  return value;
+}
+
+__inline__ __device__ uint32_t max_bf16_abs_bits(uint32_t current, uint32_t packed) {
+  const uint32_t lo = packed & 0x7fffu;
+  const uint32_t hi = (packed >> 16) & 0x7fffu;
+  return max(current, max(lo, hi));
+}
+
+__inline__ __device__ float bf16_abs_bits_to_float(uint32_t bits) {
+  return __uint_as_float(bits << 16);
+}
+
+__inline__ __device__ float bf16_bits_to_float(uint16_t bits) {
+  return __uint_as_float(static_cast<uint32_t>(bits) << 16);
+}
+
+__inline__ __device__ uint8_t encode_fp4_e2m1(float value) {
+  const bool negative = signbit(value);
+  float abs_value = fabsf(value);
+  uint8_t code = 0;
+  if (abs_value > 5.0f) {
+    code = 7;
+  } else if (abs_value >= 3.5f) {
+    code = 6;
+  } else if (abs_value > 2.5f) {
+    code = 5;
+  } else if (abs_value >= 1.75f) {
+    code = 4;
+  } else if (abs_value > 1.25f) {
+    code = 3;
+  } else if (abs_value >= 0.75f) {
+    code = 2;
+  } else if (abs_value > 0.25f) {
+    code = 1;
+  }
+  return negative ? static_cast<uint8_t>(code | 0x8u) : code;
+}
+
+__inline__ __device__ uint8_t encode_fp8_e4m3(float value) {
+  __nv_fp8_e4m3 encoded(value);
+  return *reinterpret_cast<uint8_t*>(&encoded);
+}
+
+__inline__ __device__ float decode_fp8_e4m3(uint8_t value) {
+  __nv_fp8_e4m3 encoded;
+  *reinterpret_cast<uint8_t*>(&encoded) = value;
+  return static_cast<float>(encoded);
+}
+
+__global__ void quantize_nvfp4_bf16_kernel(
+    const uint16_t* __restrict__ input,
+    const float* __restrict__ scale,
+    uint8_t* __restrict__ qdata,
+    uint8_t* __restrict__ block_scale,
+    int64_t rows,
+    int64_t cols,
+    int64_t q_rows,
+    int64_t q_cols,
+    int64_t scale_rows,
+    int64_t scale_cols) {
+  const int64_t row = blockIdx.x;
+  const float tensor_scale = scale[0];
+  for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) {
+    float local_max = 0.0f;
+    float values[16];
+    #pragma unroll
+    for (int i = 0; i < 16; ++i) {
+      const int64_t col = block_col * 16 + i;
+      float value = 0.0f;
+      if (row < rows && col < cols) {
+        value = bf16_bits_to_float(input[row * cols + col]);
+      }
+      values[i] = value;
+      local_max = fmaxf(local_max, fabsf(value));
+    }
+    uint8_t scale_byte = 0;
+    float block_scale_value = 0.0f;
+    if (local_max > 0.0f && tensor_scale > 0.0f) {
+      scale_byte = encode_fp8_e4m3(local_max / (tensor_scale * 6.0f));
+      block_scale_value = decode_fp8_e4m3(scale_byte);
+    }
+    if (row < scale_rows) {
+      int64_t scale_row = row;
+      int64_t scale_col = block_col;
+      if (scale_cols == 336 || scale_cols == 896) {
+        const int64_t row_in_tile = row % 128;
+        const int64_t group = block_col / 4;
+        const int64_t pair = group / 2;
+        const int64_t phase = group % 2;
+        const int64_t col_stride = scale_cols == 336 ? 16 : 128;
+        const int64_t phase_offset = scale_cols == 336 ? 176 : 512;
+        int64_t swizzled_col_base = pair * col_stride + phase * phase_offset;
+        int64_t swizzled_col = (swizzled_col_base % scale_cols) + (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 32) * 16;
+        scale_row = (row / 128) * 128 + pair * (scale_cols == 336 ? 3 : 1) + (scale_cols == 336 ? phase : 0) + swizzled_col_base / scale_cols + swizzled_col / scale_cols;
+        scale_col = swizzled_col % scale_cols;
+        if (scale_col >= scale_cols) {
+          scale_col -= scale_cols;
+          scale_row += 1;
+        }
+      } else if (scale_cols >= 32) {
+        const int64_t row_in_tile = row % 128;
+        scale_row = (row / 128) * 128 + (block_col / 4) * 16 + ((row_in_tile % 32) / 2);
+        scale_col = (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 2) * 16;
+      }
+      block_scale[scale_row * scale_cols + scale_col] = scale_byte;
+    }
+    if (row < q_rows && block_col * 8 < q_cols) {
+      #pragma unroll
+      for (int pair = 0; pair < 8; ++pair) {
+        const float denom = tensor_scale * block_scale_value;
+        const uint8_t even = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2] / denom) : 0;
+        const uint8_t odd = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2 + 1] / denom) : 0;
+        qdata[row * q_cols + block_col * 8 + pair] = static_cast<uint8_t>((even << 4) | odd);
+      }
+    }
+  }
+}
+
+template <typename scalar_t>
+__global__ void partial_absmax_kernel(const scalar_t* __restrict__ input, float* __restrict__ partials, int64_t numel) {
+  __shared__ float shared[kThreads];
+  const int tid = threadIdx.x;
+  const int64_t stride = static_cast<int64_t>(blockDim.x) * gridDim.x;
+  int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + tid;
+  float local_max = 0.0f;
+  while (index < numel) {
+    const float value = static_cast<float>(input[index]);
+    local_max = fmaxf(local_max, fabsf(value));
+    index += stride;
+  }
+  shared[tid] = local_max;
+  __syncthreads();
+  for (int offset = blockDim.x / 2; offset > 0; offset >>= 1) {
+    if (tid < offset) {
+      shared[tid] = fmaxf(shared[tid], shared[tid + offset]);
+    }
+    __syncthreads();
+  }
+  if (tid == 0) {
+    partials[blockIdx.x] = shared[0];
+  }
+}
+
+__global__ void final_scale_kernel(const float* __restrict__ partials, float* __restrict__ output, int64_t count, float divisor) {
+  __shared__ float shared[kThreads];
+  const int tid = threadIdx.x;
+  float local_max = 0.0f;
+  for (int64_t index = tid; index < count; index += blockDim.x) {
+    local_max = fmaxf(local_max, partials[index]);
+  }
+  shared[tid] = local_max;
+  __syncthreads();
+  for (int offset = blockDim.x / 2; offset > 0; offset >>= 1) {
+    if (tid < offset) {
+      shared[tid] = fmaxf(shared[tid], shared[tid + offset]);
+    }
+    __syncthreads();
+  }
+  if (tid == 0) {
+    output[0] = shared[0] / divisor;
+  }
+}
+
+__global__ void partial_absmax_bf16_vec_kernel(const uint4* __restrict__ input, const uint16_t* __restrict__ scalar_input, float* __restrict__ partials, int64_t vector_count, int64_t numel) {
+  const int64_t stride = static_cast<int64_t>(blockDim.x) * gridDim.x;
+  int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
+  uint32_t local_bits = 0;
+  while (index < vector_count) {
+    const uint4 values = input[index];
+    local_bits = max_bf16_abs_bits(local_bits, values.x);
+    local_bits = max_bf16_abs_bits(local_bits, values.y);
+    local_bits = max_bf16_abs_bits(local_bits, values.z);
+    local_bits = max_bf16_abs_bits(local_bits, values.w);
+    index += stride;
+  }
+  const int64_t tail_start = vector_count * 8;
+  for (int64_t tail = tail_start + static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x; tail < numel; tail += stride) {
+    local_bits = max(local_bits, static_cast<uint32_t>(scalar_input[tail] & 0x7fffu));
+  }
+  const float local_max = bf16_abs_bits_to_float(local_bits);
+  const float block_max = block_reduce_max(local_max);
+  if (threadIdx.x == 0) {
+    partials[blockIdx.x] = block_max;
+  }
+}
+
+__global__ void final_scale_warp_kernel(const float* __restrict__ partials, float* __restrict__ output, int64_t count, float divisor) {
+  float local_max = 0.0f;
+  for (int64_t index = threadIdx.x; index < count; index += blockDim.x) {
+    local_max = fmaxf(local_max, partials[index]);
+  }
+  const float block_max = block_reduce_max(local_max);
+  if (threadIdx.x == 0) {
+    output[0] = block_max / divisor;
+  }
+}
+
+template <typename scalar_t>
+__global__ void qkv_to_bshd_kernel(
+    const scalar_t* __restrict__ qkv,
+    scalar_t* __restrict__ q,
+    scalar_t* __restrict__ k,
+    scalar_t* __restrict__ v,
+    int64_t sequence,
+    int64_t heads,
+    int64_t head_dim) {
+  const int64_t inner = heads * head_dim;
+  const int64_t total = sequence * inner;
+  const int64_t stride = static_cast<int64_t>(blockDim.x) * gridDim.x;
+  for (int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x; index < total; index += stride) {
+    const int64_t s = index / inner;
+    const int64_t hd = index - s * inner;
+    const int64_t source = s * inner * 3 + hd;
+    q[index] = qkv[source];
+    k[index] = qkv[source + inner];
+    v[index] = qkv[source + inner * 2];
+  }
+}
+
+__global__ void qkv_to_bshd_vec16_kernel(
+    const uint4* __restrict__ qkv,
+    uint4* __restrict__ q,
+    uint4* __restrict__ k,
+    uint4* __restrict__ v,
+    int64_t sequence,
+    int64_t vectors_per_inner) {
+  const int64_t total = sequence * vectors_per_inner;
+  const int64_t row_stride = vectors_per_inner * 3;
+  const int64_t stride = static_cast<int64_t>(blockDim.x) * gridDim.x;
+  for (int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x; index < total; index += stride) {
+    const int64_t s = index / vectors_per_inner;
+    const int64_t offset = index - s * vectors_per_inner;
+    const int64_t source = s * row_stride + offset;
+    q[index] = qkv[source];
+    k[index] = qkv[source + vectors_per_inner];
+    v[index] = qkv[source + vectors_per_inner * 2];
+  }
+}
+
+}  // namespace
+
+torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor) {
+  c10::cuda::CUDAGuard device_guard(input.device());
+  const auto numel = input.numel();
+  TORCH_CHECK(numel > 0, "nvfp4_activation_scale input must be non-empty");
+  TORCH_CHECK(divisor > 0.0, "nvfp4_activation_scale divisor must be positive");
+
+  const int blocks = static_cast<int>(std::min<int64_t>((numel + kThreads - 1) / kThreads, 4096));
+  auto partials = torch::empty({blocks}, input.options().dtype(torch::kFloat32));
+  auto output = torch::empty({}, input.options().dtype(torch::kFloat32));
+  auto stream = at::cuda::getCurrentCUDAStream();
+
+  AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, input.scalar_type(), "vortex_nvfp4_activation_scale", [&] {
+    partial_absmax_kernel<scalar_t><<<blocks, kThreads, 0, stream>>>(
+        input.data_ptr<scalar_t>(), partials.data_ptr<float>(), numel);
+  });
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  final_scale_kernel<<<1, kThreads, 0, stream>>>(partials.data_ptr<float>(), output.data_ptr<float>(), blocks, static_cast<float>(divisor));
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  return output;
+}
+
+torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divisor, torch::Tensor partials, torch::Tensor output, int64_t blocks, int64_t threads) {
+  c10::cuda::CUDAGuard device_guard(input.device());
+  const auto numel = input.numel();
+  TORCH_CHECK(numel > 0, "nvfp4_activation_scale_into input must be non-empty");
+  TORCH_CHECK(divisor > 0.0, "nvfp4_activation_scale_into divisor must be positive");
+  TORCH_CHECK(input.scalar_type() == at::ScalarType::BFloat16, "nvfp4_activation_scale_into v1 currently expects bfloat16 input");
+  TORCH_CHECK(threads == 128 || threads == 256 || threads == 512, "nvfp4_activation_scale_into threads must be 128, 256, or 512");
+  TORCH_CHECK(blocks > 0, "nvfp4_activation_scale_into blocks must be positive");
+  TORCH_CHECK(reinterpret_cast<uintptr_t>(input.data_ptr()) % alignof(uint4) == 0, "nvfp4_activation_scale_into expects 16-byte aligned input");
+
+  const int launch_blocks = static_cast<int>(blocks);
+  const int launch_threads = static_cast<int>(threads);
+  const int64_t vector_count = numel / 8;
+  auto stream = at::cuda::getCurrentCUDAStream();
+
+  partial_absmax_bf16_vec_kernel<<<launch_blocks, launch_threads, 0, stream>>>(
+      reinterpret_cast<const uint4*>(input.data_ptr()),
+      reinterpret_cast<const uint16_t*>(input.data_ptr()),
+      partials.data_ptr<float>(),
+      vector_count,
+      numel);
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  final_scale_warp_kernel<<<1, launch_threads, 0, stream>>>(partials.data_ptr<float>(), output.data_ptr<float>(), blocks, static_cast<float>(divisor));
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  return output;
+}
+
+std::vector<torch::Tensor> quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads) {
+  c10::cuda::CUDAGuard device_guard(input.device());
+  const int64_t rows = input.size(0);
+  const int64_t cols = input.size(1);
+  TORCH_CHECK(cols % 2 == 0, "quantize_nvfp4_bf16 expects an even feature dimension");
+  const int64_t q_rows = pad_16x ? roundup(rows, 16) : rows;
+  const int64_t q_cols = pad_16x ? roundup(cols, 16) / 2 : cols / 2;
+  const int64_t scale_rows = roundup(q_rows, 128);
+  const int64_t scale_cols = roundup(q_cols / 8, 4);
+  auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8));
+  auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn);
+  auto stream = at::cuda::getCurrentCUDAStream();
+  quantize_nvfp4_bf16_kernel<<<scale_rows, static_cast<int>(threads), 0, stream>>>(
+      reinterpret_cast<const uint16_t*>(input.data_ptr()),
+      scale.data_ptr<float>(),
+      qdata.data_ptr<uint8_t>(),
+      reinterpret_cast<uint8_t*>(block_scale.data_ptr()),
+      rows,
+      cols,
+      q_rows,
+      q_cols,
+      scale_rows,
+      scale_cols);
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  return {qdata, block_scale};
+}
+
+std::vector<torch::Tensor> qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim) {
+  c10::cuda::CUDAGuard device_guard(qkv.device());
+  const int64_t sequence = qkv.size(0);
+  const int64_t inner = heads * head_dim;
+  auto q = torch::empty({1, sequence, heads, head_dim}, qkv.options());
+  auto k = torch::empty({1, sequence, heads, head_dim}, qkv.options());
+  auto v = torch::empty({1, sequence, heads, head_dim}, qkv.options());
+  const int threads = 256;
+  auto stream = at::cuda::getCurrentCUDAStream();
+  const bool can_vectorize =
+      qkv.element_size() == 2 &&
+      (inner * static_cast<int64_t>(qkv.element_size())) % static_cast<int64_t>(sizeof(uint4)) == 0 &&
+      reinterpret_cast<uintptr_t>(qkv.data_ptr()) % alignof(uint4) == 0 &&
+      reinterpret_cast<uintptr_t>(q.data_ptr()) % alignof(uint4) == 0 &&
+      reinterpret_cast<uintptr_t>(k.data_ptr()) % alignof(uint4) == 0 &&
+      reinterpret_cast<uintptr_t>(v.data_ptr()) % alignof(uint4) == 0;
+  if (can_vectorize) {
+    const int64_t vectors_per_inner = inner * static_cast<int64_t>(qkv.element_size()) / static_cast<int64_t>(sizeof(uint4));
+    const int blocks = static_cast<int>(std::min<int64_t>((sequence * vectors_per_inner + threads - 1) / threads, 4096));
+    qkv_to_bshd_vec16_kernel<<<blocks, threads, 0, stream>>>(
+        reinterpret_cast<const uint4*>(qkv.data_ptr()),
+        reinterpret_cast<uint4*>(q.data_ptr()),
+        reinterpret_cast<uint4*>(k.data_ptr()),
+        reinterpret_cast<uint4*>(v.data_ptr()),
+        sequence,
+        vectors_per_inner);
+  } else {
+    const int blocks = static_cast<int>(std::min<int64_t>((sequence * inner + threads - 1) / threads, 4096));
+    AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, qkv.scalar_type(), "h3_qkv_to_bshd", [&] {
+      qkv_to_bshd_kernel<scalar_t><<<blocks, threads, 0, stream>>>(
+          qkv.data_ptr<scalar_t>(),
+          q.data_ptr<scalar_t>(),
+          k.data_ptr<scalar_t>(),
+          v.data_ptr<scalar_t>(),
+          sequence,
+          heads,
+          head_dim);
+    });
+  }
+  C10_CUDA_KERNEL_LAUNCH_CHECK();
+  return {q, k, v};
+}
--- a/src/h3_blackwell_runtime/denoiser.py
+++ b/src/h3_blackwell_runtime/denoiser.py
@ -2,11 +2,16 @@

 import torch
 from torch import nn
+from typing import TYPE_CHECKING

+from .attention import DEFAULT_ATTENTION_BACKEND
 from .backbone import H3DenoiserBackbone
 from .checkpoint import H3Checkpoint
 from .final import H3FinalLayer

+if TYPE_CHECKING:
+    from .distributed import SequenceParallelContext
+

 class H3PackedDenoiser(nn.Module):
    """Run the H3 transformer once its Ref2VA payload has been packed into hidden rows."""
@ -17,7 +22,7 @@ class H3PackedDenoiser(nn.Module):
        self.final_layer = final_layer

    @classmethod
-    def from_checkpoint(cls, checkpoint: H3Checkpoint, *, output_dtype=torch.bfloat16, attention_backend: str = "sage2"):
+    def from_checkpoint(cls, checkpoint: H3Checkpoint, *, output_dtype=torch.bfloat16, attention_backend: str = DEFAULT_ATTENTION_BACKEND):
        return cls(
            H3DenoiserBackbone.from_checkpoint(checkpoint, output_dtype=output_dtype, attention_backend=attention_backend),
            H3FinalLayer.from_checkpoint(checkpoint, output_dtype=output_dtype),
@ -34,3 +39,67 @@ class H3PackedDenoiser(nn.Module):
    ) -> tuple[torch.Tensor, torch.Tensor]:
        hidden = self.backbone(hidden, timesteps, position_ids, segments)
        return self.final_layer(hidden, timesteps, video_segment, audio_segment)
+
+    def forward_sequence_parallel(
+        self,
+        full_hidden: torch.Tensor,
+        timesteps: torch.Tensor,
+        full_position_ids: torch.Tensor,
+        segments: list[tuple[int, int, int]],
+        video_segment: tuple[int, int, int],
+        audio_segment: tuple[int, int, int],
+        context: "SequenceParallelContext",
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """Run all 50 blocks with token-sharded activations and Ulysses attention."""
+        if full_hidden.shape[0] != context.sequence_length:
+            raise ValueError("packed hidden length does not match sequence-parallel context")
+        start, stop = context.local_token_range
+        local_hidden = full_hidden[start:stop].contiguous()
+        local_positions = full_position_ids[start:stop].contiguous()
+        del full_hidden, full_position_ids
+        local_hidden = self.backbone(
+            local_hidden,
+            timesteps,
+            local_positions,
+            segments,
+            sequence_parallel=context,
+        )
+        return self.final_layer.forward_sequence_parallel(
+            local_hidden,
+            timesteps,
+            video_segment,
+            audio_segment,
+            context,
+        )
+
+    def forward_tensor_parallel(
+        self,
+        full_hidden: torch.Tensor,
+        timesteps: torch.Tensor,
+        full_position_ids: torch.Tensor,
+        segments: list[tuple[int, int, int]],
+        video_segment: tuple[int, int, int],
+        audio_segment: tuple[int, int, int],
+        context: "SequenceParallelContext",
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """Run TP-sharded NVFP4 linears with ragged sequence-sharded residuals."""
+        if full_hidden.shape[0] != context.sequence_length:
+            raise ValueError("packed hidden length does not match tensor-parallel context")
+        start, stop = context.local_token_range
+        local_hidden = full_hidden[start:stop].contiguous()
+        local_positions = full_position_ids[start:stop].contiguous()
+        del full_hidden, full_position_ids
+        local_hidden = self.backbone(
+            local_hidden,
+            timesteps,
+            local_positions,
+            segments,
+            tensor_parallel=context,
+        )
+        return self.final_layer.forward_sequence_parallel(
+            local_hidden,
+            timesteps,
+            video_segment,
+            audio_segment,
+            context,
+        )
--- a/src/h3_blackwell_runtime/distributed.py
+++ b/src/h3_blackwell_runtime/distributed.py
@ -0,0 +1,324 @@
+"""Ragged Ulysses sequence-parallel transport for H3 inference."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+import torch.distributed as dist
+
+
+def balanced_ranges(total: int, parts: int) -> tuple[tuple[int, int], ...]:
+    """Split ``total`` ordered items into balanced contiguous non-empty ranges."""
+    if parts < 1:
+        raise ValueError("parts must be positive")
+    if total < parts:
+        raise ValueError(f"cannot split {total} items into {parts} non-empty ranges")
+    base, extra = divmod(total, parts)
+    lengths = [base + (rank < extra) for rank in range(parts)]
+    ranges = []
+    start = 0
+    for length in lengths:
+        stop = start + int(length)
+        ranges.append((start, stop))
+        start = stop
+    return tuple(ranges)
+
+
+def range_lengths(ranges: tuple[tuple[int, int], ...]) -> tuple[int, ...]:
+    return tuple(stop - start for start, stop in ranges)
+
+
+def localize_segments(
+    segments: list[tuple[int, int, int]],
+    shard_start: int,
+    shard_stop: int,
+) -> list[tuple[int, int, int]]:
+    """Clip global H3 AdaLN segments to one contiguous token shard."""
+    localized = []
+    for start, stop, row in segments:
+        local_start = max(start, shard_start)
+        local_stop = min(stop, shard_stop)
+        if local_start < local_stop:
+            localized.append((local_start - shard_start, local_stop - shard_start, row))
+    return localized
+
+
+@dataclass(frozen=True)
+class SequenceParallelContext:
+    """One rank's ragged token and attention-head ownership."""
+
+    group: dist.ProcessGroup | None
+    rank: int
+    world_size: int
+    token_ranges: tuple[tuple[int, int], ...]
+    head_ranges: tuple[tuple[int, int], ...]
+    head_dim: int
+
+    @classmethod
+    def create(
+        cls,
+        sequence_length: int,
+        heads: int,
+        head_dim: int,
+        *,
+        group: dist.ProcessGroup | None = None,
+    ) -> "SequenceParallelContext":
+        if not dist.is_initialized():
+            raise RuntimeError("torch.distributed process group is not initialized")
+        world_size = dist.get_world_size(group)
+        rank = dist.get_rank(group)
+        return cls(
+            group=group,
+            rank=rank,
+            world_size=world_size,
+            token_ranges=balanced_ranges(sequence_length, world_size),
+            head_ranges=balanced_ranges(heads, world_size),
+            head_dim=head_dim,
+        )
+
+    @property
+    def sequence_length(self) -> int:
+        return self.token_ranges[-1][1]
+
+    @property
+    def heads(self) -> int:
+        return self.head_ranges[-1][1]
+
+    @property
+    def token_lengths(self) -> tuple[int, ...]:
+        return range_lengths(self.token_ranges)
+
+    @property
+    def head_lengths(self) -> tuple[int, ...]:
+        return range_lengths(self.head_ranges)
+
+    @property
+    def local_token_range(self) -> tuple[int, int]:
+        return self.token_ranges[self.rank]
+
+    @property
+    def local_head_range(self) -> tuple[int, int]:
+        return self.head_ranges[self.rank]
+
+    @property
+    def local_token_length(self) -> int:
+        start, stop = self.local_token_range
+        return stop - start
+
+    @property
+    def local_head_count(self) -> int:
+        start, stop = self.local_head_range
+        return stop - start
+
+    def localize_segments(self, segments: list[tuple[int, int, int]]) -> list[tuple[int, int, int]]:
+        return localize_segments(segments, *self.local_token_range)
+
+    def seq_to_heads(
+        self,
+        q: torch.Tensor,
+        k: torch.Tensor,
+        v: torch.Tensor,
+    ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+        """Exchange local tokens for full-sequence Q/K/V over locally owned heads.
+
+        Inputs use BSHD layout ``[1, local_tokens, all_heads, head_dim]``. Outputs
+        use ``[1, all_tokens, local_heads, head_dim]``.
+        """
+        expected = (1, self.local_token_length, self.heads, self.head_dim)
+        if tuple(q.shape) != expected or tuple(k.shape) != expected or tuple(v.shape) != expected:
+            raise ValueError(
+                f"sequence-parallel Q/K/V must each have shape {expected}; "
+                f"got {tuple(q.shape)}, {tuple(k.shape)}, {tuple(v.shape)}"
+            )
+        if q.dtype != k.dtype or q.dtype != v.dtype or q.device != k.device or q.device != v.device:
+            raise ValueError("sequence-parallel Q/K/V must share dtype and device")
+        if self.world_size == 1:
+            return q, k, v
+
+        local_qkv = torch.stack((q[0], k[0], v[0]), dim=1)
+        send_chunks = [
+            local_qkv[:, :, start:stop, :].contiguous().view(-1)
+            for start, stop in self.head_ranges
+        ]
+        input_splits = [chunk.numel() for chunk in send_chunks]
+        send = torch.cat(send_chunks)
+
+        output_splits = [
+            token_length * 3 * self.local_head_count * self.head_dim
+            for token_length in self.token_lengths
+        ]
+        receive = torch.empty(sum(output_splits), dtype=q.dtype, device=q.device)
+        dist.all_to_all_single(
+            receive,
+            send,
+            output_split_sizes=output_splits,
+            input_split_sizes=input_splits,
+            group=self.group,
+        )
+
+        source_chunks = []
+        offset = 0
+        for token_length, count in zip(self.token_lengths, output_splits, strict=True):
+            source_chunks.append(
+                receive[offset : offset + count].view(
+                    token_length, 3, self.local_head_count, self.head_dim,
+                )
+            )
+            offset += count
+        full_qkv = torch.cat(source_chunks, dim=0)
+        full_q, full_k, full_v = full_qkv.unbind(dim=1)
+        return full_q.unsqueeze(0), full_k.unsqueeze(0), full_v.unsqueeze(0)
+
+    def heads_to_seq(self, output: torch.Tensor) -> torch.Tensor:
+        """Exchange full-sequence local-head output back to local-token all-head output.
+
+        Input is BSHD ``[1, all_tokens, local_heads, head_dim]``. The return value
+        is ``[local_tokens, all_heads, head_dim]``.
+        """
+        expected = (1, self.sequence_length, self.local_head_count, self.head_dim)
+        if tuple(output.shape) != expected:
+            raise ValueError(f"sequence-parallel output must have shape {expected}, got {tuple(output.shape)}")
+        if self.world_size == 1:
+            return output[0]
+
+        output = output[0]
+        send_chunks = []
+        input_splits = []
+        token_offset = 0
+        for token_length in self.token_lengths:
+            chunk = output[token_offset : token_offset + token_length].contiguous().view(-1)
+            send_chunks.append(chunk)
+            input_splits.append(chunk.numel())
+            token_offset += token_length
+        send = torch.cat(send_chunks)
+
+        output_splits = [
+            self.local_token_length * head_length * self.head_dim
+            for head_length in self.head_lengths
+        ]
+        receive = torch.empty(sum(output_splits), dtype=output.dtype, device=output.device)
+        dist.all_to_all_single(
+            receive,
+            send,
+            output_split_sizes=output_splits,
+            input_split_sizes=input_splits,
+            group=self.group,
+        )
+
+        head_chunks = []
+        offset = 0
+        for head_length, count in zip(self.head_lengths, output_splits, strict=True):
+            head_chunks.append(
+                receive[offset : offset + count].view(
+                    self.local_token_length, head_length, self.head_dim,
+                )
+            )
+            offset += count
+        return torch.cat(head_chunks, dim=1).contiguous()
+
+    def target_intersection(self, target_start: int, target_stop: int) -> tuple[int, int]:
+        """Return one global target span's bounds relative to this token shard."""
+        shard_start, shard_stop = self.local_token_range
+        start = max(target_start, shard_start)
+        stop = min(target_stop, shard_stop)
+        if start >= stop:
+            return (0, 0)
+        return (start - shard_start, stop - shard_start)
+
+    def target_counts(self, target_start: int, target_stop: int) -> tuple[int, ...]:
+        """Return ordered target-row counts contributed by every token rank."""
+        counts = []
+        for shard_start, shard_stop in self.token_ranges:
+            counts.append(max(0, min(target_stop, shard_stop) - max(target_start, shard_start)))
+        return tuple(counts)
+
+    def all_gather_target_rows(
+        self,
+        local_rows: torch.Tensor,
+        target_start: int,
+        target_stop: int,
+    ) -> torch.Tensor:
+        """Gather a global target span's projected rows onto every rank.
+
+        Padding is transport-only and is removed before concatenation; it is never
+        exposed to attention or model semantics.
+        """
+        counts = self.target_counts(target_start, target_stop)
+        if local_rows.ndim != 2:
+            raise ValueError("target rows must be rank-2 [rows, features]")
+        if local_rows.shape[0] != counts[self.rank]:
+            raise ValueError(
+                f"rank {self.rank} must contribute {counts[self.rank]} target rows, "
+                f"got {local_rows.shape[0]}"
+            )
+        if self.world_size == 1:
+            return local_rows
+        max_rows = max(counts)
+        padded = torch.zeros(
+            max_rows, local_rows.shape[1], dtype=local_rows.dtype, device=local_rows.device,
+        )
+        if local_rows.shape[0]:
+            padded[: local_rows.shape[0]].copy_(local_rows)
+        gathered = [torch.empty_like(padded) for _ in range(self.world_size)]
+        dist.all_gather(gathered, padded, group=self.group)
+        return torch.cat(
+            [rows[:count] for rows, count in zip(gathered, counts, strict=True) if count],
+            dim=0,
+        )
+
+    def all_gather_rows(self, local_rows: torch.Tensor) -> torch.Tensor:
+        """Gather ragged token rows on every rank without exposing padding to the model."""
+        if local_rows.shape[0] != self.local_token_length:
+            raise ValueError(
+                f"rank {self.rank} must contribute {self.local_token_length} rows, "
+                f"got {local_rows.shape[0]}"
+            )
+        if self.world_size == 1:
+            return local_rows
+        max_rows = max(self.token_lengths)
+        padded = torch.zeros(
+            (max_rows, *local_rows.shape[1:]),
+            dtype=local_rows.dtype,
+            device=local_rows.device,
+        )
+        padded[: local_rows.shape[0]].copy_(local_rows)
+        gathered = [torch.empty_like(padded) for _ in range(self.world_size)]
+        dist.all_gather(gathered, padded, group=self.group)
+        return torch.cat(
+            [rows[:count] for rows, count in zip(gathered, self.token_lengths, strict=True)],
+            dim=0,
+        )
+
+    def reduce_scatter_rows(self, partial_full_rows: torch.Tensor) -> torch.Tensor:
+        """Sum tensor-parallel partials and return this rank's ragged token rows."""
+        if partial_full_rows.shape[0] != self.sequence_length:
+            raise ValueError(
+                f"partial rows must cover sequence length {self.sequence_length}, "
+                f"got {partial_full_rows.shape[0]}"
+            )
+        if self.world_size == 1:
+            return partial_full_rows
+        trailing_shape = partial_full_rows.shape[1:]
+        row_width = partial_full_rows[0].numel()
+        send_chunks = []
+        input_splits = []
+        offset = 0
+        for token_length in self.token_lengths:
+            chunk = partial_full_rows[offset : offset + token_length].contiguous().view(-1)
+            send_chunks.append(chunk)
+            input_splits.append(chunk.numel())
+            offset += token_length
+        send = torch.cat(send_chunks)
+
+        output_splits = [self.local_token_length * row_width] * self.world_size
+        receive = torch.empty(sum(output_splits), dtype=send.dtype, device=send.device)
+        dist.all_to_all_single(
+            receive,
+            send,
+            output_split_sizes=output_splits,
+            input_split_sizes=input_splits,
+            group=self.group,
+        )
+        contributions = receive.view(self.world_size, self.local_token_length, *trailing_shape)
+        return contributions.sum(dim=0)
--- a/src/h3_blackwell_runtime/final.py
+++ b/src/h3_blackwell_runtime/final.py
@ -3,10 +3,14 @@
 import torch
 import torch.nn.functional as functional
 from torch import nn
+from typing import TYPE_CHECKING

 from .attention import rms_norm
 from .checkpoint import H3Checkpoint

+if TYPE_CHECKING:
+    from .distributed import SequenceParallelContext
+

 class H3FinalLayer(nn.Module):
    def __init__(
@ -70,3 +74,43 @@ class H3FinalLayer(nn.Module):
            functional.linear(video_hidden, self.video_weight, self.video_bias),
            functional.linear(audio_hidden, self.audio_weight, self.audio_bias),
        )
+
+    def forward_sequence_parallel(
+        self,
+        local_hidden: torch.Tensor,
+        timesteps: torch.Tensor,
+        video_segment: tuple[int, int, int],
+        audio_segment: tuple[int, int, int],
+        context: "SequenceParallelContext",
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """Project local target intersections and gather compact AV rows on every rank."""
+        position = timesteps.float().clamp(0, 1) * (self.curve_table.shape[0] - 1)
+        lower = position.floor().long().clamp(max=self.curve_table.shape[0] - 2)
+        embedding = torch.lerp(
+            self.curve_table[lower],
+            self.curve_table[lower + 1],
+            (position - lower).unsqueeze(1),
+        )
+        shift, scale = functional.linear(embedding, self.adaln_weight, self.adaln_bias).chunk(2, dim=-1)
+
+        outputs = []
+        for segment, weight, bias in (
+            (video_segment, self.video_weight, self.video_bias),
+            (audio_segment, self.audio_weight, self.audio_bias),
+        ):
+            global_start, global_stop, row = segment
+            local_start, local_stop = context.target_intersection(global_start, global_stop)
+            local_target = local_hidden[local_start:local_stop]
+            if local_target.shape[0]:
+                local_target = (
+                    rms_norm(local_target, self.norm_weight, self.eps)
+                    * (1.0 + scale[row])
+                    + shift[row]
+                ).to(torch.float32)
+                local_output = functional.linear(local_target, weight, bias)
+            else:
+                local_output = torch.empty(
+                    0, weight.shape[0], dtype=torch.float32, device=local_hidden.device,
+                )
+            outputs.append(context.all_gather_target_rows(local_output, global_start, global_stop))
+        return outputs[0], outputs[1]
--- a/src/h3_blackwell_runtime/latent_upscaler.py
+++ b/src/h3_blackwell_runtime/latent_upscaler.py
@ -0,0 +1,163 @@
+"""MiniMax H3 learned latent upscaler inference.
+
+Adapted from LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler (Apache-2.0).
+"""
+
+from pathlib import Path
+import re
+
+import torch
+from safetensors.torch import load_file
+from torch import nn
+from torch.nn import functional as F
+
+
+LATENTS_MEAN = (
+    0.858090341091156, -0.9606591463088989, 1.0661640167236328, -0.5090325474739075,
+    -0.2727581858634949, -1.3675414323806763, -0.2553254961967468, -0.26907554268836975,
+    -0.5376840829849243, -0.0464097298681736, 0.6657370328903198, 0.19690127670764923,
+    -0.5460608005523682, -0.4035342037677765, -0.23683024942874908, 0.25928452610969543,
+    -0.30133944749832153, 0.211341992020607, -1.1206848621368408, 0.3581933379173279,
+    -0.04225143790245056, 0.2604829967021942, 0.22864092886447906, 0.7056031823158264,
+)
+LATENTS_STD = (
+    1.2223774194717407, 1.2767263650894165, 1.6831774711608887, 1.7549455165863037,
+    1.5636216402053833, 2.194143533706665, 0.9653137922286987, 1.0569885969161987,
+    0.841948926448822, 0.7729952931404114, 1.8955937623977661, 0.946841835975647,
+    0.7996809482574463, 0.44988900423049927, 0.7197399735450745, 0.6936293244361877,
+    2.961095094680786, 2.7694199085235596, 3.0496184825897217, 2.1088054180145264,
+    3.276226282119751, 3.1627357006073, 2.2816812992095947, 2.6127843856811523,
+)
+
+
+def _normalization(channels: int) -> nn.GroupNorm:
+    return nn.GroupNorm(32, channels)
+
+
+class ResBlockEmb3D(nn.Module):
+    def __init__(self, channels: int, emb_channels: int, dropout: float = 0.0):
+        super().__init__()
+        self.in_layers = nn.Sequential(
+            _normalization(channels),
+            nn.SiLU(),
+            nn.Conv3d(channels, channels, 3, padding=1),
+        )
+        self.emb_layers = nn.Sequential(nn.SiLU(), nn.Linear(emb_channels, 2 * channels))
+        self.out_norm = _normalization(channels)
+        self.out_layers = nn.Sequential(
+            nn.SiLU(),
+            nn.Dropout(p=dropout),
+            nn.Conv3d(channels, channels, 3, padding=1),
+        )
+        nn.init.zeros_(self.out_layers[-1].weight)
+        nn.init.zeros_(self.out_layers[-1].bias)
+
+    def forward(self, x: torch.Tensor, emb: torch.Tensor) -> torch.Tensor:
+        hidden = self.in_layers(x)
+        scale, shift = self.emb_layers(emb).to(hidden.dtype).chunk(2, dim=1)
+        hidden = self.out_norm(hidden) * (1 + scale[:, :, None, None, None]) + shift[:, :, None, None, None]
+        return x + self.out_layers(hidden)
+
+
+class TemporalConv(nn.Module):
+    def __init__(self, channels: int, kernel_size: int = 5):
+        super().__init__()
+        self.norm = _normalization(channels)
+        self.dwconv = nn.Conv3d(
+            channels, channels, kernel_size=(kernel_size, 1, 1),
+            padding=(kernel_size // 2, 0, 0), groups=channels,
+        )
+        self.pwconv = nn.Conv3d(channels, channels, 1)
+        nn.init.zeros_(self.pwconv.weight)
+        nn.init.zeros_(self.pwconv.bias)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        hidden = self.dwconv(F.silu(self.norm(x)))
+        return x + self.pwconv(hidden)
+
+
+class H3LatentResizer3D(nn.Module):
+    def __init__(
+        self,
+        in_channels: int = 24,
+        in_blocks: int = 12,
+        out_blocks: int = 12,
+        channels: int = 512,
+        dropout: float = 0.1,
+        temporal_every: int = 2,
+        temporal_kernel: int = 5,
+    ):
+        super().__init__()
+        self.conv_in = nn.Conv3d(in_channels, channels, 3, padding=1)
+        embed_dim = 64
+        self.embed = nn.Sequential(nn.Linear(1, embed_dim), nn.SiLU(), nn.Linear(embed_dim, embed_dim))
+        self.in_blocks = self._make_blocks(in_blocks, channels, embed_dim, dropout, temporal_every, temporal_kernel)
+        self.out_blocks = self._make_blocks(out_blocks, channels, embed_dim, dropout, temporal_every, temporal_kernel)
+        self.norm_out = _normalization(channels)
+        self.conv_out = nn.Conv3d(channels, in_channels, 3, padding=1)
+
+    @staticmethod
+    def _make_blocks(count, channels, embed_dim, dropout, temporal_every, temporal_kernel):
+        blocks = nn.ModuleList()
+        for index in range(count):
+            blocks.append(ResBlockEmb3D(channels, embed_dim, dropout))
+            if temporal_every > 0 and index % temporal_every == 0:
+                blocks.append(TemporalConv(channels, temporal_kernel))
+        return blocks
+
+    def forward(self, x: torch.Tensor, *, scale: float, target_size: tuple[int, int, int]) -> torch.Tensor:
+        emb = self.embed(torch.tensor([[scale - 1]], dtype=x.dtype, device=x.device)).expand(x.shape[0], -1)
+        hidden = self.conv_in(x)
+        for block in self.in_blocks:
+            hidden = block(hidden, emb) if isinstance(block, ResBlockEmb3D) else block(hidden)
+        hidden = F.interpolate(hidden, size=target_size, mode="trilinear", align_corners=False)
+        for block in self.out_blocks:
+            hidden = block(hidden, emb) if isinstance(block, ResBlockEmb3D) else block(hidden)
+        return self.conv_out(F.silu(self.norm_out(hidden)))
+
+
+def _checkpoint_config(state: dict[str, torch.Tensor]) -> dict:
+    in_ids = {int(match.group(1)) for key in state if (match := re.match(r"in_blocks\.(\d+)\.in_layers\.", key))}
+    out_ids = {int(match.group(1)) for key in state if (match := re.match(r"out_blocks\.(\d+)\.in_layers\.", key))}
+    temporal_keys = [key for key in state if key.endswith("dwconv.weight")]
+    conv_in = state["conv_in.weight"]
+    return {
+        "in_channels": conv_in.shape[1],
+        "in_blocks": len(in_ids),
+        "out_blocks": len(out_ids),
+        "channels": conv_in.shape[0],
+        "temporal_every": 2 if temporal_keys else 0,
+        "temporal_kernel": state[temporal_keys[0]].shape[2] if temporal_keys else 5,
+    }
+
+
+def load_h3_latent_upscaler(
+    path: str | Path,
+    *,
+    device: str | torch.device = "cuda",
+    dtype: torch.dtype = torch.float16,
+) -> H3LatentResizer3D:
+    state = load_file(str(path), device="cpu")
+    if any(key.startswith("upscaler.") for key in state):
+        state = {key.removeprefix("upscaler."): value for key, value in state.items() if key.startswith("upscaler.")}
+    model = H3LatentResizer3D(**_checkpoint_config(state))
+    model.load_state_dict(state, strict=True)
+    return model.to(device=device, dtype=dtype).eval().requires_grad_(False)
+
+
+@torch.inference_mode()
+def upscale_h3_latent(model: H3LatentResizer3D, latent: torch.Tensor, *, scale: float = 2.0) -> torch.Tensor:
+    if latent.ndim != 5 or latent.shape[1] != 24:
+        raise ValueError("H3 latent must have shape [B,24,T,H,W]")
+    if not 1.0 <= scale <= 4.0:
+        raise ValueError("scale must be between 1.0 and 4.0")
+    dtype = next(model.parameters()).dtype
+    device = next(model.parameters()).device
+    source = latent.to(device=device, dtype=dtype)
+    if scale == 1.0:
+        return source
+    mean = torch.tensor(LATENTS_MEAN, device=device, dtype=dtype).view(1, 24, 1, 1, 1)
+    std = torch.tensor(LATENTS_STD, device=device, dtype=dtype).view(1, 24, 1, 1, 1)
+    target_size = (source.shape[2], round(source.shape[3] * scale), round(source.shape[4] * scale))
+    result = model((source - mean) / std, scale=scale, target_size=target_size)
+    return result.mul_(std).add_(mean)
--- a/src/h3_blackwell_runtime/lora.py
+++ b/src/h3_blackwell_runtime/lora.py
@ -0,0 +1,95 @@
+"""Resident dynamic LoRA branches for the quantized H3 denoiser and refiner."""
+
+from collections.abc import Iterator
+
+import torch
+import torch.nn.functional as functional
+from safetensors import safe_open
+from torch import nn
+
+
+class LoraBranch(nn.Module):
+    def __init__(self, down: torch.Tensor, up: torch.Tensor, alpha: float):
+        super().__init__()
+        if down.ndim != 2 or up.ndim != 2 or down.shape[0] != up.shape[1]:
+            raise ValueError(f"Invalid LoRA shapes: down={tuple(down.shape)}, up={tuple(up.shape)}")
+        self.scale = float(alpha) / down.shape[0]
+        self.register_buffer("down", down.contiguous(), persistent=False)
+        self.register_buffer("up", up.contiguous(), persistent=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = x.to(self.down.dtype)
+        return functional.linear(functional.linear(x, self.down), self.up) * self.scale
+
+
+class DynamicLoraMixin:
+    """Small mixin for linears that can host resident request-selectable LoRAs."""
+
+    def _init_dynamic_lora(self) -> None:
+        self.lora_branches = nn.ModuleDict()
+        self.active_lora: str | None = None
+        self.lora_strength = 0.0
+
+    def add_lora(self, name: str, down: torch.Tensor, up: torch.Tensor, alpha: float) -> None:
+        if name in self.lora_branches:
+            raise ValueError(f"LoRA {name!r} is already attached")
+        if down.shape[1] != self.in_features or up.shape[0] != self.out_features:
+            raise ValueError(
+                f"LoRA {name!r} dimensions {tuple(down.shape)}, {tuple(up.shape)} do not match "
+                f"linear [{self.out_features}, {self.in_features}]"
+            )
+        self.lora_branches[name] = LoraBranch(down, up, alpha)
+
+    def set_lora(self, name: str | None, strength: float = 1.0) -> None:
+        if name is not None and name not in self.lora_branches:
+            raise ValueError(f"LoRA {name!r} is not attached")
+        self.active_lora = name
+        self.lora_strength = float(strength) if name is not None else 0.0
+
+    def _apply_lora(self, x: torch.Tensor, base: torch.Tensor) -> torch.Tensor:
+        if self.active_lora is None or self.lora_strength == 0.0:
+            return base
+        delta = self.lora_branches[self.active_lora](x)
+        return base + delta.to(base.dtype) * self.lora_strength
+
+
+def iter_lora_targets(model: nn.Module, refiner: nn.Module) -> Iterator[tuple[str, DynamicLoraMixin]]:
+    for index, block in enumerate(model.backbone.blocks):
+        yield f"blocks.{index}.attn.qkv_proj", block.attention.qkv_proj
+        yield f"blocks.{index}.attn.out_proj", block.attention.out_proj
+        yield f"blocks.{index}.mlp.fc1", block.mlp.fc1
+        yield f"blocks.{index}.mlp.fc2", block.mlp.fc2
+    for index, block in enumerate(refiner.blocks):
+        yield f"token_refiner.blocks.{index}.attn.qkv_proj", block.qkv
+        yield f"token_refiner.blocks.{index}.attn.out_proj", block.out
+        yield f"token_refiner.blocks.{index}.mlp.fc1", block.fc1
+        yield f"token_refiner.blocks.{index}.mlp.fc2", block.fc2
+
+
+def load_lora_adapter(model: nn.Module, refiner: nn.Module, name: str, path: str, device: str) -> int:
+    targets = list(iter_lora_targets(model, refiner))
+    expected = {
+        f"diffusion_model.{target}.{suffix}"
+        for target, _module in targets
+        for suffix in ("alpha", "lora_A.weight", "lora_B.weight")
+    }
+    with safe_open(path, framework="pt", device=device) as checkpoint:
+        actual = set(checkpoint.keys())
+        if actual != expected:
+            missing = sorted(expected - actual)[:8]
+            unexpected = sorted(actual - expected)[:8]
+            raise ValueError(f"LoRA key mismatch: missing={missing}, unexpected={unexpected}")
+        for target, module in targets:
+            prefix = f"diffusion_model.{target}"
+            module.add_lora(
+                name,
+                checkpoint.get_tensor(f"{prefix}.lora_A.weight"),
+                checkpoint.get_tensor(f"{prefix}.lora_B.weight"),
+                checkpoint.get_tensor(f"{prefix}.alpha").item(),
+            )
+    return len(targets)
+
+
+def set_active_lora(model: nn.Module, refiner: nn.Module, name: str | None, strength: float = 1.0) -> None:
+    for _target, module in iter_lora_targets(model, refiner):
+        module.set_lora(name, strength)
--- a/src/h3_blackwell_runtime/nvfp4.py
+++ b/src/h3_blackwell_runtime/nvfp4.py
@ -7,6 +7,8 @@ import torch
 import torch.nn.functional as functional
 from torch import nn

+from .lora import DynamicLoraMixin
+

@dataclass(frozen=True)
 class Nvfp4LinearTensors:
@ -28,7 +30,7 @@ def parse_quant_sidecar(sidecar: torch.Tensor) -> dict:
    return metadata


-class Nvfp4Linear(nn.Module):
+class Nvfp4Linear(DynamicLoraMixin, nn.Module):
    """Execute a packed Comfy NVFP4 linear with Comfy Kitchen's CUDA 13 kernel."""

    def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16):
@ -49,6 +51,7 @@ class Nvfp4Linear(nn.Module):
        self.register_buffer("weight_scale_2", tensors.weight_scale_2.to(torch.float32).contiguous(), persistent=False)
        self.register_buffer("bias", tensors.bias.contiguous() if tensors.bias is not None else None, persistent=False)
        self.register_buffer("pre_quant_scale", tensors.pre_quant_scale.contiguous() if tensors.pre_quant_scale is not None else None, persistent=False)
+        self._init_dynamic_lora()

    def _packed_weight(self):
        from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
@ -73,6 +76,7 @@ class Nvfp4Linear(nn.Module):
        from comfy_kitchen.tensor import QuantizedTensor

        original_shape = x.shape[:-1]
+        original_x = x
        flat_x = x.reshape(-1, self.in_features).contiguous()
        if self.pre_quant_scale is not None:
            flat_x = flat_x * self.pre_quant_scale.to(flat_x)
@ -81,12 +85,13 @@ class Nvfp4Linear(nn.Module):
        if self.full_precision_matrix_mult:
            weight = packed_weight.dequantize().to(flat_x)
            output = functional.linear(flat_x, weight, bias)
-            return output.reshape(*original_shape, self.out_features)
-        if x.dtype == torch.float32:
-            raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.")
-        packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout")
-        output = functional.linear(packed_x, packed_weight, bias)
-        return output[:flat_x.shape[0], :self.out_features].reshape(*original_shape, self.out_features)
+        else:
+            if x.dtype == torch.float32:
+                raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.")
+            packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout")
+            output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features]
+        base = output.reshape(*original_shape, self.out_features)
+        return self._apply_lora(original_x, base)


 def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear:
--- a/src/h3_blackwell_runtime/nvfp4_quant.py
+++ b/src/h3_blackwell_runtime/nvfp4_quant.py
@ -0,0 +1,216 @@
+"""Experimental NVFP4 activation quantization helpers for profiling."""
+
+from __future__ import annotations
+
+import os
+import time
+from functools import lru_cache
+from pathlib import Path
+
+import torch
+
+
+_VORTEX_SCALE_WORKSPACES: dict[tuple[int, int], tuple[torch.Tensor, torch.Tensor]] = {}
+
+
+def _sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def _record_timing(timings: dict[str, list[float]] | None, name: str, fn):
+    if timings is None:
+        return fn()
+    _sync()
+    started = time.perf_counter()
+    value = fn()
+    _sync()
+    timings.setdefault(name, []).append(time.perf_counter() - started)
+    return value
+
+
+@lru_cache(maxsize=1)
+def _vortex_scale_extension():
+    from torch.utils.cpp_extension import load
+
+    root = Path(__file__).resolve().parent
+    return load(
+        name="h3_vortex_nvfp4_scale",
+        sources=[str(root / "csrc" / "nvfp4_scale.cpp"), str(root / "csrc" / "nvfp4_scale.cu")],
+        extra_cflags=["-O3"],
+        extra_cuda_cflags=["-O3"],
+        verbose=os.getenv("H3_NVFP4_SCALE_VERBOSE", "").lower() in {"1", "true", "yes", "on"},
+    )
+
+
+def _env_int(name: str, default: int) -> int:
+    value = os.getenv(name)
+    if value is None or value == "":
+        return default
+    return int(value)
+
+
+def _vortex_scale_geometry(numel: int) -> tuple[int, int]:
+    threads = _env_int("H3_NVFP4_SCALE_THREADS", 128)
+    if threads not in {128, 256, 512}:
+        raise ValueError(f"H3_NVFP4_SCALE_THREADS must be 128, 256, or 512, got {threads}")
+    default_blocks = min((numel + threads - 1) // threads, 256)
+    blocks = _env_int("H3_NVFP4_SCALE_BLOCKS", default_blocks)
+    if blocks <= 0:
+        raise ValueError(f"H3_NVFP4_SCALE_BLOCKS must be positive, got {blocks}")
+    return blocks, threads
+
+
+def _vortex_scale_workspace(tensor: torch.Tensor, blocks: int) -> tuple[torch.Tensor, torch.Tensor]:
+    key = (tensor.device.index or 0, blocks)
+    workspace = _VORTEX_SCALE_WORKSPACES.get(key)
+    if workspace is None or workspace[0].device != tensor.device:
+        workspace = (
+            torch.empty((blocks,), device=tensor.device, dtype=torch.float32),
+            torch.empty((), device=tensor.device, dtype=torch.float32),
+        )
+        _VORTEX_SCALE_WORKSPACES[key] = workspace
+    return workspace
+
+
+def nvfp4_activation_scale(tensor: torch.Tensor, *, timings: dict[str, list[float]] | None = None) -> torch.Tensor:
+    """Compute Comfy Kitchen's current per-tensor NVFP4 activation scale."""
+    from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX
+
+    divisor = float(F8_E4M3_MAX * F4_E2M1_MAX)
+    backend = os.getenv("H3_NVFP4_SCALE_BACKEND", "torch").lower()
+    if backend == "vortex":
+        if tensor.dim() != 2:
+            raise ValueError(f"Vortex NVFP4 scale backend requires 2D input, got {tensor.dim()}D")
+        if not tensor.is_cuda:
+            raise ValueError("Vortex NVFP4 scale backend requires CUDA input")
+        if not tensor.is_contiguous():
+            raise ValueError("Vortex NVFP4 scale backend requires contiguous input")
+        try:
+            extension = _vortex_scale_extension()
+            version = os.getenv("H3_NVFP4_SCALE_VERSION", "1").lower()
+            if version in {"1", "v1"} and tensor.dtype == torch.bfloat16 and tensor.data_ptr() % 16 == 0:
+                blocks, threads = _vortex_scale_geometry(tensor.numel())
+                partials, output = _vortex_scale_workspace(tensor, blocks)
+                scale = _record_timing(
+                    timings,
+                    "vortex_absmax_scale",
+                    lambda: extension.nvfp4_activation_scale_into(tensor, divisor, partials, output, blocks, threads),
+                )
+            else:
+                scale = _record_timing(timings, "vortex_absmax_scale", lambda: extension.nvfp4_activation_scale(tensor, divisor))
+            return _record_timing(timings, "scale_compat_cast", lambda: scale.to(tensor.dtype) if tensor.dtype != torch.float32 else scale)
+        except Exception:
+            if os.getenv("H3_NVFP4_SCALE_STRICT", "").lower() in {"1", "true", "yes", "on"}:
+                raise
+            backend = "torch"
+
+    if backend != "torch":
+        raise ValueError(f"Unsupported H3_NVFP4_SCALE_BACKEND={backend!r}")
+    amax = _record_timing(timings, "scale_absmax", lambda: torch.amax(tensor.abs()))
+    return _record_timing(timings, "scale_finalize", lambda: amax / divisor)
+
+
+def vortex_quantize_nvfp4(
+    tensor: torch.Tensor,
+    *,
+    scale: torch.Tensor | float | None = None,
+    timings: dict[str, list[float]] | None = None,
+):
+    """Create a TensorCoreNVFP4 QuantizedTensor through an explicit Vortex seam.
+
+    This prototype still uses Comfy Kitchen's low-level ``quantize_nvfp4`` CUDA op
+    for the pack/block-scale step, but bypasses ``QuantizedTensor.from_float`` and
+    lets callers provide a precomputed/global scale. It is intentionally isolated so a
+    native Vortex quantizer can replace this implementation without touching the
+    linear call sites or benchmarks.
+    """
+    if tensor.dim() != 2:
+        raise ValueError(f"NVFP4 activation quantization requires a 2D tensor, got {tensor.dim()}D")
+    if not tensor.is_contiguous():
+        raise ValueError("vortex_quantize_nvfp4 requires contiguous input; fix the caller rather than hiding a copy here")
+
+    import comfy_kitchen as ck
+    from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
+
+    orig_dtype = tensor.dtype
+    orig_shape = tuple(tensor.shape)
+    if scale is None:
+        scale = nvfp4_activation_scale(tensor, timings=timings)
+    scale = _record_timing(
+        timings,
+        "scale_to_device",
+        lambda: torch.as_tensor(scale, device=tensor.device, dtype=torch.float32),
+    )
+
+    qdata, block_scale = _record_timing(
+        timings,
+        "ck_quantize_nvfp4",
+        lambda: ck.quantize_nvfp4(
+            tensor,
+            scale,
+            pad_16x=TensorCoreNVFP4Layout.get_padded_shape(orig_shape) != orig_shape,
+        ),
+    )
+    return _record_timing(
+        timings,
+        "params_wrap",
+        lambda: QuantizedTensor(
+            qdata,
+            "TensorCoreNVFP4Layout",
+            TensorCoreNVFP4Layout.Params(
+                scale=scale,
+                orig_dtype=orig_dtype,
+                orig_shape=orig_shape,
+                block_scale=block_scale,
+            ),
+        ),
+    )
+
+
+def vortex_native_quantize_nvfp4(
+    tensor: torch.Tensor,
+    *,
+    scale: torch.Tensor | float | None = None,
+    timings: dict[str, list[float]] | None = None,
+):
+    """Prototype native BF16 activation packer for TensorCoreNVFP4Layout."""
+    if tensor.dim() != 2:
+        raise ValueError(f"NVFP4 activation quantization requires a 2D tensor, got {tensor.dim()}D")
+    if tensor.dtype != torch.bfloat16:
+        raise ValueError("vortex_native_quantize_nvfp4 currently expects BF16 input")
+    if not tensor.is_cuda or not tensor.is_contiguous():
+        raise ValueError("vortex_native_quantize_nvfp4 requires contiguous CUDA input")
+
+    from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
+
+    orig_shape = tuple(tensor.shape)
+    orig_dtype = tensor.dtype
+    if scale is None:
+        scale = nvfp4_activation_scale(tensor, timings=timings)
+    scale = _record_timing(timings, "scale_to_device", lambda: torch.as_tensor(scale, device=tensor.device, dtype=torch.float32))
+    extension = _vortex_scale_extension()
+    qdata, block_scale = _record_timing(
+        timings,
+        "vortex_quantize_nvfp4",
+        lambda: extension.quantize_nvfp4_bf16(
+            tensor,
+            scale,
+            TensorCoreNVFP4Layout.get_padded_shape(orig_shape) != orig_shape,
+            _env_int("H3_NVFP4_PACK_THREADS", 256),
+        ),
+    )
+    return _record_timing(
+        timings,
+        "params_wrap",
+        lambda: QuantizedTensor(
+            qdata,
+            "TensorCoreNVFP4Layout",
+            TensorCoreNVFP4Layout.Params(
+                scale=scale,
+                orig_dtype=orig_dtype,
+                orig_shape=orig_shape,
+                block_scale=block_scale,
+            ),
+        ),
+    )
--- a/src/h3_blackwell_runtime/packing.py
+++ b/src/h3_blackwell_runtime/packing.py
@ -34,6 +34,13 @@ def _axis(dim: int, area: float) -> torch.Tensor:
    return (torch.arange(count, dtype=torch.float64) * ratio / count + (1 - ratio) / 2) * 32


+def _frame_positions(height: int, width: int) -> torch.Tensor:
+    """(t ignored) area-normalized (h, w) grid of one latent frame's 2x2-patch rows."""
+    area = math.sqrt(height * width)
+    ys, xs = torch.meshgrid(_axis(height, area), _axis(width, area), indexing="ij")
+    return torch.stack((ys.flatten(), xs.flatten()), dim=-1)  # [frame_rows, 2]
+
+
 def _video_positions(frames: int, height: int, width: int, offset: float) -> torch.Tensor:
    area = math.sqrt(height * width)
    ys, xs = torch.meshgrid(_axis(height, area), _axis(width, area), indexing="ij")
@ -45,6 +52,18 @@ def _video_positions(frames: int, height: int, width: int, offset: float) -> tor
    return result.reshape(-1, 3)


+def _cond_positions(frames: int, cond_t: float, height: int, width: int) -> torch.Tensor:
+    spatial = _frame_positions(height, width)
+    g = torch.empty(spatial.shape[0], 3, dtype=torch.float64)
+    g[:, 0] = cond_t
+    g[:, 1:] = spatial
+    return g
+
+
+def _video_t_spans(n: int) -> list[float]:
+    return [FRAME_RESCALE * FRAME_PER_TOKEN[k % 5] for k in range(n)]
+
+
 def _audio_positions(steps: int, offset: float, width: int, height: int) -> torch.Tensor:
    area = math.sqrt(height * width)
    x_axis = _axis(width, area)
@ -65,6 +84,8 @@ class H3PromptPacker:
        self.text_weight = checkpoint.tensor("condition_proj.weight", dtype=torch.bfloat16)
        self.text_bias = checkpoint.tensor("condition_proj.bias", dtype=torch.bfloat16)

+    VISUAL_COND_TIMESTEP = 0.999
+
    def __call__(
        self,
        text: torch.Tensor,
@ -72,35 +93,125 @@ class H3PromptPacker:
        audio: torch.Tensor,
        sigma: float | torch.Tensor,
        model_timesteps: torch.Tensor | None = None,
-    ) -> tuple[torch.Tensor, torch.Tensor, list[tuple[int, int, int]], tuple[int, int, int], tuple[int, int, int]]:
+        *,
+        text_token_tags: torch.Tensor | None = None,
+        cond_latents: list[torch.Tensor] | None = None,
+        cond_frame_indices: list[int] | None = None,
+        frame_count: int | None = None,
+        seed: int = 0,
+    ) -> tuple[torch.Tensor, torch.Tensor, list[tuple[int, int, int]], torch.Tensor, tuple[int, int, int], tuple[int, int, int]]:
+        """Build ``[text | cond | audio | video]`` packed rows for (fl2va) H3.
+
+        ``text`` is the refined text span (width 5376 when already refined, 5120
+        for raw Qwen states); ``text_token_tags`` is the per-token DiT modality tag
+        (1=text, 0=video over vision pads). ``cond_latents`` are normalized keyframe
+        latents ``[1,24,1,H/16,W/16]`` and ``cond_frame_indices`` preserves each
+        keyframe's resolved first/last pixel index. They are spliced right after
+        the text as non-denoised cond rows with their own near-1 timestep. Returns
+        ``(hidden, times, segments, positions, video_seg, audio_seg)`` where
+        ``segments`` rows are ``t_row*3 + modality_tag``.
+        """
        if text.shape[-1] == 5120:
            text_rows = functional.linear(text[0].to(self.text_weight.dtype), self.text_weight, self.text_bias).to(torch.bfloat16)
        elif text.shape[-1] == 5376:
            text_rows = text[0].to(torch.bfloat16)
        else:
            raise ValueError("H3 text states must be Qwen 5120-wide or refined 5376-wide.")
+        text_length = text_rows.shape[0]
+        latent_t, latent_h, latent_w = video.shape[2], video.shape[-2], video.shape[-1]
+        frame_rows = (latent_h // 2) * (latent_w // 2)
+
        video_rows = functional.linear(patchify_video(video.to(torch.bfloat16)).float(), self.video_weight, self.video_bias).to(torch.bfloat16)
        audio_rows = functional.linear(pack_audio(audio.to(torch.bfloat16)).float(), self.audio_weight, self.audio_bias).to(torch.bfloat16)
-        text_length, audio_length = text_rows.shape[0], audio_rows.shape[0]
-        hidden = torch.cat((text_rows, audio_rows, video_rows))
+        cond_rows = None
+        cond_lengths = []
+        if cond_latents:
+            if cond_frame_indices is None or len(cond_frame_indices) != len(cond_latents):
+                raise ValueError("cond_frame_indices must match cond_latents")
+            cond_patches = []
+            # every cond video restarts the same CPU RNG stream (Comfy _cond_video_rows)
+            for idx, z in enumerate(cond_latents):
+                r = patchify_video(z.to(torch.float32))
+                if self.VISUAL_COND_TIMESTEP < 1.0:
+                    gen = torch.Generator("cpu").manual_seed(int(seed))
+                    noise = torch.randn(r.shape, generator=gen, dtype=torch.float32)
+                    r = self.VISUAL_COND_TIMESTEP * r + (1.0 - self.VISUAL_COND_TIMESTEP) * noise.to(r.device)
+                cond_patches.append(r)
+                cond_lengths.append(r.shape[0])
+            cond_rows = functional.linear(torch.cat(cond_patches, dim=0), self.video_weight.to(torch.float32), self.video_bias.to(torch.float32)).to(torch.bfloat16)
+
        if model_timesteps is None:
-            video_sigma = torch.as_tensor(sigma, device=hidden.device, dtype=torch.float32).clamp(min=1e-6)
+            video_sigma = torch.as_tensor(sigma, device=text_rows.device, dtype=torch.float32).clamp(min=1e-6)
            base = video_sigma / (12.0 + video_sigma * (1.0 - 12.0))
            audio_sigma = 3.0 * base / (1.0 + (3.0 - 1.0) * base)
            video_time, audio_time = (1.0 - video_sigma).item(), (1.0 - audio_sigma).item()
-            unique_times = sorted({video_time, audio_time})
        else:
-            times_override = model_timesteps.to(device=hidden.device, dtype=torch.float32).flatten()
-            if times_override.numel() not in (1, 2):
-                raise ValueError("Prompt-only H3 expects one or two model timesteps.")
-            unique_times = times_override.tolist()
-            video_time, audio_time = unique_times[0], unique_times[-1]
-        row = {value: index for index, value in enumerate(unique_times)}
-        video_row, audio_row = row[video_time] * 3, row[audio_time] * 3
-        times = torch.tensor(unique_times, device=hidden.device, dtype=torch.float32)
-        positions = torch.cat((torch.stack((torch.arange(text_length, dtype=torch.float64), torch.zeros(text_length), torch.zeros(text_length)), dim=-1), _audio_positions(audio.shape[-1], float(text_length), video.shape[-1], video.shape[-2]), _video_positions(video.shape[2], video.shape[-2], video.shape[-1], float(text_length))))
-        block_video_segment = (text_length + audio_length, hidden.shape[0], video_row)
-        block_audio_segment = (text_length, text_length + audio_length, audio_row + 2)
-        final_video_segment = (text_length + audio_length, hidden.shape[0], row[video_time])
-        final_audio_segment = (text_length, text_length + audio_length, row[audio_time])
-        return hidden, times, [(0, text_length, video_row + 1), block_audio_segment, block_video_segment], positions, final_video_segment, final_audio_segment
+            times_override = model_timesteps.to(device=text_rows.device, dtype=torch.float32).flatten()
+            unique_override = sorted(times_override.tolist())
+            video_time, audio_time = unique_override[0], unique_override[-1]
+
+        has_vis_cond = cond_rows is not None
+        cond_time = max(video_time, self.VISUAL_COND_TIMESTEP)
+        unique_times = sorted({video_time, audio_time} | ({cond_time} if has_vis_cond else set()))
+        t_row = {value: index for index, value in enumerate(unique_times)}
+        times = torch.tensor(unique_times, device=text_rows.device, dtype=torch.float32)
+
+        # Assemble hidden in segment order: text | cond | audio | video.
+        parts = [text_rows]
+        offsets = [0]
+        for block in ((cond_rows, "cond"), (audio_rows, "audio"), (video_rows, "video")):
+            if block[0] is None:
+                continue
+            parts.append(block[0])
+            offsets.append(offsets[-1] + block[0].shape[0])
+        hidden = torch.cat(parts, dim=0)
+
+        audio_length = audio_rows.shape[0]
+        cond_length = cond_rows.shape[0] if cond_rows is not None else 0
+
+        # Positions: text rows, cond (first/last t anchors), audio, video.
+        text_positions = torch.stack((torch.arange(text_length, dtype=torch.float64), torch.zeros(text_length), torch.zeros(text_length)), dim=-1)
+        position_blocks = [text_positions]
+        if cond_rows is not None and cond_latents:
+            spans = _video_t_spans(latent_t)
+            cond_t_values = []
+            for pixel_index in cond_frame_indices:
+                if pixel_index == 0:
+                    cond_t_values.append(float(text_length))
+                elif frame_count is not None and pixel_index == frame_count - 1:
+                    cond_t_values.append(float(text_length) + sum(spans) - FRAME_RESCALE)
+                else:
+                    raise ValueError("only first/last keyframe anchors are supported")
+            position_blocks.append(torch.cat([_cond_positions(frame_rows, cond_t, latent_h, latent_w) for cond_t in cond_t_values], dim=0))
+        position_blocks.append(_audio_positions(audio.shape[-1], float(text_length), latent_w, latent_h))
+        position_blocks.append(_video_positions(latent_t, latent_h, latent_w, float(text_length)))
+        positions = torch.cat(position_blocks, dim=0)
+
+        # mod_segments: (start, stop, t_row*3 + tag).
+        segments: list[tuple[int, int, int]] = []
+        if text_token_tags is not None:
+            tags = text_token_tags.view(-1).tolist()
+            row_base = t_row[video_time] * 3
+            run_start = 0
+            for i in range(1, text_length + 1):
+                if i == text_length or tags[i] != tags[run_start]:
+                    segments.append((run_start, i, row_base + int(tags[run_start])))
+                    run_start = i
+        else:
+            segments.append((0, text_length, t_row[video_time] * 3 + 1))
+
+        cursor_start = text_length
+        if cond_rows is not None:
+            for length in cond_lengths:
+                segments.append((cursor_start, cursor_start + length, t_row[cond_time] * 3 + 0))
+                cursor_start += length
+        segments.append((cursor_start, cursor_start + audio_length, t_row[audio_time] * 3 + 2))
+        cursor_start += audio_length
+        video_start = cursor_start
+        video_stop = video_start + video_rows.shape[0]
+        segments.append((video_start, video_stop, t_row[video_time] * 3 + 0))
+
+        video_segment = (video_start, video_stop, t_row[video_time])
+        audio_start = text_length + cond_length
+        audio_segment = (audio_start, audio_start + audio_length, t_row[audio_time])
+        return hidden, times, segments, positions, video_segment, audio_segment
--- a/src/h3_blackwell_runtime/qwen3vl_text.py
+++ b/src/h3_blackwell_runtime/qwen3vl_text.py
@ -42,6 +42,24 @@ class _RMSNorm(nn.Module):
        return F.rms_norm(x, self.weight.shape, weight=self.weight.to(x), eps=self.eps)


+def _apply_rope(query: torch.Tensor, key: torch.Tensor, freqs) -> tuple[torch.Tensor, torch.Tensor]:
+    """Direct PyTorch port of Comfy's `apply_rope` fed `freqs=(cos, sin, neg_sin)`.
+
+    ``query``/``key`` are ``[batch, heads, seq, head_dim]``; ``freqs`` broadcast over
+    that shape. This is the exact split-half kernel the reference uses for both the
+    plain text rope and the Qwen3-VL interleaved mrope.
+    """
+    cosine, sine, negative_sine = freqs
+    split = query.shape[-1] // 2
+    q = query * cosine
+    q[..., :split].addcmul_(query[..., split:], negative_sine)
+    q[..., split:].addcmul_(query[..., :split], sine)
+    k = key * cosine
+    k[..., :split].addcmul_(key[..., split:], negative_sine)
+    k[..., split:].addcmul_(key[..., :split], sine)
+    return q.to(query.dtype), k.to(key.dtype)
+
+
 def _rope(query: torch.Tensor, key: torch.Tensor, theta: float) -> tuple[torch.Tensor, torch.Tensor]:
    """Direct PyTorch port of Comfy's text-only `precompute_freqs_cis` / `apply_rope`."""
    sequence, head_dim = query.shape[-2:]
@ -54,16 +72,7 @@ def _rope(query: torch.Tensor, key: torch.Tensor, theta: float) -> tuple[torch.T
    sine = embedding.sin().unsqueeze(1)
    negative_sine = -sine[..., sine.shape[-1] // 2 :]
    sine = sine[..., : sine.shape[-1] // 2]
-
-    query_output = query * cosine
-    split = query_output.shape[-1] // 2
-    query_output[..., :split].addcmul_(query[..., split:], negative_sine)
-    query_output[..., split:].addcmul_(query[..., :split], sine)
-    key_output = key * cosine
-    split = key_output.shape[-1] // 2
-    key_output[..., :split].addcmul_(key[..., split:], negative_sine)
-    key_output[..., split:].addcmul_(key[..., :split], sine)
-    return query_output.to(query.dtype), key_output.to(key.dtype)
+    return _apply_rope(query, key, (cosine, sine, negative_sine))


 def _qwen_attention(query: torch.Tensor, key: torch.Tensor, value: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
@ -105,7 +114,7 @@ class _Qwen3VLBlock(nn.Module):
        self.up_proj = checkpoint.nvfp4_linear(f"{prefix}.mlp.up_proj", output_dtype=dtype)
        self.down_proj = checkpoint.nvfp4_linear(f"{prefix}.mlp.down_proj", output_dtype=dtype)

-    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+    def forward(self, hidden_states: torch.Tensor, freqs_cis=None) -> torch.Tensor:
        residual = hidden_states
        x = self.input_layernorm(hidden_states)
        batch, sequence, _ = x.shape
@ -114,7 +123,10 @@ class _Qwen3VLBlock(nn.Module):
        value = self.v_proj(x).view(batch, sequence, self.config.num_key_value_heads, self.config.head_dim).transpose(1, 2)
        query = self.q_norm(query)
        key = self.k_norm(key)
-        query, key = _rope(query, key, self.config.rope_theta)
+        if freqs_cis is None:
+            query, key = _rope(query, key, self.config.rope_theta)
+        else:
+            query, key = _apply_rope(query, key, freqs_cis)
        # Comfy selects its small-input SDPA path for Qwen, with an explicit causal mask.
        causal_mask = torch.full(
            (sequence, sequence),
@ -178,6 +190,44 @@ class Qwen3VL32BTextEncoder(nn.Module):
                + ", ".join(missing)
            )

+    def _embed_rows(self, input_ids: torch.Tensor) -> torch.Tensor:
+        """Scaled token embeds ``[batch, tokens, 5120]`` in fp32 (pre-decoder)."""
+        token_rows = F.embedding(input_ids.to(self.embed_tokens.device), self.embed_tokens).to(torch.float32)
+        token_scales = F.embedding(input_ids.to(self.embed_scale.device), self.embed_scale)
+        return (token_rows * token_scales).to(torch.bfloat16).to(torch.float32)
+
+    def _run_layers(
+        self,
+        hidden_states: torch.Tensor,
+        *,
+        position_ids: torch.Tensor | None = None,
+        visual_pos_masks: torch.Tensor | None = None,
+        deepstack_embeds: list[torch.Tensor] | None = None,
+    ) -> torch.Tensor:
+        """Run all 50 decoder blocks over an fp32 ``[batch, tokens, 5120]`` state.
+
+        With ``position_ids`` (``[3, seq]``) the Qwen3-VL interleaved mrope is used
+        instead of the plain 1D text rope; with ``visual_pos_masks`` +
+        ``deepstack_embeds`` the three DeepStack features are added at the visual
+        positions of the first three decoder layers (Comfy ``Llama2_.forward``).
+        """
+        hidden_states = hidden_states.to(self.dtype)
+        freqs_cis = None
+        if position_ids is not None:
+            from .qwen3vl_vision import mrope_freqs_cis
+
+            freqs_cis = mrope_freqs_cis(position_ids.to(hidden_states.device))
+        for layer_index, layer in enumerate(self.layers):
+            hidden_states = layer(hidden_states, freqs_cis)
+            if (
+                deepstack_embeds is not None
+                and visual_pos_masks is not None
+                and layer_index < len(deepstack_embeds)
+            ):
+                mask = visual_pos_masks.to(hidden_states.device)
+                hidden_states[mask] = hidden_states[mask] + deepstack_embeds[layer_index].to(hidden_states)
+        return hidden_states
+
    @torch.inference_mode()
    def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
        """Return unnormalized `[batch, tokens, 5120]` output after decoder layer 50."""
@ -185,12 +235,7 @@ class Qwen3VL32BTextEncoder(nn.Module):
            raise ValueError(f"input_ids must have shape [batch, tokens], got {tuple(input_ids.shape)}")
        if input_ids.numel() == 0:
            raise ValueError("input_ids must contain at least one token")
-        token_rows = F.embedding(input_ids.to(self.embed_tokens.device), self.embed_tokens).to(torch.float32)
-        token_scales = F.embedding(input_ids.to(self.embed_scale.device), self.embed_scale)
-        hidden_states = (token_rows * token_scales).to(self.dtype)
-        for layer in self.layers:
-            hidden_states = layer(hidden_states)
-        return hidden_states
+        return self._run_layers(self._embed_rows(input_ids))


 class Qwen3VLPromptConditioner:
--- a/src/h3_blackwell_runtime/qwen3vl_vision.py
+++ b/src/h3_blackwell_runtime/qwen3vl_vision.py
@ -0,0 +1,762 @@
+"""Qwen3-VL vision conditioning for the direct H3 runtime (first/last keyframes).
+
+Standalone port of the Comfy reference's Qwen3.5/Qwen3-VL visual stack and the
+MiniMax H3 presentation mechanism, built only on ``torch``:
+
+- ``Qwen35VisionModel`` (visual.pos_embed, visual.patch_embed, 27 visual.blocks
+  with 2D rotary attention, visual.merger) plus the three
+  ``Qwen3VLDeepstackMerger`` (visual.deepstack_merger_list).
+- ``process_qwen2vl_images`` image preprocessing (Qwen grid/resize/normalize).
+- ``qwen2vl_mrope_position_ids`` (H3 mrope ids from ``embeds_info``),
+  ``precompute_mrope_freqs_cis`` (interleaved Qwen3-VL text mrope), and
+  ``token_tags_from_embeds_info`` (DiT per-token modality tags).
+- Comfy ``Llama2_.forward`` DeepStack plumbing: per-decoder-layer additions at
+  visual positions for the first three decoder layers.
+
+The visual tower is loaded from the same ``qwen3vl_32b_minimax_h3_nvfp4_awq``
+safetensors the text encoder uses; its keys are all plain (bf16) tensors under
+the ``visual.*`` prefix (verified at 1.1 GB, no quantized sub-tensors).
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+from torch.nn.attention import SDPBackend, sdpa_kernel
+from safetensors import safe_open
+from torch import nn
+from torch.nn import functional as F
+
+
+# H3 presentation sentinels (see upstream_qwen3vl.py / upstream_text.py).
+VISION_START = 151652
+VISION_END = 151653
+IMAGE_EMBED_TOKEN = 151655
+
+# Qwen3-VL-32B visual geometry (QWEN3VL_VISION in upstream_qwen3vl.py).
+VISION_HIDDEN = 1152
+VISION_INTERMEDIATE = 4304
+VISION_DEPTH = 27
+VISION_HEADS = 16
+VISION_HEAD_DIM = VISION_HIDDEN // VISION_HEADS  # 72
+VISION_PATCH = 16
+VISION_TEMPORAL = 2
+VISION_MERGE = 2
+VISION_POSITION_EMBEDS = 2304
+DEEPSTACK_VISUAL_INDEXES = (8, 16, 24)
+
+# Qwen3-VL text mrope geometry (Qwen3VL_32BConfig in llama.py).
+TEXT_ROPE_DIMS = (24, 20, 20)
+TEXT_ROPE_THETA = 5_000_000.0
+TEXT_HEAD_DIM = 128
+
+# Qwen image preprocessing policy (process_qwen2vl_images, H3 mean/std 0.5).
+QWEN_IMAGE_MEAN = (0.5, 0.5, 0.5)
+QWEN_IMAGE_STD = (0.5, 0.5, 0.5)
+QWEN_MIN_PIXELS = 3136
+QWEN_MAX_PIXELS = 12845056
+
+
+def process_image(
+    image: torch.Tensor,
+    *,
+    min_pixels: int = QWEN_MIN_PIXELS,
+    max_pixels: int = QWEN_MAX_PIXELS,
+    patch_size: int = VISION_PATCH,
+    temporal_patch_size: int = VISION_TEMPORAL,
+    merge_size: int = VISION_MERGE,
+    image_mean: tuple[float, ...] = QWEN_IMAGE_MEAN,
+    image_std: tuple[float, ...] = QWEN_IMAGE_STD,
+) -> tuple[torch.Tensor, torch.Tensor]:
+    """Qwen image preprocessing (process_qwen2vl_images): a single
+    ``[1, H, W, 3]`` float image (``[0, 1]``) -> (flatten_patches
+    ``[grid_h*grid_w, C*tp*ps*ps]``, image_grid_thw ``[1, grid_h, grid_w]``).
+    """
+    if image.ndim != 4 or image.shape[0] != 1:
+        raise ValueError("process_image expects a single [1, H, W, 3] tensor")
+    batch, height, width, _ = image.shape
+    device = image.device
+
+    images = image.permute(0, 3, 1, 2)
+    img = images[0]
+
+    factor = patch_size * merge_size
+    h_bar = round(height / factor) * factor
+    w_bar = round(width / factor) * factor
+
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    img_resized = F.interpolate(img.unsqueeze(0), size=(h_bar, w_bar), mode="bilinear", align_corners=False).squeeze(0)
+
+    normalized = img_resized.clone()
+    for c in range(3):
+        normalized[c] = (img_resized[c] - image_mean[c]) / image_std[c]
+
+    grid_h = h_bar // patch_size
+    grid_w = w_bar // patch_size
+    grid_thw = torch.tensor([[1, grid_h, grid_w]], device=device, dtype=torch.long)
+
+    pixel_values = normalized
+    channel = pixel_values.shape[0]
+    grid_t = 1
+    pixel_values = pixel_values.unsqueeze(0).repeat(2, 1, 1, 1)
+
+    patches = pixel_values.reshape(
+        grid_t,
+        temporal_patch_size,
+        channel,
+        grid_h // merge_size,
+        merge_size,
+        patch_size,
+        grid_w // merge_size,
+        merge_size,
+        patch_size,
+    )
+    patches = patches.permute(0, 3, 6, 4, 7, 2, 1, 5, 8)
+    flatten = patches.reshape(grid_t * grid_h * grid_w, channel * temporal_patch_size * patch_size * patch_size)
+    return flatten, grid_thw
+
+
+def mrope_position_ids(embeds_info: list[dict], seq_len: int, device) -> torch.Tensor | None:
+    """(T, H, W) mrope ids for a sequence with spliced visual blocks (reference
+    ``qwen2vl_mrope_position_ids``). ``embeds_info`` entries carry
+    ``index``/``size`` spans and an ``extra`` dict with a ``grid`` tensor
+    ``[1, grid_h, grid_w]``. Returns ``[3, seq_len]`` or ``None``."""
+    position_ids = None
+    offset = 0
+    for e in embeds_info:
+        if e.get("type") != "image":
+            continue
+        extra = e.get("extra", None)
+        grid = extra["grid"] if isinstance(extra, dict) else extra
+        start = e.get("index")
+        if position_ids is None:
+            position_ids = torch.zeros((3, seq_len), device=device)
+            position_ids[:, :start] = torch.arange(0, start, device=device)
+        end = e.get("size") + start
+        len_max = int(grid.max()) // 2
+        start_next = len_max + start
+        position_ids[:, end:] = torch.arange(start_next + offset, start_next + (seq_len - end) + offset, device=device)
+        position_ids[0, start:end] = start + offset
+        max_d = int(grid[0][1]) // 2
+        position_ids[1, start:end] = (
+            torch.arange(start + offset, start + max_d + offset, device=device)
+            .unsqueeze(1)
+            .repeat(1, math.ceil((end - start) / max_d))
+            .flatten(0)[: end - start]
+        )
+        max_d = int(grid[0][2]) // 2
+        position_ids[2, start:end] = (
+            torch.arange(start + offset, start + max_d + offset, device=device)
+            .unsqueeze(0)
+            .repeat(math.ceil((end - start) / max_d), 1)
+            .flatten(0)[: end - start]
+        )
+        offset += len_max - (end - start)
+    return position_ids
+
+
+def token_tags(seq_len: int, embeds_info: list[dict], device) -> torch.Tensor:
+    """DiT per-token AdaLN tags: 0 (video) inside a visual block including the
+    flanking sentinels, 1 (text) elsewhere (reference
+    ``token_tags_from_embeds_info``)."""
+    tags = torch.ones(seq_len, dtype=torch.long, device=device)
+    for e in embeds_info:
+        if e.get("type") == "image":
+            start = max(0, e["index"] - 1)
+            stop = e["index"] + e["size"] + 1
+            tags[start:stop] = 0
+    return tags
+
+
+def mrope_freqs_cis(position_ids: torch.Tensor, *, theta: float = TEXT_ROPE_THETA, head_dim: int = TEXT_HEAD_DIM, rope_dims: tuple[int, ...] = TEXT_ROPE_DIMS) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+    """Interleaved Qwen3-VL text mrope (rope_dims=(24,20,20)).
+
+    ``position_ids`` is ``[3, seq]``; returns ``(cos, sin, neg_sin)`` matching
+    the reference ``precompute_freqs_cis`` + ``apply_rope`` convention (shape
+    ``[1, seq, head_dim]``; the ``neg_sin`` entry is ``-sin[..., half:]``).
+    """
+    if position_ids.shape[0] < 1:
+        position_ids = position_ids.unsqueeze(0)
+    inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=position_ids.device, dtype=torch.float32) / head_dim))
+    inv_freq_expanded = inv_freq[None, :, None].expand(position_ids.shape[0], -1, 1)
+    position_ids_expanded = position_ids[:, None, :].to(torch.float32)
+    freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)
+    freqs_inter = freqs[0].clone()
+    for axis_idx, offset in ((1, 1), (2, 2)):
+        length = rope_dims[axis_idx] * 3
+        idx = slice(offset, length, 3)
+        freqs_inter[..., idx] = freqs[axis_idx, ..., idx]
+    emb = torch.cat((freqs_inter, freqs_inter), dim=-1)
+    cos = emb.cos().unsqueeze(0)
+    sin = emb.sin().unsqueeze(0)
+    sine = sin[..., : sin.shape[-1] // 2]
+    negative_sine = -sin[..., sin.shape[-1] // 2 :]
+    return cos, sine, negative_sine
+
+
+class _VisionPatchEmbed(nn.Module):
+    def __init__(self, weight: torch.Tensor, bias: torch.Tensor):
+        super().__init__()
+        self.register_buffer("weight", weight, persistent=False)
+        self.register_buffer("bias", bias, persistent=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        target = self.weight.dtype
+        x = x.view(-1, 3, VISION_TEMPORAL, VISION_PATCH, VISION_PATCH)
+        s = (VISION_TEMPORAL, VISION_PATCH, VISION_PATCH)
+        x = x.to(target)
+        if x.is_cuda and target in (torch.float16, torch.bfloat16):
+            # Match Comfy's NVIDIA Conv3d workaround dispatch exactly.
+            output = torch.cudnn_convolution(
+                x,
+                self.weight,
+                (0, 0, 0),
+                s,
+                (1, 1, 1),
+                1,
+                benchmark=False,
+                deterministic=False,
+                allow_tf32=True,
+            )
+            output += self.bias.view(1, -1, 1, 1, 1)
+        else:
+            output = F.conv3d(x, self.weight, self.bias, stride=s)
+        return output.view(-1, self.weight.shape[0])
+
+
+class _VisionMLP(nn.Module):
+    def __init__(self, fc1_w: torch.Tensor, fc1_b: torch.Tensor, fc2_w: torch.Tensor, fc2_b: torch.Tensor):
+        super().__init__()
+        self.register_buffer("fc1_weight", fc1_w, persistent=False)
+        self.register_buffer("fc1_bias", fc1_b, persistent=False)
+        self.register_buffer("fc2_weight", fc2_w, persistent=False)
+        self.register_buffer("fc2_bias", fc2_b, persistent=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return F.linear(F.gelu(F.linear(x, self.fc1_weight, self.fc1_bias), approximate="tanh"), self.fc2_weight, self.fc2_bias)
+
+
+def _rotate_half(x: torch.Tensor) -> torch.Tensor:
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def _apply_rope_vision(q: torch.Tensor, k: torch.Tensor, freqs) -> tuple[torch.Tensor, torch.Tensor]:
+    """Reference ``apply_rope`` (split-half) fed ``freqs=(cos, sin, neg_sin)``.
+
+    The vision rotary is built from the doubled-angle embedding so that
+    ``sin[..., half:]`` already equals ``-sin[..., :half]``; the caller passes it
+    through unchanged, making this the rotate-half equivalent and keeping q/k at
+    their native dtype.
+    """
+    cos, sin, neg_sin = freqs
+    original_q = q
+    q = original_q * cos
+    split = q.shape[-1] // 2
+    q[..., :split].addcmul_(original_q[..., split:], neg_sin)
+    q[..., split:].addcmul_(original_q[..., :split], sin)
+    original_k = k
+    k = original_k * cos
+    k[..., :split].addcmul_(original_k[..., split:], neg_sin)
+    k[..., split:].addcmul_(original_k[..., :split], sin)
+    return q, k
+
+
+class _VisionAttention(nn.Module):
+    def __init__(self, qkv_w: torch.Tensor, qkv_b: torch.Tensor, proj_w: torch.Tensor, proj_b: torch.Tensor, *, num_heads: int, head_dim: int):
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = head_dim
+        self.register_buffer("qkv_weight", qkv_w, persistent=False)
+        self.register_buffer("qkv_bias", qkv_b, persistent=False)
+        self.register_buffer("proj_weight", proj_w, persistent=False)
+        self.register_buffer("proj_bias", proj_b, persistent=False)
+
+    def forward(self, x: torch.Tensor, cu_seqlens: torch.Tensor, position_embeddings: torch.Tensor) -> torch.Tensor:
+        seq_length = x.shape[0]
+        # Cast qkv to a common dtype (the weights may be bf16 or fp32).
+        qkv_dtype = self.qkv_weight.dtype
+        qkv = F.linear(x.to(qkv_dtype), self.qkv_weight, self.qkv_bias)
+        query_states, key_states, value_states = (
+            qkv.reshape(seq_length, 3, self.num_heads, self.head_dim).permute(1, 0, 2, 3).unbind(0)
+        )
+        # RoPE in fp32 to avoid bf16 precision loss, then cast back.
+        orig_dtype = query_states.dtype
+        cos, sin, neg_sin = position_embeddings
+        query_states, key_states = _apply_rope_vision(
+            query_states.float(), key_states.float(),
+            (cos.float(), sin.float(), neg_sin.float()),
+        )
+        query_states, key_states = query_states.to(orig_dtype), key_states.to(orig_dtype)
+        lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
+        attn_outputs = []
+        for q, k, v in zip(
+            torch.split(query_states, lengths, dim=0),
+            torch.split(key_states, lengths, dim=0),
+            torch.split(value_states, lengths, dim=0),
+        ):
+            with sdpa_kernel(
+                [
+                    SDPBackend.FLASH_ATTENTION,
+                    SDPBackend.CUDNN_ATTENTION,
+                    SDPBackend.EFFICIENT_ATTENTION,
+                    SDPBackend.MATH,
+                ],
+                set_priority=True,
+            ):
+                output = F.scaled_dot_product_attention(
+                    q.transpose(0, 1).unsqueeze(0),
+                    k.transpose(0, 1).unsqueeze(0),
+                    v.transpose(0, 1).unsqueeze(0),
+                )
+            attn_outputs.append(output.transpose(1, 2).reshape(1, q.shape[0], -1))
+        attn_output = torch.cat(attn_outputs, dim=1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        return F.linear(attn_output, self.proj_weight, self.proj_bias)
+
+
+class _VisionBlock(nn.Module):
+    def __init__(self, norm1_w: torch.Tensor, norm1_b: torch.Tensor, attn: _VisionAttention, norm2_w: torch.Tensor, norm2_b: torch.Tensor, mlp: _VisionMLP):
+        super().__init__()
+        self.attn = attn
+        self.mlp = mlp
+        self.register_buffer("norm1_weight", norm1_w, persistent=False)
+        self.register_buffer("norm1_bias", norm1_b, persistent=False)
+        self.register_buffer("norm2_weight", norm2_w, persistent=False)
+        self.register_buffer("norm2_bias", norm2_b, persistent=False)
+
+    def forward(self, x: torch.Tensor, cu_seqlens: torch.Tensor, position_embeddings: torch.Tensor) -> torch.Tensor:
+        x = x + self.attn(F.layer_norm(x, (x.shape[-1],), weight=self.norm1_weight, bias=self.norm1_bias, eps=1e-6), cu_seqlens=cu_seqlens, position_embeddings=position_embeddings)
+        return x + self.mlp(F.layer_norm(x, (x.shape[-1],), weight=self.norm2_weight, bias=self.norm2_bias, eps=1e-6))
+
+
+class _VisionPatchMerger(nn.Module):
+    """Qwen3-VL spatial-merge projector (main or deepstack).
+
+    The main merger applies LayerNorm over ``hidden_size`` (1152) BEFORE the
+    2x2 spatial merge; the deepstack merger applies LayerNorm over
+    ``merge_dim`` (4608) AFTER the merge. This is controlled by ``norm_dim``.
+    """
+
+    def __init__(self, norm_w: torch.Tensor, norm_b: torch.Tensor, fc1_w: torch.Tensor, fc1_b: torch.Tensor, fc2_w: torch.Tensor, fc2_b: torch.Tensor, *, merge_size: int, out_hidden_size: int, norm_dim: int | None = None):
+        super().__init__()
+        self.merge_dim = VISION_HIDDEN * (merge_size ** 2)
+        # Default: norm_dim = merge_dim (deepstack style).  Main merger overrides.
+        self.norm_dim = norm_dim if norm_dim is not None else self.merge_dim
+        self.register_buffer("norm_weight", norm_w, persistent=False)
+        self.register_buffer("norm_bias", norm_b, persistent=False)
+        self.register_buffer("fc1_weight", fc1_w, persistent=False)
+        self.register_buffer("fc1_bias", fc1_b, persistent=False)
+        self.register_buffer("fc2_weight", fc2_w, persistent=False)
+        self.register_buffer("fc2_bias", fc2_b, persistent=False)
+        self.out_hidden_size = out_hidden_size
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        # x is the existing 2x2-block-major [t*h*w, hidden] patch stream.
+        if self.norm_dim == self.merge_dim:
+            # DeepStack merger: group the existing block-major patch stream first.
+            x = x.view(-1, self.merge_dim)
+            x = F.layer_norm(x, (self.merge_dim,), weight=self.norm_weight, bias=self.norm_bias, eps=1e-6)
+        else:
+            # Main merger: per-patch LayerNorm over hidden, then group 2x2 into merge_dim.
+            x = F.layer_norm(x, (x.shape[-1],), weight=self.norm_weight, bias=self.norm_bias, eps=1e-6)
+            x = x.view(-1, self.merge_dim)
+        return F.linear(F.gelu(F.linear(x, self.fc1_weight, self.fc1_bias)), self.fc2_weight, self.fc2_bias)
+
+
+def resize_keyframe(image: torch.Tensor, width: int, height: int, *, crop: str = "disabled") -> torch.Tensor:
+    """Resize a ``[1, H, W, 3]`` float ``[0, 1]`` keyframe to the ``width x height``
+    canvas the way the Comfy reference does (``common_upscale(..., "lanczos", crop)``):
+
+    - ``crop="disabled"`` (first frame) stretches to the canvas.
+    - ``crop="center"`` (last frame) aspect-covers then center-crops.
+
+    Returns ``[1, height, width, 3]`` float ``[0, 1]``.
+    """
+    if image.ndim != 4 or image.shape[0] != 1:
+        raise ValueError("resize_keyframe expects a single [1, H, W, 3] image")
+    samples = image[:, :, :, :3].movedim(-1, 1)  # [1, 3, H, W]
+    if crop == "center":
+        old_h, old_w = samples.shape[-2], samples.shape[-1]
+        old_aspect = old_w / old_h
+        new_aspect = width / height
+        x = 0
+        y = 0
+        if old_aspect > new_aspect:
+            x = round((old_w - old_w * (new_aspect / old_aspect)) / 2)
+        elif old_aspect < new_aspect:
+            y = round((old_h - old_h * (old_aspect / new_aspect)) / 2)
+        samples = samples.narrow(-2, y, old_h - 2 * y).narrow(-1, x, old_w - 2 * x)
+    # F.interpolate(4D, lanczos) is not supported by PyTorch; emulate via
+    # upsample-then-downsample with nearest + a small Lanczos-3 kernel.
+    # (Simpler: just use PIL's LANCZOS via numpy for the single image.)
+    import numpy as np
+    from PIL import Image as PILImage
+    arr = samples[0].permute(1, 2, 0).cpu().numpy()  # [H, W, 3]
+    arr = (arr * 255.0).astype(np.uint8)
+    img = PILImage.fromarray(arr, mode="RGB")
+    img = img.resize((width, height), PILImage.LANCZOS)
+    samples = torch.from_numpy(np.asarray(img).astype(np.float32) / 255.0).permute(2, 0, 1).unsqueeze(0)
+    return samples.clamp(0.0, 1.0).movedim(1, -1)  # [1, H, W, 3]
+
+
+def _text_run_ids(tokenizer, prompt: str) -> list[int]:
+    """Token ids for a raw text run (``add_special_tokens=False``, no template)."""
+    raw_tokenizer = getattr(tokenizer, "tokenizer", None)
+    if raw_tokenizer is not None:
+        ids = raw_tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids
+    else:
+        ids = tokenizer(prompt, device="cpu")
+    return [int(t) for t in ids.reshape(-1).tolist()]
+
+
+@dataclass
+class Fl2vaPresentation:
+    """Expanded first/last keyframe prompt for the H3 DiT.
+
+    Carries the Qwen layer-50 text states (with spliced vision pads), the per-token
+    DiT modality tags, the mrope ids, and the DiT-level keyframe anchors used to
+    build the cond rows.
+    """
+
+    input_ids: torch.Tensor
+    text_states: torch.Tensor
+    text_token_tags: torch.Tensor
+    embeds_info: list[dict]
+    keyframes: list[dict]
+    frame_count: int
+
+
+def build_fl2va_presentation(
+    prompt: str,
+    first_frame: torch.Tensor | None,
+    last_frame: torch.Tensor | None,
+    *,
+    width: int,
+    height: int,
+    frame_count: int,
+    tokenizer,
+    vision: Qwen3VL32BVision,
+    text_encoder: "Qwen3VL32BTextEncoder",
+    device,
+) -> Fl2vaPresentation:
+    """Build the fl2va presentation and run the Qwen text conditioner over it.
+
+    ``first_frame`` / ``last_frame`` are ``[1, H, W, 3]`` ``[0, 1]`` images (already
+    aligned to the target canvas by the caller when needed). The presentation
+    sequence is ``[Picture1 <img> Picture2? <img> prompt]``: raw text ids with the
+    ``VISION_START`` sentinel, a vision pad run, and ``VISION_END`` per keyframe,
+    followed by the raw prompt ids (no chat template).
+    """
+    images: list[torch.Tensor] = []
+    keyframes: list[dict] = []
+    if first_frame is not None:
+        img = resize_keyframe(first_frame, width, height, crop="disabled")
+        images.append(img)
+        keyframes.append({"resolved_frame_index": 0, "image": img})
+    if last_frame is not None:
+        img = resize_keyframe(last_frame, width, height, crop="center")
+        images.append(img)
+        keyframes.append({"resolved_frame_index": frame_count - 1, "image": img})
+
+    # Build the entry list: (token_id/"text") runs and image placeholders.
+    entries: list = []
+    for i in range(len(images)):
+        entries.extend((tid, "text") for tid in _text_run_ids(tokenizer, f"<Picture {i + 1}>: "))
+        entries.append((VISION_START, "text"))
+        entries.append((i, "image"))
+        entries.append((VISION_END, "text"))
+    entries.extend((tid, "text") for tid in _text_run_ids(tokenizer, prompt))
+    if not any(kind == "text" for _, kind in entries):
+        entries = [(151643, "text")]
+
+    # Expand images through the vision tower up front (needs the vision tower); the
+    # merged output becomes the pad run that the token sequence points at.
+    vision_outputs: list[tuple[torch.Tensor, torch.Tensor, torch.Tensor, list[torch.Tensor], int]] = []
+    for i in range(len(images)):
+        flatten, grid = process_image(images[i].to(device))
+        merged, deepstack = vision(flatten, grid.to(device))
+        vision_outputs.append((flatten, grid, merged, deepstack, merged.shape[0]))
+
+    # Expand into a flat token list + embeds_info.
+    token_ids: list[int] = []
+    embeds_info: list[dict] = []
+    vision_index = 0
+    for tok, kind in entries:
+        if kind == "image":
+            flatten, grid, merged, deepstack, size = vision_outputs[vision_index]
+            size = int(size)
+            start = len(token_ids)
+            token_ids.extend([0] * size)  # placeholder ids, overwritten by the merged rows
+            embeds_info.append({"type": "image", "index": start, "size": size, "extra": {"grid": grid, "deepstack": deepstack, "merged": merged}})
+            vision_index += 1
+        else:
+            token_ids.append(int(tok))
+
+    seq_len = len(token_ids)
+    input_ids = torch.tensor([int(t) for t in token_ids], dtype=torch.long, device=device).unsqueeze(0)  # [1, seq]
+
+    # Build the Qwen hidden-state input: scaled token embeds, with the merged vision
+    # rows spliced over the pad positions.
+    base_embeds = text_encoder._embed_rows(input_ids)  # [1, seq, 5120] fp32
+    visual_pos_masks = torch.zeros((1, seq_len), dtype=torch.bool, device=device)
+    for e in embeds_info:
+        merged = e["extra"]["merged"]
+        base_embeds[0, e["index"]: e["index"] + e["size"], :] = merged.to(base_embeds.dtype)
+        visual_pos_masks[0, e["index"]: e["index"] + e["size"]] = True
+
+    # DeepStack features: one concatenated tensor per vision-layer index, spanning all
+    # spliced blocks in sequence order.
+    merged_deepstack_per_index: dict = {}
+    for e in embeds_info:
+        for i, ds in enumerate(e["extra"]["deepstack"]):
+            merged_deepstack_per_index.setdefault(i, []).append(ds)
+    deepstack_embeds = [torch.cat(v, dim=0) for i, v in sorted(merged_deepstack_per_index.items())]
+    position_ids = mrope_position_ids(embeds_info, seq_len, device)
+
+    text_states = text_encoder._run_layers(
+        base_embeds.float(),
+        position_ids=position_ids,
+        visual_pos_masks=visual_pos_masks,
+        deepstack_embeds=deepstack_embeds,
+    )
+    tags = token_tags(seq_len, embeds_info, device)
+    return Fl2vaPresentation(
+        input_ids=input_ids,
+        text_states=text_states,
+        text_token_tags=tags,
+        embeds_info=embeds_info,
+        keyframes=keyframes,
+        frame_count=frame_count,
+    )
+
+
+class Qwen3VL32BVision(nn.Module):
+    """H3's Qwen3-VL-32B visual tower (+ DeepStack mergers), from the text-encoder safetensors."""
+
+    def __init__(self, checkpoint_path: str | Path, *, device, dtype: torch.dtype):
+        super().__init__()
+        self.device = device
+        self.dtype = dtype
+        self.spatial_merge_size = VISION_MERGE
+        self.spatial_merge_unit = VISION_MERGE * VISION_MERGE
+        self.patch_size = VISION_PATCH
+        self.hidden_size = VISION_HIDDEN
+        self.num_heads = VISION_HEADS
+        self.num_position_embeddings = VISION_POSITION_EMBEDS
+        self.num_grid_per_side = int(self.num_position_embeddings ** 0.5)
+        self.depth = VISION_DEPTH
+        self.deepstack_visual_indexes = list(DEEPSTACK_VISUAL_INDEXES)
+        self.out_hidden_size = 5120
+
+        checkpoint_path = str(checkpoint_path)
+        required = {
+            "visual.pos_embed.weight",
+            "visual.patch_embed.proj.weight",
+            "visual.patch_embed.proj.bias",
+            "visual.merger.norm.weight",
+            "visual.merger.norm.bias",
+            "visual.merger.linear_fc1.weight",
+            "visual.merger.linear_fc1.bias",
+            "visual.merger.linear_fc2.weight",
+            "visual.merger.linear_fc2.bias",
+        }
+        for i in range(self.depth):
+            required.update({
+                f"visual.blocks.{i}.norm1.weight",
+                f"visual.blocks.{i}.norm1.bias",
+                f"visual.blocks.{i}.attn.qkv.weight",
+                f"visual.blocks.{i}.attn.qkv.bias",
+                f"visual.blocks.{i}.attn.proj.weight",
+                f"visual.blocks.{i}.attn.proj.bias",
+                f"visual.blocks.{i}.norm2.weight",
+                f"visual.blocks.{i}.norm2.bias",
+                f"visual.blocks.{i}.mlp.linear_fc1.weight",
+                f"visual.blocks.{i}.mlp.linear_fc1.bias",
+                f"visual.blocks.{i}.mlp.linear_fc2.weight",
+                f"visual.blocks.{i}.mlp.linear_fc2.bias",
+            })
+        for i in range(len(self.deepstack_visual_indexes)):
+            required.update({
+                f"visual.deepstack_merger_list.{i}.norm.weight",
+                f"visual.deepstack_merger_list.{i}.norm.bias",
+                f"visual.deepstack_merger_list.{i}.linear_fc1.weight",
+                f"visual.deepstack_merger_list.{i}.linear_fc1.bias",
+                f"visual.deepstack_merger_list.{i}.linear_fc2.weight",
+                f"visual.deepstack_merger_list.{i}.linear_fc2.bias",
+            })
+        with safe_open(checkpoint_path, framework="pt", device=device) as checkpoint:
+            names = set(checkpoint.keys())
+            missing = sorted(required - names)
+            if missing:
+                raise ValueError("Not a MiniMax H3 Qwen3-VL vision checkpoint; missing: " + ", ".join(missing[:12]))
+
+            self._init_modules(
+                device,
+                dtype,
+                lambda name: checkpoint.get_tensor(name).to(device=device, dtype=dtype),
+            )
+            # Comfy's Embedding is not dynamically cast to the FP32 vision stream.
+            self.pos_embed = checkpoint.get_tensor("visual.pos_embed.weight").to(device=device)
+
+    def _init_modules(self, device, dtype, get) -> None:
+        self.register_buffer("pos_embed", get("visual.pos_embed.weight"), persistent=False)
+        self.patch_embed = _VisionPatchEmbed(get("visual.patch_embed.proj.weight"), get("visual.patch_embed.proj.bias"))
+
+        self.merger = _VisionPatchMerger(
+            get("visual.merger.norm.weight"), get("visual.merger.norm.bias"),
+            get("visual.merger.linear_fc1.weight"), get("visual.merger.linear_fc1.bias"),
+            get("visual.merger.linear_fc2.weight"), get("visual.merger.linear_fc2.bias"),
+            merge_size=self.spatial_merge_size, out_hidden_size=self.out_hidden_size,
+            norm_dim=VISION_HIDDEN,  # main merger: LayerNorm over hidden before 2x2 merge
+        )
+        self.deepstack_merger_list = nn.ModuleList([
+            _VisionPatchMerger(
+                get(f"visual.deepstack_merger_list.{i}.norm.weight"), get(f"visual.deepstack_merger_list.{i}.norm.bias"),
+                get(f"visual.deepstack_merger_list.{i}.linear_fc1.weight"), get(f"visual.deepstack_merger_list.{i}.linear_fc1.bias"),
+                get(f"visual.deepstack_merger_list.{i}.linear_fc2.weight"), get(f"visual.deepstack_merger_list.{i}.linear_fc2.bias"),
+                merge_size=self.spatial_merge_size, out_hidden_size=self.out_hidden_size,
+            )
+            for i in range(len(self.deepstack_visual_indexes))
+        ])
+        self.blocks = nn.ModuleList([
+            _VisionBlock(
+                get(f"visual.blocks.{i}.norm1.weight"), get(f"visual.blocks.{i}.norm1.bias"),
+                _VisionAttention(
+                    get(f"visual.blocks.{i}.attn.qkv.weight"), get(f"visual.blocks.{i}.attn.qkv.bias"),
+                    get(f"visual.blocks.{i}.attn.proj.weight"), get(f"visual.blocks.{i}.attn.proj.bias"),
+                    num_heads=self.num_heads, head_dim=VISION_HEAD_DIM,
+                ),
+                get(f"visual.blocks.{i}.norm2.weight"), get(f"visual.blocks.{i}.norm2.bias"),
+                _VisionMLP(
+                    get(f"visual.blocks.{i}.mlp.linear_fc1.weight"), get(f"visual.blocks.{i}.mlp.linear_fc1.bias"),
+                    get(f"visual.blocks.{i}.mlp.linear_fc2.weight"), get(f"visual.blocks.{i}.mlp.linear_fc2.bias"),
+                ),
+            )
+            for i in range(self.depth)
+        ])
+        self.rotary_pos_emb = _VisionRotary(VISION_HIDDEN // self.num_heads // 2, device, dtype)
+
+    def rot_pos_emb(self, grid_thw: torch.Tensor) -> torch.Tensor:
+        """Reference ``rot_pos_emb``: (row, col) coords -> 2D rotary angles per token."""
+        merge_size = self.spatial_merge_size
+        grid_thw_list = grid_thw.tolist()
+        max_hw = max(max(h, w) for _, h, w in grid_thw_list)
+        freq_table = self.rotary_pos_emb(max_hw).to(grid_thw.device)
+        device = freq_table.device
+        total_tokens = sum(int(t * h * w) for t, h, w in grid_thw_list)
+        pos_ids = torch.empty((total_tokens, 2), dtype=torch.long, device=device)
+        offset = 0
+        for num_frames, height, width in grid_thw_list:
+            num_frames, height, width = int(num_frames), int(height), int(width)
+            merged_h, merged_w = height // merge_size, width // merge_size
+            block_rows = torch.arange(merged_h, device=device)
+            block_cols = torch.arange(merged_w, device=device)
+            intra_row = torch.arange(merge_size, device=device)
+            intra_col = torch.arange(merge_size, device=device)
+            row_idx = (block_rows[:, None, None, None] * merge_size + intra_row[None, None, :, None]).expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
+            col_idx = (block_cols[None, :, None, None] * merge_size + intra_col[None, None, None, :]).expand(merged_h, merged_w, merge_size, merge_size).reshape(-1)
+            coords = torch.stack((row_idx, col_idx), dim=-1)
+            if num_frames > 1:
+                coords = coords.repeat(num_frames, 1)
+            num_tokens = coords.shape[0]
+            pos_ids[offset:offset + num_tokens] = coords
+            offset += num_tokens
+        return freq_table[pos_ids].flatten(1)
+
+    def fast_pos_embed_interpolate(self, grid_thw: torch.Tensor) -> torch.Tensor:
+        """Reference 4-tap bilinear interpolation of the learned 48x48 grid."""
+        grid_ts = [int(row[0]) for row in grid_thw.tolist()]
+        grid_hs = [int(row[1]) for row in grid_thw.tolist()]
+        grid_ws = [int(row[2]) for row in grid_thw.tolist()]
+        device = self.pos_embed.device
+        idx_list: list[list] = [[] for _ in range(4)]
+        weight_list: list[list] = [[] for _ in range(4)]
+        for t, h, w in zip(grid_ts, grid_hs, grid_ws):
+            h, w = int(h), int(w)
+            h_idxs = torch.linspace(0, self.num_grid_per_side - 1, h, device=device)
+            w_idxs = torch.linspace(0, self.num_grid_per_side - 1, w, device=device)
+            h_floor = h_idxs.int()
+            w_floor = w_idxs.int()
+            h_ceil = (h_idxs + 1).int().clamp(max=self.num_grid_per_side - 1)
+            w_ceil = (w_idxs + 1).int().clamp(max=self.num_grid_per_side - 1)
+            dh = h_idxs - h_floor
+            dw = w_idxs - w_floor
+            base_h = h_floor * self.num_grid_per_side
+            base_h_ceil = h_ceil * self.num_grid_per_side
+            indices = [
+                (base_h[None].T + w_floor[None]).flatten(),
+                (base_h[None].T + w_ceil[None]).flatten(),
+                (base_h_ceil[None].T + w_floor[None]).flatten(),
+                (base_h_ceil[None].T + w_ceil[None]).flatten(),
+            ]
+            weights = [
+                ((1 - dh)[None].T * (1 - dw)[None]).flatten(),
+                ((1 - dh)[None].T * dw[None]).flatten(),
+                (dh[None].T * (1 - dw)[None]).flatten(),
+                (dh[None].T * dw[None]).flatten(),
+            ]
+            for j in range(4):
+                idx_list[j].extend(indices[j].tolist())
+                weight_list[j].extend(weights[j].tolist())
+        idx_tensor = torch.tensor(idx_list, dtype=torch.long, device=device)
+        weight_tensor = torch.tensor(weight_list, dtype=self.pos_embed.dtype, device=device)
+        pos_embeds = self.pos_embed[idx_tensor] * weight_tensor[:, :, None]
+        patch_pos_embeds = pos_embeds[0] + pos_embeds[1] + pos_embeds[2] + pos_embeds[3]
+        patch_pos_embeds = patch_pos_embeds.split([h * w for h, w in zip(grid_hs, grid_ws)])
+        patch_pos_embeds_permute = []
+        merge_size = self.spatial_merge_size
+        for pos_embed, t, h, w in zip(patch_pos_embeds, grid_ts, grid_hs, grid_ws):
+            pos_embed = pos_embed.repeat(t, 1)
+            pos_embed = pos_embed.view(t, h // merge_size, merge_size, w // merge_size, merge_size, -1).permute(0, 1, 3, 2, 4, 5).flatten(0, 4)
+            patch_pos_embeds_permute.append(pos_embed)
+        return torch.cat(patch_pos_embeds_permute)
+
+    def forward(self, flatten_patches: torch.Tensor, grid_thw: torch.Tensor) -> tuple[torch.Tensor, list[torch.Tensor]]:
+        """Run the visual tower -> (merged, deepstack)."""
+        x = self.patch_embed(flatten_patches.to(self.dtype).to(self.device))
+        x = x + self.fast_pos_embed_interpolate(grid_thw).to(x.device)
+        x = x.reshape(x.shape[0], -1)
+        rotary = self.rot_pos_emb(grid_thw.to(x.device)).to(x.device).reshape(x.shape[0], -1)
+        emb = torch.cat((rotary, rotary), dim=-1)
+        cos = emb.cos().unsqueeze(-2)
+        sin = emb.sin().unsqueeze(-2)
+        sin_split = sin.shape[-1] // 2
+        position_embeddings = (cos, sin[..., :sin_split], -sin[..., sin_split:])
+        cu_seqlens = F.pad(
+            torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(0, dtype=torch.int32),
+            (1, 0), value=0,
+        )
+        deepstack_features = []
+        for layer_num, block in enumerate(self.blocks):
+            x = block(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings)
+            # x: [t*h*w, hidden] (unmerged patches).
+            if layer_num in self.deepstack_visual_indexes:
+                deepstack_features.append(
+                    self.deepstack_merger_list[self.deepstack_visual_indexes.index(layer_num)](x)
+                )
+        return self.merger(x), deepstack_features
+
+
+class _VisionRotary(nn.Module):
+    def __init__(self, dim: int, device, dtype):
+        super().__init__()
+        # Comfy constructs this buffer on CPU, then moves the model to CUDA.
+        inv_freq = 1.0 / (10000.0 ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
+        self.register_buffer("inv_freq", inv_freq.to(device), persistent=False)
+
+    def forward(self, seqlen: int) -> torch.Tensor:
+        seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
+        return torch.outer(seq, self.inv_freq)
--- a/src/h3_blackwell_runtime/runtime.py
+++ b/src/h3_blackwell_runtime/runtime.py
@ -0,0 +1,411 @@
+"""Resident prompt-only H3 runtime used by the hot service and tests."""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import time
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+
+from .audio_vae_decoder import MiniMaxH3AudioVAE
+from .attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND
+from .block import configure_mlp_chunking
+from .checkpoint import H3Checkpoint
+from .denoiser import H3PackedDenoiser
+from .lora import load_lora_adapter, set_active_lora
+from .latent_upscaler import load_h3_latent_upscaler, upscale_h3_latent
+from .packing import H3PromptPacker
+from .qwen3vl_text import Qwen3VLPromptConditioner
+from .qwen3vl_vision import build_fl2va_presentation, Qwen3VL32BVision
+from .sampler import sample_video_res_multistep, sample_video_turbo
+from .t2v import random_av_latents
+from .token_refiner import H3TokenRefiner
+from .vae_decoder import MiniMaxH3VideoVAE, dtype_from_name
+from .vae_encoder import MiniMaxH3VideoVAEEncoder
+
+
+TURBO_VARIANTS = {
+    "4step": {"steps": 4, "video_shift": 6.0},
+    "8step": {"steps": 8, "video_shift": 12.0},
+}
+
+
+@dataclass(frozen=True)
+class RuntimeConfig:
+    model_path: str = "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors"
+    text_encoder_path: str = "/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors"
+    tokenizer_path: str = "/opt/h3-blackwell-runtime/src/h3_blackwell_runtime/qwen25_tokenizer"
+    video_vae_path: str = "/vae/minimax_h3_video_vae_fp16.safetensors"
+    audio_vae_path: str = "/vae/minimax_h3_audio_vae_fp32.safetensors"
+    attention: str = DEFAULT_ATTENTION_BACKEND
+    vae_dtype: str = "float16"
+    vae_tile_size: int = 256
+    mlp_chunks: int = 1
+    mlp_chunk_threshold: int = 4096
+    turbo_loras: tuple[tuple[str, str], ...] = ()
+    latent_upscaler_path: str | None = None
+    device: str = "cuda"
+
+
+def _sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def _ffmpeg_command(loglevel: str, *parts: str) -> list[str]:
+    return ["ffmpeg", "-hide_banner", "-loglevel", loglevel, *parts]
+
+
+def _refiner_attention_backend(attention: str) -> str:
+    if attention != "sol_attn":
+        return attention
+    fallback = os.getenv("H3_SOL_FALLBACK", "sage2")
+    return fallback if fallback in AVAILABLE_BACKENDS and fallback != "sol_attn" else "sage2"
+
+
+def normalize_upscale(value) -> float | None:
+    if value is None or value == "" or value == "none" or value == 1 or value == 1.0:
+        return None
+    try:
+        scale = float(value)
+    except (TypeError, ValueError) as exc:
+        raise ValueError("upscale must be a number between 1.0 and 4.0") from exc
+    if not 1.0 < scale <= 4.0:
+        raise ValueError("upscale must be greater than 1.0 and at most 4.0")
+    return scale
+
+
+class H3HotRuntime:
+    """Keep all prompt-only H3 models resident for repeated requests."""
+
+    def __init__(self, config: RuntimeConfig):
+        self.config = config
+        self.attention = config.attention
+        self.turbo: str | None = None
+        self.loaded_at = time.time()
+        self.load_stages: list[dict] = []
+
+        self.checkpoint = H3Checkpoint(config.model_path, device=config.device)
+        self.conditioner = self._timed_load(
+            "qwen_loaded",
+            lambda: Qwen3VLPromptConditioner(config.text_encoder_path, config.tokenizer_path),
+        )
+        self.model = self._timed_load(
+            "h3_loaded",
+            self._load_h3,
+        )
+        self.refiner = self._timed_load(
+            "token_refiner_loaded",
+            lambda: H3TokenRefiner(self.checkpoint, attention_backend=_refiner_attention_backend(config.attention)).eval(),
+        )
+        self.turbo_loras = dict(config.turbo_loras)
+        unknown_turbos = sorted(set(self.turbo_loras) - set(TURBO_VARIANTS))
+        if unknown_turbos:
+            raise ValueError(f"Unknown Turbo variants: {unknown_turbos}")
+        for name, path in self.turbo_loras.items():
+            self._timed_load(
+                f"turbo_{name}_loaded",
+                lambda name=name, path=path: load_lora_adapter(self.model, self.refiner, name, path, config.device),
+            )
+        self.packer = H3PromptPacker(self.checkpoint)
+        self.video_vae = self._timed_load(
+            "video_vae_loaded",
+            self._load_video_vae,
+        )
+        self.audio_vae = self._timed_load(
+            "audio_vae_loaded",
+            lambda: MiniMaxH3AudioVAE.from_safetensors(config.audio_vae_path, device=config.device).eval(),
+        )
+        self.vae_encoder = self._timed_load(
+            "vae_encoder_loaded",
+            lambda: MiniMaxH3VideoVAEEncoder.from_safetensors(config.video_vae_path, device=config.device).eval(),
+        )
+        self.vision_tower = self._timed_load(
+            "vision_tower_loaded",
+            lambda: Qwen3VL32BVision(config.text_encoder_path, device=config.device, dtype=torch.float32),
+        )
+        self.latent_upscaler = None
+        if config.latent_upscaler_path is not None:
+            self.latent_upscaler = self._timed_load(
+                "latent_upscaler_loaded",
+                lambda: load_h3_latent_upscaler(config.latent_upscaler_path, device=config.device, dtype=torch.float16),
+            )
+
+    def _timed_load(self, stage: str, fn):
+        _sync()
+        start = time.perf_counter()
+        value = fn()
+        _sync()
+        self.load_stages.append({"stage": stage, "seconds": time.perf_counter() - start})
+        return value
+
+    def _load_video_vae(self) -> MiniMaxH3VideoVAE:
+        vae = MiniMaxH3VideoVAE.from_safetensors(
+            self.config.video_vae_path,
+            device=self.config.device,
+            dtype=dtype_from_name(self.config.vae_dtype),
+        ).eval()
+        vae.tile_size = self.config.vae_tile_size
+        return vae
+
+    def _load_h3(self) -> H3PackedDenoiser:
+        model = H3PackedDenoiser.from_checkpoint(self.checkpoint, attention_backend=self.config.attention).eval()
+        configure_mlp_chunking(model, self.config.mlp_chunks, self.config.mlp_chunk_threshold)
+        return model
+
+    def status(self) -> dict:
+        return {
+            "ready": True,
+            "initial_attention": self.config.attention,
+            "current_attention": self.attention,
+            "available_turbos": sorted(self.turbo_loras),
+            "current_turbo": self.turbo,
+            "latent_upscaler_loaded": self.latent_upscaler is not None,
+            "vae_dtype": self.config.vae_dtype,
+            "vae_tile_size": self.config.vae_tile_size,
+            "mlp_chunks": self.config.mlp_chunks,
+            "mlp_chunk_threshold": self.config.mlp_chunk_threshold,
+            "loaded_at": self.loaded_at,
+            "load_stages": self.load_stages,
+        }
+
+    @torch.inference_mode()
+    def set_attention(self, attention: str) -> None:
+        if attention not in AVAILABLE_BACKENDS:
+            raise ValueError(f"Unsupported attention backend: {attention}")
+        if attention == self.attention:
+            return
+        for module in self.model.modules():
+            if hasattr(module, "backend"):
+                module.backend = attention
+        for block in self.refiner.blocks:
+            block.attention_backend = _refiner_attention_backend(attention)
+        self.attention = attention
+
+    @torch.inference_mode()
+    def set_turbo(self, turbo: str | None) -> None:
+        turbo = None if turbo in {None, "", "none"} else turbo
+        if turbo is not None and turbo not in self.turbo_loras:
+            raise ValueError(f"Turbo variant {turbo!r} is not loaded")
+        if turbo == self.turbo:
+            return
+        set_active_lora(self.model, self.refiner, turbo)
+        self.turbo = turbo
+
+    def _build_fl2va(self, prompt: str, first_frame: torch.Tensor | None, last_frame: torch.Tensor | None, width: int, height: int, aligned_frames: int):
+        from .qwen3vl_vision import build_fl2va_presentation, resize_keyframe
+
+        device = self.config.device
+        first = self._image_to_uint8_nhwc(first_frame.to(device)).float() / 255.0 if first_frame is not None else None
+        last = self._image_to_uint8_nhwc(last_frame.to(device)).float() / 255.0 if last_frame is not None else None
+        presentation = build_fl2va_presentation(
+            prompt,
+            first,
+            last,
+            width=width,
+            height=height,
+            frame_count=aligned_frames,
+            tokenizer=self.conditioner.tokenizer,
+            vision=self.vision_tower,
+            text_encoder=self.conditioner.encoder,
+            device=device,
+        )
+        # resize each keyframe to the canvas and VAE-encode it (the DiT cond latent).
+        cond_latents = []
+        for kf in presentation.keyframes:
+            resized = resize_keyframe(kf["image"].to(device), width, height, crop="disabled" if kf["resolved_frame_index"] == 0 else "center")
+            pix = resized.movedim(-1, 1).to(device, dtype=torch.float32).mul(2.0).sub(1.0)
+            cond_latents.append(self.vae_encoder.encode(pix))
+        return presentation, cond_latents, aligned_frames
+
+    def _image_to_uint8_nhwc(self, img: torch.Tensor) -> torch.Tensor:
+        """Normalize a [1,3,H,W] image in [0,1], [-1,1], or [0,255] to NHWC uint8."""
+        x = img.float()
+        if x.numel() == 0:
+            return x
+        if x.max() > 1.0:
+            x = x / 255.0  # already 0..255
+        elif x.min() < 0.0:
+            x = (x.clamp(-1, 1) + 1) * 0.5  # -1..1 -> 0..1
+        return (x.movedim(1, -1).clamp(0, 1) * 255).to(torch.uint8)
+
+    @torch.inference_mode()
+    def generate(
+        self,
+        *,
+        prompt: str,
+        output: str | Path,
+        width: int,
+        height: int,
+        frames: int,
+        steps: int,
+        seed: int,
+        attention: str | None = None,
+        turbo: str | None = None,
+        upscale: float | None = None,
+        first_frame: torch.Tensor | None = None,
+        last_frame: torch.Tensor | None = None,
+        mux_audio: bool = True,
+        keep_intermediates: bool = False,
+        ffmpeg_loglevel: str = "error",
+        save_latent: str | Path | None = None,
+        cache_mode: str | None = None,
+        cache_threshold: float = 0.0,
+        cache_start_percent: float = 0.0,
+        cache_end_percent: float = 1.0,
+        cache_subsample_factor: int = 2,
+    ) -> dict:
+        stages: list[dict] = []
+        cache_stats: dict = {}
+
+        def timed(stage: str, fn):
+            _sync()
+            start = time.perf_counter()
+            value = fn()
+            _sync()
+            stages.append({"stage": stage, "seconds": time.perf_counter() - start})
+            return value
+
+        output = Path(output)
+        output.parent.mkdir(parents=True, exist_ok=True)
+        requested_turbo = None if turbo in {None, "", "none"} else turbo
+        upscale_scale = normalize_upscale(upscale)
+        if upscale_scale is not None and self.latent_upscaler is None:
+            raise ValueError("H3 latent upscaler is not loaded")
+        if requested_turbo is not None:
+            if requested_turbo not in self.turbo_loras:
+                raise ValueError(f"Turbo variant {requested_turbo!r} is not loaded")
+            expected_steps = TURBO_VARIANTS[requested_turbo]["steps"]
+            if steps != expected_steps:
+                raise ValueError(f"Turbo {requested_turbo} requires exactly {expected_steps} steps")
+            if cache_mode not in {None, "", "disabled", "none"}:
+                raise ValueError("Turbo sampling does not support denoiser caching")
+        if attention is not None:
+            self.set_attention(attention)
+        self.set_turbo(requested_turbo)
+
+        video, audio, aligned_frames = timed(
+            "latents_initialized",
+            lambda: random_av_latents(width, height, frames, seed, device=self.config.device),
+        )
+        use_fl2va = first_frame is not None or last_frame is not None
+        if use_fl2va:
+            # Build the fl2va Qwen presentation, encode the keyframes to cond latents,
+            # and refine the text span (vision-aware).
+            presentation, cond_latents, frame_count = timed(
+                "fl2va_conditioned",
+                lambda: self._build_fl2va(prompt, first_frame, last_frame, width, height, aligned_frames),
+            )
+            text = timed("text_conditioned", lambda: self.refiner(presentation.text_states))
+            pack_kwargs = {
+                "text_token_tags": presentation.text_token_tags,
+                "cond_latents": cond_latents,
+                "cond_frame_indices": [kf["resolved_frame_index"] for kf in presentation.keyframes],
+                "frame_count": frame_count,
+            }
+        else:
+            text = timed("text_conditioned", lambda: self.refiner(self.conditioner(prompt)))
+            pack_kwargs = {}
+        if self.turbo is None:
+            sample = lambda: sample_video_res_multistep(
+                self.model, self.packer, text, video, audio, steps=steps, seed=seed,
+                return_audio=mux_audio, cache_mode=cache_mode, cache_threshold=cache_threshold,
+                cache_start_percent=cache_start_percent, cache_end_percent=cache_end_percent,
+                cache_subsample_factor=cache_subsample_factor, cache_stats=cache_stats, **pack_kwargs,
+            )
+        else:
+            sample = lambda: sample_video_turbo(
+                self.model, self.packer, text, video, audio, steps=steps,
+                video_shift=TURBO_VARIANTS[self.turbo]["video_shift"], seed=seed,
+                return_audio=mux_audio, **pack_kwargs,
+            )
+        sampled = timed("sampled", sample)
+        if mux_audio:
+            latent, audio_latent = sampled
+        else:
+            latent, audio_latent = sampled, None
+        source_width, source_height = width, height
+        if upscale_scale is not None:
+            latent = timed(
+                "latent_upscaled",
+                lambda: upscale_h3_latent(self.latent_upscaler, latent, scale=upscale_scale),
+            )
+            width = latent.shape[-1] * 16
+            height = latent.shape[-2] * 16
+        if save_latent is not None:
+            latent_path = Path(save_latent)
+            latent_path.parent.mkdir(parents=True, exist_ok=True)
+            state = {"latent": latent.detach().cpu(), "frames": aligned_frames, "width": width, "height": height, "prompt": prompt, "seed": seed}
+            if audio_latent is not None:
+                state["audio_latent"] = audio_latent.detach().cpu()
+            torch.save(state, latent_path)
+        pixels = timed("vae_decoded", lambda: self.video_vae.decode(latent.to(next(self.video_vae.parameters()).dtype))[:, :, :aligned_frames])
+        pixels = timed("pixels_cpu", lambda: ((pixels[0].permute(1, 2, 3, 0).clamp(-1, 1) + 1) * 127.5).to(torch.uint8).cpu())
+
+        raw = output.with_suffix(".rgb")
+        video_output = output.with_name(output.stem + ".video.mp4") if mux_audio else output
+        timed("raw_write", lambda: pixels.numpy().tofile(raw))
+        timed(
+            "video_encode",
+            lambda: subprocess.run(
+                _ffmpeg_command(
+                    ffmpeg_loglevel,
+                    "-y",
+                    "-f",
+                    "rawvideo",
+                    "-pixel_format",
+                    "rgb24",
+                    "-video_size",
+                    f"{pixels.shape[2]}x{pixels.shape[1]}",
+                    "-framerate",
+                    "24",
+                    "-i",
+                    str(raw),
+                    "-an",
+                    "-c:v",
+                    "libx264",
+                    "-pix_fmt",
+                    "yuv420p",
+                    str(video_output),
+                ),
+                check=True,
+            ),
+        )
+        raw.unlink()
+
+        audio_output = None
+        if mux_audio:
+            audio_output = output.with_suffix(".wav")
+            waveform = timed("audio_decoded", lambda: self.audio_vae.decode(audio_latent.to(next(self.audio_vae.parameters()).dtype)).clamp(-1, 1).cpu()[0])
+            audio_raw = audio_output.with_suffix(".f32le")
+            timed("audio_raw_write", lambda: waveform.transpose(0, 1).contiguous().numpy().tofile(audio_raw))
+            timed("audio_encode", lambda: subprocess.run(_ffmpeg_command(ffmpeg_loglevel, "-y", "-f", "f32le", "-ar", "32000", "-ac", "2", "-i", str(audio_raw), str(audio_output)), check=True))
+            audio_raw.unlink()
+            timed("mux", lambda: subprocess.run(_ffmpeg_command(ffmpeg_loglevel, "-y", "-i", str(video_output), "-i", str(audio_output), "-c:v", "copy", "-c:a", "aac", "-shortest", str(output)), check=True))
+            if not keep_intermediates:
+                video_output.unlink()
+                audio_output.unlink()
+                audio_output = None
+
+        return {
+            "output": str(output),
+            "audio_output": str(audio_output) if audio_output is not None else None,
+            "frames": aligned_frames,
+            "width": width,
+            "height": height,
+            "source_width": source_width,
+            "source_height": source_height,
+            "seed": seed,
+            "attention": self.attention,
+            "turbo": self.turbo,
+            "upscale": upscale_scale,
+            "keep_intermediates": keep_intermediates,
+            "vae_dtype": self.config.vae_dtype,
+            "vae_tile_size": self.config.vae_tile_size,
+            "stages": stages,
+            "cache": cache_stats,
+            "request_seconds": sum(stage["seconds"] for stage in stages),
+        }
--- a/src/h3_blackwell_runtime/sampler.py
+++ b/src/h3_blackwell_runtime/sampler.py
@ -14,13 +14,28 @@ def shifted_sigma(base: torch.Tensor, shift: float) -> torch.Tensor:

 def beta_sigmas(steps: int, *, device: torch.device | str, alpha: float = 0.6, beta: float = 0.6) -> torch.Tensor:
    """Comfy's discrete beta scheduler over H3's 1,000-entry shift-12 table."""
+    import numpy as np
    from scipy.stats import beta as beta_distribution

-    table = shifted_sigma(torch.arange(1, 1001, device=device, dtype=torch.float32) / 1000, 12.0)
-    fractions = 1.0 - torch.arange(steps, device=device, dtype=torch.float64).cpu().numpy() / steps
-    indices = torch.from_numpy((999 * beta_distribution.ppf(fractions, alpha, beta)).round().astype("int64")).to(device)
-    indices = torch.unique_consecutive(indices)
-    return torch.cat((table[indices], table.new_zeros(1)))
+    timesteps = (torch.arange(1, 1001, 1) / 1000) * 1000
+    table = shifted_sigma(timesteps / 1000, 12.0)
+    fractions = 1.0 - np.linspace(0, 1, steps, endpoint=False)
+    indices = np.rint(beta_distribution.ppf(fractions, alpha, beta) * 999)
+    sigmas = []
+    last_index = -1
+    for index in indices:
+        if index != last_index:
+            sigmas.append(float(table[int(index)]))
+        last_index = index
+    return torch.FloatTensor([*sigmas, 0.0]).to(device)
+
+
+def turbo_sigmas(steps: int, shift: float, *, device: torch.device | str) -> torch.Tensor:
+    """Uniform shifted sigma grid used to train the distilled Turbo adapters."""
+    if steps < 1 or shift <= 0:
+        raise ValueError("Turbo steps and shift must be positive")
+    base = torch.linspace(1.0, 0.0, steps + 1, dtype=torch.float32, device="cpu")
+    return shifted_sigma(base, shift).to(device)


 def res_multistep_update(x: torch.Tensor, denoised: torch.Tensor, sigma: torch.Tensor, sigma_down: torch.Tensor, old_denoised: torch.Tensor | None, old_sigma_down: torch.Tensor | None, previous_sigma: torch.Tensor | None) -> torch.Tensor:
@ -57,6 +72,14 @@ def _decode_audio_latent(audio_carried: torch.Tensor, *, shift_video: float = 12
    return audio_carried * (shift_audio / shift_video)


+def _cache_sample(x: torch.Tensor, factor: int) -> torch.Tensor:
+    if factor <= 1:
+        return x
+    if x.ndim == 5:
+        return x[..., ::factor, ::factor]
+    return x[..., ::factor]
+
+
@torch.inference_mode()
 def sample_video_res_multistep(
    model,
@ -69,6 +92,18 @@ def sample_video_res_multistep(
    model_timesteps: list[torch.Tensor] | tuple[torch.Tensor, ...] | None = None,
    return_audio: bool = False,
    progress: bool = False,
+    seed: int = 0,
+    text_token_tags: torch.Tensor | None = None,
+    cond_latents: list[torch.Tensor] | None = None,
+    cond_frame_indices: list[int] | None = None,
+    frame_count: int | None = None,
+    cache_mode: str | None = None,
+    cache_threshold: float = 0.0,
+    cache_start_percent: float = 0.0,
+    cache_end_percent: float = 1.0,
+    cache_subsample_factor: int = 2,
+    cache_stats: dict | None = None,
+    audio_step_trace: list[dict] | None = None,
 ) -> torch.Tensor:
    """Direct H3 beta/RES sampling with Comfy-equivalent joint AV carry semantics."""
    sigmas = beta_sigmas(steps, device=video.device)
@ -77,28 +112,93 @@ def sample_video_res_multistep(
    video_history_sigma = audio_history_sigma = None
    total_steps = len(sigmas) - 1
    started = time.perf_counter()
+    cache_mode = None if cache_mode in {None, "", "disabled", "none"} else cache_mode
+    if cache_mode not in {None, "easycache", "h3_cache"}:
+        raise ValueError(f"Unsupported cache mode: {cache_mode}")
+    if cache_stats is not None:
+        cache_stats.update({"mode": cache_mode, "threshold": cache_threshold, "skipped_steps": 0, "rates": []})
+    cache = {
+        "video_diff": None,
+        "audio_diff": None,
+        "video_prev": None,
+        "audio_prev": None,
+        "prev_norm": None,
+        "cumulative_rate": 0.0,
+    }
    for index, sigma in enumerate(sigmas[:-1], start=1):
        step_started = time.perf_counter()
        previous_index = index - 1
        sigma_down = sigmas[index]
-        sigma_audio = _audio_sigma(sigma)
-        carry = sigma_audio / sigma
-        native_audio = audio_carried.to(torch.bfloat16) * carry
-        step_timesteps = None if model_timesteps is None else model_timesteps[previous_index]
-        hidden, times, segments, positions, video_segment, audio_segment = packer(text, video, native_audio, _model_sigma(sigma), step_timesteps)
-        raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
-        raw_video = raw_video.to(torch.bfloat16).float()
-        raw_audio = raw_audio.to(torch.bfloat16)
-        velocity_video = -unpatchify_video(raw_video, video.shape[2], video.shape[-2], video.shape[-1])
-        velocity_audio = (
-            (1.0 - 4.0) * (audio_carried.to(torch.bfloat16) * carry.to(torch.bfloat16))
-            + (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(raw_audio))
-        ).float()
-        video_denoised = video - sigma * velocity_video
-        audio_denoised = audio_carried - sigma * velocity_audio
+        audio_before = audio_carried if audio_step_trace is None else audio_carried.detach().cpu()
+        current_percent = previous_index / total_steps
+        can_cache = cache_mode is not None and cache_threshold > 0 and cache_start_percent <= current_percent <= cache_end_percent and cache["video_diff"] is not None
+        skipped = False
+        if can_cache:
+            video_now = _cache_sample(video, cache_subsample_factor)
+            audio_now = _cache_sample(audio_carried, cache_subsample_factor)
+            input_change = (video_now - cache["video_prev"]).flatten().abs().mean() + (audio_now - cache["audio_prev"]).flatten().abs().mean()
+            input_norm = cache["prev_norm"].clamp_min(1e-8)
+            rate = (input_change / input_norm).item()
+            if cache_mode == "easycache":
+                cache["cumulative_rate"] += rate
+                skipped = cache["cumulative_rate"] < cache_threshold
+                if not skipped:
+                    cache["cumulative_rate"] = 0.0
+            else:
+                skipped = rate < cache_threshold
+            if cache_stats is not None:
+                cache_stats["rates"].append({"step": previous_index, "rate": rate, "skipped": skipped})
+        if skipped:
+            video_denoised = video + cache["video_diff"]
+            audio_denoised = audio_carried + cache["audio_diff"]
+            if cache_stats is not None:
+                cache_stats["skipped_steps"] += 1
+        else:
+            sigma_audio = _audio_sigma(sigma)
+            carry = sigma_audio / sigma
+            native_audio = audio_carried.to(torch.bfloat16) * carry
+            step_timesteps = None if model_timesteps is None else model_timesteps[previous_index]
+            hidden, times, segments, positions, video_segment, audio_segment = packer(
+                text,
+                video,
+                native_audio,
+                _model_sigma(sigma),
+                step_timesteps,
+                text_token_tags=text_token_tags,
+                cond_latents=cond_latents,
+                cond_frame_indices=cond_frame_indices,
+                frame_count=frame_count,
+                seed=seed,
+            )
+            raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
+            raw_video = raw_video.to(torch.bfloat16).float()
+            raw_audio = raw_audio.to(torch.bfloat16)
+            velocity_video = -unpatchify_video(raw_video, video.shape[2], video.shape[-2], video.shape[-1])
+            velocity_audio = (
+                (1.0 - 4.0) * (audio_carried.to(torch.bfloat16) * carry.to(torch.bfloat16))
+                + (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(raw_audio))
+            ).float()
+            video_denoised = video - sigma * velocity_video
+            audio_denoised = audio_carried - sigma * velocity_audio
+            if cache_mode is not None:
+                cache["video_diff"] = (video_denoised - video).detach()
+                cache["audio_diff"] = (audio_denoised - audio_carried).detach()
+                cache["video_prev"] = _cache_sample(video, cache_subsample_factor).detach().clone()
+                cache["audio_prev"] = _cache_sample(audio_carried, cache_subsample_factor).detach().clone()
+                cache["prev_norm"] = video.flatten().abs().mean() + audio_carried.flatten().abs().mean()
        previous_sigma = sigmas[previous_index - 1] if previous_index else None
        video = res_multistep_update(video, video_denoised, sigma, sigma_down, video_history, video_history_sigma, previous_sigma)
        audio_carried = res_multistep_update(audio_carried, audio_denoised, sigma, sigma_down, audio_history, audio_history_sigma, previous_sigma)
+        if audio_step_trace is not None:
+            audio_step_trace.append({
+                "step": index,
+                "video_sigma": float(sigma),
+                "audio_sigma": float(_audio_sigma(sigma)),
+                "video_sigma_down": float(sigma_down),
+                "audio_before": audio_before,
+                "audio_denoised": audio_denoised.detach().cpu(),
+                "audio_after": audio_carried.detach().cpu(),
+            })
        video_history, audio_history = video_denoised, audio_denoised
        video_history_sigma = audio_history_sigma = sigma_down
        if progress:
@ -112,6 +212,51 @@ def sample_video_res_multistep(
    return (video, _decode_audio_latent(audio_carried)) if return_audio else video


+@torch.inference_mode()
+def sample_video_turbo(
+    model,
+    packer: H3PromptPacker,
+    text: torch.Tensor,
+    video: torch.Tensor,
+    audio: torch.Tensor,
+    *,
+    steps: int,
+    video_shift: float,
+    audio_shift: float = 3.0,
+    return_audio: bool = False,
+    seed: int = 0,
+    text_token_tags: torch.Tensor | None = None,
+    cond_latents: list[torch.Tensor] | None = None,
+    cond_frame_indices: list[int] | None = None,
+    frame_count: int | None = None,
+) -> torch.Tensor:
+    """Run the distilled H3 training-Euler schedule on independent AV clocks."""
+    video_schedule = turbo_sigmas(steps, video_shift, device=video.device)
+    audio_schedule = turbo_sigmas(steps, audio_shift, device=audio.device)
+    for index in range(steps):
+        video_sigma = video_schedule[index]
+        audio_sigma = audio_schedule[index]
+        model_timesteps = torch.stack((1.0 - video_sigma, 1.0 - audio_sigma))
+        hidden, times, segments, positions, video_segment, audio_segment = packer(
+            text,
+            video,
+            audio,
+            video_sigma,
+            model_timesteps,
+            text_token_tags=text_token_tags,
+            cond_latents=cond_latents,
+            cond_frame_indices=cond_frame_indices,
+            frame_count=frame_count,
+            seed=seed,
+        )
+        raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
+        video_output = unpatchify_video(raw_video.to(torch.bfloat16).float(), video.shape[2], video.shape[-2], video.shape[-1])
+        audio_output = _unpack_audio(raw_audio.to(torch.bfloat16)).float()
+        video = video.float() + (video_sigma - video_schedule[index + 1]) * video_output
+        audio = audio.float() + (audio_sigma - audio_schedule[index + 1]) * audio_output
+    return (video, audio) if return_audio else video
+
+
@torch.inference_mode()
 def sample_video_euler(model, packer: H3PromptPacker, text: torch.Tensor, video: torch.Tensor, audio: torch.Tensor, *, steps: int = 2) -> torch.Tensor:
    """Use Euler updates to obtain a visual-only H3 preview, not parity sampling."""
--- a/src/h3_blackwell_runtime/tensor_parallel.py
+++ b/src/h3_blackwell_runtime/tensor_parallel.py
@ -0,0 +1,101 @@
+"""True NVFP4 tensor-parallel sharding for the H3 denoiser."""
+
+from __future__ import annotations
+
+import torch
+
+from .distributed import SequenceParallelContext, balanced_ranges
+from .nvfp4 import Nvfp4Linear, Nvfp4LinearTensors
+
+
+def aligned_balanced_ranges(total: int, parts: int, alignment: int) -> tuple[tuple[int, int], ...]:
+    """Balance ranges in indivisible alignment-sized groups."""
+    if alignment < 1 or total % alignment:
+        raise ValueError(f"total {total} must be divisible by alignment {alignment}")
+    groups = balanced_ranges(total // alignment, parts)
+    return tuple((start * alignment, stop * alignment) for start, stop in groups)
+
+
+def _reject_lora(linear: Nvfp4Linear) -> None:
+    if len(linear.lora_branches):
+        raise ValueError("tensor-parallel sharding must occur before loading Turbo LoRA branches")
+
+
+def select_nvfp4_outputs(linear: Nvfp4Linear, ranges: tuple[tuple[int, int], ...]) -> Nvfp4Linear:
+    """Create one column-parallel NVFP4 linear from selected output-row ranges."""
+    _reject_lora(linear)
+    indices = torch.cat([
+        torch.arange(start, stop, device=linear.weight.device)
+        for start, stop in ranges
+    ])
+    bias = None if linear.bias is None else linear.bias.index_select(0, indices)
+    tensors = Nvfp4LinearTensors(
+        weight=linear.weight.index_select(0, indices).contiguous(),
+        weight_scale=linear.weight_scale.index_select(0, indices).contiguous(),
+        weight_scale_2=linear.weight_scale_2,
+        bias=bias.contiguous() if bias is not None else None,
+        pre_quant_scale=linear.pre_quant_scale,
+        full_precision_matrix_mult=linear.full_precision_matrix_mult,
+        in_features=linear.in_features,
+        out_features=indices.numel(),
+    )
+    return Nvfp4Linear(tensors, output_dtype=linear.output_dtype)
+
+
+def slice_nvfp4_inputs(linear: Nvfp4Linear, start: int, stop: int) -> tuple[Nvfp4Linear, torch.Tensor | None]:
+    """Create one row-parallel NVFP4 linear and return its once-only output bias."""
+    _reject_lora(linear)
+    if start < 0 or stop > linear.in_features or start >= stop:
+        raise ValueError(f"invalid input shard [{start}, {stop}) for width {linear.in_features}")
+    if start % 32 or stop % 32:
+        raise ValueError("NVFP4 input shards must align to 32 features")
+    bias = linear.bias
+    tensors = Nvfp4LinearTensors(
+        weight=linear.weight[:, start // 2 : stop // 2].contiguous(),
+        weight_scale=linear.weight_scale[:, start // 16 : stop // 16].contiguous(),
+        weight_scale_2=linear.weight_scale_2,
+        bias=None,
+        pre_quant_scale=(
+            None if linear.pre_quant_scale is None
+            else linear.pre_quant_scale[start:stop].contiguous()
+        ),
+        full_precision_matrix_mult=linear.full_precision_matrix_mult,
+        in_features=stop - start,
+        out_features=linear.out_features,
+    )
+    return Nvfp4Linear(tensors, output_dtype=linear.output_dtype), bias
+
+
+def configure_h3_tensor_parallel(model, context: SequenceParallelContext) -> None:
+    """Shard all denoiser attention and MLP linears in place across ranks."""
+    for block in model.backbone.blocks:
+        attention = block.attention
+        global_heads = attention.heads
+        if global_heads != context.heads or attention.head_dim != context.head_dim:
+            raise ValueError("tensor-parallel context does not match H3 attention dimensions")
+        head_start, head_stop = context.local_head_range
+        inner = global_heads * attention.head_dim
+        local_start = head_start * attention.head_dim
+        local_stop = head_stop * attention.head_dim
+        attention.qkv_proj = select_nvfp4_outputs(attention.qkv_proj, (
+            (local_start, local_stop),
+            (inner + local_start, inner + local_stop),
+            (2 * inner + local_start, 2 * inner + local_stop),
+        ))
+        attention.out_proj, output_bias = slice_nvfp4_inputs(
+            attention.out_proj, local_start, local_stop,
+        )
+        attention.heads = context.local_head_count
+        attention.register_buffer("tensor_parallel_output_bias", output_bias, persistent=False)
+
+        mlp = block.mlp
+        intermediate = mlp.fc2.in_features
+        mlp_ranges = aligned_balanced_ranges(intermediate, context.world_size, 32)
+        mlp_start, mlp_stop = mlp_ranges[context.rank]
+        mlp.fc1 = select_nvfp4_outputs(mlp.fc1, (
+            (mlp_start, mlp_stop),
+            (intermediate + mlp_start, intermediate + mlp_stop),
+        ))
+        mlp.fc2, output_bias = slice_nvfp4_inputs(mlp.fc2, mlp_start, mlp_stop)
+        mlp.register_buffer("tensor_parallel_output_bias", output_bias, persistent=False)
+        mlp.tensor_parallel_intermediate_ranges = mlp_ranges
--- a/src/h3_blackwell_runtime/token_refiner.py
+++ b/src/h3_blackwell_runtime/token_refiner.py
@ -6,15 +6,19 @@ from torch import nn

 from .attention import rms_norm, run_attention
 from .checkpoint import H3Checkpoint
+from .lora import DynamicLoraMixin


-class _Linear(nn.Module):
+class _Linear(DynamicLoraMixin, nn.Module):
    def __init__(self, checkpoint: H3Checkpoint, prefix: str, dtype: torch.dtype):
        super().__init__()
        self.register_buffer("weight", checkpoint.tensor(f"{prefix}.weight", dtype=dtype), persistent=False)
+        self.in_features = self.weight.shape[1]
+        self.out_features = self.weight.shape[0]
+        self._init_dynamic_lora()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
-        return functional.linear(x, self.weight)
+        return self._apply_lora(x, functional.linear(x, self.weight))


 class _RefinerBlock(nn.Module):
--- a/src/h3_blackwell_runtime/vae_decoder.py
+++ b/src/h3_blackwell_runtime/vae_decoder.py
@ -11,6 +11,11 @@ from safetensors import safe_open
 from torch import nn
 from torch.nn import functional as F

+try:
+    import comfy_kitchen  # noqa: F401
+except Exception:
+    comfy_kitchen = None
+

 IMAGENET_MEAN = (0.485, 0.456, 0.406)
 IMAGENET_STD = (0.229, 0.224, 0.225)
@ -18,9 +23,30 @@ LATENTS_MEAN = (0.858090341091156, -0.9606591463088989, 1.0661640167236328, -0.5
 LATENTS_STD = (1.2223774194717407, 1.2767263650894165, 1.68317747116088865, 1.7549455165863037, 1.5636216402053833, 2.194143533706665, 0.96531379222869875, 1.05698859691619875, 0.841948926448822, 0.7729952931404114, 1.8955937623977661, 0.946841835975647, 0.7996809482574463, 0.44988900423049925, 0.7197399735450745, 0.69362932443618775, 2.961095094680786, 2.7694199085235595, 3.0496184825897215, 2.1088054180145265, 3.276226282119751, 3.1627357006073, 2.28168129920959475, 2.6127843856811525)


+def dtype_from_name(name: str) -> torch.dtype:
+    return {
+        "float32": torch.float32,
+        "float16": torch.float16,
+        "bfloat16": torch.bfloat16,
+    }[name]
+
+
 def _rms_norm(x: torch.Tensor, weight: torch.Tensor | None, eps: float) -> torch.Tensor:
-    result = x * torch.rsqrt(x.float().square().mean(dim=-1, keepdim=True) + eps).to(x.dtype)
-    return result if weight is None else result * weight.to(dtype=x.dtype)
+    if weight is None:
+        return F.rms_norm(x, (x.shape[-1],), eps=eps)
+    return F.rms_norm(x, weight.shape, weight=weight.to(device=x.device, dtype=x.dtype), eps=eps)
+
+
+def _conv3d(x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor | None) -> torch.Tensor:
+    if x.is_cuda and weight.dtype in (torch.float16, torch.bfloat16):
+        output = torch.cudnn_convolution(
+            x, weight, (0, 0, 0), (1, 1, 1), (1, 1, 1), 1,
+            benchmark=False, deterministic=False, allow_tf32=True,
+        )
+        if bias is not None:
+            output += bias.reshape(1, -1, 1, 1, 1)
+        return output
+    return F.conv3d(x, weight, bias)


 def create_token_ids(patch_dims: tuple[int, int, int], device: torch.device, dtype: torch.dtype) -> torch.Tensor:
@ -33,16 +59,13 @@ class RotaryEmbeddingND(nn.Module):
        super().__init__()
        self.rotary_base = rotary_base
        self.step = 2 * n_dim / dim
-        inv_freq = 1 / rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32, device=device)
+        inv_freq = 1 / rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.angle_scale = 2.0 * math.pi

    def forward(self, img_ids: torch.Tensor) -> torch.Tensor:
        inv_freq = self.inv_freq
-        if inv_freq.device.type == "meta":
-            inv_freq = 1 / self.rotary_base ** torch.arange(0, 1, self.step, dtype=torch.float32, device=img_ids.device)
-        else:
-            inv_freq = inv_freq.to(img_ids.device)
+        inv_freq = inv_freq.to(device=img_ids.device, dtype=img_ids.dtype)
        angles = self.angle_scale * img_ids[:, :, :, None].float() * inv_freq[None, None, None, :]
        angles = angles.flatten(2, 3)
        cos, sin = torch.cos(angles), torch.sin(angles)
@ -72,6 +95,10 @@ class FeedForward(nn.Module):

 def _apply_rope_split_half(x: torch.Tensor, table: torch.Tensor) -> torch.Tensor:
    """Apply the reference split-half RoPE layout to leading rotary channels."""
+    try:
+        return torch.ops.comfy_kitchen.apply_rope_split_half1(x, table)
+    except Exception:
+        pass
    pairs = table.shape[-3]
    rot = pairs * 2
    first, second = x[..., :pairs], x[..., pairs:rot]
@ -95,7 +122,13 @@ class Attention(nn.Module):
        qkv = self.to_qkv(x).view(batch, sequence, self.heads, 3 * self.dim_head)
        query, key, value = qkv.chunk(3, dim=-1)
        query, key = self.norm_q(query), self.norm_k(key)
-        query, key = _apply_rope_split_half(query, rotary_pos_emb), _apply_rope_split_half(key, rotary_pos_emb)
+        try:
+            rot = rotary_pos_emb.shape[-3] * 2
+            query_rot, key_rot = torch.ops.comfy_kitchen.apply_rope_split_half(query[..., :rot], key[..., :rot], rotary_pos_emb)
+            query = torch.cat((query_rot, query[..., rot:]), dim=-1)
+            key = torch.cat((key_rot, key[..., rot:]), dim=-1)
+        except Exception:
+            query, key = _apply_rope_split_half(query, rotary_pos_emb), _apply_rope_split_half(key, rotary_pos_emb)
        query, key, value = query.transpose(1, 2), key.transpose(1, 2), value.transpose(1, 2)
        try:
            from comfy.ldm.modules.attention import optimized_attention
@ -175,7 +208,7 @@ class MiniMaxH3VideoVAE(nn.Module):
        self.register_buffer("pixel_std", torch.tensor(IMAGENET_STD, device=device).view(1, 3, 1, 1, 1), persistent=False)

    @classmethod
-    def from_safetensors(cls, path: str | Path, *, device: str | torch.device = "cuda", tiling: bool = True, dtype: torch.dtype = torch.float32) -> "MiniMaxH3VideoVAE":
+    def from_safetensors(cls, path: str | Path, *, device: str | torch.device = "cuda", tiling: bool = True, dtype: torch.dtype = torch.float16) -> "MiniMaxH3VideoVAE":
        model = cls(device="meta", tiling=tiling)
        expected = model.state_dict()
        if os.getenv("H3_FAST_SAFETENSORS", "").lower() in {"1", "true", "yes", "on"}:
@ -219,7 +252,7 @@ class MiniMaxH3VideoVAE(nn.Module):
        return model

    def _decode_pixels(self, z: torch.Tensor) -> torch.Tensor:
-        return self.decoder(self.post_quant_conv(z))
+        return self.decoder(_conv3d(z, self.post_quant_conv.weight, self.post_quant_conv.bias))

    def split_tiles(self, length: int) -> tuple[list[int], list[int], list[int]]:
        if self.tile_size >= length:
--- a/src/h3_blackwell_runtime/vae_encoder.py
+++ b/src/h3_blackwell_runtime/vae_encoder.py
@ -0,0 +1,327 @@
+"""Direct, encoder-only MiniMax H3 video VAE implementation.
+
+Mirrors the encoder half of ``upstream_vae.py`` so keyframe/reference images can
+be encoded without ComfyUI. The encoder runs in FP16 like Comfy's keyframe VAE
+path, and latent moments are upcast for mean/std normalization.
+
+Causal-conv semantics: spatial padding is reflect; temporal padding is causal
+(front-only zeros) with a stride grid that starts at the first input frame
+(Comfy autopad "same" / ``causal``). For a single input frame the temporal taps
+of the kernel are truncated (Comfy ``autopad="causal_zero"``) so a keyframe is
+never convolved against zero frames.
+
+Weights are read straight from the checkpoint's ``encoder.*`` / ``quant_conv.*``
+keys (a direct name-for-name copy into plain tensors) and applied by the
+stateless kernels below.
+"""
+
+from __future__ import annotations
+
+import math
+import os
+from pathlib import Path
+
+import torch
+from safetensors import safe_open
+from torch import nn
+from torch.nn import functional as F
+
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+LATENTS_MEAN = (0.858090341091156, -0.9606591463088989, 1.0661640167236328, -0.5090325474739075, -0.2727581858634949, -1.3675414323806763, -0.2553254961967468, -0.26907554268836975, -0.5376840829849243, -0.0464097298681736, 0.6657370328903198, 0.19690127670764923, -0.5460608005523682, -0.4035342037677765, -0.23683024942874908, 0.25928452610969543, -0.30133944749832153, 0.211341992020607, -1.1206848621368408, 0.3581933379173279, -0.04225143790245056, 0.2604829967021942, 0.22864092886447906, 0.7056031823158264)
+LATENTS_STD = (1.2223774194717407, 1.2767263650894165, 1.68317747116088865, 1.7549455165863037, 1.5636216402053833, 2.194143533706665, 0.96531379222869875, 1.05698859691619875, 0.841948926448822, 0.7729952931404114, 1.8955937623977661, 0.946841835975647, 0.7996809482574463, 0.44988900423049925, 0.7197399735450745, 0.69362932443618775, 2.961095094680786, 2.7694199085235595, 3.0496184825897215, 2.1088054180145265, 3.276226282119751, 3.1627357006073, 2.28168129920959475, 2.6127843856811525)
+
+CH = 128
+CH_MULT = (1, 2, 2, 4, 4, 8)
+SPACE_DOWN = (2, 2, 2, 2, 1, 1)
+TIME_DOWN = (1, 2, 2, 1, 1, 1)
+NUM_RES_BLOCKS = 2
+VAE_RATIO = 16
+NIN_LEVELS = frozenset({1, 3, 5})
+DOWNSAMPLE_LEVELS = frozenset({0, 1, 2, 3})
+
+
+def _conv3d(x, weight, bias, stride=(1, 1, 1), padding=(0, 0, 0)):
+    if x.is_cuda and weight.dtype in (torch.float16, torch.bfloat16):
+        output = torch.cudnn_convolution(
+            x, weight, padding, stride, (1, 1, 1), 1,
+            benchmark=False, deterministic=False, allow_tf32=True,
+        )
+        if bias is not None:
+            output += bias.reshape(1, -1, 1, 1, 1)
+        return output
+    return F.conv3d(x, weight, bias, stride, padding)
+
+
+def _causal_conv3d(x, weight, bias, *, kernel_size, stride, spatial_padding, temporal_causal):
+    """Causal 3D conv (matches upstream_vae.CausalConv3d).
+
+    - ``spatial_padding > 0``: reflect H and W by ``spatial_padding`` on each side.
+    - ``temporal_causal``: front-zero T by ``kernel_size - 1`` (the reference's
+      ``causal_padding[0] * 2`` for ``causal_padding[0]=1``, which is every
+      3D-causal conv in the H3 VAE).  A single-frame input truncates the
+      temporal taps to the final input-aligned tap instead of convolving zero rows.
+    - If neither applies: no padding at all (reference early-return).
+    """
+    if x.shape[2] == 1:
+        # Keyframe path (matches reference's `autopad="causal_zero"`):
+        # apply spatial-reflect pad (if any), then run an effective 2D conv
+        # by slicing the kernel to its final input-aligned temporal tap.
+        if spatial_padding > 0:
+            x = F.pad(x, (spatial_padding, spatial_padding, spatial_padding, spatial_padding, 0, 0), mode="reflect")
+        kernel_5d = weight[:, :, -1:, :, :]
+        return _conv3d(x, kernel_5d, bias, (1, stride[1], stride[2]))
+    if spatial_padding > 0:
+        x = F.pad(x, (spatial_padding, spatial_padding, spatial_padding, spatial_padding, 0, 0), mode="reflect")
+    if temporal_causal:
+        x = F.pad(x, (0, 0, 0, 0, kernel_size - 1, 0))
+    return _conv3d(x, weight, bias, stride)
+
+
+def _group_norm_3d(x, weight, bias):
+    """GroupNorm (32 groups, eps 1e-6) with per-frame statistics."""
+    b, c, t, h, w = x.shape
+    y = F.group_norm(x.permute(0, 2, 1, 3, 4).contiguous().view(b * t, c, 1, h, w), 32, weight, bias, 1e-6)
+    return y.view(b, t, c, h, w).permute(0, 2, 1, 3, 4).contiguous()
+
+
+def _resnet(x, p):
+    # nin_shortcut uses CausalConv3d(k=1, padding=1) in the reference.
+    residual = x if p["nin"] is None else _conv3d(x, p["nin"][0], p["nin"][1])
+    h = _causal_conv3d(F.silu(_group_norm_3d(x, p["norm1_w"], p["norm1_b"])), p["conv1_w"], p["conv1_b"], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+    h = _causal_conv3d(F.silu(_group_norm_3d(h, p["norm2_w"], p["norm2_b"])), p["conv2_w"], p["conv2_b"], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+    return h.add_(residual)
+
+
+def _downsample(x, p):
+    if p["space"] == 2:
+        # Reference Downsample3D pads only the right and bottom edges.
+        x = F.pad(x, (0, 1, 0, 1, 0, 0), mode="reflect")
+    # Conv uses padding=(1,0,0) -> causal_padding=(1,0,0), so spatial pad=0,
+    # temporal front-zero is applied.
+    return _causal_conv3d(x, p["w"], p["b"], kernel_size=3, stride=(p["time"], p["space"], p["space"]), spatial_padding=0, temporal_causal=True)
+
+
+def _encoder_run(x, E):
+    h = _causal_conv3d(x, E["conv_in"][0], E["conv_in"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+    for level in E["down"]:
+        for blk in level["blocks"]:
+            h = _resnet(h, blk)
+        if level["down"] is not None:
+            h = _downsample(h, level["down"])
+    h = F.silu(_group_norm_3d(h, E["norm_out_w"], E["norm_out_b"]))
+    return _causal_conv3d(h, E["conv_out"][0], E["conv_out"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+
+
+class MiniMaxH3VideoVAEEncoder(nn.Module):
+    """Encoder-only H3 VAE. ``encode`` matches the public contract of upstream_vae.py.
+
+    Weights are plain tensors loaded from the checkpoint by canonical name into
+    ``self.W`` (a dict), so no ``nn.Module`` sub-hierarchy is needed.
+    """
+
+    def __init__(self, *, tiling: bool = True, compute_dtype: torch.dtype = torch.float16):
+        super().__init__()
+        self.compute_dtype = compute_dtype
+        self.vae_ratio, self.vae_ratio_t = VAE_RATIO, 4
+        self.clip_length, self.token_drop = 17, 3
+        self.frame_pre_padding = (-self.clip_length) % self.vae_ratio_t
+        self.tiling, self.tile_size, self.tile_overlap_min = tiling, 256, 64
+        self.quant_conv = nn.Conv3d(48, 48, 1)
+        self.register_buffer("latents_mean", torch.tensor(LATENTS_MEAN), persistent=False)
+        self.register_buffer("latents_std", torch.tensor(LATENTS_STD), persistent=False)
+        self.register_buffer("pixel_mean", torch.tensor(IMAGENET_MEAN).view(1, 3, 1, 1, 1), persistent=False)
+        self.register_buffer("pixel_std", torch.tensor(IMAGENET_STD).view(1, 3, 1, 1, 1), persistent=False)
+
+    def _required_encoder_names(self) -> list[str]:
+        names = [
+            "encoder.conv_in.weight", "encoder.conv_in.bias",
+            "encoder.norm_out.weight", "encoder.norm_out.bias",
+            "encoder.conv_out.weight", "encoder.conv_out.bias",
+            "quant_conv.weight", "quant_conv.bias",
+            "latents_mean", "latents_std",
+        ]
+        for i in range(len(CH_MULT)):
+            for b in range(NUM_RES_BLOCKS):
+                base = f"encoder.down.{i}.block.{b}."
+                names += [
+                    base + "conv1.weight", base + "conv1.bias",
+                    base + "conv2.weight", base + "conv2.bias",
+                    base + "norm1.weight", base + "norm1.bias",
+                    base + "norm2.weight", base + "norm2.bias",
+                ]
+                if b == 0 and i in NIN_LEVELS:
+                    names += [base + "nin_shortcut.weight", base + "nin_shortcut.bias"]
+            if i in DOWNSAMPLE_LEVELS:
+                names += [f"encoder.down.{i}.downsample.conv.weight", f"encoder.down.{i}.downsample.conv.bias"]
+        return names
+
+    @classmethod
+    def from_safetensors(cls, path: str | Path, *, device: str | torch.device = "cuda", tiling: bool = True, dtype: torch.dtype = torch.float16) -> "MiniMaxH3VideoVAEEncoder":
+        model = cls(tiling=tiling, compute_dtype=dtype)
+        names = model._required_encoder_names()
+        if os.getenv("H3_FAST_SAFETENSORS", "").lower() in {"1", "true", "yes", "on"}:
+            from fastsafetensors import fastsafe_open
+
+            fast_device = "cuda:0" if str(device) == "cuda" else str(device)
+            with fastsafe_open(filenames=[str(path)], nogds=True, device=fast_device) as ck:
+                available = set(ck.keys())
+                missing = [n for n in names if n not in available]
+                if missing:
+                    raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
+                W = {n: ck.get_tensor(n).clone().detach().to(dtype=dtype) for n in names}
+        elif os.getenv("H3_DISABLE_MMAP", "").lower() in {"1", "true", "yes", "on"}:
+            from safetensors.torch import load
+
+            with open(path, "rb") as file:
+                available_weights = load(file.read())
+            available = set(available_weights)
+            missing = [n for n in names if n not in available]
+            if missing:
+                raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
+            W = {n: available_weights[n].to(device=device, dtype=dtype) for n in names}
+            del available_weights
+        else:
+            with safe_open(str(path), framework="pt", device=str(device)) as ck:
+                available = set(ck.keys())
+                missing = [n for n in names if n not in available]
+                if missing:
+                    raise ValueError(f"incompatible H3 VAE checkpoint; missing: {', '.join(sorted(missing)[:16])}")
+                W = {n: ck.get_tensor(n).to(dtype=dtype).to(device) for n in names}
+
+        # Build the structured params dict.
+        down = []
+        for i in range(len(CH_MULT)):
+            mid = CH * CH_MULT[i]
+            blocks = []
+            for b in range(NUM_RES_BLOCKS):
+                base = f"encoder.down.{i}.block.{b}."
+                blk = {
+                    "conv1_w": W[base + "conv1.weight"], "conv1_b": W[base + "conv1.bias"],
+                    "conv2_w": W[base + "conv2.weight"], "conv2_b": W[base + "conv2.bias"],
+                    "norm1_w": W[base + "norm1.weight"], "norm1_b": W[base + "norm1.bias"],
+                    "norm2_w": W[base + "norm2.weight"], "norm2_b": W[base + "norm2.bias"],
+                    "nin": None,
+                }
+                if base + "nin_shortcut.weight" in W:
+                    blk["nin"] = (W[base + "nin_shortcut.weight"], W[base + "nin_shortcut.bias"])
+                blocks.append(blk)
+            level_down = None
+            if i in DOWNSAMPLE_LEVELS:
+                ds = f"encoder.down.{i}.downsample.conv."
+                level_down = {"w": W[ds + "weight"], "b": W[ds + "bias"], "time": TIME_DOWN[i], "space": SPACE_DOWN[i]}
+            down.append({"blocks": blocks, "down": level_down})
+
+        E = {
+            "conv_in": (W["encoder.conv_in.weight"], W["encoder.conv_in.bias"]),
+            "down": down,
+            "norm_out_w": W["encoder.norm_out.weight"], "norm_out_b": W["encoder.norm_out.bias"],
+            "conv_out": (W["encoder.conv_out.weight"], W["encoder.conv_out.bias"]),
+        }
+        model.W = E
+        model.quant_conv.to(device, dtype)
+        with torch.no_grad():
+            model.quant_conv.weight.copy_(W["quant_conv.weight"])
+            model.quant_conv.bias.copy_(W["quant_conv.bias"])
+            model.latents_mean.copy_(W["latents_mean"].float().cpu())
+            model.latents_std.copy_(W["latents_std"].float().cpu())
+        for b in ("latents_mean", "latents_std", "pixel_mean", "pixel_std"):
+            getattr(model, b).to(device)
+        return model
+
+    @torch.inference_mode()
+    def _encode_moments(self, x: torch.Tensor) -> torch.Tensor:
+        return _conv3d(_encoder_run(x.to(self.compute_dtype), self.W), self.quant_conv.weight, self.quant_conv.bias)
+
+    def _adaptive_encode(self, x: torch.Tensor) -> torch.Tensor:
+        if self.tiling:
+            return self.tiled_encode(x)
+        return self._encode_moments(x)
+
+    def split_tiles(self, length: int) -> tuple[list[int], list[int], list[int]]:
+        if self.tile_size >= length:
+            return [0], [length], []
+        count = math.ceil(length / self.tile_size)
+        while self.tile_size * count - self.tile_overlap_min * (count - 1) < length:
+            count += 1
+        overlaps = [self.tile_overlap_min] * (count - 1)
+        for index in range((self.tile_size * count - sum(overlaps) - length) // self.vae_ratio):
+            overlaps[index % len(overlaps)] += self.vae_ratio
+        starts = [0]
+        for overlap in overlaps:
+            starts.append(starts[-1] + self.tile_size - overlap)
+        return starts, [self.tile_size] * count, overlaps
+
+    @staticmethod
+    def blend(a: torch.Tensor, b: torch.Tensor, extent: int, dim: int) -> torch.Tensor:
+        extent = min(a.shape[dim], b.shape[dim], extent)
+        positions = torch.arange(extent, device=b.device, dtype=b.dtype)
+        weight_a = 1 - positions / extent
+        weight_b = positions / extent
+        shape = [1] * a.ndim
+        shape[dim] = extent
+        weight_a = weight_a.view(shape)
+        weight_b = weight_b.view(shape)
+        slice_a = [slice(None)] * a.ndim
+        slice_a[dim] = slice(-extent, None)
+        slice_b = [slice(None)] * a.ndim
+        slice_b[dim] = slice(0, extent)
+        blended = a[tuple(slice_a)] * weight_a + b[tuple(slice_b)] * weight_b
+        if extent < b.shape[dim]:
+            slice_b_rest = [slice(None)] * b.ndim
+            slice_b_rest[dim] = slice(extent, None)
+            return torch.cat((blended, b[tuple(slice_b_rest)]), dim=dim)
+        return blended
+
+    def tiled_encode(self, x: torch.Tensor) -> torch.Tensor:
+        height, width = x.shape[-2], x.shape[-1]
+        y_idx, y_len, y_overlap = self.split_tiles(height)
+        x_idx, x_len, x_overlap = self.split_tiles(width)
+        rows = [[self._encode_moments(x[..., i_pos:i_pos + i_len, j_pos:j_pos + j_len]) for j_pos, j_len in zip(x_idx, x_len)] for i_pos, i_len in zip(y_idx, y_len)]
+        latent_y_overlap = [o // self.vae_ratio for o in y_overlap]
+        latent_x_overlap = [o // self.vae_ratio for o in x_overlap]
+        result_rows = []
+        for i, row in enumerate(rows):
+            result_row = []
+            for j, tile in enumerate(row):
+                if i > 0:
+                    tile = self.blend(rows[i - 1][j], tile, latent_y_overlap[i - 1], dim=-2)
+                if j > 0:
+                    tile = self.blend(row[j - 1], tile, latent_x_overlap[j - 1], dim=-1)
+                if i < len(rows) - 1:
+                    tile = tile[..., :-latent_y_overlap[i], :]
+                if j < len(row) - 1:
+                    tile = tile[..., :, :-latent_x_overlap[j]]
+                result_row.append(tile)
+            result_rows.append(torch.cat(result_row, dim=-1))
+        return torch.cat(result_rows, dim=-2)
+
+    def encode_temporal(self, x: torch.Tensor) -> torch.Tensor:
+        if x.shape[2] % self.clip_length != 0:
+            pad_size = (-x.shape[2]) % self.clip_length
+            x = torch.cat([x, x[:, :, -1:].repeat(1, 1, pad_size, 1, 1)], dim=2)
+        num_chunks = x.shape[2] // self.clip_length
+        z_list = [self._adaptive_encode(x[:, :, i * self.clip_length : (i + 1) * self.clip_length, :, :]) for i in range(num_chunks)]
+        z = torch.cat(z_list, dim=2)
+        if self.token_drop > 0:
+            z = z[:, :, :-self.token_drop]
+        return z
+
+    def encode(self, x: torch.Tensor) -> torch.Tensor:
+        """``[B,3,H,W]`` or ``[B,3,T,H,W]`` pixels in ``[-1, 1]`` -> normalized latents ``[B,24,T_lat,H//16,W//16]``."""
+        if x.ndim == 4:
+            x = x.unsqueeze(2)
+        # Comfy's VAE wrapper casts before entering the model, so image
+        # normalization rounds in the VAE compute dtype as well.
+        x = x.to(self.compute_dtype)
+        x = x.add(1.0).mul_(0.5).sub_(self.pixel_mean.to(x)).div_(self.pixel_std.to(x))
+        if x.shape[2] == 1:
+            moments = self._adaptive_encode(x)
+            moments = moments[:, :, -1:, :, :]
+        else:
+            moments = self.encode_temporal(x)
+        mean = torch.chunk(moments.float(), 2, dim=1)[0]
+        latents_mean = self.latents_mean.view(1, -1, 1, 1, 1).to(mean)
+        latents_std = self.latents_std.view(1, -1, 1, 1, 1).to(mean)
+        return (mean - latents_mean) / latents_std
--- a/tests/test_distributed.py
+++ b/tests/test_distributed.py
@ -0,0 +1,262 @@
+import tempfile
+from pathlib import Path
+import unittest
+
+import torch
+import torch.distributed as dist
+import torch.multiprocessing as mp
+from torch.nn import functional as F
+
+from h3_blackwell_runtime.distributed import (
+    SequenceParallelContext,
+    balanced_ranges,
+    localize_segments,
+    range_lengths,
+)
+from h3_blackwell_runtime.final import H3FinalLayer
+from h3_blackwell_runtime.nvfp4 import Nvfp4Linear, Nvfp4LinearTensors
+from h3_blackwell_runtime.tensor_parallel import (
+    aligned_balanced_ranges,
+    select_nvfp4_outputs,
+    slice_nvfp4_inputs,
+)
+
+
+def _init_gloo(rank: int, world_size: int, init_file: str) -> None:
+    dist.init_process_group(
+        "gloo",
+        init_method=f"file://{init_file}",
+        rank=rank,
+        world_size=world_size,
+    )
+
+
+def _transport_identity_worker(rank: int, world_size: int, init_file: str) -> None:
+    _init_gloo(rank, world_size, init_file)
+    try:
+        sequence, heads, head_dim = 17, 56, 3
+        context = SequenceParallelContext.create(sequence, heads, head_dim)
+        start, stop = context.local_token_range
+        full = torch.arange(sequence * heads * head_dim, dtype=torch.float32).reshape(1, sequence, heads, head_dim)
+        q = full[:, start:stop].contiguous()
+        k = q + 1_000_000
+        v = q + 2_000_000
+        full_q, full_k, full_v = context.seq_to_heads(q, k, v)
+        torch.testing.assert_close(context.heads_to_seq(full_q), q[0], rtol=0, atol=0)
+        torch.testing.assert_close(context.heads_to_seq(full_k), k[0], rtol=0, atol=0)
+        torch.testing.assert_close(context.heads_to_seq(full_v), v[0], rtol=0, atol=0)
+    finally:
+        dist.destroy_process_group()
+
+
+def _attention_parity_worker(rank: int, world_size: int, init_file: str) -> None:
+    _init_gloo(rank, world_size, init_file)
+    try:
+        torch.manual_seed(440420)
+        sequence, heads, head_dim = 19, 56, 8
+        context = SequenceParallelContext.create(sequence, heads, head_dim)
+        q = torch.randn(1, sequence, heads, head_dim)
+        k = torch.randn_like(q)
+        v = torch.randn_like(q)
+        start, stop = context.local_token_range
+        local_q, local_k, local_v = context.seq_to_heads(
+            q[:, start:stop].contiguous(),
+            k[:, start:stop].contiguous(),
+            v[:, start:stop].contiguous(),
+        )
+        local_heads = F.scaled_dot_product_attention(
+            local_q.transpose(1, 2),
+            local_k.transpose(1, 2),
+            local_v.transpose(1, 2),
+        ).transpose(1, 2)
+        actual = context.heads_to_seq(local_heads)
+        expected = F.scaled_dot_product_attention(
+            q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
+        ).transpose(1, 2)[0, start:stop]
+        torch.testing.assert_close(actual, expected, rtol=1e-5, atol=1e-6)
+    finally:
+        dist.destroy_process_group()
+
+
+def _final_projection_worker(rank: int, world_size: int, init_file: str) -> None:
+    _init_gloo(rank, world_size, init_file)
+    try:
+        sequence, hidden_size = 9, 4
+        context = SequenceParallelContext.create(sequence, heads=4, head_dim=2)
+        hidden = torch.arange(sequence * hidden_size, dtype=torch.float32).reshape(sequence, hidden_size) / 10
+        timesteps = torch.tensor([0.25, 0.75])
+        layer = H3FinalLayer(
+            torch.zeros(1025, 2),
+            torch.ones(hidden_size),
+            torch.zeros(2 * hidden_size, 2),
+            torch.zeros(2 * hidden_size),
+            torch.arange(3 * hidden_size, dtype=torch.float32).reshape(3, hidden_size) / 10,
+            torch.tensor([0.1, 0.2, 0.3]),
+            torch.arange(2 * hidden_size, dtype=torch.float32).reshape(2, hidden_size) / 20,
+            torch.tensor([-0.1, 0.1]),
+            hidden_size=hidden_size,
+        )
+        video_segment = (3, 9, 0)
+        audio_segment = (0, 3, 1)
+        expected_video, expected_audio = layer(hidden, timesteps, video_segment, audio_segment)
+        start, stop = context.local_token_range
+        actual_video, actual_audio = layer.forward_sequence_parallel(
+            hidden[start:stop], timesteps, video_segment, audio_segment, context,
+        )
+        torch.testing.assert_close(actual_video, expected_video)
+        torch.testing.assert_close(actual_audio, expected_audio)
+    finally:
+        dist.destroy_process_group()
+
+
+def _ragged_row_collectives_worker(rank: int, world_size: int, init_file: str) -> None:
+    _init_gloo(rank, world_size, init_file)
+    try:
+        sequence, features = 17, 5
+        context = SequenceParallelContext.create(sequence, heads=56, head_dim=2)
+        full = torch.arange(sequence * features, dtype=torch.float32).reshape(sequence, features)
+        start, stop = context.local_token_range
+        gathered = context.all_gather_rows(full[start:stop].contiguous())
+        torch.testing.assert_close(gathered, full, rtol=0, atol=0)
+        partial = full * float(rank + 1)
+        reduced = context.reduce_scatter_rows(partial)
+        expected = full[start:stop] * sum(range(1, world_size + 1))
+        torch.testing.assert_close(reduced, expected, rtol=0, atol=0)
+    finally:
+        dist.destroy_process_group()
+
+
+def _tensor_parallel_math_worker(rank: int, world_size: int, init_file: str) -> None:
+    _init_gloo(rank, world_size, init_file)
+    try:
+        torch.manual_seed(440421)
+        sequence, hidden, heads, head_dim, intermediate = 17, 16, 14, 4, 224
+        context = SequenceParallelContext.create(sequence, heads=heads, head_dim=head_dim)
+        full_x = torch.randn(sequence, hidden)
+        start, stop = context.local_token_range
+        gathered_x = context.all_gather_rows(full_x[start:stop].contiguous())
+
+        inner = heads * head_dim
+        qkv_weight = torch.randn(3 * inner, hidden)
+        output_weight = torch.randn(hidden, inner)
+        output_bias = torch.randn(hidden)
+        head_start, head_stop = context.local_head_range
+        feature_start, feature_stop = head_start * head_dim, head_stop * head_dim
+        indices = torch.cat((
+            torch.arange(feature_start, feature_stop),
+            torch.arange(inner + feature_start, inner + feature_stop),
+            torch.arange(2 * inner + feature_start, 2 * inner + feature_stop),
+        ))
+        local_qkv = F.linear(gathered_x, qkv_weight.index_select(0, indices))
+        local_inner = context.local_head_count * head_dim
+        local_q, local_k, local_v = local_qkv.split(local_inner, dim=-1)
+        local_q = local_q.view(1, sequence, context.local_head_count, head_dim)
+        local_k = local_k.view_as(local_q)
+        local_v = local_v.view_as(local_q)
+        local_attention = F.scaled_dot_product_attention(
+            local_q.transpose(1, 2), local_k.transpose(1, 2), local_v.transpose(1, 2),
+        ).transpose(1, 2).reshape(sequence, local_inner)
+        partial_attention = F.linear(
+            local_attention, output_weight[:, feature_start:feature_stop],
+        )
+        actual_attention = context.reduce_scatter_rows(partial_attention) + output_bias
+
+        full_q, full_k, full_v = F.linear(full_x, qkv_weight).split(inner, dim=-1)
+        full_q = full_q.view(1, sequence, heads, head_dim)
+        full_k = full_k.view_as(full_q)
+        full_v = full_v.view_as(full_q)
+        expected_attention = F.linear(
+            F.scaled_dot_product_attention(
+                full_q.transpose(1, 2), full_k.transpose(1, 2), full_v.transpose(1, 2),
+            ).transpose(1, 2).reshape(sequence, inner),
+            output_weight,
+            output_bias,
+        )[start:stop]
+        torch.testing.assert_close(actual_attention, expected_attention, rtol=2e-5, atol=2e-5)
+
+        fc1_weight = torch.randn(2 * intermediate, hidden)
+        fc2_weight = torch.randn(hidden, intermediate)
+        fc2_bias = torch.randn(hidden)
+        mlp_ranges = aligned_balanced_ranges(intermediate, world_size, 32)
+        mlp_start, mlp_stop = mlp_ranges[rank]
+        local_fc1_weight = torch.cat((
+            fc1_weight[mlp_start:mlp_stop],
+            fc1_weight[intermediate + mlp_start:intermediate + mlp_stop],
+        ))
+        gate, up = F.linear(gathered_x, local_fc1_weight).chunk(2, dim=-1)
+        partial_mlp = F.linear(F.silu(gate) * up, fc2_weight[:, mlp_start:mlp_stop])
+        actual_mlp = context.reduce_scatter_rows(partial_mlp) + fc2_bias
+        full_gate, full_up = F.linear(full_x, fc1_weight).chunk(2, dim=-1)
+        expected_mlp = F.linear(F.silu(full_gate) * full_up, fc2_weight, fc2_bias)[start:stop]
+        # TP reduction changes FP32 accumulation order across rank partials.
+        torch.testing.assert_close(actual_mlp, expected_mlp, rtol=1e-4, atol=2e-4)
+    finally:
+        dist.destroy_process_group()
+
+
+def _run_distributed(worker, world_size: int) -> None:
+    with tempfile.TemporaryDirectory() as directory:
+        init_file = str(Path(directory) / "process-group")
+        mp.spawn(worker, args=(world_size, init_file), nprocs=world_size, join=True)
+
+
+class DistributedPartitionContracts(unittest.TestCase):
+    def test_balanced_ragged_ranges(self):
+        ranges = balanced_ranges(56, 6)
+        self.assertEqual(range_lengths(ranges), (10, 10, 9, 9, 9, 9))
+        self.assertEqual(ranges[0], (0, 10))
+        self.assertEqual(ranges[-1], (47, 56))
+
+    def test_ranges_reject_empty_partitions(self):
+        with self.assertRaisesRegex(ValueError, "non-empty"):
+            balanced_ranges(3, 4)
+
+    def test_segments_are_clipped_and_rebased(self):
+        segments = [(0, 4, 1), (4, 10, 2), (10, 15, 3)]
+        self.assertEqual(localize_segments(segments, 3, 12), [(0, 1, 1), (1, 7, 2), (7, 9, 3)])
+
+    def test_transport_identity_for_planned_world_sizes(self):
+        for world_size in (2, 4, 6, 8):
+            with self.subTest(world_size=world_size):
+                _run_distributed(_transport_identity_worker, world_size)
+
+    def test_two_rank_sdpa_matches_single_process(self):
+        _run_distributed(_attention_parity_worker, 2)
+
+    def test_distributed_final_projection_matches_single_process(self):
+        _run_distributed(_final_projection_worker, 2)
+
+    def test_ragged_all_gather_and_reduce_scatter(self):
+        _run_distributed(_ragged_row_collectives_worker, 6)
+
+    def test_tensor_parallel_attention_and_mlp_match_dense_math(self):
+        for world_size in (2, 6):
+            with self.subTest(world_size=world_size):
+                _run_distributed(_tensor_parallel_math_worker, world_size)
+
+    def test_nvfp4_column_and_row_shards_preserve_layout(self):
+        tensors = Nvfp4LinearTensors(
+            weight=torch.arange(96 * 32, dtype=torch.int32).to(torch.uint8).reshape(96, 32),
+            weight_scale=torch.arange(96 * 4, dtype=torch.float32).to(torch.float8_e4m3fn).reshape(96, 4),
+            weight_scale_2=torch.tensor(0.5),
+            bias=torch.arange(96, dtype=torch.bfloat16),
+            pre_quant_scale=torch.arange(64, dtype=torch.bfloat16),
+            full_precision_matrix_mult=False,
+            in_features=64,
+            out_features=96,
+        )
+        linear = Nvfp4Linear(tensors)
+        column = select_nvfp4_outputs(linear, ((0, 16), (32, 48)))
+        self.assertEqual(tuple(column.weight.shape), (32, 32))
+        self.assertEqual(tuple(column.weight_scale.shape), (32, 4))
+        self.assertEqual(column.out_features, 32)
+        row, bias = slice_nvfp4_inputs(linear, 32, 64)
+        self.assertEqual(tuple(row.weight.shape), (96, 16))
+        self.assertEqual(tuple(row.weight_scale.shape), (96, 2))
+        self.assertEqual(row.in_features, 32)
+        self.assertIsNone(row.bias)
+        torch.testing.assert_close(bias, tensors.bias)
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/tests/test_fl2va_contracts.py
+++ b/tests/test_fl2va_contracts.py
@ -0,0 +1,270 @@
+import math
+import sys
+from types import SimpleNamespace
+import unittest
+from unittest.mock import patch
+
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from h3_blackwell_runtime.packing import FRAME_RESCALE, H3PromptPacker, _video_t_spans
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, run_attention
+from h3_blackwell_runtime.qwen3vl_vision import (
+    TEXT_HEAD_DIM,
+    TEXT_ROPE_DIMS,
+    TEXT_ROPE_THETA,
+    VISION_HIDDEN,
+    Qwen3VL32BVision,
+    _VisionAttention,
+    _VisionPatchMerger,
+    _apply_rope_vision,
+    _text_run_ids,
+    mrope_freqs_cis,
+)
+from h3_blackwell_runtime.runtime import H3HotRuntime
+from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder, _downsample
+
+
+class Fl2vaVAEContracts(unittest.TestCase):
+    def test_quant_conv_is_a_required_checkpoint_weight(self):
+        names = MiniMaxH3VideoVAEEncoder()._required_encoder_names()
+        self.assertIn("quant_conv.weight", names)
+        self.assertIn("quant_conv.bias", names)
+
+    def test_single_frame_is_encoded_without_temporal_prepad(self):
+        encoder = MiniMaxH3VideoVAEEncoder(tiling=False)
+        seen = []
+
+        def fake_encode(x):
+            seen.append(tuple(x.shape))
+            return torch.zeros((x.shape[0], 48, x.shape[2], 1, 1), device=x.device)
+
+        encoder._adaptive_encode = fake_encode
+        result = encoder.encode(torch.zeros(1, 3, 8, 8))
+        self.assertEqual(seen, [(1, 3, 1, 8, 8)])
+        self.assertEqual(tuple(result.shape), (1, 24, 1, 1, 1))
+
+    def test_downsample_pads_only_right_and_bottom(self):
+        x = torch.arange(16, dtype=torch.float32).reshape(1, 1, 1, 4, 4)
+        weight = torch.ones(1, 1, 3, 3, 3)
+        params = {"w": weight, "b": torch.zeros(1), "time": 1, "space": 2}
+        actual = _downsample(x, params)
+        padded = F.pad(x, (0, 1, 0, 1, 0, 0), mode="reflect")
+        expected = F.conv3d(padded, weight[:, :, -1:], params["b"], stride=(1, 2, 2))
+        torch.testing.assert_close(actual, expected)
+
+    def test_hot_runtime_preserves_zero_to_one_images(self):
+        runtime = H3HotRuntime.__new__(H3HotRuntime)
+        image = torch.tensor([[[[0.0, 0.5, 1.0]]]])
+        converted = runtime._image_to_uint8_nhwc(image)
+        self.assertEqual(converted.flatten().tolist(), [0, 127, 255])
+
+
+class AttentionBackendContracts(unittest.TestCase):
+    def test_hot_backends_include_benchmark_candidates(self):
+        self.assertTrue({"sage2", "cudnn_sdpa", "ck_int8", "flash4"}.issubset(AVAILABLE_BACKENDS))
+
+    def test_flash4_dispatches_bshd_and_restores_hnd(self):
+        q = torch.randn(1, 2, 3, 128, dtype=torch.bfloat16)
+        expected_bshd = torch.randn(1, 3, 2, 128, dtype=torch.bfloat16)
+        flash = unittest.mock.MagicMock(return_value=(expected_bshd, torch.empty(0)))
+        package = SimpleNamespace(cute=SimpleNamespace(flash_attn_func=flash))
+        with (
+            patch.dict(sys.modules, {"flash_attn": package, "flash_attn.cute": package.cute}),
+            patch.object(torch.Tensor, "is_cuda", new_callable=unittest.mock.PropertyMock, return_value=True),
+        ):
+            actual = run_attention(q, q, q, backend="flash4", is_causal=False)
+        torch.testing.assert_close(actual, expected_bshd.transpose(1, 2))
+        flash.assert_called_once()
+        called_q, called_k, called_v = flash.call_args.args
+        self.assertEqual(called_q.shape, (1, 3, 2, 128))
+        self.assertTrue(called_q.is_contiguous())
+        self.assertTrue(called_k.is_contiguous())
+        self.assertTrue(called_v.is_contiguous())
+        self.assertFalse(flash.call_args.kwargs["causal"])
+
+    def test_cudnn_backend_is_forced_without_fallback(self):
+        q = torch.randn(1, 2, 3, 4)
+        expected = torch.randn_like(q)
+        context = unittest.mock.MagicMock()
+        with (
+            patch("torch.nn.attention.sdpa_kernel", return_value=context) as kernel,
+            patch("h3_blackwell_runtime.attention.functional.scaled_dot_product_attention", return_value=expected) as sdpa,
+        ):
+            actual = run_attention(q, q, q, backend="cudnn_sdpa", is_causal=False)
+        self.assertIs(actual, expected)
+        self.assertEqual(kernel.call_args.args[0], [torch.nn.attention.SDPBackend.CUDNN_ATTENTION])
+        sdpa.assert_called_once_with(q, q, q, is_causal=False)
+
+    def test_comfy_kitchen_int8_backend_dispatches_hnd_tensors(self):
+        q = torch.randn(1, 2, 3, 4)
+        expected = torch.randn_like(q)
+        kitchen = SimpleNamespace(int8_attention=unittest.mock.MagicMock(return_value=expected))
+        with patch.dict(sys.modules, {"comfy_kitchen": kitchen}):
+            actual = run_attention(q, q, q, backend="ck_int8", is_causal=False)
+        self.assertIs(actual, expected)
+        kitchen.int8_attention.assert_called_once_with(q, q, q)
+
+    def test_comfy_kitchen_int8_rejects_causal_attention(self):
+        q = torch.randn(1, 2, 3, 4)
+        with self.assertRaisesRegex(ValueError, "does not support causal"):
+            run_attention(q, q, q, backend="ck_int8", is_causal=True)
+
+
+class Fl2vaVisionContracts(unittest.TestCase):
+    def test_visual_rotary_coordinates_are_block_major(self):
+        class CoordinateTable(nn.Module):
+            def forward(self, length):
+                return torch.arange(length, dtype=torch.float32).unsqueeze(1)
+
+        vision = Qwen3VL32BVision.__new__(Qwen3VL32BVision)
+        nn.Module.__init__(vision)
+        vision.spatial_merge_size = 2
+        vision.rotary_pos_emb = CoordinateTable()
+        coordinates = vision.rot_pos_emb(torch.tensor([[1, 4, 4]])).tolist()
+        self.assertEqual(coordinates[:8], [
+            [0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0],
+            [0.0, 2.0], [0.0, 3.0], [1.0, 2.0], [1.0, 3.0],
+        ])
+
+    def test_sdpa_output_is_restored_to_token_major_layout(self):
+        torch.manual_seed(7)
+        sequence, heads, head_dim = 3, 2, 2
+        hidden = heads * head_dim
+        qkv_weight = torch.randn(hidden * 3, hidden)
+        qkv_bias = torch.randn(hidden * 3)
+        proj_weight = torch.randn(hidden, hidden)
+        proj_bias = torch.randn(hidden)
+        module = _VisionAttention(qkv_weight, qkv_bias, proj_weight, proj_bias, num_heads=heads, head_dim=head_dim)
+        x = torch.randn(sequence, hidden)
+        cos = torch.ones(sequence, 1, head_dim)
+        sin = torch.zeros(sequence, 1, head_dim // 2)
+        actual = module(x, torch.tensor([0, sequence], dtype=torch.int32), (cos, sin, sin))
+
+        qkv = F.linear(x, qkv_weight, qkv_bias)
+        query, key, value = qkv.reshape(sequence, 3, heads, head_dim).permute(1, 0, 2, 3).unbind(0)
+        output = F.scaled_dot_product_attention(
+            query.transpose(0, 1).unsqueeze(0),
+            key.transpose(0, 1).unsqueeze(0),
+            value.transpose(0, 1).unsqueeze(0),
+        )
+        expected = F.linear(output.transpose(1, 2).reshape(sequence, hidden), proj_weight, proj_bias)
+        torch.testing.assert_close(actual, expected)
+
+    def test_vision_rope_uses_original_halves(self):
+        q = torch.tensor([[[1.0, 2.0, 3.0, 4.0]]])
+        k = q + 4
+        cos = torch.full_like(q, 0.5)
+        sin = torch.full_like(q[..., :2], 0.25)
+        neg_sin = -sin
+        actual_q, actual_k = _apply_rope_vision(q, k, (cos, sin, neg_sin))
+
+        def expected(x):
+            return torch.cat((x[..., :2] * 0.5 + x[..., 2:] * -0.25,
+                              x[..., 2:] * 0.5 + x[..., :2] * 0.25), dim=-1)
+
+        torch.testing.assert_close(actual_q, expected(q))
+        torch.testing.assert_close(actual_k, expected(k))
+
+    def test_mrope_uses_reference_section_boundaries(self):
+        positions = torch.stack((torch.arange(8), torch.arange(8) + 10, torch.arange(8) + 20))
+        actual = mrope_freqs_cis(positions)[0]
+        inv_freq = 1.0 / (
+            TEXT_ROPE_THETA ** (torch.arange(0, TEXT_HEAD_DIM, 2, dtype=torch.float32) / TEXT_HEAD_DIM)
+        )
+        freqs = (inv_freq[None, :, None].expand(3, -1, 1) @ positions[:, None, :].float()).transpose(1, 2)
+        interleaved = freqs[0].clone()
+        for axis, offset in ((1, 1), (2, 2)):
+            index = slice(offset, TEXT_ROPE_DIMS[axis] * 3, 3)
+            interleaved[..., index] = freqs[axis, ..., index]
+        expected = torch.cat((interleaved, interleaved), dim=-1).cos().unsqueeze(0)
+        torch.testing.assert_close(actual, expected)
+
+    def test_mergers_preserve_existing_block_major_order(self):
+        x = torch.arange(4 * VISION_HIDDEN, dtype=torch.float32).reshape(4, VISION_HIDDEN)
+        passthrough = lambda value, *args, **kwargs: value
+        with (
+            patch("h3_blackwell_runtime.qwen3vl_vision.F.layer_norm", side_effect=passthrough),
+            patch("h3_blackwell_runtime.qwen3vl_vision.F.linear", side_effect=passthrough),
+            patch("h3_blackwell_runtime.qwen3vl_vision.F.gelu", side_effect=passthrough),
+        ):
+            main = _VisionPatchMerger(*(torch.empty(1) for _ in range(6)), merge_size=2,
+                                      out_hidden_size=1, norm_dim=VISION_HIDDEN)
+            deepstack = _VisionPatchMerger(*(torch.empty(1) for _ in range(6)), merge_size=2,
+                                           out_hidden_size=1)
+            torch.testing.assert_close(main(x), x.reshape(1, -1))
+            torch.testing.assert_close(deepstack(x), x.reshape(1, -1))
+
+    def test_configured_tokenizer_preserves_empty_prompt(self):
+        calls = []
+
+        class RawTokenizer:
+            def __call__(self, text, **kwargs):
+                calls.append((text, kwargs))
+                return SimpleNamespace(input_ids=torch.empty((1, 0), dtype=torch.long))
+
+        self.assertEqual(_text_run_ids(SimpleNamespace(tokenizer=RawTokenizer()), ""), [])
+        self.assertEqual(calls[0][0], "")
+        self.assertFalse(calls[0][1]["add_special_tokens"])
+
+
+class _FakeCheckpoint:
+    def tensor(self, name, dtype=None):
+        if name == "video_patch_proj.weight":
+            value = torch.zeros(5376, 96)
+        elif name == "video_patch_proj.bias":
+            value = torch.zeros(5376)
+        elif name == "audio_patch_proj.weight":
+            value = torch.zeros(5376, 32)
+        elif name == "audio_patch_proj.bias":
+            value = torch.zeros(5376)
+        else:
+            value = torch.empty(0)
+        return value.to(dtype=dtype) if dtype is not None else value
+
+
+class Fl2vaPackingContracts(unittest.TestCase):
+    def test_each_keyframe_keeps_its_own_condition_segment(self):
+        packer = H3PromptPacker(_FakeCheckpoint())
+        text = torch.zeros(1, 3, 5376)
+        video = torch.zeros(1, 24, 2, 2, 2)
+        audio = torch.zeros(1, 32, 2, 2)
+        keyframes = [torch.zeros(1, 24, 1, 2, 2) for _ in range(2)]
+        _, _, segments, _, _, _ = packer(
+            text,
+            video,
+            audio,
+            0.5,
+            cond_latents=keyframes,
+            cond_frame_indices=[0, 21],
+            frame_count=22,
+        )
+        self.assertEqual(segments[1][:2], (3, 4))
+        self.assertEqual(segments[2][:2], (4, 5))
+        self.assertEqual(segments[1][2], segments[2][2])
+
+    def test_last_only_anchor_and_targets_share_reference_cursor(self):
+        packer = H3PromptPacker(_FakeCheckpoint())
+        text = torch.zeros(1, 3, 5376)
+        video = torch.zeros(1, 24, 2, 2, 2)
+        audio = torch.zeros(1, 32, 2, 2)
+        last = torch.zeros(1, 24, 1, 2, 2)
+        _, _, _, positions, _, _ = packer(
+            text,
+            video,
+            audio,
+            0.5,
+            cond_latents=[last],
+            cond_frame_indices=[21],
+            frame_count=22,
+            seed=1,
+        )
+        expected_last_t = 3.0 + sum(_video_t_spans(2)) - FRAME_RESCALE
+        self.assertTrue(math.isclose(float(positions[3, 0]), expected_last_t))
+        self.assertEqual(float(positions[4, 0]), 3.0)  # target audio
+        self.assertEqual(float(positions[8, 0]), 3.0)  # target video
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/tests/test_latent_upscaler.py
+++ b/tests/test_latent_upscaler.py
@ -0,0 +1,52 @@
+import unittest
+
+import torch
+
+from h3_blackwell_runtime.latent_upscaler import H3LatentResizer3D, _checkpoint_config, upscale_h3_latent
+from h3_blackwell_runtime.runtime import normalize_upscale
+
+
+class LatentUpscalerContracts(unittest.TestCase):
+    def test_request_upscale_normalization(self):
+        self.assertIsNone(normalize_upscale(None))
+        self.assertIsNone(normalize_upscale("none"))
+        self.assertIsNone(normalize_upscale(1))
+        self.assertEqual(normalize_upscale("2"), 2.0)
+        with self.assertRaisesRegex(ValueError, "greater than 1.0"):
+            normalize_upscale(0.5)
+        with self.assertRaisesRegex(ValueError, "at most 4.0"):
+            normalize_upscale(4.1)
+        with self.assertRaisesRegex(ValueError, "must be a number"):
+            normalize_upscale({"scale": 2})
+
+    def test_3d_model_preserves_time_and_scales_space(self):
+        model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, dropout=0, temporal_every=0).eval()
+        latent = torch.randn(1, 24, 3, 2, 4)
+        output = upscale_h3_latent(model, latent, scale=2)
+        self.assertEqual(output.shape, (1, 24, 3, 4, 8))
+
+    def test_invalid_latent_shape_is_rejected(self):
+        model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, temporal_every=0).eval()
+        with self.assertRaisesRegex(ValueError, r"\[B,24,T,H,W\]"):
+            upscale_h3_latent(model, torch.randn(1, 16, 3, 2, 4))
+
+    def test_scale_one_preserves_latent_exactly(self):
+        model = H3LatentResizer3D(in_blocks=1, out_blocks=1, channels=32, temporal_every=0).eval()
+        latent = torch.randn(1, 24, 3, 2, 4)
+        self.assertTrue(torch.equal(upscale_h3_latent(model, latent, scale=1), latent))
+
+    def test_checkpoint_architecture_detection_matches_module_layout(self):
+        model = H3LatentResizer3D(in_blocks=3, out_blocks=2, channels=32, temporal_every=2, temporal_kernel=3)
+        config = _checkpoint_config(model.state_dict())
+        self.assertEqual(config, {
+            "in_channels": 24,
+            "in_blocks": 3,
+            "out_blocks": 2,
+            "channels": 32,
+            "temporal_every": 2,
+            "temporal_kernel": 3,
+        })
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/tests/test_turbo.py
+++ b/tests/test_turbo.py
@ -0,0 +1,120 @@
+import unittest
+from unittest.mock import patch
+
+import torch
+from torch import nn
+
+from h3_blackwell_runtime.lora import DynamicLoraMixin
+from h3_blackwell_runtime.sampler import sample_video_res_multistep, sample_video_turbo, turbo_sigmas
+
+
+class _Linear(DynamicLoraMixin, nn.Module):
+    def __init__(self, weight):
+        super().__init__()
+        self.register_buffer("weight", weight)
+        self.in_features = weight.shape[1]
+        self.out_features = weight.shape[0]
+        self._init_dynamic_lora()
+
+    def forward(self, x):
+        return self._apply_lora(x, torch.nn.functional.linear(x, self.weight))
+
+
+class TurboLoraContracts(unittest.TestCase):
+    def test_dynamic_lora_matches_unmerged_formula(self):
+        base_weight = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
+        down = torch.tensor([[2.0, -1.0]])
+        up = torch.tensor([[3.0], [-2.0]])
+        x = torch.tensor([[0.5, -1.0]])
+        linear = _Linear(base_weight)
+        linear.add_lora("turbo", down, up, alpha=0.5)
+        linear.set_lora("turbo", strength=0.75)
+        expected = torch.nn.functional.linear(x, base_weight)
+        expected += 0.75 * 0.5 * torch.nn.functional.linear(torch.nn.functional.linear(x, down), up)
+        torch.testing.assert_close(linear(x), expected)
+
+    def test_disabled_lora_returns_exact_base_result(self):
+        weight = torch.randn(3, 2)
+        x = torch.randn(4, 2)
+        linear = _Linear(weight)
+        linear.add_lora("turbo", torch.randn(1, 2), torch.randn(3, 1), alpha=1.0)
+        expected = torch.nn.functional.linear(x, weight)
+        self.assertTrue(torch.equal(linear(x), expected))
+        linear.set_lora("turbo", strength=0.0)
+        self.assertTrue(torch.equal(linear(x), expected))
+
+    def test_unknown_lora_is_rejected(self):
+        linear = _Linear(torch.randn(3, 2))
+        with self.assertRaisesRegex(ValueError, "not attached"):
+            linear.set_lora("missing")
+
+    def test_four_step_shift_six_schedule(self):
+        actual = turbo_sigmas(4, 6.0, device="cpu")
+        expected = torch.tensor([1.0, 0.9473684211, 0.8571428571, 0.6666666667, 0.0])
+        torch.testing.assert_close(actual, expected)
+
+    def test_eight_step_shift_twelve_schedule_has_nine_grid_points(self):
+        actual = turbo_sigmas(8, 12.0, device="cpu")
+        self.assertEqual(actual.shape, (9,))
+        self.assertEqual(actual[0].item(), 1.0)
+        self.assertEqual(actual[-1].item(), 0.0)
+        self.assertTrue(bool(torch.all(actual[:-1] > actual[1:])))
+
+    def test_turbo_sampler_uses_positive_dataward_updates_and_independent_clocks(self):
+        video = torch.zeros(1, 1, 1, 1, 1)
+        audio = torch.zeros(1, 1, 1, 1)
+        calls = []
+
+        def packer(text, current_video, current_audio, sigma, model_timesteps, **kwargs):
+            calls.append((current_video.clone(), current_audio.clone(), model_timesteps.clone()))
+            return (None, None, None, None, None, None)
+
+        def model(*args):
+            return torch.ones(1), torch.ones(1)
+
+        with (
+            patch("h3_blackwell_runtime.sampler.unpatchify_video", return_value=torch.full_like(video, 2.0)),
+            patch("h3_blackwell_runtime.sampler._unpack_audio", return_value=torch.full_like(audio, 3.0)),
+        ):
+            result_video, result_audio = sample_video_turbo(
+                model, packer, torch.empty(0), video, audio,
+                steps=2, video_shift=6.0, audio_shift=3.0, return_audio=True,
+            )
+
+        torch.testing.assert_close(result_video, torch.full_like(video, 2.0))
+        torch.testing.assert_close(result_audio, torch.full_like(audio, 3.0))
+        torch.testing.assert_close(calls[1][0], torch.full_like(video, 2.0 / 7.0))
+        torch.testing.assert_close(calls[1][1], torch.full_like(audio, 0.75))
+        torch.testing.assert_close(calls[1][2], torch.tensor([1.0 / 7.0, 0.25]))
+
+    def test_base_sampler_records_opt_in_audio_step_trace(self):
+        video = torch.zeros(1, 1, 1, 1, 1)
+        audio = torch.zeros(1, 32, 2, 1)
+        trace = []
+
+        def packer(*args, **kwargs):
+            return (None, None, None, None, None, None)
+
+        def model(*args):
+            return torch.ones(1), torch.ones(1)
+
+        with (
+            patch("h3_blackwell_runtime.sampler.beta_sigmas", return_value=torch.tensor([1.0, 0.0])),
+            patch("h3_blackwell_runtime.sampler.unpatchify_video", return_value=torch.zeros_like(video)),
+            patch("h3_blackwell_runtime.sampler._unpack_audio", return_value=torch.ones_like(audio)),
+        ):
+            _video, final_audio = sample_video_res_multistep(
+                model, packer, torch.empty(0), video, audio,
+                steps=1, return_audio=True, audio_step_trace=trace,
+            )
+
+        self.assertEqual(len(trace), 1)
+        self.assertEqual(trace[0]["step"], 1)
+        torch.testing.assert_close(trace[0]["audio_before"], torch.zeros_like(audio))
+        torch.testing.assert_close(trace[0]["audio_denoised"], torch.full_like(audio, 4.0))
+        torch.testing.assert_close(trace[0]["audio_after"], torch.full_like(audio, 4.0))
+        torch.testing.assert_close(final_audio, torch.ones_like(audio))
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/tools/analyze_audio_boundary.py
+++ b/tools/analyze_audio_boundary.py
@ -0,0 +1,123 @@
+"""Compare H3 audio-latent and lossless-waveform boundaries."""
+
+import argparse
+import json
+import math
+import wave
+from pathlib import Path
+
+import numpy as np
+import torch
+
+
+def dbfs(value: float) -> float:
+    return 20.0 * math.log10(max(value, 1e-20))
+
+
+def load_audio_latent(path: Path) -> torch.Tensor:
+    state = torch.load(path, map_location="cpu", weights_only=False)
+    latent = state.get("audio_latent") if isinstance(state, dict) else state
+    if latent is None or latent.ndim != 4:
+        raise ValueError(f"{path} does not contain a [B,C,S,T] audio latent")
+    return latent.float()
+
+
+def latent_metrics(latent: torch.Tensor) -> dict:
+    frames = latent.movedim(-1, 0).flatten(1)
+    frame_rms = frames.square().mean(1).sqrt()
+    frame_mean = frames.mean(1)
+    frame_max = frames.abs().amax(1)
+    deltas = frames[1:] - frames[:-1]
+    delta_rms = deltas.square().mean(1).sqrt()
+    adjacent_cosine = torch.nn.functional.cosine_similarity(frames[:-1], frames[1:], dim=1)
+
+    block_frames = min(4, frames.shape[0] // 2)
+    first = frames[:block_frames].flatten()
+    last = frames[-block_frames:].flatten()
+    first_last_cosine = torch.nn.functional.cosine_similarity(first, last, dim=0)
+
+    first_count = min(20, frames.shape[0])
+    return {
+        "shape": list(latent.shape),
+        "dtype": str(latent.dtype),
+        "first_20_frame_rms": frame_rms[:first_count].tolist(),
+        "first_20_frame_mean": frame_mean[:first_count].tolist(),
+        "first_20_frame_max_abs": frame_max[:first_count].tolist(),
+        "first_19_delta_rms": delta_rms[: max(0, first_count - 1)].tolist(),
+        "first_19_adjacent_cosine": adjacent_cosine[: max(0, first_count - 1)].tolist(),
+        "first_4_rms": float(frames[:block_frames].square().mean().sqrt()),
+        "frames_4_20_rms": float(frames[block_frames:first_count].square().mean().sqrt()),
+        "remaining_rms": float(frames[first_count:].square().mean().sqrt()),
+        "first_4_vs_last_4_cosine": float(first_last_cosine),
+        "largest_delta_frame": int(delta_rms.argmax().item() + 1),
+        "largest_delta_rms": float(delta_rms.max()),
+    }
+
+
+def load_wav(path: Path) -> tuple[np.ndarray, int]:
+    with wave.open(str(path), "rb") as source:
+        if source.getsampwidth() != 2:
+            raise ValueError(f"{path} must be PCM S16")
+        channels = source.getnchannels()
+        sample_rate = source.getframerate()
+        samples = np.frombuffer(source.readframes(source.getnframes()), dtype="<i2")
+    return samples.reshape(-1, channels).astype(np.float32) / 32768.0, sample_rate
+
+
+def waveform_metrics(samples: np.ndarray, sample_rate: int) -> dict:
+    first_half_second = samples[: sample_rate // 2]
+    mono = first_half_second.mean(1)
+    window_samples = sample_rate // 100
+    windows = []
+    for start in range(0, len(first_half_second), window_samples):
+        block = first_half_second[start : start + window_samples]
+        if len(block) == 0:
+            continue
+        windows.append({
+            "start_ms": start * 1000.0 / sample_rate,
+            "peak_dbfs": dbfs(float(np.max(np.abs(block)))),
+            "rms_dbfs": dbfs(float(np.sqrt(np.mean(block * block)))),
+            "mean": float(block.mean()),
+        })
+
+    spectrum_samples = min(sample_rate // 10, len(mono))
+    windowed = mono[:spectrum_samples] * np.hanning(spectrum_samples)
+    magnitudes = np.abs(np.fft.rfft(windowed))
+    frequencies = np.fft.rfftfreq(spectrum_samples, 1.0 / sample_rate)
+    dominant = np.argsort(magnitudes[1:])[-8:][::-1] + 1
+    derivatives = np.max(np.abs(np.diff(first_half_second, axis=0)), axis=1)
+    return {
+        "sample_rate": sample_rate,
+        "samples": len(samples),
+        "first_sample": samples[0].tolist(),
+        "first_sample_dbfs": [dbfs(float(abs(value))) for value in samples[0]],
+        "first_500ms_peak_dbfs": dbfs(float(np.max(np.abs(first_half_second)))),
+        "first_500ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_half_second**2)))),
+        "largest_derivative": float(derivatives.max()),
+        "largest_derivative_ms": float((derivatives.argmax() + 1) * 1000.0 / sample_rate),
+        "dominant_first_100ms_hz": [float(frequencies[index]) for index in dominant],
+        "windows_10ms": windows,
+    }
+
+
+def analyze(latent_path: Path, wav_path: Path) -> dict:
+    samples, sample_rate = load_wav(wav_path)
+    return {
+        "latent_path": str(latent_path),
+        "wav_path": str(wav_path),
+        "latent": latent_metrics(load_audio_latent(latent_path)),
+        "waveform": waveform_metrics(samples, sample_rate),
+    }
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--affected-latent", type=Path, required=True)
+parser.add_argument("--affected-wav", type=Path, required=True)
+parser.add_argument("--clean-latent", type=Path, required=True)
+parser.add_argument("--clean-wav", type=Path, required=True)
+args = parser.parse_args()
+
+print(json.dumps({
+    "affected": analyze(args.affected_latent, args.affected_wav),
+    "clean": analyze(args.clean_latent, args.clean_wav),
+}, indent=2))
--- a/tools/benchmark_ulysses.py
+++ b/tools/benchmark_ulysses.py
@ -0,0 +1,143 @@
+"""Benchmark ragged H3 Ulysses transport and attention under torchrun."""
+
+import argparse
+import json
+import os
+import statistics
+import time
+from pathlib import Path
+
+import torch
+import torch.distributed as dist
+
+from h3_blackwell_runtime.attention import run_attention, run_flash4_attention_bshd, run_sol_attention_bshd
+from h3_blackwell_runtime.distributed import SequenceParallelContext
+
+
+def synchronize(device: torch.device) -> None:
+    if device.type == "cuda":
+        torch.cuda.synchronize(device)
+
+
+def run_backend(backend: str, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
+    if backend == "flash4":
+        return run_flash4_attention_bshd(q, k, v, is_causal=False)
+    if backend == "sol_attn":
+        return run_sol_attention_bshd(q, k, v, is_causal=False)
+    return run_attention(
+        q.transpose(1, 2).contiguous(),
+        k.transpose(1, 2).contiguous(),
+        v.transpose(1, 2).contiguous(),
+        backend=backend,
+        is_causal=False,
+    ).transpose(1, 2)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--sequence", type=int, default=20000)
+parser.add_argument("--heads", type=int, default=56)
+parser.add_argument("--head-dim", type=int, default=128)
+parser.add_argument("--backend", default="sdpa")
+parser.add_argument("--warmup", type=int, default=3)
+parser.add_argument("--iterations", type=int, default=10)
+parser.add_argument("--output", type=Path)
+args = parser.parse_args()
+
+local_rank = int(os.environ.get("LOCAL_RANK", "0"))
+use_cuda = torch.cuda.is_available()
+device = torch.device(f"cuda:{local_rank}" if use_cuda else "cpu")
+if use_cuda:
+    torch.cuda.set_device(device)
+dist.init_process_group(backend="nccl" if use_cuda else "gloo", device_id=device if use_cuda else None)
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+context = SequenceParallelContext.create(args.sequence, args.heads, args.head_dim)
+
+generator = torch.Generator(device=device).manual_seed(440420 + rank)
+shape = (1, context.local_token_length, args.heads, args.head_dim)
+dtype = torch.bfloat16 if use_cuda else torch.float32
+q = torch.randn(shape, generator=generator, dtype=dtype, device=device)
+k = torch.randn(shape, generator=generator, dtype=dtype, device=device)
+v = torch.randn(shape, generator=generator, dtype=dtype, device=device)
+
+
+def iteration() -> tuple[float, float, float, torch.Tensor]:
+    dist.barrier()
+    synchronize(device)
+    started = time.perf_counter()
+    full_q, full_k, full_v = context.seq_to_heads(q, k, v)
+    synchronize(device)
+    after_forward = time.perf_counter()
+    head_output = run_backend(args.backend, full_q, full_k, full_v)
+    synchronize(device)
+    after_attention = time.perf_counter()
+    local_output = context.heads_to_seq(head_output)
+    synchronize(device)
+    finished = time.perf_counter()
+    return (
+        after_forward - started,
+        after_attention - after_forward,
+        finished - after_attention,
+        local_output,
+    )
+
+
+for _ in range(args.warmup):
+    *_timings, output = iteration()
+    del output
+
+rank_timings = []
+for _ in range(args.iterations):
+    forward, attention, inverse, output = iteration()
+    rank_timings.append((forward, attention, inverse, forward + attention + inverse))
+    del output
+
+timings = torch.tensor(rank_timings, dtype=torch.float64, device=device)
+gathered = [torch.empty_like(timings) for _ in range(world_size)]
+dist.all_gather(gathered, timings)
+
+if use_cuda:
+    peak_memory = torch.tensor([torch.cuda.max_memory_allocated(device)], dtype=torch.int64, device=device)
+else:
+    peak_memory = torch.tensor([0], dtype=torch.int64, device=device)
+memory_by_rank = [torch.empty_like(peak_memory) for _ in range(world_size)]
+dist.all_gather(memory_by_rank, peak_memory)
+
+if rank == 0:
+    stacked = torch.stack(gathered).cpu()
+    stage_names = ("forward_all_to_all", "attention", "inverse_all_to_all", "total")
+    stages = {}
+    for index, name in enumerate(stage_names):
+        maximum_rank = stacked[:, :, index].amax(dim=0).tolist()
+        stages[name] = {
+            "median_seconds": statistics.median(maximum_rank),
+            "minimum_seconds": min(maximum_rank),
+            "maximum_seconds": max(maximum_rank),
+        }
+    element_size = q.element_size()
+    report = {
+        "world_size": world_size,
+        "backend": args.backend,
+        "device": torch.cuda.get_device_name(device) if use_cuda else "cpu",
+        "torch": torch.__version__,
+        "sequence": args.sequence,
+        "heads": args.heads,
+        "head_dim": args.head_dim,
+        "token_lengths": list(context.token_lengths),
+        "head_lengths": list(context.head_lengths),
+        "dtype": str(dtype),
+        "iterations": args.iterations,
+        "aggregate_transport_bytes_per_iteration": {
+            "forward_qkv": 3 * args.sequence * args.heads * args.head_dim * element_size,
+            "inverse_output": args.sequence * args.heads * args.head_dim * element_size,
+        },
+        "stages": stages,
+        "peak_allocated_bytes_by_rank": [int(value.item()) for value in memory_by_rank],
+    }
+    serialized = json.dumps(report, indent=2)
+    if args.output is not None:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(serialized + "\n", encoding="utf-8")
+    print(serialized)
+
+dist.destroy_process_group()
--- a/tools/benchmark_video_vae_decode.py
+++ b/tools/benchmark_video_vae_decode.py
@ -0,0 +1,167 @@
+"""Benchmark MiniMax H3 video VAE decode variants on a saved latent."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import time
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.vae_decoder import dtype_from_name
+
+
+def sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def timed(stage: str, rows: list[dict], fn):
+    sync()
+    start = time.perf_counter()
+    value = fn()
+    sync()
+    elapsed = time.perf_counter() - start
+    rows.append({"stage": stage, "seconds": elapsed})
+    return value
+
+
+def ffmpeg_command(loglevel: str, *parts: str) -> list[str]:
+    return ["ffmpeg", "-hide_banner", "-loglevel", loglevel, *parts]
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--latent", type=Path, required=True)
+    parser.add_argument("--output", type=Path)
+    parser.add_argument("--metrics", type=Path, required=True)
+    parser.add_argument("--implementation", choices=("direct", "upstream"), default="direct")
+    parser.add_argument("--dtype", choices=("float32", "float16", "bfloat16"), default="float16")
+    parser.add_argument("--no-tiling", action="store_true")
+    parser.add_argument("--tile-size", type=int)
+    parser.add_argument("--tile-overlap", type=int)
+    parser.add_argument("--trace-calls", action="store_true")
+    parser.add_argument("--skip-video", action="store_true")
+    parser.add_argument("--ffmpeg-loglevel", default="error")
+    args = parser.parse_args()
+
+    rows: list[dict] = []
+    call_rows: list[dict] = []
+    dtype = dtype_from_name(args.dtype)
+
+    state = timed("latent_load", rows, lambda: torch.load(args.latent, map_location="cuda", weights_only=False))
+    latent = state["latent"].to("cuda") if isinstance(state, dict) else state.to("cuda")
+    frames = int(state.get("frames", latent.shape[2] * 4)) if isinstance(state, dict) else latent.shape[2] * 4
+
+    if args.implementation == "direct":
+        from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE
+
+        vae = timed(
+            "vae_load",
+            rows,
+            lambda: MiniMaxH3VideoVAE.from_safetensors(
+                "/vae/minimax_h3_video_vae_fp16.safetensors",
+                device="cuda",
+                tiling=not args.no_tiling,
+                dtype=dtype,
+            ).eval(),
+        )
+    else:
+        from safetensors.torch import load_file
+
+        from h3_blackwell_runtime.upstream_vae import MiniMaxH3VideoVAE
+
+        def load_upstream():
+            model = MiniMaxH3VideoVAE(tiling=not args.no_tiling).to("cuda").eval()
+            checkpoint = load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda")
+            checkpoint = {key: value.to(dtype=dtype) for key, value in checkpoint.items()}
+            model.load_state_dict(checkpoint, strict=True)
+            return model
+
+        vae = timed("vae_load", rows, load_upstream)
+
+    if args.tile_size is not None:
+        vae.tile_size = args.tile_size
+    if args.tile_overlap is not None:
+        vae.tile_overlap_min = args.tile_overlap
+
+    if args.trace_calls:
+        original_decode_pixels = vae._decode_pixels
+
+        def traced_decode_pixels(z):
+            sync()
+            start = time.perf_counter()
+            result = original_decode_pixels(z)
+            sync()
+            call_rows.append({
+                "index": len(call_rows),
+                "latent_shape": tuple(z.shape),
+                "seconds": time.perf_counter() - start,
+            })
+            return result
+
+        vae._decode_pixels = traced_decode_pixels
+
+    with torch.inference_mode():
+        decoded = timed("vae_decode", rows, lambda: vae.decode(latent.to(dtype))[:, :, :frames])
+        pixels = timed(
+            "pixelize_cpu",
+            rows,
+            lambda: ((decoded[0].permute(1, 2, 3, 0).clamp(-1, 1) + 1) * 127.5).to(torch.uint8).cpu(),
+        )
+
+    if args.output is not None and not args.skip_video:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        raw = args.output.with_suffix(".rgb")
+        timed("raw_write", rows, lambda: pixels.numpy().tofile(raw))
+        timed(
+            "ffmpeg_encode",
+            rows,
+            lambda: subprocess.run(
+                ffmpeg_command(
+                    args.ffmpeg_loglevel,
+                    "-y",
+                    "-f",
+                    "rawvideo",
+                    "-pixel_format",
+                    "rgb24",
+                    "-video_size",
+                    f"{pixels.shape[2]}x{pixels.shape[1]}",
+                    "-framerate",
+                    "24",
+                    "-i",
+                    str(raw),
+                    "-an",
+                    "-c:v",
+                    "libx264",
+                    "-pix_fmt",
+                    "yuv420p",
+                    str(args.output),
+                ),
+                check=True,
+            ),
+        )
+        raw.unlink()
+
+    metrics = {
+        "implementation": args.implementation,
+        "dtype": args.dtype,
+        "tiling": not args.no_tiling,
+        "tile_size": vae.tile_size,
+        "tile_overlap_min": vae.tile_overlap_min,
+        "frames": frames,
+        "latent_shape": tuple(latent.shape),
+        "output": str(args.output) if args.output is not None else None,
+        "stages": rows,
+        "decode_pixel_calls": call_rows,
+        "total_seconds": sum(row["seconds"] for row in rows),
+    }
+    args.metrics.parent.mkdir(parents=True, exist_ok=True)
+    args.metrics.write_text(json.dumps(metrics, indent=2), encoding="utf-8")
+    print(json.dumps(metrics, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/compare_attention_backends.py
+++ b/tools/compare_attention_backends.py
@ -4,8 +4,8 @@ import argparse
 import time

 import torch
-import torch.nn.functional as functional

+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, run_attention
 from h3_blackwell_runtime.checkpoint import H3Checkpoint
 from h3_blackwell_runtime.denoiser import H3PackedDenoiser

@ -21,20 +21,14 @@ q, k, v = payload["q"], payload["k"], payload["v"]
 model = H3PackedDenoiser.from_checkpoint(H3Checkpoint(args.model)).eval()
 out_proj = model.backbone.blocks[0].attention.out_proj

-for name in ("sdpa", "sage2", "sage3"):
+for name in AVAILABLE_BACKENDS:
    torch.cuda.synchronize()
    start = time.perf_counter()
-    if name == "sdpa":
-        output = functional.scaled_dot_product_attention(q, k, v, is_causal=False)
-    elif name == "sage2":
-        from sageattention import sageattn
-
-        output = sageattn(q, k, v, is_causal=False, tensor_layout="HND", smooth_k=False)
-    else:
-        from sageattn3 import sageattn3_blackwell
-
-        output = sageattn3_blackwell(q, k, v, is_causal=False)
-    torch.cuda.synchronize()
-    output = out_proj(output.transpose(1, 2).reshape(q.shape[0], q.shape[2], -1).reshape(q.shape[2], -1).contiguous())
-    delta = (output.float() - expected.float()).abs()
-    print(f"{name} elapsed_s={time.perf_counter() - start:.3f} max_abs={delta.max().item():.6g} mean_abs={delta.mean().item():.6g}")
+    try:
+        output = run_attention(q, k, v, backend=name, is_causal=False)
+        torch.cuda.synchronize()
+        output = out_proj(output.transpose(1, 2).reshape(q.shape[0], q.shape[2], -1).reshape(q.shape[2], -1).contiguous())
+        delta = (output.float() - expected.float()).abs()
+        print(f"{name} elapsed_s={time.perf_counter() - start:.3f} max_abs={delta.max().item():.6g} mean_abs={delta.mean().item():.6g}")
+    except Exception as exc:
+        print(f"{name} error={type(exc).__name__}: {exc}")
--- a/tools/compare_audio_pcm.py
+++ b/tools/compare_audio_pcm.py
@ -0,0 +1,64 @@
+"""Compare decoded audio streams using aligned float PCM arrays."""
+
+import argparse
+import json
+import subprocess
+from pathlib import Path
+
+import numpy as np
+
+
+def decode(path: Path) -> np.ndarray:
+    raw = subprocess.check_output([
+        "ffmpeg", "-hide_banner", "-loglevel", "error", "-i", str(path),
+        "-map", "0:a:0", "-f", "f32le", "-acodec", "pcm_f32le", "-",
+    ])
+    return np.frombuffer(raw, dtype="<f4").reshape(-1, 2)
+
+
+def metrics(reference: np.ndarray, candidate: np.ndarray) -> dict:
+    count = min(len(reference), len(candidate))
+    reference = reference[:count]
+    candidate = candidate[:count]
+    error = candidate - reference
+    signal_power = np.maximum(np.mean(reference**2, axis=0), 1e-30)
+    noise_power = np.maximum(np.mean(error**2, axis=0), 1e-30)
+    first_250ms = error[:8000]
+    first_signal_power = np.maximum(np.mean(reference[:8000] ** 2, axis=0), 1e-30)
+    first_noise_power = np.maximum(np.mean(first_250ms**2, axis=0), 1e-30)
+    return {
+        "reference_samples": len(reference),
+        "candidate_samples": len(candidate),
+        "compared_samples": count,
+        "snr_db_by_channel": (10.0 * np.log10(signal_power / noise_power)).tolist(),
+        "rmse": float(np.sqrt(np.mean(error**2))),
+        "first_250ms_snr_db_by_channel": (
+            10.0 * np.log10(first_signal_power / first_noise_power)
+        ).tolist(),
+        "first_250ms_rmse": float(np.sqrt(np.mean(first_250ms**2))),
+        "max_abs_error": float(np.max(np.abs(error))),
+    }
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--reference", type=Path, required=True)
+parser.add_argument("--candidate", action="append", default=[], metavar="NAME=PATH")
+parser.add_argument("--output", type=Path)
+args = parser.parse_args()
+
+reference = decode(args.reference)
+report = {"reference": str(args.reference), "candidates": {}}
+for value in args.candidate:
+    if "=" not in value:
+        raise ValueError(f"candidate must be NAME=PATH, got {value!r}")
+    name, raw_path = value.split("=", 1)
+    report["candidates"][name] = {
+        "path": raw_path,
+        **metrics(reference, decode(Path(raw_path))),
+    }
+
+serialized = json.dumps(report, indent=2)
+if args.output is not None:
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(serialized + "\n", encoding="utf-8")
+print(serialized)
--- a/tools/compare_fl2va_free_run.py
+++ b/tools/compare_fl2va_free_run.py
@ -5,6 +5,7 @@ import argparse
 import torch

 from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS
 from h3_blackwell_runtime.denoiser import H3PackedDenoiser
 from h3_blackwell_runtime.packing import H3PromptPacker, unpatchify_video
 from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, _unpack_audio, res_multistep_update
@ -18,7 +19,7 @@ parser.add_argument("--width", type=int, default=320)
 parser.add_argument("--height", type=int, default=192)
 parser.add_argument("--frames", type=int, default=22)
 parser.add_argument("--seed", type=int, default=440204)
-parser.add_argument("--attention", choices=("sage2", "sdpa", "sage3"), default="sage2")
+parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sage2")
 parser.add_argument("--oracle-timesteps", action="store_true")
 args = parser.parse_args()

--- a/tools/compare_fl2va_key_sampler.py
+++ b/tools/compare_fl2va_key_sampler.py
@ -0,0 +1,128 @@
+"""Replay a matched Comfy keyframe sampler capture with exact static conditioning."""
+
+import argparse
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.denoiser import H3PackedDenoiser
+from h3_blackwell_runtime.packing import H3PromptPacker, unpatchify_video
+from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, _unpack_audio, beta_sigmas, res_multistep_update
+from h3_blackwell_runtime.t2v import random_av_latents
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--sampler", type=Path, required=True)
+parser.add_argument("--dit", type=Path, required=True)
+parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+parser.add_argument("--result-latent", type=Path)
+parser.add_argument("--production-trace", type=Path)
+args = parser.parse_args()
+
+checkpoint = H3Checkpoint(args.model)
+model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend="sage2").eval()
+packer = H3PromptPacker(checkpoint)
+captured_input = torch.load(args.dit / "input.pt", map_location="cuda", weights_only=False)
+text_length = next(start for start, _, code in captured_input["segments"] if code == 3)
+prefix_stop = next(start for start, _, code in captured_input["segments"] if code % 3 == 2)
+text = captured_input["hidden"][:text_length].unsqueeze(0).cuda()
+prefix = captured_input["hidden"][:prefix_stop].cuda()
+text_tags = torch.ones(text_length, dtype=torch.long, device="cuda")
+for start, stop, code in captured_input["segments"]:
+    if stop <= text_length:
+        text_tags[start:stop] = code % 3
+
+video, audio, frame_count = random_av_latents(384, 384, 22, 440207)
+video_shape, audio_shape = video.shape, audio.shape
+video_count, audio_count = video.numel(), audio.numel()
+zero_cond = [torch.zeros(1, 24, 1, 24, 24, device="cuda") for _ in range(2)]
+sigmas = torch.load(args.sampler / "initial.pt", map_location="cuda", weights_only=False)["sigmas"].cuda()
+direct_sigmas = beta_sigmas(12, device="cuda")
+sigma_delta = (direct_sigmas.float() - sigmas.float()).abs()
+print({"stage": "sigmas", "direct": direct_sigmas.tolist(), "comfy": sigmas.tolist(), "mean_delta": float(sigma_delta.mean()), "max_delta": float(sigma_delta.max())}, flush=True)
+video_history = audio_history = history_sigma = None
+
+for index, sigma in enumerate(sigmas[:-1]):
+    reference = torch.load(args.sampler / f"step_{index:02d}.pt", map_location="cuda", weights_only=False)
+    reference_x = reference["x"].cuda().reshape(-1)
+    reference_video = reference_x[:video_count].reshape(video_shape)
+    reference_audio = reference_x[video_count:video_count + audio_count].reshape(audio_shape)
+    pre_video = (video.float() - reference_video.float()).abs()
+    pre_audio = (audio.float() - reference_audio.float()).abs()
+
+    sigma_audio = _audio_sigma(sigma)
+    carry = sigma_audio / sigma
+    hidden, times, segments, positions, video_segment, audio_segment = packer(
+        text,
+        video,
+        audio.to(torch.bfloat16) * carry,
+        _model_sigma(sigma),
+        text_token_tags=text_tags,
+        cond_latents=zero_cond,
+        cond_frame_indices=[0, frame_count - 1],
+        frame_count=frame_count,
+        seed=440207,
+    )
+    hidden[:prefix_stop] = prefix.to(hidden)
+    input_hidden = hidden.detach().clone()
+    with torch.inference_mode():
+        raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment)
+    raw_video = raw_video.to(torch.bfloat16).float()
+    raw_audio = raw_audio.to(torch.bfloat16)
+    video_denoised = video + sigma * unpatchify_video(raw_video, video.shape[2], video.shape[-2], video.shape[-1])
+    audio_model_output = (
+        (1.0 - 4.0) * (audio.to(torch.bfloat16) * carry.to(torch.bfloat16))
+        + (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(raw_audio))
+    ).float()
+    audio_denoised = audio - sigma * audio_model_output
+
+    reference_denoised = reference["denoised"].cuda().reshape(-1)
+    reference_video_denoised = reference_denoised[:video_count].reshape(video_shape)
+    reference_audio_denoised = reference_denoised[video_count:video_count + audio_count].reshape(audio_shape)
+    denoised_video = (video_denoised.float() - reference_video_denoised.float()).abs()
+    denoised_audio = (audio_denoised.float() - reference_audio_denoised.float()).abs()
+    production = torch.load(args.production_trace / f"step_{index:02d}.pt", map_location="cuda", weights_only=False) if args.production_trace else None
+    production_video = (production["video"].float() - reference_video.float()).abs() if production else None
+    production_audio = (production["audio"].float() - reference_audio.float()).abs() if production else None
+    production_denoised = (production["video_denoised"].float() - reference_video_denoised.float()).abs() if production else None
+    production_hidden = (production["hidden"].float() - input_hidden.float()).abs() if production and "hidden" in production else None
+    production_raw = (production["raw_video"].float() - raw_video.float()).abs() if production and "raw_video" in production else None
+    production_segments = [
+        (start, stop, code, float((production["hidden"][start:stop].float() - input_hidden[start:stop].float()).abs().mean()))
+        for start, stop, code in segments
+    ] if production and "hidden" in production else None
+    print({
+        "step": index,
+        "pre_video_mean": float(pre_video.mean()),
+        "pre_video_max": float(pre_video.max()),
+        "pre_audio_mean": float(pre_audio.mean()),
+        "pre_audio_max": float(pre_audio.max()),
+        "denoised_video_mean": float(denoised_video.mean()),
+        "denoised_video_max": float(denoised_video.max()),
+        "denoised_audio_mean": float(denoised_audio.mean()),
+        "denoised_audio_max": float(denoised_audio.max()),
+        "production_video_mean": float(production_video.mean()) if production_video is not None else None,
+        "production_video_max": float(production_video.max()) if production_video is not None else None,
+        "production_audio_mean": float(production_audio.mean()) if production_audio is not None else None,
+        "production_audio_max": float(production_audio.max()) if production_audio is not None else None,
+        "production_denoised_mean": float(production_denoised.mean()) if production_denoised is not None else None,
+        "production_denoised_max": float(production_denoised.max()) if production_denoised is not None else None,
+        "production_hidden_mean": float(production_hidden.mean()) if production_hidden is not None else None,
+        "production_hidden_max": float(production_hidden.max()) if production_hidden is not None else None,
+        "production_raw_mean": float(production_raw.mean()) if production_raw is not None else None,
+        "production_raw_max": float(production_raw.max()) if production_raw is not None else None,
+        "production_segments": production_segments,
+    }, flush=True)
+
+    previous_sigma = sigmas[index - 1] if index else None
+    sigma_down = sigmas[index + 1]
+    video = res_multistep_update(video, video_denoised, sigma, sigma_down, video_history, history_sigma, previous_sigma)
+    audio = res_multistep_update(audio, audio_denoised, sigma, sigma_down, audio_history, history_sigma, previous_sigma)
+    video_history, audio_history, history_sigma = video_denoised, audio_denoised, sigma_down
+
+if args.result_latent:
+    result = torch.load(args.result_latent, map_location="cuda", weights_only=False)
+    result_video = result["latent"] if isinstance(result, dict) else result
+    delta = (video.float() - result_video.cuda().float()).abs()
+    print({"stage": "final_video", "mean_delta": float(delta.mean()), "max_delta": float(delta.max())}, flush=True)
--- a/tools/compare_fl2va_keyframe_input.py
+++ b/tools/compare_fl2va_keyframe_input.py
@ -0,0 +1,193 @@
+"""Compare direct keyframe FL2VA packing with a matched Comfy DiT capture."""
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+from PIL import Image
+import torch
+
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.denoiser import H3PackedDenoiser
+from h3_blackwell_runtime.packing import H3PromptPacker
+from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
+from h3_blackwell_runtime.qwen3vl_vision import Qwen3VL32BVision, build_fl2va_presentation, resize_keyframe
+from h3_blackwell_runtime.qwen3vl_vision import mrope_freqs_cis, mrope_position_ids
+from h3_blackwell_runtime.sampler import _model_sigma, beta_sigmas
+from h3_blackwell_runtime.t2v import random_av_latents
+from h3_blackwell_runtime.token_refiner import H3TokenRefiner
+from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--capture", required=True)
+parser.add_argument("--first", required=True)
+parser.add_argument("--last", required=True)
+parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+parser.add_argument("--qwen", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
+parser.add_argument("--vae", default="/vae/minimax_h3_video_vae_fp16.safetensors")
+parser.add_argument("--tokenizer", default="/opt/h3-blackwell-runtime/src/h3_blackwell_runtime/qwen25_tokenizer")
+parser.add_argument("--reference-vision-first")
+parser.add_argument("--reference-vision-last")
+parser.add_argument("--qwen-capture-dir")
+parser.add_argument("--sampler-capture-dir")
+parser.add_argument("--oracle-qwen-input", action="store_true")
+parser.add_argument("--vae-dtype", choices=("float16", "bfloat16", "float32"), default="float32")
+parser.add_argument("--vae-no-tiling", action="store_true")
+parser.add_argument("--load-dit-first", action="store_true")
+parser.add_argument("--preview-order", action="store_true")
+args = parser.parse_args()
+
+prompt = "A studio time-lapse of the same pink peony bud opening into the same fully bloomed pink peony, fixed camera, cream background."
+width = height = 384
+requested_frames = 22
+seed = 440207
+
+
+def load_image(path):
+    image = Image.open(path).convert("RGB")
+    return torch.from_numpy(np.asarray(image).copy()).unsqueeze(0).cuda().float().div(255.0)
+
+
+def report(name, actual, expected):
+    expected = expected.to(actual.device)
+    delta = (actual.float() - expected.float()).abs()
+    print({
+        "stage": name,
+        "shape": tuple(actual.shape),
+        "mean_delta": float(delta.mean()),
+        "max_delta": float(delta.max()),
+    }, flush=True)
+
+
+capture = torch.load(args.capture, map_location="cuda", weights_only=False)
+checkpoint = H3Checkpoint(args.model)
+dit_probe = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend="sage2").eval() if args.load_dit_first else None
+early_refiner = H3TokenRefiner(checkpoint, attention_backend="sage2") if args.preview_order else None
+early_packer = H3PromptPacker(checkpoint) if args.preview_order else None
+conditioner = Qwen3VLPromptConditioner(args.qwen, args.tokenizer)
+vision = Qwen3VL32BVision(args.qwen, device="cuda", dtype=torch.float32)
+if args.reference_vision_first and args.reference_vision_last:
+    class CapturedVision:
+        def __init__(self, paths):
+            self.outputs = [torch.load(path, map_location="cuda", weights_only=False) for path in paths]
+
+        def __call__(self, flatten, grid):
+            output = self.outputs.pop(0)
+            return output["merged"].cuda(), [value.cuda() for value in output["deepstack"]]
+
+    vision = CapturedVision([args.reference_vision_first, args.reference_vision_last])
+video, audio, frame_count = random_av_latents(width, height, requested_frames, seed)
+if args.sampler_capture_dir:
+    initial = torch.load(Path(args.sampler_capture_dir) / "initial.pt", map_location="cuda", weights_only=False)
+    direct_initial = torch.cat((video.reshape(-1), audio.reshape(-1)))
+    report("sampler_initial", direct_initial, initial["initial_x"].reshape(-1))
+presentation = build_fl2va_presentation(
+    prompt,
+    load_image(args.first),
+    load_image(args.last),
+    width=width,
+    height=height,
+    frame_count=frame_count,
+    tokenizer=conditioner.tokenizer,
+    vision=vision,
+    text_encoder=conditioner.encoder,
+    device="cuda",
+)
+if args.qwen_capture_dir:
+    qwen_capture = Path(args.qwen_capture_dir)
+    expected_ids = torch.load(qwen_capture / "qwen_input_ids.pt", map_location="cuda", weights_only=False)
+    print({"stage": "qwen_input_ids", "equal": torch.equal(presentation.input_ids, expected_ids), "direct_shape": tuple(presentation.input_ids.shape), "comfy_shape": tuple(expected_ids.shape)}, flush=True)
+    direct_embeds = conditioner.encoder._embed_rows(presentation.input_ids)
+    visual_mask = torch.zeros((1, direct_embeds.shape[1]), dtype=torch.bool, device="cuda")
+    deepstack_by_index = {}
+    for embed in presentation.embeds_info:
+        start = embed["index"]
+        end = start + embed["size"]
+        direct_embeds[0, start:end] = embed["extra"]["merged"].to(direct_embeds)
+        visual_mask[0, start:end] = True
+        for index, value in enumerate(embed["extra"]["deepstack"]):
+            deepstack_by_index.setdefault(index, []).append(value)
+    compact_ids_path = qwen_capture / "qwen_compact_token_ids.pt"
+    if compact_ids_path.exists():
+        compact_ids = torch.load(compact_ids_path, map_location="cuda", weights_only=False)
+        print({"stage": "qwen_compact_token_ids", "equal": torch.equal(presentation.input_ids[~visual_mask], compact_ids.reshape(-1)), "direct": presentation.input_ids[~visual_mask].tolist(), "comfy": compact_ids.reshape(-1).tolist()}, flush=True)
+    direct_deepstack = [torch.cat(values, dim=0) for _, values in sorted(deepstack_by_index.items())]
+    expected_embeds = torch.load(qwen_capture / "qwen_input_embeds.pt", map_location="cuda", weights_only=False)
+    raw_rows = torch.nn.functional.embedding(presentation.input_ids, conditioner.encoder.embed_tokens)
+    raw_scales = torch.nn.functional.embedding(presentation.input_ids, conditioner.encoder.embed_scale)
+    fp16_embeds = (raw_rows.to(torch.float16) * raw_scales.to(torch.float16)).float()
+    bf16_embeds = (raw_rows.to(torch.bfloat16) * raw_scales.to(torch.bfloat16)).float()
+    fp32_to_fp16_embeds = (raw_rows.float() * raw_scales.float()).half().float()
+    fp32_to_bf16_embeds = (raw_rows.float() * raw_scales.float()).bfloat16().float()
+    report("qwen_input_text_rows_fp16", fp16_embeds[~visual_mask], expected_embeds[~visual_mask])
+    report("qwen_input_text_rows_bf16", bf16_embeds[~visual_mask], expected_embeds[~visual_mask])
+    report("qwen_input_text_rows_fp32_to_fp16", fp32_to_fp16_embeds[~visual_mask], expected_embeds[~visual_mask])
+    report("qwen_input_text_rows_fp32_to_bf16", fp32_to_bf16_embeds[~visual_mask], expected_embeds[~visual_mask])
+    report("qwen_input_embeds", direct_embeds, expected_embeds)
+    report("qwen_input_text_rows", direct_embeds[~visual_mask], expected_embeds.to(direct_embeds.device)[~visual_mask])
+    report("qwen_input_visual_rows", direct_embeds[visual_mask], expected_embeds.to(direct_embeds.device)[visual_mask])
+    position_ids = mrope_position_ids(presentation.embeds_info, direct_embeds.shape[1], "cuda")
+    freqs = mrope_freqs_cis(position_ids)
+    hidden = (expected_embeds if args.oracle_qwen_input else direct_embeds).to(conditioner.encoder.dtype)
+    for index, layer in enumerate(conditioner.encoder.layers):
+        hidden = layer(hidden, freqs)
+        expected_layer = torch.load(qwen_capture / "qwen_layers" / f"{index:02d}.pt", map_location="cuda", weights_only=False)
+        report(f"qwen_layer_{index:02d}", hidden, expected_layer)
+        if index < len(direct_deepstack):
+            hidden[visual_mask] = hidden[visual_mask] + direct_deepstack[index].to(hidden)
+    expected_layer50 = torch.load(qwen_capture / "qwen_layer50.pt", map_location="cuda", weights_only=False)
+    report("qwen_layer50", presentation.text_states, expected_layer50)
+vae_dtype = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[args.vae_dtype]
+vae = MiniMaxH3VideoVAEEncoder.from_safetensors(args.vae, device="cuda", dtype=vae_dtype, tiling=not args.vae_no_tiling).eval()
+cond_latents = []
+cond_images = []
+for keyframe in presentation.keyframes:
+    resized = resize_keyframe(
+        keyframe["image"],
+        width,
+        height,
+        crop="disabled" if keyframe["resolved_frame_index"] == 0 else "center",
+    )
+    cond_images.append(resized)
+    pixels = resized.movedim(-1, 1).cuda().float().mul(2.0).sub(1.0)
+    cond_latents.append(vae.encode(pixels))
+if args.qwen_capture_dir:
+    for index, latent in enumerate(cond_latents):
+        captured_vae = torch.load(Path(args.qwen_capture_dir) / f"vae_keyframe_{index}.pt", map_location="cuda", weights_only=False)
+        print({"stage": f"vae_meta_{index}", **captured_vae.get("meta", {})}, flush=True)
+        report(f"vae_image_{index}", cond_images[index], captured_vae["image"])
+        report(f"vae_keyframe_{index}", latent, captured_vae["latent"])
+
+text = (early_refiner or H3TokenRefiner(checkpoint))(presentation.text_states)
+packer = early_packer or H3PromptPacker(checkpoint)
+sigma = beta_sigmas(12, device=video.device)[0]
+hidden, times, segments, positions, _, _ = packer(
+    text,
+    video,
+    audio,
+    _model_sigma(sigma),
+    text_token_tags=presentation.text_token_tags,
+    cond_latents=cond_latents,
+    cond_frame_indices=[keyframe["resolved_frame_index"] for keyframe in presentation.keyframes],
+    frame_count=frame_count,
+    seed=seed,
+)
+
+expected_hidden = capture["hidden"]
+text_length = text.shape[1]
+frame_rows = (video.shape[-2] // 2) * (video.shape[-1] // 2)
+direct_first = hidden[text_length:text_length + frame_rows]
+direct_last = hidden[text_length + frame_rows:text_length + 2 * frame_rows]
+comfy_first = expected_hidden[text_length:text_length + frame_rows]
+comfy_last = expected_hidden[text_length + frame_rows:text_length + 2 * frame_rows]
+
+print({"stage": "lengths", "text": text_length, "cond_each": frame_rows, "direct_total": hidden.shape[0], "comfy_total": expected_hidden.shape[0]}, flush=True)
+report("text_rows", hidden[:text_length], expected_hidden[:text_length])
+report("cond_first_to_first", direct_first, comfy_first)
+report("cond_first_to_last", direct_first, comfy_last)
+report("cond_last_to_last", direct_last, comfy_last)
+report("cond_last_to_first", direct_last, comfy_first)
+report("timesteps", times, capture["timesteps"])
+report("positions", positions, capture["position_ids"])
+print({"stage": "segments", "direct": segments, "comfy": capture["segments"]}, flush=True)
--- a/tools/compare_generation_latents.py
+++ b/tools/compare_generation_latents.py
@ -0,0 +1,32 @@
+"""Compare saved hot-runtime video and audio latents against one reference run."""
+
+import argparse
+import json
+
+import torch
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("reference")
+parser.add_argument("candidates", nargs="+")
+args = parser.parse_args()
+
+reference = torch.load(args.reference, map_location="cpu", weights_only=False)
+results = {}
+for path in args.candidates:
+    candidate = torch.load(path, map_location="cpu", weights_only=False)
+    metrics = {}
+    for name in ("latent", "audio_latent"):
+        expected = reference[name].float()
+        actual = candidate[name].float()
+        delta = actual - expected
+        metrics[name] = {
+            "max_abs": delta.abs().max().item(),
+            "mean_abs": delta.abs().mean().item(),
+            "rmse": delta.square().mean().sqrt().item(),
+            "relative_rmse": (delta.square().mean().sqrt() / expected.square().mean().sqrt()).item(),
+            "cosine": torch.nn.functional.cosine_similarity(actual.flatten(), expected.flatten(), dim=0).item(),
+        }
+    results[path] = metrics
+
+print(json.dumps({"reference": args.reference, "results": results}, indent=2))
--- a/tools/compare_h3_keyframe_capture.py
+++ b/tools/compare_h3_keyframe_capture.py
@ -0,0 +1,55 @@
+"""Replay a matched Comfy keyframe DiT capture through the direct H3 model."""
+
+import argparse
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.denoiser import H3PackedDenoiser
+from h3_blackwell_runtime.packing import unpatchify_video
+from h3_blackwell_runtime.rope import h3_rope_rotation
+from h3_blackwell_runtime.sampler import _unpack_audio
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--capture", type=Path, required=True)
+parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+parser.add_argument("--attention", default="sage2")
+args = parser.parse_args()
+
+
+def report(name, actual, expected):
+    expected = expected.to(actual.device)
+    delta = (actual.float() - expected.float()).abs()
+    print({"stage": name, "shape": tuple(actual.shape), "mean_delta": float(delta.mean()), "max_delta": float(delta.max())}, flush=True)
+
+
+captured_input = torch.load(args.capture / "input.pt", map_location="cuda", weights_only=False)
+captured_output = torch.load(args.capture / "output.pt", map_location="cuda", weights_only=False)
+checkpoint = H3Checkpoint(args.model)
+model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend=args.attention).eval()
+
+hidden = captured_input["hidden"].cuda()
+timesteps = captured_input["timesteps"].cuda()
+positions = captured_input["position_ids"].cuda()
+segments = captured_input["segments"]
+rotation = h3_rope_rotation(positions, model.backbone.inv_freq, hidden.dtype)
+with torch.inference_mode():
+    for index, (block, adaln) in enumerate(zip(model.backbone.blocks, model.backbone.adaln, strict=True)):
+        hidden = block(hidden, rotation, *adaln(timesteps), segments)
+        expected = torch.load(args.capture / "blocks" / f"{index:02d}.pt", map_location="cuda", weights_only=False)
+        report(f"block_{index:02d}", hidden, expected)
+
+    video_rows, audio_rows = model.final_layer(
+        hidden,
+        timesteps,
+        tuple(captured_output["video_segment"]),
+        tuple(captured_output["audio_segment"]),
+    )
+    expected_video = captured_output["video"].cuda()
+    expected_audio = captured_output["audio"].cuda()
+    video = unpatchify_video(video_rows, expected_video.shape[2], expected_video.shape[3], expected_video.shape[4])
+    audio = _unpack_audio(audio_rows)
+    report("video_output", video, expected_video)
+    report("audio_output", audio, expected_audio)
--- a/tools/compare_qwen_vision.py
+++ b/tools/compare_qwen_vision.py
@ -0,0 +1,212 @@
+"""Compare direct and Comfy Qwen3-VL vision outputs on one keyframe."""
+
+import argparse
+from pathlib import Path
+import sys
+
+import numpy as np
+from PIL import Image
+from safetensors import safe_open
+import torch
+from torch.nn import functional as F
+
+from h3_blackwell_runtime.qwen3vl_vision import (
+    Qwen3VL32BVision,
+    _apply_rope_vision,
+    process_image,
+    resize_keyframe,
+)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--image", required=True)
+parser.add_argument("--checkpoint", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
+parser.add_argument("--width", type=int, default=384)
+parser.add_argument("--height", type=int, default=384)
+parser.add_argument("--comfy-path", default="/opt/ComfyUI")
+parser.add_argument("--save-reference")
+parser.add_argument("--dtype", choices=("float16", "bfloat16", "float32"), default="bfloat16")
+parser.add_argument("--captured-reference")
+args = parser.parse_args()
+
+sys.path.insert(0, args.comfy_path)
+import comfy.ops  # noqa: E402
+from comfy.ldm.modules.attention import optimized_attention_for_device  # noqa: E402
+from comfy.text_encoders.qwen3vl import (  # noqa: E402
+    QWEN3VL_VISION,
+    QWEN3VL_VISION_COMMON,
+    Qwen3VLVisionModel,
+)
+from comfy.text_encoders.qwen_vl import process_qwen2vl_images  # noqa: E402
+from comfy.text_encoders.llama import apply_rope  # noqa: E402
+
+
+def report(name, actual, expected):
+    actual = actual.detach()
+    expected = expected.detach().to(actual.device)
+    delta = (actual.float() - expected.float()).abs()
+    print({
+        "stage": name,
+        "shape": tuple(actual.shape),
+        "actual_dtype": str(actual.dtype),
+        "expected_dtype": str(expected.dtype),
+        "mean_delta": float(delta.mean()),
+        "max_delta": float(delta.max()),
+    }, flush=True)
+
+
+image = Image.open(args.image).convert("RGB")
+pixels = torch.from_numpy(np.asarray(image).copy()).unsqueeze(0).cuda().float().div(255.0)
+pixels = resize_keyframe(pixels, args.width, args.height)
+
+direct_flatten, direct_grid = process_image(pixels)
+reference_flatten, reference_grid = process_qwen2vl_images(
+    pixels,
+    patch_size=16,
+    image_mean=[0.5, 0.5, 0.5],
+    image_std=[0.5, 0.5, 0.5],
+)
+report("flatten_patches", direct_flatten, reference_flatten)
+print({"stage": "grid", "direct": direct_grid.tolist(), "reference": reference_grid.tolist()}, flush=True)
+
+dtype = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[args.dtype]
+config = {
+    **QWEN3VL_VISION_COMMON,
+    **QWEN3VL_VISION["qwen3vl_32b"],
+    "out_hidden_size": 5120,
+}
+reference = Qwen3VLVisionModel(
+    config,
+    device="cuda",
+    dtype=dtype,
+    ops=comfy.ops.disable_weight_init,
+).to("cuda").eval()
+with safe_open(args.checkpoint, framework="pt", device="cuda") as checkpoint:
+    print({
+        "stage": "checkpoint_dtypes",
+        "text_norm": str(checkpoint.get_tensor("model.layers.0.input_layernorm.weight").dtype),
+        "vision_norm": str(checkpoint.get_tensor("visual.blocks.0.norm1.weight").dtype),
+        "vision_patch": str(checkpoint.get_tensor("visual.patch_embed.proj.weight").dtype),
+    }, flush=True)
+    visual_state = {
+        name.removeprefix("visual."): checkpoint.get_tensor(name).to(dtype)
+        for name in checkpoint.keys()
+        if name.startswith("visual.")
+    }
+reference.load_state_dict(visual_state, strict=True)
+del visual_state
+
+direct = Qwen3VL32BVision(args.checkpoint, device="cuda", dtype=dtype).eval()
+with torch.inference_mode():
+    direct_x = direct.patch_embed(direct_flatten.cuda().to(dtype))
+    direct_patch_embed = direct_x
+    reference_x = reference.patch_embed(reference_flatten.cuda().to(dtype))
+    report("patch_embed", direct_x, reference_x)
+    direct_pos = direct.fast_pos_embed_interpolate(direct_grid).to(direct_x.device)
+    reference_pos = reference.fast_pos_embed_interpolate(reference_grid).to(reference_x.device)
+    report("position_embed", direct_pos, reference_pos)
+    direct_x = direct_x + direct_pos
+    direct_vision_input = direct_x
+    reference_x = reference_x + reference_pos
+    report("vision_input", direct_x, reference_x)
+
+    direct_rotary = direct.rot_pos_emb(direct_grid.to(direct_x.device)).reshape(direct_x.shape[0], -1)
+    reference_rotary = reference.rot_pos_emb(reference_grid).to(reference_x.device).reshape(reference_x.shape[0], -1)
+    report("rotary", direct_rotary, reference_rotary)
+
+    def position_tuple(rotary):
+        embedding = torch.cat((rotary, rotary), dim=-1)
+        cosine = embedding.cos().unsqueeze(-2)
+        sine = embedding.sin().unsqueeze(-2)
+        split = sine.shape[-1] // 2
+        return cosine, sine[..., :split], -sine[..., split:]
+
+    direct_position = position_tuple(direct_rotary)
+    reference_position = position_tuple(reference_rotary)
+    cu_seqlens = F.pad(
+        torch.repeat_interleave(direct_grid[:, 1] * direct_grid[:, 2], direct_grid[:, 0]).cumsum(0, dtype=torch.int32),
+        (1, 0),
+        value=0,
+    )
+    optimized_attention = optimized_attention_for_device(reference_x.device, mask=False, small_input=True)
+
+    direct_block0 = direct.blocks[0]
+    reference_block0 = reference.blocks[0]
+    direct_norm = F.layer_norm(
+        direct_x,
+        (direct_x.shape[-1],),
+        weight=direct_block0.norm1_weight,
+        bias=direct_block0.norm1_bias,
+        eps=1e-6,
+    )
+    reference_norm = reference_block0.norm1(reference_x)
+    report("block0_norm1", direct_norm, reference_norm)
+    direct_qkv = F.linear(direct_norm, direct_block0.attn.qkv_weight, direct_block0.attn.qkv_bias)
+    reference_qkv = reference_block0.attn.qkv(reference_norm)
+    report("block0_qkv", direct_qkv, reference_qkv)
+    direct_q, direct_k, direct_v = direct_qkv.reshape(direct_x.shape[0], 3, 16, 72).permute(1, 0, 2, 3).unbind(0)
+    reference_q, reference_k, reference_v = reference_qkv.reshape(reference_x.shape[0], 3, 16, 72).permute(1, 0, 2, 3).unbind(0)
+    direct_q, direct_k = _apply_rope_vision(direct_q.float(), direct_k.float(), direct_position)
+    direct_q, direct_k = direct_q.to(dtype), direct_k.to(dtype)
+    reference_q, reference_k = apply_rope(reference_q, reference_k, reference_position)
+    report("block0_rope_q", direct_q, reference_q)
+    report("block0_rope_k", direct_k, reference_k)
+    direct_attention_heads = F.scaled_dot_product_attention(
+        direct_q.transpose(0, 1).unsqueeze(0),
+        direct_k.transpose(0, 1).unsqueeze(0),
+        direct_v.transpose(0, 1).unsqueeze(0),
+    )
+    direct_attention = direct_attention_heads.transpose(1, 2).reshape(1, direct_x.shape[0], -1)
+    reference_attention = optimized_attention(
+        reference_q.transpose(0, 1).unsqueeze(0),
+        reference_k.transpose(0, 1).unsqueeze(0),
+        reference_v.transpose(0, 1).unsqueeze(0),
+        16,
+        skip_reshape=True,
+    )
+    report("block0_attention", direct_attention, reference_attention)
+    direct_projected = F.linear(direct_attention[0], direct_block0.attn.proj_weight, direct_block0.attn.proj_bias)
+    reference_projected = reference_block0.attn.proj(reference_attention)[0]
+    report("block0_projected", direct_projected, reference_projected)
+
+    direct_deepstack = []
+    direct_blocks = []
+    reference_deepstack = []
+    for index, (direct_block, reference_block) in enumerate(zip(direct.blocks, reference.blocks)):
+        direct_x = direct_block(direct_x, cu_seqlens, direct_position)
+        direct_blocks.append(direct_x)
+        reference_x = reference_block(
+            reference_x,
+            cu_seqlens,
+            reference_position,
+            optimized_attention=optimized_attention,
+        )
+        report(f"block_{index:02d}", direct_x, reference_x)
+        if index in direct.deepstack_visual_indexes:
+            merger_index = direct.deepstack_visual_indexes.index(index)
+            direct_deepstack.append(direct.deepstack_merger_list[merger_index](direct_x))
+            reference_deepstack.append(reference.deepstack_merger_list[merger_index](reference_x))
+    direct_merged = direct.merger(direct_x)
+    reference_merged = reference.merger(reference_x)
+report("merged", direct_merged, reference_merged)
+for index, (actual, expected) in enumerate(zip(direct_deepstack, reference_deepstack)):
+    report(f"deepstack_{index}", actual, expected)
+if args.captured_reference:
+    captured = torch.load(args.captured_reference, map_location="cuda", weights_only=False)
+    report("loaded_comfy_pixel_values", direct_flatten, captured["pixel_values"])
+    print({"stage": "loaded_comfy_grid", "direct": direct_grid.tolist(), "expected": captured["grid"].tolist()})
+    trace_path = Path(args.captured_reference).with_name(Path(args.captured_reference).name.replace("qwen_vision_", "qwen_vision_trace_"))
+    trace = torch.load(trace_path, map_location="cuda", weights_only=False)
+    report("loaded_comfy_patch_embed", direct_patch_embed, trace["patch_embed"])
+    report("loaded_comfy_position_embed", direct_pos, trace["position_embed"])
+    report("loaded_comfy_vision_input", direct_vision_input, trace["vision_input"])
+    for index, block_output in enumerate(direct_blocks):
+        report(f"loaded_comfy_block_{index:02d}", block_output, trace[f"block_{index:02d}"])
+    report("loaded_comfy_merged", direct_merged, captured["merged"])
+    for index, (actual, expected) in enumerate(zip(direct_deepstack, captured["deepstack"])):
+        report(f"loaded_comfy_deepstack_{index}", actual, expected)
+if args.save_reference:
+    torch.save({
+        "merged": reference_merged.detach().cpu(),
+        "deepstack": [value.detach().cpu() for value in reference_deepstack],
+    }, args.save_reference)
--- a/tools/compare_vae_decoder_clip.py
+++ b/tools/compare_vae_decoder_clip.py
@ -7,7 +7,7 @@ from pathlib import Path
 import torch
 from safetensors.torch import load_file

-from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE as DirectVAE
+from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE as DirectVAE, _conv3d


 parser = argparse.ArgumentParser()
@ -30,7 +30,7 @@ state = torch.load(args.latent, map_location="cuda", weights_only=False)
 latent = state["latent"].to("cuda") if isinstance(state, dict) else state.to("cuda")

 direct = DirectVAE.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda", tiling=False).eval()
-upstream = UpstreamVAE(tiling=False).to("cuda").eval()
+upstream = UpstreamVAE(tiling=False).to("cuda", dtype=torch.float16).eval()
 upstream.load_state_dict(load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda"), strict=True)

 with torch.inference_mode():
@ -38,7 +38,7 @@ with torch.inference_mode():
    z_u = z_d.clone().to(next(upstream.parameters()).dtype)
    z_d = z_d * direct.latents_std.view(1, -1, 1, 1, 1).to(z_d) + direct.latents_mean.view(1, -1, 1, 1, 1).to(z_d)
    z_u = z_u * upstream.latents_std.view(1, -1, 1, 1, 1).to(z_u) + upstream.latents_mean.view(1, -1, 1, 1, 1).to(z_u)
-    z_d = direct.post_quant_conv(z_d)
+    z_d = _conv3d(z_d, direct.post_quant_conv.weight, direct.post_quant_conv.bias)
    z_u = upstream.post_quant_conv(z_u)
    stats("post_quant_conv", z_d, z_u)

--- a/tools/compare_vae_encoder.py
+++ b/tools/compare_vae_encoder.py
@ -0,0 +1,110 @@
+"""Compare direct and upstream MiniMax H3 VAE encoding on one keyframe."""
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from PIL import Image
+import torch
+
+from h3_blackwell_runtime.vae_encoder import (
+    MiniMaxH3VideoVAEEncoder as DirectVAEEncoder,
+    _causal_conv3d,
+    _downsample,
+    _group_norm_3d,
+    _resnet,
+)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--image", required=True)
+parser.add_argument("--vae", default="/vae/minimax_h3_video_vae_fp16.safetensors")
+parser.add_argument("--comfy-path", default="/opt/ComfyUI")
+parser.add_argument("--tiling", action="store_true")
+args = parser.parse_args()
+
+sys.path.insert(0, args.comfy_path)
+from h3_blackwell_runtime.upstream_vae import MiniMaxH3VideoVAE as UpstreamVAE  # noqa: E402
+
+
+def load_checkpoint(path):
+    if os.getenv("H3_DISABLE_MMAP", "").lower() in {"1", "true", "yes", "on"}:
+        from safetensors.torch import load
+
+        with open(path, "rb") as file:
+            return load(file.read())
+    from safetensors.torch import load_file
+
+    return load_file(path, device="cuda")
+
+
+def report(name, actual, expected):
+    actual = actual.detach()
+    expected = expected.detach().to(actual.device)
+    delta = (actual.float() - expected.float()).abs()
+    print({
+        "stage": name,
+        "shape": tuple(actual.shape),
+        "actual_min": float(actual.min()),
+        "actual_max": float(actual.max()),
+        "expected_min": float(expected.min()),
+        "expected_max": float(expected.max()),
+        "mean_delta": float(delta.mean()),
+        "max_delta": float(delta.max()),
+    }, flush=True)
+
+
+image = Image.open(args.image).convert("RGB")
+pixels = torch.from_numpy(np.asarray(image).copy()).permute(2, 0, 1).unsqueeze(0).cuda().float()
+pixels = pixels.div(127.5).sub(1.0)
+
+direct = DirectVAEEncoder.from_safetensors(args.vae, device="cuda", tiling=args.tiling).eval()
+upstream = UpstreamVAE(tiling=args.tiling).to("cuda").eval()
+upstream = upstream.to(dtype=direct.compute_dtype)
+state = load_checkpoint(args.vae)
+upstream.load_state_dict(state, strict=True)
+del state
+
+trace = {}
+trace_names = {"encoder.conv_in", "encoder.norm_out", "encoder.conv_out", "quant_conv"}
+for level in range(6):
+    trace_names.update(f"encoder.down.{level}.block.{block}" for block in range(2))
+    if level < 4:
+        trace_names.add(f"encoder.down.{level}.downsample")
+for name, module in upstream.named_modules():
+    if name in trace_names:
+        module.register_forward_hook(lambda _module, _inputs, output, name=name: trace.__setitem__(name, output.detach().cpu()))
+
+report("quant_conv_weight", direct.quant_conv.weight, upstream.quant_conv.weight)
+report("quant_conv_bias", direct.quant_conv.bias, upstream.quant_conv.bias)
+with torch.inference_mode():
+    direct_latent = direct.encode(pixels.clone())
+    upstream_latent = upstream.encode(pixels.to(direct.compute_dtype))
+report("normalized_latent", direct_latent, upstream_latent)
+
+with torch.inference_mode():
+    x = pixels.unsqueeze(2)
+    x = (x + 1.0) * 0.5
+    x = (x - direct.pixel_mean.to(x)) / direct.pixel_std.to(x)
+    params = direct.W
+    x = _causal_conv3d(x, params["conv_in"][0], params["conv_in"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+    report("encoder.conv_in", x, trace.pop("encoder.conv_in"))
+    for level_index, level in enumerate(params["down"]):
+        for block_index, block in enumerate(level["blocks"]):
+            x = _resnet(x, block)
+            name = f"encoder.down.{level_index}.block.{block_index}"
+            report(name, x, trace.pop(name))
+        if level["down"] is not None:
+            x = _downsample(x, level["down"])
+            name = f"encoder.down.{level_index}.downsample"
+            report(name, x, trace.pop(name))
+    x = _group_norm_3d(x, params["norm_out_w"], params["norm_out_b"])
+    report("encoder.norm_out", x, trace.pop("encoder.norm_out"))
+    x = torch.nn.functional.silu(x)
+    x = _causal_conv3d(x, params["conv_out"][0], params["conv_out"][1], kernel_size=3, stride=(1, 1, 1), spatial_padding=1, temporal_causal=True)
+    report("encoder.conv_out", x, trace.pop("encoder.conv_out"))
+    x = torch.nn.functional.conv3d(x, direct.quant_conv.weight, direct.quant_conv.bias)
+    report("quant_conv", x, trace.pop("quant_conv"))
+    report("latents_mean", direct.latents_mean, upstream.latents_mean)
+    report("latents_std", direct.latents_std, upstream.latents_std)
--- a/tools/compare_vae_full_decode.py
+++ b/tools/compare_vae_full_decode.py
@ -51,7 +51,7 @@ with torch.inference_mode():
    torch.cuda.empty_cache()
    gc.collect()

-    upstream = UpstreamVAE().to("cuda").eval()
+    upstream = UpstreamVAE().to("cuda", dtype=torch.float16).eval()
    upstream.load_state_dict(load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda"), strict=True)
    upstream_pixels = pixelize(upstream.decode(latent.to(next(upstream.parameters()).dtype)), frames)

--- a/tools/decode_audio_latent.py
+++ b/tools/decode_audio_latent.py
@ -18,9 +18,9 @@ args = parser.parse_args()

 state = torch.load(args.latent, map_location="cuda", weights_only=False)
 if isinstance(state, dict):
-    latent = state.get("audio_latent", state.get("latent"))
+    latent = state.get("audio_latent", state.get("final_audio", state.get("latent")))
    if latent is None:
-        raise ValueError("saved state does not contain 'audio_latent' or 'latent'")
+        raise ValueError("saved state does not contain 'audio_latent', 'final_audio', or 'latent'")
 else:
    latent = state
 latent = latent.to("cuda")
--- a/tools/decode_video_latent.py
+++ b/tools/decode_video_latent.py
@ -6,7 +6,7 @@ from pathlib import Path

 import torch

-from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE
+from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE, dtype_from_name


 parser = argparse.ArgumentParser()
@ -17,6 +17,8 @@ parser.add_argument("--implementation", choices=("direct", "upstream"), default=
 parser.add_argument("--frames-dir", type=Path)
 parser.add_argument("--ffmpeg-loglevel", default="error")
 parser.add_argument("--quiet", action="store_true")
+parser.add_argument("--vae-dtype", choices=("float32", "float16", "bfloat16"), default="float16")
+parser.add_argument("--vae-tile-size", type=int, default=256)
 args = parser.parse_args()

 state = torch.load(args.latent, map_location="cuda", weights_only=False)
@ -28,7 +30,9 @@ if args.implementation == "direct":
        "/vae/minimax_h3_video_vae_fp16.safetensors",
        device="cuda",
        tiling=not args.no_tiling,
+        dtype=dtype_from_name(args.vae_dtype),
    ).eval()
+    vae.tile_size = args.vae_tile_size
 else:
    from safetensors.torch import load_file

@ -36,7 +40,10 @@ else:

    vae = UpstreamMiniMaxH3VideoVAE(tiling=not args.no_tiling).to("cuda").eval()
    checkpoint = load_file("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda")
+    checkpoint = {key: value.to(dtype=dtype_from_name(args.vae_dtype)) for key, value in checkpoint.items()}
    missing, unexpected = vae.load_state_dict(checkpoint, strict=False)
+    vae = vae.to(dtype=dtype_from_name(args.vae_dtype))
+    vae.tile_size = args.vae_tile_size
    if not args.quiet:
        print({"upstream_missing": len(missing), "upstream_unexpected": len(unexpected)}, flush=True)
 with torch.inference_mode():
@ -61,4 +68,4 @@ subprocess.run([
 ], check=True)
 raw.unlink()
 if not args.quiet:
-    print({"output": str(args.output), "frames": frames, "shape": tuple(pixels.shape), "tiling": not args.no_tiling, "implementation": args.implementation})
+    print({"output": str(args.output), "frames": frames, "shape": tuple(pixels.shape), "tiling": not args.no_tiling, "implementation": args.implementation, "vae_dtype": args.vae_dtype, "vae_tile_size": vae.tile_size})
--- a/tools/direct_t2v_preview.py
+++ b/tools/direct_t2v_preview.py
@ -7,6 +7,7 @@ import subprocess
 import time
 from datetime import datetime, timezone
 import warnings
+import numpy as np

 warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)

@ -14,13 +15,17 @@ import torch

 from h3_blackwell_runtime.checkpoint import H3Checkpoint
 from h3_blackwell_runtime.audio_vae_decoder import MiniMaxH3AudioVAE
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND
+from h3_blackwell_runtime.block import configure_mlp_chunking
 from h3_blackwell_runtime.denoiser import H3PackedDenoiser
 from h3_blackwell_runtime.packing import H3PromptPacker
 from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
 from h3_blackwell_runtime.sampler import sample_video_res_multistep
 from h3_blackwell_runtime.t2v import random_av_latents
+from h3_blackwell_runtime.qwen3vl_vision import build_fl2va_presentation, resize_keyframe
 from h3_blackwell_runtime.token_refiner import H3TokenRefiner
-from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE
+from h3_blackwell_runtime.vae_decoder import MiniMaxH3VideoVAE, dtype_from_name
+from h3_blackwell_runtime.vae_encoder import MiniMaxH3VideoVAEEncoder


 parser = argparse.ArgumentParser()
@ -31,7 +36,9 @@ parser.add_argument("--height", type=int, default=192)
 parser.add_argument("--frames", type=int, default=22)
 parser.add_argument("--steps", type=int, default=12)
 parser.add_argument("--seed", type=int, default=440204)
-parser.add_argument("--attention", choices=("sage2", "sdpa", "sage3"), default="sage2")
+parser.add_argument("--first-frame", type=Path, help="First keyframe image (fl2va), PNG/JPG path.")
+parser.add_argument("--last-frame", type=Path, help="Last keyframe image (fl2va), PNG/JPG path.")
+parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default=DEFAULT_ATTENTION_BACKEND)
 parser.add_argument("--model-timesteps-capture", type=Path, help="Directory containing captured input_XX.pt H3 timesteps for strict parity checks.")
 parser.add_argument("--progress", action="store_true", help="Print per-step sampler progress.")
 parser.add_argument("--profile-memory", action="store_true")
@ -42,6 +49,15 @@ parser.add_argument("--save-audio-latent", type=Path)
 parser.add_argument("--audio-output", type=Path)
 parser.add_argument("--mux-audio", action="store_true")
 parser.add_argument("--skip-decode", action="store_true")
+parser.add_argument("--vae-dtype", choices=("float32", "float16", "bfloat16"), default=os.getenv("H3_VAE_DTYPE", "float16"))
+parser.add_argument("--vae-tile-size", type=int, default=int(os.getenv("H3_VAE_TILE_SIZE", "256")))
+parser.add_argument("--mlp-chunks", type=int, default=int(os.getenv("H3_MLP_CHUNKS", "1")))
+parser.add_argument("--mlp-chunk-threshold", type=int, default=int(os.getenv("H3_MLP_CHUNK_THRESHOLD", "4096")))
+parser.add_argument("--cache-mode", choices=("disabled", "easycache", "h3_cache"), default="disabled")
+parser.add_argument("--cache-threshold", type=float, default=0.0)
+parser.add_argument("--cache-start-percent", type=float, default=0.0)
+parser.add_argument("--cache-end-percent", type=float, default=1.0)
+parser.add_argument("--cache-subsample-factor", type=int, default=2)
 args = parser.parse_args()
 started = time.perf_counter()
 last_report = started
@ -84,9 +100,8 @@ conditioner = Qwen3VLPromptConditioner(
 report_memory("qwen_loaded")
 video, audio, frames = random_av_latents(args.width, args.height, args.frames, args.seed)
 model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend=args.attention).eval()
+configure_mlp_chunking(model, args.mlp_chunks, args.mlp_chunk_threshold)
 report_memory("h3_loaded")
-text = H3TokenRefiner(checkpoint, attention_backend=args.attention)(conditioner(args.prompt))
-report_memory("text_conditioned")
 model_timesteps = None
 if args.model_timesteps_capture is not None:
    model_timesteps = [
@ -94,7 +109,93 @@ if args.model_timesteps_capture is not None:
        for index in range(args.steps)
    ]
 want_audio = args.save_audio_latent is not None or args.audio_output is not None or args.mux_audio
-sampled = sample_video_res_multistep(model, H3PromptPacker(checkpoint), text, video, audio, steps=args.steps, model_timesteps=model_timesteps, return_audio=want_audio, progress=args.progress)
+cache_stats = {}
+refiner = H3TokenRefiner(checkpoint, attention_backend=args.attention)
+packer = H3PromptPacker(checkpoint)
+
+if args.first_frame is not None or args.last_frame is not None:
+    from PIL import Image
+
+    def load_image(path: Path) -> torch.Tensor:
+        img = Image.open(path).convert("RGB")
+        return torch.from_numpy(np.array(img)).unsqueeze(0).float() / 255.0
+
+    first = load_image(args.first_frame) if args.first_frame is not None else None
+    last = load_image(args.last_frame) if args.last_frame is not None else None
+    vae_encoder = MiniMaxH3VideoVAEEncoder.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda").eval()
+    report_memory("vae_encoder_loaded")
+    from h3_blackwell_runtime.qwen3vl_vision import Qwen3VL32BVision
+
+    vision_tower = Qwen3VL32BVision(
+        "/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", device="cuda", dtype=torch.float32
+    )
+    report_memory("vision_tower_loaded")
+    presentation = build_fl2va_presentation(
+        args.prompt,
+        first,
+        last,
+        width=args.width,
+        height=args.height,
+        frame_count=frames,
+        tokenizer=conditioner.tokenizer,
+        vision=vision_tower,
+        text_encoder=conditioner.encoder,
+        device="cuda",
+    )
+    cond_latents = []
+    for kf in presentation.keyframes:
+        resized = resize_keyframe(kf["image"].cuda(), args.width, args.height, crop="disabled" if kf["resolved_frame_index"] == 0 else "center")
+        pixels = resized.movedim(-1, 1).cuda().float().mul(2.0).sub(1.0)
+        cond_latents.append(vae_encoder.encode(pixels))
+    report_memory("fl2va_conditioned")
+    text = refiner(presentation.text_states)
+    report_memory("text_conditioned")
+    seed = args.seed
+    sampled = sample_video_res_multistep(
+        model,
+        packer,
+        text,
+        video,
+        audio,
+        steps=args.steps,
+        model_timesteps=model_timesteps,
+        return_audio=want_audio,
+        progress=args.progress,
+        seed=seed,
+        text_token_tags=presentation.text_token_tags,
+        cond_latents=cond_latents,
+        cond_frame_indices=[kf["resolved_frame_index"] for kf in presentation.keyframes],
+        frame_count=frames,
+        cache_mode=args.cache_mode,
+        cache_threshold=args.cache_threshold,
+        cache_start_percent=args.cache_start_percent,
+        cache_end_percent=args.cache_end_percent,
+        cache_subsample_factor=args.cache_subsample_factor,
+        cache_stats=cache_stats,
+    )
+else:
+    text = refiner(conditioner(args.prompt))
+    report_memory("text_conditioned")
+    sampled = sample_video_res_multistep(
+        model,
+        packer,
+        text,
+        video,
+        audio,
+        steps=args.steps,
+        model_timesteps=model_timesteps,
+        return_audio=want_audio,
+        progress=args.progress,
+        seed=args.seed,
+        cache_mode=args.cache_mode,
+        cache_threshold=args.cache_threshold,
+        cache_start_percent=args.cache_start_percent,
+        cache_end_percent=args.cache_end_percent,
+        cache_subsample_factor=args.cache_subsample_factor,
+        cache_stats=cache_stats,
+    )
+if cache_stats:
+    report({"cache": cache_stats})
 if want_audio:
    latent, audio_latent = sampled
 else:
@ -115,7 +216,8 @@ if args.save_audio_latent is not None:
    report({"audio_latent": str(args.save_audio_latent)})
 if args.skip_decode:
    raise SystemExit(0)
-vae = MiniMaxH3VideoVAE.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda").eval()
+vae = MiniMaxH3VideoVAE.from_safetensors("/vae/minimax_h3_video_vae_fp16.safetensors", device="cuda", dtype=dtype_from_name(args.vae_dtype)).eval()
+vae.tile_size = args.vae_tile_size
 report_memory("vae_loaded")
 with torch.inference_mode():
    pixels = vae.decode(latent.to(next(vae.parameters()).dtype))[:, :, :frames]
--- a/tools/distributed_t2va.py
+++ b/tools/distributed_t2va.py
@ -0,0 +1,185 @@
+"""Run prompt-only H3 T2VA with Ulysses or TP+sequence parallelism."""
+
+import argparse
+import json
+import os
+import time
+from pathlib import Path
+
+import torch
+import torch.distributed as dist
+
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.denoiser import H3PackedDenoiser
+from h3_blackwell_runtime.distributed import SequenceParallelContext
+from h3_blackwell_runtime.packing import H3PromptPacker
+from h3_blackwell_runtime.qwen3vl_text import Qwen3VLPromptConditioner
+from h3_blackwell_runtime.sampler import sample_video_res_multistep
+from h3_blackwell_runtime.t2v import random_av_latents
+from h3_blackwell_runtime.tensor_parallel import configure_h3_tensor_parallel
+from h3_blackwell_runtime.token_refiner import H3TokenRefiner
+
+
+class ParallelDenoiser:
+    def __init__(self, model: H3PackedDenoiser, mode: str):
+        self.model = model
+        self.mode = mode
+
+    def __call__(self, hidden, timesteps, positions, segments, video_segment, audio_segment):
+        context = SequenceParallelContext.create(hidden.shape[0], heads=56, head_dim=128)
+        if self.mode == "ulysses":
+            return self.model.forward_sequence_parallel(
+                hidden, timesteps, positions, segments, video_segment, audio_segment, context,
+            )
+        return self.model.forward_tensor_parallel(
+            hidden, timesteps, positions, segments, video_segment, audio_segment, context,
+        )
+
+
+def synchronize(device: torch.device) -> None:
+    torch.cuda.synchronize(device)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--benchmark", type=Path, required=True)
+parser.add_argument("--mode", choices=("ulysses", "tensor"), required=True)
+parser.add_argument("--attention", default="sdpa")
+parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+parser.add_argument("--text-encoder", default="/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors")
+parser.add_argument("--save-latent", type=Path)
+parser.add_argument("--report", type=Path)
+args = parser.parse_args()
+
+local_rank = int(os.environ["LOCAL_RANK"])
+torch.cuda.set_device(local_rank)
+device = torch.device(f"cuda:{local_rank}")
+dist.init_process_group("nccl", device_id=device)
+rank = dist.get_rank()
+world_size = dist.get_world_size()
+benchmark = json.loads(args.benchmark.read_text(encoding="utf-8"))
+
+load_started = time.perf_counter()
+checkpoint = H3Checkpoint(args.model, device=device)
+model = H3PackedDenoiser.from_checkpoint(
+    checkpoint, output_dtype=torch.bfloat16, attention_backend=args.attention,
+).eval()
+packer = H3PromptPacker(checkpoint)
+if args.mode == "tensor":
+    partition_context = SequenceParallelContext.create(world_size, heads=56, head_dim=128)
+    configure_h3_tensor_parallel(model, partition_context)
+synchronize(device)
+model_load_seconds = time.perf_counter() - load_started
+model_load_peak = torch.cuda.max_memory_allocated(device)
+torch.cuda.reset_peak_memory_stats(device)
+
+conditioning_started = time.perf_counter()
+if rank == 0:
+    conditioner = Qwen3VLPromptConditioner(args.text_encoder, device=device, dtype=torch.float32)
+    refiner = H3TokenRefiner(checkpoint, attention_backend="sdpa").eval()
+    text = refiner(conditioner(benchmark["prompt"])).to(torch.bfloat16)
+    text_length = torch.tensor([text.shape[1]], dtype=torch.int64, device=device)
+    del conditioner, refiner
+else:
+    text = None
+    text_length = torch.zeros(1, dtype=torch.int64, device=device)
+checkpoint.release_cache()
+torch.cuda.empty_cache()
+dist.broadcast(text_length, src=0)
+if rank != 0:
+    text = torch.empty(1, int(text_length.item()), 5376, dtype=torch.bfloat16, device=device)
+dist.broadcast(text, src=0)
+synchronize(device)
+conditioning_seconds = time.perf_counter() - conditioning_started
+conditioning_peak = torch.cuda.max_memory_allocated(device)
+torch.cuda.reset_peak_memory_stats(device)
+
+width, height = benchmark["resolution"]
+video, audio, aligned_frames = random_av_latents(
+    width, height, benchmark["frames"], benchmark["seed"], device=device,
+)
+dist.barrier()
+synchronize(device)
+sampling_started = time.perf_counter()
+video, audio = sample_video_res_multistep(
+    ParallelDenoiser(model, args.mode),
+    packer,
+    text,
+    video,
+    audio,
+    steps=benchmark["steps"],
+    seed=benchmark["seed"],
+    return_audio=True,
+    progress=rank == 0,
+)
+synchronize(device)
+sampling_seconds = time.perf_counter() - sampling_started
+
+timing = torch.tensor(
+    [model_load_seconds, conditioning_seconds, sampling_seconds],
+    dtype=torch.float64,
+    device=device,
+)
+timings = [torch.empty_like(timing) for _ in range(world_size)]
+dist.all_gather(timings, timing)
+checksums = torch.stack((video.float().sum(), audio.float().sum())).to(torch.float64)
+all_checksums = [torch.empty_like(checksums) for _ in range(world_size)]
+dist.all_gather(all_checksums, checksums)
+checksum_stack = torch.stack(all_checksums)
+if not torch.allclose(checksum_stack, checksum_stack[0].expand_as(checksum_stack), rtol=0, atol=1e-5):
+    raise RuntimeError(f"rank outputs diverged: {checksum_stack.cpu().tolist()}")
+
+peak_memory = torch.tensor(
+    [model_load_peak, conditioning_peak, torch.cuda.max_memory_allocated(device)],
+    dtype=torch.int64,
+    device=device,
+)
+memory = [torch.empty_like(peak_memory) for _ in range(world_size)]
+dist.all_gather(memory, peak_memory)
+
+if rank == 0:
+    if args.save_latent is not None:
+        args.save_latent.parent.mkdir(parents=True, exist_ok=True)
+        torch.save({
+            "latent": video.cpu(),
+            "audio_latent": audio.cpu(),
+            "frames": aligned_frames,
+            "width": width,
+            "height": height,
+            "prompt": benchmark["prompt"],
+            "seed": benchmark["seed"],
+            "distributed_mode": args.mode,
+            "world_size": world_size,
+            "attention": args.attention,
+        }, args.save_latent)
+    timing_stack = torch.stack(timings).cpu()
+    report = {
+        "mode": args.mode,
+        "world_size": world_size,
+        "attention": args.attention,
+        "device": torch.cuda.get_device_name(device),
+        "torch": torch.__version__,
+        "benchmark": str(args.benchmark),
+        "resolution": [width, height],
+        "frames": aligned_frames,
+        "steps": benchmark["steps"],
+        "seed": benchmark["seed"],
+        "timings_max_rank_seconds": {
+            "model_load": float(timing_stack[:, 0].max()),
+            "conditioning": float(timing_stack[:, 1].max()),
+            "sampling": float(timing_stack[:, 2].max()),
+        },
+        "peak_allocated_bytes_by_rank": {
+            "model_load": [int(value[0].item()) for value in memory],
+            "conditioning": [int(value[1].item()) for value in memory],
+            "sampling": [int(value[2].item()) for value in memory],
+        },
+        "checksums": checksum_stack[0].cpu().tolist(),
+        "latent": str(args.save_latent) if args.save_latent is not None else None,
+    }
+    serialized = json.dumps(report, indent=2)
+    if args.report is not None:
+        args.report.parent.mkdir(parents=True, exist_ok=True)
+        args.report.write_text(serialized + "\n", encoding="utf-8")
+    print(serialized)
+
+dist.destroy_process_group()
--- a/tools/inspect_checkpoint_tensors.py
+++ b/tools/inspect_checkpoint_tensors.py
@ -1,10 +1,18 @@
 """List selected H3 checkpoint tensor shapes."""

+import argparse
+
 from safetensors import safe_open


-path = "/models/minimax_h3_ref2va_pruned_nvfp4.safetensors"
-with safe_open(path, framework="pt", device="cpu") as checkpoint:
+parser = argparse.ArgumentParser()
+parser.add_argument("--checkpoint", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+parser.add_argument("--prefix", action="append", default=[])
+args = parser.parse_args()
+prefixes = tuple(args.prefix) or ("adaln_t_table", "blocks.0.adaln_proj", "final_layer.adaln_proj")
+
+with safe_open(args.checkpoint, framework="pt", device="cpu") as checkpoint:
    for name in checkpoint.keys():
-        if name == "adaln_t_table" or name.startswith("blocks.0.adaln_proj") or name.startswith("final_layer.adaln_proj"):
-            print(name, tuple(checkpoint.get_tensor(name).shape))
+        if any(name == prefix or name.startswith(prefix) for prefix in prefixes):
+            tensor = checkpoint.get_tensor(name)
+            print(name, tuple(tensor.shape), tensor.dtype)
--- a/tools/patch_comfy_qwen_vision_capture.py
+++ b/tools/patch_comfy_qwen_vision_capture.py
@ -0,0 +1,74 @@
+"""Capture actual loaded-Comfy Qwen merged and DeepStack vision tensors."""
+
+from pathlib import Path
+
+
+path = Path("/opt/ComfyUI/comfy/text_encoders/qwen3vl.py")
+source = path.read_text(encoding="utf-8")
+if "import os\n" not in source:
+    source = source.replace("import os\n", "import os\n", 1) if "import os\n" in source else "import os\n" + source
+if "import traceback\n" not in source:
+    source = "import traceback\n" + source
+old = (
+    "            merged, deepstack = self.visual(image.to(device, dtype=torch.float32), grid)\n"
+    "            return merged, {\"grid\": grid, \"deepstack\": deepstack}\n"
+)
+new = (
+    "            merged, deepstack = self.visual(image.to(device, dtype=torch.float32), grid)\n"
+    "            capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n"
+    "            if capture_dir:\n"
+    "                os.makedirs(capture_dir, exist_ok=True)\n"
+    "                capture_index = getattr(self, \"_h3_vision_capture_index\", 0)\n"
+    "                torch.save({\"merged\": merged.detach().cpu(), \"deepstack\": [value.detach().cpu() for value in deepstack], \"pixel_values\": image.detach().cpu(), \"grid\": grid.detach().cpu(), \"stack\": traceback.format_stack()}, os.path.join(capture_dir, f\"qwen_vision_{capture_index}.pt\"))\n"
+    "                open(os.path.join(capture_dir, \"qwen_vision_stack.txt\"), \"w\", encoding=\"utf-8\").writelines(traceback.format_stack())\n"
+    "                self._h3_vision_capture_index = capture_index + 1\n"
+    "            return merged, {\"grid\": grid, \"deepstack\": deepstack}\n"
+)
+if source.count(old) != 1:
+    raise RuntimeError("Unable to locate Qwen3-VL vision preprocess block.")
+path.write_text(source.replace(old, new), encoding="utf-8")
+
+vision_path = Path("/opt/ComfyUI/comfy/text_encoders/qwen35.py")
+vision_source = vision_path.read_text(encoding="utf-8")
+vision_source = vision_source.replace(
+    "        x = self.patch_embed(x)\n        pos_embeds = self.fast_pos_embed_interpolate(grid_thw).to(x.device)\n        x = x + pos_embeds\n",
+    "        x = self.patch_embed(x)\n        capture_trace = {\"patch_embed\": x.detach().cpu()}\n        pos_embeds = self.fast_pos_embed_interpolate(grid_thw).to(x.device)\n        capture_trace[\"position_embed\"] = pos_embeds.detach().cpu()\n        x = x + pos_embeds\n        capture_trace[\"vision_input\"] = x.detach().cpu()\n",
+    1,
+)
+vision_source = vision_source.replace(
+    "            x = blk(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings, optimized_attention=optimized_attention)\n",
+    "            x = blk(x, cu_seqlens=cu_seqlens, position_embeddings=position_embeddings, optimized_attention=optimized_attention)\n            capture_trace[f\"block_{layer_num:02d}\"] = x.detach().cpu()\n",
+    1,
+)
+vision_source = vision_source.replace(
+    "        merged = self.merger(x)\n        if self.deepstack_merger_list is not None:\n",
+    "        merged = self.merger(x)\n        capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n        if capture_dir:\n            trace_index = getattr(self, \"_h3_vision_trace_index\", 0)\n            torch.save(capture_trace, os.path.join(capture_dir, f\"qwen_vision_trace_{trace_index}.pt\"))\n            self._h3_vision_trace_index = trace_index + 1\n        if self.deepstack_merger_list is not None:\n",
+    1,
+)
+vision_path.write_text(vision_source, encoding="utf-8")
+
+clip_path = Path("/opt/ComfyUI/comfy/sd1_clip.py")
+clip_source = clip_path.read_text(encoding="utf-8")
+if "import os\n" not in clip_source:
+    clip_source = "import os\n" + clip_source
+clip_source = clip_source.replace(
+    "            tokens_embed = torch.tensor([tokens_temp], device=device, dtype=torch.long)\n            tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)\n",
+    "            tokens_embed = torch.tensor([tokens_temp], device=device, dtype=torch.long)\n            capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n            if capture_dir:\n                torch.save(tokens_embed.detach().cpu(), os.path.join(capture_dir, \"qwen_compact_token_ids.pt\"))\n            tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)\n",
+    1,
+)
+clip_path.write_text(clip_source, encoding="utf-8")
+
+nodes_path = Path("/opt/ComfyUI/comfy_extras/nodes_minimax_h3.py")
+nodes_source = nodes_path.read_text(encoding="utf-8")
+if "import os\n" not in nodes_source:
+    nodes_source = "import os\n" + nodes_source
+vae_encode_line = "kf[\"latent\"] = vae.encode(kf.pop(\"image\"))"
+if nodes_source.count(vae_encode_line) != 1:
+    raise RuntimeError("Unable to locate MiniMax H3 keyframe VAE encode call.")
+nodes_source = nodes_source.replace(
+    vae_encode_line,
+    "keyframe_image = kf.pop(\"image\")\n                kf[\"latent\"] = vae.encode(keyframe_image)\n                capture_dir = os.getenv(\"H3_CAPTURE_DIR\")\n                if capture_dir:\n                    capture_index = getattr(vae, \"_h3_vae_capture_index\", 0)\n                    vae_model = vae.first_stage_model\n                    vae_meta = {\"parameter_dtype\": str(next(vae_model.parameters()).dtype), \"tiling\": vae_model.tiling, \"tile_size\": vae_model.tile_size, \"tile_overlap_min\": vae_model.tile_overlap_min}\n                    torch.save({\"image\": keyframe_image.detach().cpu(), \"latent\": kf[\"latent\"].detach().cpu(), \"meta\": vae_meta}, os.path.join(capture_dir, f\"vae_keyframe_{capture_index}.pt\"))\n                    vae._h3_vae_capture_index = capture_index + 1",
+    1,
+)
+nodes_path.write_text(nodes_source, encoding="utf-8")
+print("Applied Qwen3-VL vision capture patch.")
--- a/tools/poll_qwen38_health_spark.sh
+++ b/tools/poll_qwen38_health_spark.sh
@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -u
+
+for i in $(seq 0 39); do
+  status=$(docker ps -a --filter name=qwen38-vllm --format '{{.Status}}')
+  listening=$(ss -ltnp | grep -c ':8002 ' || true)
+
+  if curl -fsS --max-time 5 http://127.0.0.1:8002/health >/tmp/qwen38-health.out 2>/tmp/qwen38-health.err; then
+    echo "HEALTHY poll=$i status=$status listening=$listening"
+    docker logs --tail 60 qwen38-vllm
+    exit 0
+  fi
+
+  if [[ $status != Up* ]]; then
+    echo "NOT_RUNNING poll=$i status=$status listening=$listening"
+    docker logs --tail 160 qwen38-vllm
+    exit 2
+  fi
+
+  echo "poll=$i status=$status listening=$listening"
+  docker logs --tail 10 qwen38-vllm
+  sleep 30
+done
+
+echo "TIMEOUT"
+docker logs --tail 200 qwen38-vllm
+exit 1
--- a/tools/probe_audio_vae_boundary.py
+++ b/tools/probe_audio_vae_boundary.py
@ -0,0 +1,182 @@
+"""Decode controlled H3 audio-latent boundary variants for diagnosis."""
+
+import argparse
+import json
+import math
+import subprocess
+from pathlib import Path
+
+import numpy as np
+import torch
+
+from h3_blackwell_runtime.audio_vae_decoder import MiniMaxH3AudioVAE
+
+
+SAMPLE_RATE = 32000
+SAMPLES_PER_LATENT = 800
+
+
+def load_latent(path: Path) -> torch.Tensor:
+    state = torch.load(path, map_location="cpu", weights_only=False)
+    latent = state.get("audio_latent") if isinstance(state, dict) else state
+    if latent is None or latent.ndim != 4:
+        raise ValueError(f"{path} does not contain a [B,C,S,T] audio latent")
+    return latent
+
+
+def dbfs(value: float) -> float:
+    return 20.0 * math.log10(max(value, 1e-20))
+
+
+def waveform_metrics(waveform: torch.Tensor) -> dict:
+    samples = waveform.float().numpy().T
+    first_100ms = samples[: SAMPLE_RATE // 10]
+    first_500ms = samples[: SAMPLE_RATE // 2]
+    derivatives = np.max(np.abs(np.diff(first_500ms, axis=0)), axis=1)
+    return {
+        "samples": len(samples),
+        "first_sample": samples[0].tolist(),
+        "first_sample_dbfs": [dbfs(float(abs(value))) for value in samples[0]],
+        "first_100ms_peak_dbfs": dbfs(float(np.max(np.abs(first_100ms)))),
+        "first_100ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_100ms**2)))),
+        "first_500ms_peak_dbfs": dbfs(float(np.max(np.abs(first_500ms)))),
+        "first_500ms_rms_dbfs": dbfs(float(np.sqrt(np.mean(first_500ms**2)))),
+        "largest_derivative": float(derivatives.max()),
+        "largest_derivative_ms": float((derivatives.argmax() + 1) * 1000.0 / SAMPLE_RATE),
+    }
+
+
+def comparison(reference: torch.Tensor, candidate: torch.Tensor) -> dict:
+    count = min(reference.shape[-1], candidate.shape[-1])
+    reference = reference[..., :count].float()
+    candidate = candidate[..., :count].float()
+
+    def region_metrics(samples: int) -> dict:
+        ref = reference[..., :samples]
+        test = candidate[..., :samples]
+        error = test - ref
+        signal_power = ref.square().mean(dim=-1)
+        noise_power = error.square().mean(dim=-1)
+        psnr = 10.0 * torch.log10(signal_power.clamp_min(1e-30) / noise_power.clamp_min(1e-30))
+        return {
+            "rmse": float(error.square().mean().sqrt()),
+            "max_abs": float(error.abs().max()),
+            "psnr_db_by_channel": psnr.flatten().tolist(),
+        }
+
+    return {
+        "first_100ms": region_metrics(SAMPLE_RATE // 10),
+        "first_500ms": region_metrics(SAMPLE_RATE // 2),
+        "full": region_metrics(count),
+    }
+
+
+def write_waveform(path: Path, waveform: torch.Tensor) -> None:
+    raw = path.with_suffix(".f32le")
+    waveform.transpose(0, 1).contiguous().numpy().tofile(raw)
+    subprocess.run([
+        "ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
+        "-f", "f32le", "-ar", str(SAMPLE_RATE), "-ac", "2", "-i", str(raw),
+        "-c:a", "pcm_f32le", str(path),
+    ], check=True)
+    raw.unlink()
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--affected-latent", type=Path, required=True)
+parser.add_argument("--clean-latent", type=Path, required=True)
+parser.add_argument("--vae", type=Path, default=Path("/vae/minimax_h3_audio_vae_fp32.safetensors"))
+parser.add_argument("--output-dir", type=Path, required=True)
+parser.add_argument("--report", type=Path)
+args = parser.parse_args()
+
+affected = load_latent(args.affected_latent)
+clean = load_latent(args.clean_latent)
+if affected.shape != clean.shape:
+    raise ValueError(f"latent shapes differ: {tuple(affected.shape)} != {tuple(clean.shape)}")
+
+boundary_frames = 4
+variants = {
+    "affected-original": (affected, 0),
+    "clean-original": (clean, 0),
+    "zero-normalized-latent": (torch.zeros_like(affected), 0),
+    "affected-repeat-frame0": (affected[..., :1].expand_as(affected).clone(), 0),
+    "affected-repeat-frame4": (affected[..., 4:5].expand_as(affected).clone(), 0),
+}
+
+silent_carrier = affected[..., 4:5].expand_as(affected).clone()
+carrier_start = silent_carrier.clone()
+carrier_start[..., :boundary_frames] = affected[..., :boundary_frames]
+variants["carrier-affected-first4-at-start"] = (carrier_start, 0)
+
+interior_frame = 40
+carrier_interior = silent_carrier.clone()
+carrier_interior[..., interior_frame : interior_frame + boundary_frames] = affected[..., :boundary_frames]
+variants["carrier-affected-first4-at-frame40"] = (carrier_interior, 0)
+
+replaced_with_frame4 = affected.clone()
+replaced_with_frame4[..., :boundary_frames] = affected[..., 4:5]
+variants["affected-first4-repeat-frame4"] = (replaced_with_frame4, 0)
+
+affected_with_clean = affected.clone()
+affected_with_clean[..., :boundary_frames] = clean[..., :boundary_frames]
+variants["affected-first4-from-clean"] = (affected_with_clean, 0)
+
+clean_with_affected = clean.clone()
+clean_with_affected[..., :boundary_frames] = affected[..., :boundary_frames]
+variants["clean-first4-from-affected"] = (clean_with_affected, 0)
+
+prefix_repeat = affected[..., :1].expand(*affected.shape[:-1], boundary_frames)
+variants["affected-prefix-repeat-frame0"] = (
+    torch.cat((prefix_repeat, affected), dim=-1),
+    boundary_frames * SAMPLES_PER_LATENT,
+)
+variants["affected-prefix-own-first4"] = (
+    torch.cat((affected[..., :boundary_frames], affected), dim=-1),
+    boundary_frames * SAMPLES_PER_LATENT,
+)
+
+vae = MiniMaxH3AudioVAE.from_safetensors(args.vae, device="cuda").eval()
+args.output_dir.mkdir(parents=True, exist_ok=True)
+decoded = {}
+report = {"boundary_frames": boundary_frames, "variants": {}}
+with torch.inference_mode():
+    for name, (latent, crop_start) in variants.items():
+        waveform = vae.decode(latent.to("cuda", dtype=next(vae.parameters()).dtype)).cpu()[0]
+        waveform = waveform[:, crop_start : crop_start + affected.shape[-1] * SAMPLES_PER_LATENT]
+        decoded[name] = waveform
+        output = args.output_dir / f"{name}.wav"
+        write_waveform(output, waveform)
+        report["variants"][name] = {
+            "output": str(output),
+            "crop_start_samples": crop_start,
+            "metrics": waveform_metrics(waveform),
+        }
+
+affected_reference = decoded["affected-original"]
+for name, waveform in decoded.items():
+    if name != "affected-original":
+        report["variants"][name]["difference_from_affected_original"] = comparison(
+            affected_reference, waveform,
+        )
+
+interior_start = interior_frame * SAMPLES_PER_LATENT
+segment_samples = boundary_frames * SAMPLES_PER_LATENT
+report["interior_placement"] = {
+    "frame": interior_frame,
+    "start_seconds": interior_start / SAMPLE_RATE,
+    "affected_onset_vs_carrier_interior_event": comparison(
+        affected_reference[..., :segment_samples],
+        decoded["carrier-affected-first4-at-frame40"][..., interior_start : interior_start + segment_samples],
+    ),
+    "carrier_start_event_vs_carrier_interior_event": comparison(
+        decoded["carrier-affected-first4-at-start"][..., :segment_samples],
+        decoded["carrier-affected-first4-at-frame40"][..., interior_start : interior_start + segment_samples],
+    ),
+}
+
+serialized = json.dumps(report, indent=2)
+if args.report is not None:
+    args.report.parent.mkdir(parents=True, exist_ok=True)
+    args.report.write_text(serialized + "\n", encoding="utf-8")
+print(serialized)
--- a/tools/profile_attention_path.py
+++ b/tools/profile_attention_path.py
@ -0,0 +1,252 @@
+"""Microbenchmark H3 attention kernels and Q/K/V layout costs from captured real block tensors."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+import warnings
+from pathlib import Path
+
+warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)
+
+import torch
+
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, qkv_to_bshd, rms_rope_split_half_, run_attention, run_sol_attention_bshd
+from h3_blackwell_runtime.adaln import H3CurveAdaLN
+from h3_blackwell_runtime.block import H3DiTBlock, modulate_segments
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.packing import H3PromptPacker
+from h3_blackwell_runtime.rope import h3_rope_rotation
+from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas
+from h3_blackwell_runtime.t2v import random_av_latents
+from h3_blackwell_runtime.attention import rms_norm
+
+
+def sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def summarize(values: list[float]) -> dict[str, float]:
+    ordered = sorted(values)
+
+    def percentile(percent: float) -> float:
+        if len(ordered) == 1:
+            return ordered[0]
+        rank = (len(ordered) - 1) * percent
+        low = int(rank)
+        high = min(low + 1, len(ordered) - 1)
+        weight = rank - low
+        return ordered[low] * (1.0 - weight) + ordered[high] * weight
+
+    return {
+        "count": len(values),
+        "mean_s": sum(values) / len(values),
+        "p50_s": percentile(0.50),
+        "p90_s": percentile(0.90),
+        "p95_s": percentile(0.95),
+        "p99_s": percentile(0.99),
+        "min_s": ordered[0],
+        "max_s": ordered[-1],
+    }
+
+
+def timed(stats: dict[str, list[float]], name: str, fn):
+    sync()
+    started = time.perf_counter()
+    value = fn()
+    sync()
+    stats.setdefault(name, []).append(time.perf_counter() - started)
+    return value
+
+
+def prepare_qkv(block, x: torch.Tensor, rotation: torch.Tensor, segment: tuple[int, int, int] | None):
+    attention = block.attention
+    sequence = x.shape[0]
+    inner = attention.heads * attention.head_dim
+    qkv = attention.qkv_proj(x)
+    q, k, v = qkv.split(inner, dim=-1)
+    q = q.view(1, sequence, attention.heads, attention.head_dim)
+    k = k.view(1, sequence, attention.heads, attention.head_dim)
+    v = v.view(1, sequence, attention.heads, attention.head_dim)
+    q, k = rms_rope_split_half_(q, k, rotation, attention.q_norm_weight, attention.k_norm_weight, attention.eps)
+    full_qkv = qkv
+    if segment is not None:
+        start, end, _kind = segment
+        q = q[:, start:end].contiguous()
+        k = k[:, start:end].contiguous()
+        v = v[:, start:end].contiguous()
+        full_qkv = None
+    return q, k, v, full_qkv
+
+
+def representative_attention_inputs(args: argparse.Namespace):
+    torch.manual_seed(args.seed)
+    checkpoint = H3Checkpoint(args.model_path, device=args.device)
+    block = H3DiTBlock.from_checkpoint(checkpoint, args.block_index, attention_backend=args.attention).eval()
+    adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{args.block_index}.adaln_proj").eval()
+    packer = H3PromptPacker(checkpoint)
+
+    video, audio, aligned_frames = random_av_latents(args.width, args.height, args.frames, args.seed, device=args.device)
+    sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1]
+    native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma)
+    text = torch.randn(1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16)
+    hidden, timesteps, segments, positions, _, _ = packer(text, video, native_audio, _model_sigma(sigma))
+    rotation = h3_rope_rotation(positions.to(args.device), checkpoint.tensor("rope.inv_freq", dtype=torch.float32), hidden.dtype)
+    shift_msa, scale_msa, _gate_msa, _shift_mlp, _scale_mlp, _gate_mlp = adaln(timesteps)
+    with torch.inference_mode():
+        h_msa = modulate_segments(rms_norm(hidden, block.norm1_weight, block.norm_eps), shift_msa, scale_msa, segments)
+    metadata = {
+        "width": args.width,
+        "height": args.height,
+        "frames": aligned_frames,
+        "steps": args.steps,
+        "sampler_step": args.sampler_step,
+        "seed": args.seed,
+        "text_tokens": args.text_tokens,
+        "block_index": args.block_index,
+        "attention": args.attention,
+        "hidden_shape": list(hidden.shape),
+        "h_msa_shape": list(h_msa.shape),
+        "segments": segments,
+    }
+    return block, h_msa, rotation, segments, metadata
+
+
+def run_path(q_src: torch.Tensor, k_src: torch.Tensor, v_src: torch.Tensor, backend: str, stats: dict[str, list[float]] | None = None):
+    sequence = q_src.shape[1]
+    inner = q_src.shape[2] * q_src.shape[3]
+    q = timed(stats, "q_transpose_contiguous", lambda: q_src.transpose(1, 2).contiguous()) if stats is not None else q_src.transpose(1, 2).contiguous()
+    k = timed(stats, "k_transpose_contiguous", lambda: k_src.transpose(1, 2).contiguous()) if stats is not None else k_src.transpose(1, 2).contiguous()
+    v = timed(stats, "v_transpose_contiguous", lambda: v_src.transpose(1, 2).contiguous()) if stats is not None else v_src.transpose(1, 2).contiguous()
+    out = timed(stats, "attention_kernel", lambda: run_attention(q, k, v, backend=backend, is_causal=False)) if stats is not None else run_attention(q, k, v, backend=backend, is_causal=False)
+    rows = timed(stats, "output_reshape", lambda: out.transpose(1, 2).reshape(sequence, inner).contiguous()) if stats is not None else out.transpose(1, 2).reshape(sequence, inner).contiguous()
+    return rows
+
+
+def run_sol_native_path(q_src: torch.Tensor, k_src: torch.Tensor, v_src: torch.Tensor, stats: dict[str, list[float]] | None = None):
+    sequence = q_src.shape[1]
+    inner = q_src.shape[2] * q_src.shape[3]
+    q = timed(stats, "q_bshd_contiguous", lambda: q_src.contiguous()) if stats is not None else q_src.contiguous()
+    k = timed(stats, "k_bshd_contiguous", lambda: k_src.contiguous()) if stats is not None else k_src.contiguous()
+    v = timed(stats, "v_bshd_contiguous", lambda: v_src.contiguous()) if stats is not None else v_src.contiguous()
+    out = timed(stats, "attention_kernel", lambda: run_sol_attention_bshd(q, k, v, is_causal=False)) if stats is not None else run_sol_attention_bshd(q, k, v, is_causal=False)
+    return timed(stats, "output_reshape", lambda: out.reshape(sequence, inner).contiguous()) if stats is not None else out.reshape(sequence, inner).contiguous()
+
+
+def run_sol_fused_path(qkv: torch.Tensor, heads: int, head_dim: int, stats: dict[str, list[float]] | None = None):
+    sequence = qkv.shape[0]
+    inner = heads * head_dim
+    q, k, v = timed(stats, "qkv_to_bshd", lambda: qkv_to_bshd(qkv, heads, head_dim)) if stats is not None else qkv_to_bshd(qkv, heads, head_dim)
+    out = timed(stats, "attention_kernel", lambda: run_sol_attention_bshd(q, k, v, is_causal=False)) if stats is not None else run_sol_attention_bshd(q, k, v, is_causal=False)
+    return timed(stats, "output_reshape", lambda: out.reshape(sequence, inner).contiguous()) if stats is not None else out.reshape(sequence, inner).contiguous()
+
+
+def layout_timing_names(layout_mode: str) -> tuple[str, ...]:
+    if layout_mode == "sol_fused":
+        return ("qkv_to_bshd", "attention_kernel", "output_reshape")
+    if layout_mode == "sol_native":
+        return ("q_bshd_contiguous", "k_bshd_contiguous", "v_bshd_contiguous", "attention_kernel", "output_reshape")
+    return ("q_transpose_contiguous", "k_transpose_contiguous", "v_transpose_contiguous", "attention_kernel", "output_reshape")
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+    parser.add_argument("--output", type=Path, default=Path("/output/h3-blackwell-runtime/benchmarks/attention-path-profile.json"))
+    parser.add_argument("--width", type=int, default=960)
+    parser.add_argument("--height", type=int, default=544)
+    parser.add_argument("--frames", type=int, default=124)
+    parser.add_argument("--steps", type=int, default=12)
+    parser.add_argument("--sampler-step", type=int, default=1)
+    parser.add_argument("--seed", type=int, default=440407)
+    parser.add_argument("--text-tokens", type=int, default=93)
+    parser.add_argument("--block-index", type=int, default=24)
+    parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sage2", help="Backend used only while building representative upstream tensors.")
+    parser.add_argument("--backends", nargs="+", choices=AVAILABLE_BACKENDS, default=("sol_attn", "sage2", "sage3", "sage3_mean", "kj_sage_fp8", "kj_sage_fp8pp", "sdpa"))
+    parser.add_argument("--sol-layout", choices=("hnd", "native", "fused", "both", "all"), default="hnd", help="Compare generic HND Sol path with direct BSHD and fused QKV layout paths.")
+    parser.add_argument("--segments", nargs="+", choices=("all", "text", "secondary", "video"), default=("all",))
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--iterations", type=int, default=100)
+    parser.add_argument("--device", default="cuda")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    block, x, rotation, segments, metadata = representative_attention_inputs(args)
+    segment_map = {"all": None, "text": segments[0], "secondary": segments[1], "video": segments[2]}
+
+    results = []
+    with torch.inference_mode():
+        for segment_name in args.segments:
+            q_src, k_src, v_src, qkv_src = prepare_qkv(block, x, rotation, segment_map[segment_name])
+            reference = None
+            reference_backend = None
+            for backend in args.backends:
+                layout_modes = ["hnd"]
+                if backend == "sol_attn" and args.sol_layout != "hnd":
+                    layout_modes = {
+                        "native": ["sol_native"],
+                        "fused": ["sol_fused"],
+                        "both": ["hnd", "sol_native"],
+                        "all": ["hnd", "sol_native", "sol_fused"],
+                    }[args.sol_layout]
+                for layout_mode in layout_modes:
+                  try:
+                    if layout_mode == "sol_fused" and qkv_src is None:
+                        raise ValueError("sol_fused layout currently requires the full unsegmented QKV tensor")
+                    for _ in range(args.warmup):
+                        if layout_mode == "sol_fused":
+                            run_sol_fused_path(qkv_src, block.attention.heads, block.attention.head_dim)
+                        elif layout_mode == "sol_native":
+                            run_sol_native_path(q_src, k_src, v_src)
+                        else:
+                            run_path(q_src, k_src, v_src, backend)
+                    stats: dict[str, list[float]] = {}
+                    output = None
+                    for _ in range(args.iterations):
+                        if layout_mode == "sol_fused":
+                            output = run_sol_fused_path(qkv_src, block.attention.heads, block.attention.head_dim, stats)
+                        elif layout_mode == "sol_native":
+                            output = run_sol_native_path(q_src, k_src, v_src, stats)
+                        else:
+                            output = run_path(q_src, k_src, v_src, backend, stats)
+                    if reference is None:
+                        reference = output
+                        reference_backend = f"{backend}:{layout_mode}"
+                        diff = {"max": 0.0, "mean": 0.0}
+                    else:
+                        delta = (output.float() - reference.float()).abs()
+                        diff = {"max": delta.max().item(), "mean": delta.mean().item()}
+                    summarized = {name: summarize(values) for name, values in stats.items()}
+                    total_mean = sum(summarized[name]["mean_s"] for name in layout_timing_names(layout_mode))
+                    results.append(
+                        {
+                            "segment": segment_name,
+                            "segment_tuple": segment_map[segment_name],
+                            "backend": backend,
+                            "layout_mode": layout_mode,
+                            "q_shape": list(q_src.shape),
+                            "output_shape": list(output.shape),
+                            "timings": summarized,
+                            "layout_attention_total_mean_s": total_mean,
+                            "reference_backend": reference_backend,
+                            "reference_diff": diff,
+                            "status": "ok",
+                        }
+                    )
+                    print(segment_name, backend, layout_mode, "attn_ms", round(summarized["attention_kernel"]["mean_s"] * 1000, 3), "total_ms", round(total_mean * 1000, 3), flush=True)
+                  except Exception as exc:
+                    results.append({"segment": segment_name, "backend": backend, "layout_mode": layout_mode, "status": "failed", "error": repr(exc)})
+                    print(segment_name, backend, layout_mode, "FAILED", repr(exc), flush=True)
+
+    output = {"metadata": metadata, "segments": segments, "warmup": args.warmup, "iterations": args.iterations, "results": results}
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(output, indent=2), encoding="utf-8")
+    print(json.dumps(output, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/profile_h3_block.py
+++ b/tools/profile_h3_block.py
@ -0,0 +1,247 @@
+"""Profile one representative H3 DiT block's dense/attention hot paths."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+import warnings
+from pathlib import Path
+
+warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)
+
+import torch
+import torch.nn.functional as functional
+
+from h3_blackwell_runtime.adaln import H3CurveAdaLN
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, rms_rope_split_half_, rms_norm, run_attention, run_sol_attention_bshd
+from h3_blackwell_runtime.block import H3DiTBlock, gate_segments, modulate_segments
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.nvfp4 import Nvfp4Linear
+from h3_blackwell_runtime.nvfp4_quant import nvfp4_activation_scale
+from h3_blackwell_runtime.packing import H3PromptPacker
+from h3_blackwell_runtime.rope import h3_rope_rotation
+from h3_blackwell_runtime.sampler import beta_sigmas, _audio_sigma, _model_sigma
+from h3_blackwell_runtime.t2v import random_av_latents
+
+
+CAT_PROMPT = (
+    "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. "
+    "The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a "
+    "sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. "
+    "Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation."
+)
+
+
+def sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def timed(stats: dict[str, list[float]], name: str, fn):
+    sync()
+    started = time.perf_counter()
+    value = fn()
+    sync()
+    stats.setdefault(name, []).append(time.perf_counter() - started)
+    return value
+
+
+def summarize(values: list[float]) -> dict[str, float]:
+    ordered = sorted(values)
+
+    def percentile(percent: float) -> float:
+        if len(ordered) == 1:
+            return ordered[0]
+        rank = (len(ordered) - 1) * percent
+        low = int(rank)
+        high = min(low + 1, len(ordered) - 1)
+        weight = rank - low
+        return ordered[low] * (1.0 - weight) + ordered[high] * weight
+
+    return {
+        "count": len(values),
+        "mean_s": sum(values) / len(values),
+        "p50_s": percentile(0.50),
+        "p90_s": percentile(0.90),
+        "p95_s": percentile(0.95),
+        "p99_s": percentile(0.99),
+        "min_s": ordered[0],
+        "max_s": ordered[-1],
+    }
+
+
+def profiled_nvfp4_linear(stats: dict[str, list[float]], prefix: str, module: Nvfp4Linear, x: torch.Tensor) -> torch.Tensor:
+    from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
+
+    original_shape = x.shape[:-1]
+    flat_x = timed(stats, f"{prefix}.flatten_contiguous", lambda: x.reshape(-1, module.in_features).contiguous())
+    if module.pre_quant_scale is not None:
+        flat_x = timed(stats, f"{prefix}.pre_quant_scale", lambda: flat_x * module.pre_quant_scale.to(flat_x))
+    else:
+        stats.setdefault(f"{prefix}.pre_quant_scale", []).append(0.0)
+    packed_weight = timed(stats, f"{prefix}.packed_weight_wrapper", module._packed_weight)
+    bias = timed(stats, f"{prefix}.bias_cast", lambda: module.bias.to(flat_x) if module.bias is not None else None)
+    if module.full_precision_matrix_mult:
+        weight = timed(stats, f"{prefix}.weight_dequantize", lambda: packed_weight.dequantize().to(flat_x))
+        output = timed(stats, f"{prefix}.gemm", lambda: functional.linear(flat_x, weight, bias))
+        return timed(stats, f"{prefix}.slice_reshape", lambda: output.reshape(*original_shape, module.out_features))
+    if flat_x.dtype == torch.float32:
+        raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.")
+    orig_shape = tuple(flat_x.shape)
+    scale = timed(stats, f"{prefix}.activation_scale", lambda: nvfp4_activation_scale(flat_x))
+    scale = timed(stats, f"{prefix}.scale_to_device", lambda: torch.as_tensor(scale, device=flat_x.device, dtype=torch.float32))
+    qdata, block_scale = timed(
+        stats,
+        f"{prefix}.activation_quant_pack",
+        lambda: __import__("comfy_kitchen").quantize_nvfp4(
+            flat_x,
+            scale,
+            pad_16x=TensorCoreNVFP4Layout.get_padded_shape(orig_shape) != orig_shape,
+        ),
+    )
+    packed_x = timed(
+        stats,
+        f"{prefix}.activation_quant_wrap",
+        lambda: QuantizedTensor(
+            qdata,
+            "TensorCoreNVFP4Layout",
+            TensorCoreNVFP4Layout.Params(
+                scale=scale,
+                orig_dtype=flat_x.dtype,
+                orig_shape=orig_shape,
+                block_scale=block_scale,
+            ),
+        ),
+    )
+    output = timed(stats, f"{prefix}.gemm", lambda: functional.linear(packed_x, packed_weight, bias))
+    return timed(stats, f"{prefix}.slice_reshape", lambda: output[:flat_x.shape[0], :module.out_features].reshape(*original_shape, module.out_features))
+
+
+def profile_block(
+    block: H3DiTBlock,
+    hidden: torch.Tensor,
+    rotation: torch.Tensor,
+    adaln_values: tuple[torch.Tensor, ...],
+    segments: list[tuple[int, int, int]],
+    *,
+    iterations: int,
+) -> tuple[dict[str, dict[str, float]], torch.Tensor]:
+    stats: dict[str, list[float]] = {}
+    shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = adaln_values
+    result = hidden
+
+    for _ in range(iterations):
+        x = hidden.clone()
+
+        def run_once() -> torch.Tensor:
+            nonlocal x
+            h = timed(stats, "norm1", lambda: rms_norm(x, block.norm1_weight, block.norm_eps))
+            h = timed(stats, "modulate_msa", lambda: modulate_segments(h, shift_msa, scale_msa, segments))
+
+            attention = block.attention
+            sequence = h.shape[0]
+            inner = attention.heads * attention.head_dim
+            qkv = timed(stats, "attn_qkv_proj", lambda: profiled_nvfp4_linear(stats, "linear.attn_qkv_proj", attention.qkv_proj, h))
+            q, k, v = timed(stats, "attn_qkv_split_view", lambda: tuple(t.view(1, sequence, attention.heads, attention.head_dim) for t in qkv.split(inner, dim=-1)))
+            q, k = timed(stats, "attn_qk_rms_rope", lambda: rms_rope_split_half_(q, k, rotation, attention.q_norm_weight, attention.k_norm_weight, attention.eps))
+            if attention.backend == "sol_attn":
+                q = timed(stats, "attn_q_bshd_contiguous", lambda: q.contiguous())
+                k = timed(stats, "attn_k_bshd_contiguous", lambda: k.contiguous())
+                v = timed(stats, "attn_v_bshd_contiguous", lambda: v.contiguous())
+                attn_out = timed(stats, "attention_kernel", lambda: run_sol_attention_bshd(q, k, v, is_causal=False))
+                attn_rows = timed(stats, "attn_output_reshape", lambda: attn_out.reshape(sequence, inner).contiguous())
+            else:
+                q = timed(stats, "attn_q_transpose_contiguous", lambda: q.transpose(1, 2).contiguous())
+                k = timed(stats, "attn_k_transpose_contiguous", lambda: k.transpose(1, 2).contiguous())
+                v = timed(stats, "attn_v_transpose_contiguous", lambda: v.transpose(1, 2).contiguous())
+                attn_out = timed(stats, "attention_kernel", lambda: run_attention(q, k, v, backend=attention.backend, is_causal=False))
+                attn_rows = timed(stats, "attn_output_reshape", lambda: attn_out.transpose(1, 2).reshape(sequence, inner).contiguous())
+            attn_update = timed(stats, "attn_out_proj", lambda: profiled_nvfp4_linear(stats, "linear.attn_out_proj", attention.out_proj, attn_rows))
+            x = timed(stats, "gate_msa", lambda: gate_segments(x, attn_update, gate_msa, segments))
+
+            h2 = timed(stats, "norm2", lambda: rms_norm(x, block.norm2_weight, block.norm_eps))
+            h2 = timed(stats, "modulate_mlp", lambda: modulate_segments(h2, shift_mlp, scale_mlp, segments))
+            gate, up = timed(stats, "mlp_fc1", lambda: profiled_nvfp4_linear(stats, "linear.mlp_fc1", block.mlp.fc1, h2).chunk(2, dim=-1))
+            activated = timed(stats, "mlp_swiglu", lambda: torch.nn.functional.silu(gate).mul_(up))
+            mlp_update = timed(stats, "mlp_fc2", lambda: profiled_nvfp4_linear(stats, "linear.mlp_fc2", block.mlp.fc2, activated))
+            x = timed(stats, "gate_mlp", lambda: gate_segments(x, mlp_update, gate_mlp, segments))
+            return x
+
+        result = timed(stats, "block_total", run_once)
+
+    return {name: summarize(values) for name, values in stats.items()}, result
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+    parser.add_argument("--output", type=Path, default=Path("/output/h3-blackwell-runtime/benchmarks/h3-block-profile-3d0c093.json"))
+    parser.add_argument("--width", type=int, default=960)
+    parser.add_argument("--height", type=int, default=544)
+    parser.add_argument("--frames", type=int, default=124)
+    parser.add_argument("--steps", type=int, default=12)
+    parser.add_argument("--sampler-step", type=int, default=1, help="One-indexed sampler step used to build representative timesteps.")
+    parser.add_argument("--seed", type=int, default=440407)
+    parser.add_argument("--text-tokens", type=int, default=93, help="Synthetic refined-text token count; avoids Qwen/refiner load.")
+    parser.add_argument("--block-index", type=int, default=24)
+    parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sol_attn")
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--iterations", type=int, default=50)
+    parser.add_argument("--device", default="cuda")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    if not 0 <= args.block_index < 50:
+        raise ValueError("--block-index must be in [0, 49]")
+    if args.sampler_step < 1 or args.sampler_step > args.steps:
+        raise ValueError("--sampler-step must be between 1 and --steps")
+
+    torch.manual_seed(args.seed)
+    checkpoint = H3Checkpoint(args.model_path, device=args.device)
+    block = H3DiTBlock.from_checkpoint(checkpoint, args.block_index, attention_backend=args.attention).eval()
+    adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{args.block_index}.adaln_proj").eval()
+    packer = H3PromptPacker(checkpoint)
+
+    video, audio, aligned_frames = random_av_latents(args.width, args.height, args.frames, args.seed, device=args.device)
+    sigmas = beta_sigmas(args.steps, device=args.device)
+    sigma = sigmas[args.sampler_step - 1]
+    native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma)
+    text = torch.randn(1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16)
+    hidden, timesteps, segments, positions, _, _ = packer(text, video, native_audio, _model_sigma(sigma))
+    rotation = h3_rope_rotation(positions.to(args.device), checkpoint.tensor("rope.inv_freq", dtype=torch.float32), hidden.dtype)
+    adaln_values = tuple(value.detach() for value in adaln(timesteps))
+
+    with torch.inference_mode():
+        for _ in range(args.warmup):
+            profile_block(block, hidden, rotation, adaln_values, segments, iterations=1)
+        stats, output = profile_block(block, hidden, rotation, adaln_values, segments, iterations=args.iterations)
+
+    result = {
+        "prompt": CAT_PROMPT,
+        "model_path": args.model_path,
+        "width": args.width,
+        "height": args.height,
+        "frames": aligned_frames,
+        "steps": args.steps,
+        "sampler_step": args.sampler_step,
+        "seed": args.seed,
+        "text_tokens": args.text_tokens,
+        "block_index": args.block_index,
+        "attention": args.attention,
+        "warmup": args.warmup,
+        "iterations": args.iterations,
+        "hidden_shape": list(hidden.shape),
+        "output_shape": list(output.shape),
+        "segments": segments,
+        "timings": stats,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2), encoding="utf-8")
+    print(json.dumps(result, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/profile_nvfp4_linear.py
+++ b/tools/profile_nvfp4_linear.py
@ -0,0 +1,304 @@
+"""Profile current NVFP4 linear execution stages and CUDA kernels."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+import warnings
+from pathlib import Path
+
+warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)
+
+import torch
+import torch.nn.functional as functional
+from torch.profiler import ProfilerActivity, profile
+
+from h3_blackwell_runtime.adaln import H3CurveAdaLN
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, rms_norm, rms_rope_split_half_, run_attention
+from h3_blackwell_runtime.block import H3DiTBlock, gate_segments, modulate_segments
+from h3_blackwell_runtime.checkpoint import H3Checkpoint
+from h3_blackwell_runtime.nvfp4 import Nvfp4Linear
+from h3_blackwell_runtime.nvfp4_quant import nvfp4_activation_scale, vortex_native_quantize_nvfp4, vortex_quantize_nvfp4
+from h3_blackwell_runtime.packing import H3PromptPacker
+from h3_blackwell_runtime.rope import h3_rope_rotation
+from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas
+from h3_blackwell_runtime.t2v import random_av_latents
+
+
+def sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def timed(stats: dict[str, list[float]], name: str, fn):
+    sync()
+    started = time.perf_counter()
+    value = fn()
+    sync()
+    stats.setdefault(name, []).append(time.perf_counter() - started)
+    return value
+
+
+def summarize(values: list[float]) -> dict[str, float]:
+    ordered = sorted(values)
+    def percentile(percent: float) -> float:
+        if len(ordered) == 1:
+            return ordered[0]
+        rank = (len(ordered) - 1) * percent
+        low = int(rank)
+        high = min(low + 1, len(ordered) - 1)
+        weight = rank - low
+        return ordered[low] * (1.0 - weight) + ordered[high] * weight
+
+    return {
+        "count": len(values),
+        "mean_s": sum(values) / len(values),
+        "p50_s": percentile(0.50),
+        "p90_s": percentile(0.90),
+        "p95_s": percentile(0.95),
+        "p99_s": percentile(0.99),
+        "min_s": ordered[0],
+        "max_s": ordered[-1],
+    }
+
+
+def quantize_activation(flat_x: torch.Tensor, quantizer: str, scale: torch.Tensor | None, timings: dict[str, list[float]] | None = None):
+    from comfy_kitchen.tensor import QuantizedTensor
+
+    if quantizer == "comfy":
+        return QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout")
+    if quantizer == "vortex_recalculate":
+        return vortex_quantize_nvfp4(flat_x, timings=timings)
+    if quantizer == "vortex_precomputed_scale":
+        if scale is None:
+            raise ValueError("vortex_precomputed_scale requires a precomputed scale")
+        return vortex_quantize_nvfp4(flat_x, scale=scale, timings=timings)
+    if quantizer == "vortex_native":
+        return vortex_native_quantize_nvfp4(flat_x, timings=timings)
+    raise ValueError(f"Unsupported quantizer: {quantizer}")
+
+
+def profile_linear_stages(module: Nvfp4Linear, x: torch.Tensor, *, iterations: int, quantizer: str) -> dict[str, dict[str, float]]:
+
+    stats: dict[str, list[float]] = {}
+    precomputed_scale = None
+    if quantizer == "vortex_precomputed_scale":
+        precomputed_scale = timed(stats, "precomputed_scale_calibration", lambda: nvfp4_activation_scale(x.reshape(-1, module.in_features).contiguous()))
+    for _ in range(iterations):
+        original_shape = x.shape[:-1]
+        flat_x = timed(stats, "flatten_contiguous", lambda: x.reshape(-1, module.in_features).contiguous())
+        if module.pre_quant_scale is not None:
+            flat_x = timed(stats, "pre_quant_scale", lambda: flat_x * module.pre_quant_scale.to(flat_x))
+        else:
+            stats.setdefault("pre_quant_scale", []).append(0.0)
+        packed_weight = timed(stats, "packed_weight_wrapper", module._packed_weight)
+        bias = timed(stats, "bias_cast", lambda: module.bias.to(flat_x) if module.bias is not None else None)
+        if module.full_precision_matrix_mult:
+            weight = timed(stats, "weight_dequantize", lambda: packed_weight.dequantize().to(flat_x))
+            output = timed(stats, "linear", lambda: functional.linear(flat_x, weight, bias))
+        else:
+            packed_x = timed(stats, "activation_quantize", lambda: quantize_activation(flat_x, quantizer, precomputed_scale, stats))
+            output = timed(stats, "linear", lambda: functional.linear(packed_x, packed_weight, bias))
+        timed(stats, "slice_reshape", lambda: output[:flat_x.shape[0], :module.out_features].reshape(*original_shape, module.out_features))
+    return {name: summarize(values) for name, values in stats.items()}
+
+
+def run_linear_with_quantizer(module: Nvfp4Linear, x: torch.Tensor, quantizer: str, precomputed_scale: torch.Tensor | None = None) -> torch.Tensor:
+    original_shape = x.shape[:-1]
+    flat_x = x.reshape(-1, module.in_features).contiguous()
+    if module.pre_quant_scale is not None:
+        flat_x = flat_x * module.pre_quant_scale.to(flat_x)
+    packed_weight = module._packed_weight()
+    bias = module.bias.to(flat_x) if module.bias is not None else None
+    if module.full_precision_matrix_mult:
+        output = functional.linear(flat_x, packed_weight.dequantize().to(flat_x), bias)
+    else:
+        output = functional.linear(quantize_activation(flat_x, quantizer, precomputed_scale), packed_weight, bias)
+    return output[:flat_x.shape[0], :module.out_features].reshape(*original_shape, module.out_features)
+
+
+def profile_cuda_kernels(module: Nvfp4Linear, x: torch.Tensor, *, warmup: int, iterations: int, row_limit: int, quantizer: str) -> list[dict]:
+    with torch.inference_mode():
+        precomputed_scale = nvfp4_activation_scale(x.reshape(-1, module.in_features).contiguous()) if quantizer == "vortex_precomputed_scale" else None
+        for _ in range(warmup):
+            run_linear_with_quantizer(module, x, quantizer, precomputed_scale)
+        sync()
+        activities = [ProfilerActivity.CPU]
+        if torch.cuda.is_available():
+            activities.append(ProfilerActivity.CUDA)
+        with profile(activities=activities, record_shapes=True) as prof:
+            for _ in range(iterations):
+                run_linear_with_quantizer(module, x, quantizer, precomputed_scale)
+        sync()
+    rows = []
+    for event in prof.key_averages(group_by_input_shape=True):
+        cpu_us = float(getattr(event, "cpu_time_total", 0.0) or 0.0)
+        cuda_us = float(getattr(event, "cuda_time_total", 0.0) or 0.0)
+        rows.append(
+            {
+                "key": event.key,
+                "count": int(event.count),
+                "cpu_time_total_us": cpu_us,
+                "cuda_time_total_us": cuda_us,
+                "input_shapes": str(getattr(event, "input_shapes", "")),
+            }
+        )
+    rows.sort(key=lambda item: (item["cuda_time_total_us"], item["cpu_time_total_us"]), reverse=True)
+    return rows[:row_limit]
+
+
+def representative_inputs(args: argparse.Namespace) -> tuple[H3DiTBlock, dict[str, torch.Tensor], dict]:
+    torch.manual_seed(args.seed)
+    checkpoint = H3Checkpoint(args.model_path, device=args.device)
+    block = H3DiTBlock.from_checkpoint(checkpoint, args.block_index, attention_backend=args.attention).eval()
+    adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{args.block_index}.adaln_proj").eval()
+    packer = H3PromptPacker(checkpoint)
+
+    video, audio, aligned_frames = random_av_latents(args.width, args.height, args.frames, args.seed, device=args.device)
+    sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1]
+    native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma)
+    text = torch.randn(1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16)
+    hidden, timesteps, segments, positions, _, _ = packer(text, video, native_audio, _model_sigma(sigma))
+    rotation = h3_rope_rotation(positions.to(args.device), checkpoint.tensor("rope.inv_freq", dtype=torch.float32), hidden.dtype)
+    shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, _gate_mlp = adaln(timesteps)
+
+    with torch.inference_mode():
+        h_msa = modulate_segments(rms_norm(hidden, block.norm1_weight, block.norm_eps), shift_msa, scale_msa, segments)
+        sequence = h_msa.shape[0]
+        inner = block.attention.heads * block.attention.head_dim
+        qkv = block.attention.qkv_proj(h_msa)
+        q, k, v = qkv.split(inner, dim=-1)
+        q = q.view(1, sequence, block.attention.heads, block.attention.head_dim)
+        k = k.view(1, sequence, block.attention.heads, block.attention.head_dim)
+        v = v.view(1, sequence, block.attention.heads, block.attention.head_dim)
+        q, k = rms_rope_split_half_(q, k, rotation, block.attention.q_norm_weight, block.attention.k_norm_weight, block.attention.eps)
+        attn_out = run_attention(q.transpose(1, 2).contiguous(), k.transpose(1, 2).contiguous(), v.transpose(1, 2).contiguous(), backend=block.attention.backend, is_causal=False)
+        out_proj_input = attn_out.transpose(1, 2).reshape(sequence, inner).contiguous()
+        x_after_attn = gate_segments(hidden, block.attention.out_proj(out_proj_input), gate_msa, segments)
+        h_mlp = modulate_segments(rms_norm(x_after_attn, block.norm2_weight, block.norm_eps), shift_mlp, scale_mlp, segments)
+        gate, up = block.mlp.fc1(h_mlp).chunk(2, dim=-1)
+        fc2_input = torch.nn.functional.silu(gate).mul_(up)
+
+    inputs = {
+        "attn_qkv_proj": h_msa,
+        "attn_out_proj": out_proj_input,
+        "mlp_fc1": h_mlp,
+        "mlp_fc2": fc2_input,
+    }
+    metadata = {
+        "width": args.width,
+        "height": args.height,
+        "frames": aligned_frames,
+        "steps": args.steps,
+        "sampler_step": args.sampler_step,
+        "seed": args.seed,
+        "text_tokens": args.text_tokens,
+        "hidden_shape": list(hidden.shape),
+        "segments": segments,
+    }
+    return block, inputs, metadata
+
+
+def module_for_name(block: H3DiTBlock, name: str) -> Nvfp4Linear:
+    modules = {
+        "attn_qkv_proj": block.attention.qkv_proj,
+        "attn_out_proj": block.attention.out_proj,
+        "mlp_fc1": block.mlp.fc1,
+        "mlp_fc2": block.mlp.fc2,
+    }
+    return modules[name]
+
+
+def module_info(module: Nvfp4Linear, x: torch.Tensor) -> dict:
+    return {
+        "class": type(module).__name__,
+        "in_features": module.in_features,
+        "out_features": module.out_features,
+        "output_dtype": str(module.output_dtype),
+        "full_precision_matrix_mult": module.full_precision_matrix_mult,
+        "weight_dtype": str(module.weight.dtype),
+        "weight_shape": list(module.weight.shape),
+        "weight_scale_dtype": str(module.weight_scale.dtype),
+        "weight_scale_shape": list(module.weight_scale.shape),
+        "weight_scale_2_dtype": str(module.weight_scale_2.dtype),
+        "weight_scale_2_shape": list(module.weight_scale_2.shape),
+        "bias_dtype": str(module.bias.dtype) if module.bias is not None else None,
+        "bias_shape": list(module.bias.shape) if module.bias is not None else None,
+        "pre_quant_scale": module.pre_quant_scale is not None,
+        "input_dtype": str(x.dtype),
+        "input_shape": list(x.shape),
+        "input_is_contiguous": x.is_contiguous(),
+        "input_stride": list(x.stride()),
+    }
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+    parser.add_argument("--output", type=Path, default=Path("/output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-7cc03f3.json"))
+    parser.add_argument("--width", type=int, default=960)
+    parser.add_argument("--height", type=int, default=544)
+    parser.add_argument("--frames", type=int, default=124)
+    parser.add_argument("--steps", type=int, default=12)
+    parser.add_argument("--sampler-step", type=int, default=1)
+    parser.add_argument("--seed", type=int, default=440407)
+    parser.add_argument("--text-tokens", type=int, default=93)
+    parser.add_argument("--block-index", type=int, default=24)
+    parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sage2")
+    parser.add_argument("--linears", nargs="+", choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1", "mlp_fc2"), default=("attn_qkv_proj", "attn_out_proj", "mlp_fc1", "mlp_fc2"))
+    parser.add_argument("--quantizers", nargs="+", choices=("comfy", "vortex_recalculate", "vortex_precomputed_scale", "vortex_native"), default=("comfy", "vortex_recalculate", "vortex_precomputed_scale"))
+    parser.add_argument("--warmup", type=int, default=2)
+    parser.add_argument("--iterations", type=int, default=5)
+    parser.add_argument("--profiler-iterations", type=int, default=2)
+    parser.add_argument("--profiler-row-limit", type=int, default=30)
+    parser.add_argument("--device", default="cuda")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    block, inputs, metadata = representative_inputs(args)
+    results = []
+    with torch.inference_mode():
+        for name in args.linears:
+            module = module_for_name(block, name)
+            x = inputs[name]
+            reference = run_linear_with_quantizer(module, x, "comfy")
+            for quantizer in args.quantizers:
+                precomputed_scale = nvfp4_activation_scale(x.reshape(-1, module.in_features).contiguous()) if quantizer == "vortex_precomputed_scale" else None
+                for _ in range(args.warmup):
+                    run_linear_with_quantizer(module, x, quantizer, precomputed_scale)
+                stage_timings = profile_linear_stages(module, x, iterations=args.iterations, quantizer=quantizer)
+                candidate = run_linear_with_quantizer(module, x, quantizer, precomputed_scale)
+                diff = (candidate.float() - reference.float()).abs()
+                kernels = profile_cuda_kernels(module, x, warmup=args.warmup, iterations=args.profiler_iterations, row_limit=args.profiler_row_limit, quantizer=quantizer)
+                results.append(
+                    {
+                        "name": name,
+                        "quantizer": quantizer,
+                        "module": module_info(module, x),
+                        "stage_timings": stage_timings,
+                        "reference_diff": {"max": diff.max().item(), "mean": diff.mean().item()},
+                        "profiler_top_events": kernels,
+                    }
+                )
+
+    output = {
+        "model_path": args.model_path,
+        "block_index": args.block_index,
+        "attention": args.attention,
+        "warmup": args.warmup,
+        "iterations": args.iterations,
+        "profiler_iterations": args.profiler_iterations,
+        "quantizers": args.quantizers,
+        "metadata": metadata,
+        "results": results,
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(output, indent=2), encoding="utf-8")
+    print(json.dumps(output, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/profile_nvfp4_pack.py
+++ b/tools/profile_nvfp4_pack.py
@ -0,0 +1,160 @@
+"""Profile NVFP4 activation pack kernels without GEMM timing noise."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+import warnings
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.*", category=UserWarning)
+
+import torch
+
+import comfy_kitchen as ck
+from comfy_kitchen.tensor import TensorCoreNVFP4Layout
+
+from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension, nvfp4_activation_scale
+from tools.profile_nvfp4_linear import module_for_name, representative_inputs
+
+
+def sync() -> None:
+    if torch.cuda.is_available():
+        torch.cuda.synchronize()
+
+
+def summarize(values: list[float]) -> dict[str, float]:
+    ordered = sorted(values)
+
+    def percentile(percent: float) -> float:
+        if len(ordered) == 1:
+            return ordered[0]
+        rank = (len(ordered) - 1) * percent
+        low = int(rank)
+        high = min(low + 1, len(ordered) - 1)
+        weight = rank - low
+        return ordered[low] * (1.0 - weight) + ordered[high] * weight
+
+    return {
+        "count": len(values),
+        "mean_s": sum(values) / len(values),
+        "p50_s": percentile(0.50),
+        "p90_s": percentile(0.90),
+        "p95_s": percentile(0.95),
+        "p99_s": percentile(0.99),
+        "min_s": ordered[0],
+        "max_s": ordered[-1],
+    }
+
+
+def timed(fn, iterations: int) -> dict[str, float]:
+    values = []
+    for _ in range(iterations):
+        sync()
+        started = time.perf_counter()
+        fn()
+        sync()
+        values.append(time.perf_counter() - started)
+    return summarize(values)
+
+
+def bytes_touched(tensor: torch.Tensor, qdata: torch.Tensor, block_scale: torch.Tensor) -> int:
+    return tensor.numel() * tensor.element_size() + qdata.numel() * qdata.element_size() + block_scale.numel()
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
+    parser.add_argument("--output", type=Path, default=Path("/output/h3-blackwell-runtime/benchmarks/nvfp4-pack-profile.json"))
+    parser.add_argument("--width", type=int, default=960)
+    parser.add_argument("--height", type=int, default=544)
+    parser.add_argument("--frames", type=int, default=124)
+    parser.add_argument("--steps", type=int, default=12)
+    parser.add_argument("--sampler-step", type=int, default=1)
+    parser.add_argument("--seed", type=int, default=440407)
+    parser.add_argument("--text-tokens", type=int, default=93)
+    parser.add_argument("--block-index", type=int, default=24)
+    parser.add_argument("--attention", default="sage2")
+    parser.add_argument("--linears", nargs="+", choices=("mlp_fc1", "mlp_fc2", "attn_qkv_proj", "attn_out_proj"), default=("mlp_fc1", "mlp_fc2"))
+    parser.add_argument("--threads", nargs="+", type=int, default=(64, 128, 256, 512))
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--iterations", type=int, default=80)
+    parser.add_argument("--device", default="cuda")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    os.environ.setdefault("H3_NVFP4_SCALE_BACKEND", "vortex")
+    os.environ.setdefault("H3_NVFP4_SCALE_VERSION", "1")
+
+    block, inputs, metadata = representative_inputs(args)
+    extension = _vortex_scale_extension()
+    results = []
+    with torch.inference_mode():
+        for name in args.linears:
+            module = module_for_name(block, name)
+            tensor = inputs[name].reshape(-1, module.in_features).contiguous()
+            scale = nvfp4_activation_scale(tensor).float()
+            pad = TensorCoreNVFP4Layout.get_padded_shape(tuple(tensor.shape)) != tuple(tensor.shape)
+            q_ref, b_ref = ck.quantize_nvfp4(tensor, scale, pad_16x=pad)
+            touched = bytes_touched(tensor, q_ref, b_ref)
+
+            for _ in range(args.warmup):
+                ck.quantize_nvfp4(tensor, scale, pad_16x=pad)
+            ck_timing = timed(lambda: ck.quantize_nvfp4(tensor, scale, pad_16x=pad), args.iterations)
+            results.append(
+                {
+                    "name": name,
+                    "packer": "ck",
+                    "threads": None,
+                    "input_shape": list(tensor.shape),
+                    "qdata_shape": list(q_ref.shape),
+                    "block_scale_shape": list(b_ref.shape),
+                    "bytes_touched": touched,
+                    "effective_gbps_p50": touched / ck_timing["p50_s"] / 1e9,
+                    "effective_gbps_p95": touched / ck_timing["p95_s"] / 1e9,
+                    "timing": ck_timing,
+                    "qdata_diff": 0,
+                    "block_scale_diff": 0,
+                }
+            )
+
+            for threads in args.threads:
+                for _ in range(args.warmup):
+                    extension.quantize_nvfp4_bf16(tensor, scale, pad, threads)
+                q_native, b_native = extension.quantize_nvfp4_bf16(tensor, scale, pad, threads)
+                qdiff = int((q_ref != q_native).sum().item())
+                bdiff = int((b_ref.view(torch.uint8) != b_native.view(torch.uint8)).sum().item())
+                timing = timed(lambda: extension.quantize_nvfp4_bf16(tensor, scale, pad, threads), args.iterations)
+                results.append(
+                    {
+                        "name": name,
+                        "packer": "vortex_native",
+                        "threads": threads,
+                        "input_shape": list(tensor.shape),
+                        "qdata_shape": list(q_native.shape),
+                        "block_scale_shape": list(b_native.shape),
+                        "bytes_touched": touched,
+                        "effective_gbps_p50": touched / timing["p50_s"] / 1e9,
+                        "effective_gbps_p95": touched / timing["p95_s"] / 1e9,
+                        "timing": timing,
+                        "qdata_diff": qdiff,
+                        "block_scale_diff": bdiff,
+                    }
+                )
+                print(name, "threads", threads, "p50_ms", round(timing["p50_s"] * 1000, 3), "p95_ms", round(timing["p95_s"] * 1000, 3), "qdiff", qdiff, "bdiff", bdiff, flush=True)
+
+    output = {"metadata": metadata, "results": results}
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(output, indent=2), encoding="utf-8")
+    print(json.dumps(output, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/report_attention_path.py
+++ b/tools/report_attention_path.py
@ -0,0 +1,112 @@
+"""Generate a markdown summary for attention path benchmark JSON output."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+def ms(value: float) -> float:
+    return value * 1000.0
+
+
+def ok_results(data: dict) -> list[dict]:
+    return [item for item in data["results"] if item.get("status") == "ok"]
+
+
+def timing_ms(result: dict, name: str, field: str = "mean_s") -> float:
+    return ms(result["timings"][name][field])
+
+
+def label(result: dict) -> str:
+    layout = result.get("layout_mode", "")
+    return f"{result['backend']}:{layout}" if layout else result["backend"]
+
+
+def timing_names(result: dict) -> tuple[str, ...]:
+    layout = result.get("layout_mode")
+    if layout == "sol_fused":
+        return ("qkv_to_bshd", "attention_kernel", "output_reshape")
+    if layout == "sol_native":
+        return ("q_bshd_contiguous", "k_bshd_contiguous", "v_bshd_contiguous", "attention_kernel", "output_reshape")
+    return ("q_transpose_contiguous", "k_transpose_contiguous", "v_transpose_contiguous", "attention_kernel", "output_reshape")
+
+
+def table(results: list[dict]) -> list[str]:
+    lines = ["| Segment | Backend | Kernel mean ms | Total mean ms | Layout mean ms | Kernel p50 ms | Total p50 ms |", "| --- | --- | ---: | ---: | ---: | ---: | ---: |"]
+    for item in sorted(results, key=lambda row: (row["segment"], row["layout_attention_total_mean_s"])):
+        total = ms(item["layout_attention_total_mean_s"])
+        kernel = timing_ms(item, "attention_kernel")
+        layout = total - kernel
+        p50_total = sum(timing_ms(item, name, "p50_s") for name in timing_names(item))
+        lines.append(f"| {item['segment']} | {label(item)} | {kernel:.3f} | {total:.3f} | {layout:.3f} | {timing_ms(item, 'attention_kernel', 'p50_s'):.3f} | {p50_total:.3f} |")
+    return lines
+
+
+def fastest_by_segment(results: list[dict]) -> list[str]:
+    segments = sorted({item["segment"] for item in results})
+    lines = []
+    for segment in segments:
+        segment_results = [item for item in results if item["segment"] == segment]
+        best = min(segment_results, key=lambda item: item["layout_attention_total_mean_s"])
+        best_kernel = min(segment_results, key=lambda item: item["timings"]["attention_kernel"]["mean_s"])
+        lines.append(f"- `{segment}` fastest total: `{label(best)}` at {ms(best['layout_attention_total_mean_s']):.3f} ms; fastest kernel: `{label(best_kernel)}` at {timing_ms(best_kernel, 'attention_kernel'):.3f} ms.")
+    return lines
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--whole", type=Path, required=True)
+    parser.add_argument("--segments", type=Path, required=True)
+    parser.add_argument("--extra", type=Path, action="append", default=[])
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+
+    whole = json.loads(args.whole.read_text(encoding="utf-8"))
+    segments = json.loads(args.segments.read_text(encoding="utf-8"))
+    extras = [json.loads(path.read_text(encoding="utf-8")) for path in args.extra]
+    whole_ok = ok_results(whole)
+    segment_ok = ok_results(segments)
+    extra_ok = [item for data in extras for item in ok_results(data)]
+    extra_lines = table(extra_ok) if extra_ok else ["No extra runs."]
+    failures = [item for item in segments["results"] + whole["results"] + [item for data in extras for item in data["results"]] if item.get("status") != "ok"]
+
+    lines = [
+        "# H3 Attention Path Profile",
+        "",
+        f"Whole-sequence input shape: `{whole['metadata']['h_msa_shape']}`; segments: `{whole['segments']}`.",
+        f"Measurements: `{whole['warmup']}` warmups and `{whole['iterations']}` iterations per backend.",
+        "",
+        "## Key Findings",
+        "",
+        *fastest_by_segment(whole_ok + segment_ok + extra_ok),
+        "- Sol-native BSHD removes the generic HND round trip and keeps exact Sol output parity.",
+        "- The current fused QKV split/layout prototype is opt-in because it is not faster than three PyTorch contiguous copies yet.",
+        "- Full-sequence Sol-Attn stays under the initial 50-55 ms target for attention plus layout on this captured H3 shape.",
+        "- Q/K/V layout remains the next memory-bandwidth target; the Sage-style output reshape is not worth attacking for Sol.",
+        "- `sdpa` is fastest for tiny text/secondary segments but loses heavily on the video-dominated path, so it is only interesting for segment-specialized dispatch.",
+        "",
+        "## Whole Sequence",
+        "",
+        *table(whole_ok),
+        "",
+        "## Extra Whole-Sequence Layout Runs",
+        "",
+        *extra_lines,
+        "",
+        "## Segment Sweep",
+        "",
+        *table(segment_ok),
+    ]
+    if failures:
+        lines.extend(["", "## Expected Failures", ""])
+        lines.extend(f"- `{item['segment']}` / `{label(item)}`: `{item['error']}`" for item in failures)
+    lines.extend(["", "## Source Files", "", f"- Whole JSON: `{args.whole}`", f"- Segment JSON: `{args.segments}`"])
+    lines.extend(f"- Extra JSON: `{path}`" for path in args.extra)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/report_h3_block.py
+++ b/tools/report_h3_block.py
@ -0,0 +1,120 @@
+"""Generate a markdown summary for representative H3 block profiles."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+MAIN_TIMINGS = (
+    "norm1",
+    "modulate_msa",
+    "attn_qkv_proj",
+    "attn_qkv_split_view",
+    "attn_qk_rms_rope",
+    "attn_q_bshd_contiguous",
+    "attn_k_bshd_contiguous",
+    "attn_v_bshd_contiguous",
+    "attention_kernel",
+    "attn_output_reshape",
+    "attn_out_proj",
+    "gate_msa",
+    "norm2",
+    "modulate_mlp",
+    "mlp_fc1",
+    "mlp_swiglu",
+    "mlp_fc2",
+    "gate_mlp",
+    "block_total",
+)
+
+LINEARS = ("attn_qkv_proj", "attn_out_proj", "mlp_fc1", "mlp_fc2")
+LINEAR_STAGES = (
+    "activation_scale",
+    "activation_quant_pack",
+    "gemm",
+    "pre_quant_scale",
+    "packed_weight_wrapper",
+    "bias_cast",
+    "scale_to_device",
+    "activation_quant_wrap",
+    "flatten_contiguous",
+    "slice_reshape",
+)
+
+
+def ms(value: float) -> float:
+    return value * 1000.0
+
+
+def row(timings: dict, name: str) -> tuple[float, float, float]:
+    item = timings[name]
+    return ms(item["mean_s"]), ms(item["p50_s"]), ms(item["p95_s"])
+
+
+def add_table(lines: list[str], timings: dict, names: tuple[str, ...]) -> None:
+    lines.extend(["| Stage | Mean ms | P50 ms | P95 ms |", "| --- | ---: | ---: | ---: |"])
+    for name in names:
+        if name not in timings:
+            continue
+        mean, p50, p95 = row(timings, name)
+        lines.append(f"| `{name}` | {mean:.3f} | {p50:.3f} | {p95:.3f} |")
+
+
+def linear_table(lines: list[str], timings: dict) -> None:
+    lines.extend(["| Linear | Scale P50 ms | Pack P50 ms | GEMM P50 ms | Total P50 ms |", "| --- | ---: | ---: | ---: | ---: |"])
+    for linear in LINEARS:
+        prefix = f"linear.{linear}"
+        scale = ms(timings[f"{prefix}.activation_scale"]["p50_s"])
+        pack = ms(timings[f"{prefix}.activation_quant_pack"]["p50_s"])
+        gemm = ms(timings[f"{prefix}.gemm"]["p50_s"])
+        total = ms(timings[linear]["p50_s"])
+        lines.append(f"| `{linear}` | {scale:.3f} | {pack:.3f} | {gemm:.3f} | {total:.3f} |")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--input", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+
+    data = json.loads(args.input.read_text(encoding="utf-8"))
+    timings = data["timings"]
+    block_p50 = ms(timings["block_total"]["p50_s"])
+    attention_layout_p50 = sum(ms(timings[name]["p50_s"]) for name in ("attn_q_bshd_contiguous", "attn_k_bshd_contiguous", "attn_v_bshd_contiguous", "attention_kernel", "attn_output_reshape"))
+    mlp_linear_p50 = sum(ms(timings[name]["p50_s"]) for name in ("mlp_fc1", "mlp_fc2"))
+    norm_mod_gate_p50 = sum(ms(timings[name]["p50_s"]) for name in ("norm1", "modulate_msa", "gate_msa", "norm2", "modulate_mlp", "gate_mlp"))
+    qk_rope_p50 = ms(timings["attn_qk_rms_rope"]["p50_s"])
+
+    lines = [
+        "# H3 Block Profile Baseline",
+        "",
+        f"Input: block `{data['block_index']}`, hidden shape `{data['hidden_shape']}`, segments `{data['segments']}`.",
+        f"Config: attention `{data['attention']}`, `{data['warmup']}` warmups, `{data['iterations']}` iterations.",
+        "",
+        "## Summary",
+        "",
+        f"- Block total p50: `{block_p50:.3f} ms`; p95: `{ms(timings['block_total']['p95_s']):.3f} ms`.",
+        f"- Sol attention plus BSHD layout p50: `{attention_layout_p50:.3f} ms`.",
+        f"- MLP linears p50: `{mlp_linear_p50:.3f} ms`; SwiGLU p50: `{ms(timings['mlp_swiglu']['p50_s']):.3f} ms`.",
+        f"- Norm/modulate/gate p50: `{norm_mod_gate_p50:.3f} ms`.",
+        f"- Q/K RMS+RoPE p50: `{qk_rope_p50:.3f} ms`.",
+        "- The next likely targets are Q/K RMS+RoPE, Q/K/V contiguous extraction, and the modulation/gating helpers; Sol output reshape remains negligible.",
+        "",
+        "## Main Stages",
+        "",
+    ]
+    add_table(lines, timings, MAIN_TIMINGS)
+    lines.extend(["", "## NVFP4 Linear Breakdown", ""])
+    linear_table(lines, timings)
+    for linear in LINEARS:
+        lines.extend(["", f"## `{linear}` Stages", ""])
+        add_table(lines, timings, tuple(f"linear.{linear}.{stage}" for stage in LINEAR_STAGES))
+    lines.extend(["", "## Source", "", f"- JSON: `{args.input}`"])
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text("\n".join(lines) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/run_distributed_matrix.sh
+++ b/tools/run_distributed_matrix.sh
@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+available="$(python -c 'import torch; print(torch.cuda.device_count())')"
+if [[ "$available" -lt 1 ]]; then
+  echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
+  exit 1
+fi
+
+read -ra counts <<< "${H3_GPU_COUNTS:-1 2 4 6 8}"
+read -ra modes <<< "${H3_DISTRIBUTED_MODES:-ulysses tensor}"
+for count in "${counts[@]}"; do
+  if [[ "$count" -gt "$available" ]]; then
+    echo "Skipping ${count} GPUs; only ${available} are visible." >&2
+    continue
+  fi
+  for mode in "${modes[@]}"; do
+    tools/run_distributed_t2va.sh "$mode" "$count" "${H3_ATTENTION:-sdpa}"
+  done
+done
--- a/tools/run_distributed_t2va.sh
+++ b/tools/run_distributed_t2va.sh
@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+mode="${1:?usage: run_distributed_t2va.sh ulysses|tensor [WORLD_SIZE] [ATTENTION]}"
+world_size="${2:-${H3_WORLD_SIZE:-}}"
+if [[ -z "$world_size" ]]; then
+  world_size="$(python -c 'import torch; print(torch.cuda.device_count())')"
+fi
+if [[ "$world_size" -lt 1 ]]; then
+  echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
+  exit 1
+fi
+attention="${3:-sdpa}"
+benchmark="${H3_DISTRIBUTED_BENCHMARK:-benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json}"
+output_root="${H3_DISTRIBUTED_OUTPUT:-/output/h3-baselines}"
+stem="distributed-${mode}-${world_size}gpu-${attention}"
+model="${H3_MODEL_PATH:-/models/minimax_h3_fl2va_pruned_nvfp4.safetensors}"
+text_encoder="${H3_TEXT_ENCODER_PATH:-/text-encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors}"
+extra_args=()
+if [[ "${H3_SAVE_LATENTS:-1}" == "1" ]]; then
+  extra_args+=(--save-latent "$output_root/$stem.latent.pt")
+fi
+
+torchrun \
+  --standalone \
+  --nnodes=1 \
+  --nproc-per-node="$world_size" \
+  tools/distributed_t2va.py \
+  --benchmark "$benchmark" \
+  --mode "$mode" \
+  --attention "$attention" \
+  --model "$model" \
+  --text-encoder "$text_encoder" \
+  --report "$output_root/$stem.json" \
+  "${extra_args[@]}"
--- a/tools/run_ulysses_benchmark.sh
+++ b/tools/run_ulysses_benchmark.sh
@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+world_size="${1:-${H3_WORLD_SIZE:-}}"
+if [[ -z "$world_size" ]]; then
+  world_size="$(python -c 'import torch; print(torch.cuda.device_count())')"
+fi
+if [[ "$world_size" -lt 1 ]]; then
+  echo "No CUDA devices are visible. Launch the container with --gpus all." >&2
+  exit 1
+fi
+backend="${2:-sdpa}"
+sequence="${3:-20000}"
+output="${H3_BENCHMARK_OUTPUT:-/output/h3-baselines/ulysses-${world_size}gpu-${backend}-${sequence}t.json}"
+
+torchrun \
+  --standalone \
+  --nnodes=1 \
+  --nproc-per-node="$world_size" \
+  tools/benchmark_ulysses.py \
+  --sequence "$sequence" \
+  --backend "$backend" \
+  --output "$output"
--- a/tools/runpod_api.py
+++ b/tools/runpod_api.py
@ -0,0 +1,100 @@
+"""Minimal RunPod API v2 client for the H3 single-node benchmark pod."""
+
+import argparse
+import json
+import os
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+
+API = "https://api.runpod.io/v2"
+BLACKWELL_GPUS = (
+    "NVIDIA RTX PRO 6000 Blackwell Server Edition",
+    "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
+    "NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition",
+)
+DEFAULT_IMAGE = "runpod/pytorch:1.1.0-cu1300-torch291-ubuntu2404"
+
+
+def request(method: str, path: str, body=None, query=None):
+    key = os.environ.get("RUNPOD_API_KEY")
+    if not key:
+        raise SystemExit("RUNPOD_API_KEY is required")
+    url = f"{API}{path}"
+    if query:
+        url += "?" + urllib.parse.urlencode(query)
+    data = None if body is None else json.dumps(body).encode("utf-8")
+    call = urllib.request.Request(url, data=data, method=method)
+    call.add_header("Authorization", f"Bearer {key}")
+    call.add_header("Accept", "application/json")
+    if data is not None:
+        call.add_header("Content-Type", "application/json")
+    try:
+        with urllib.request.urlopen(call, timeout=60) as response:
+            return json.load(response)
+    except urllib.error.HTTPError as error:
+        detail = error.read().decode("utf-8", errors="replace")
+        raise SystemExit(f"RunPod API returned HTTP {error.code}: {detail}") from error
+
+
+parser = argparse.ArgumentParser()
+commands = parser.add_subparsers(dest="command", required=True)
+
+catalog_parser = commands.add_parser("catalog")
+catalog_parser.add_argument("--count", type=int, default=8)
+catalog_parser.add_argument("--cloud", choices=("SECURE", "COMMUNITY"), default="SECURE")
+
+create_parser = commands.add_parser("create")
+create_parser.add_argument("--gpu", choices=BLACKWELL_GPUS, default=BLACKWELL_GPUS[0])
+create_parser.add_argument("--count", type=int, default=8)
+create_parser.add_argument("--cloud", choices=("SECURE", "COMMUNITY"), default="SECURE")
+create_parser.add_argument("--datacenter")
+create_parser.add_argument("--image", default=DEFAULT_IMAGE)
+create_parser.add_argument("--disk", type=int, default=100)
+create_parser.add_argument("--volume", type=int, default=100)
+create_parser.add_argument("--yes", action="store_true")
+
+get_parser = commands.add_parser("get")
+get_parser.add_argument("pod_id")
+
+terminate_parser = commands.add_parser("terminate")
+terminate_parser.add_argument("pod_id")
+terminate_parser.add_argument("--yes", action="store_true")
+
+args = parser.parse_args()
+if args.command == "catalog":
+    response = request("GET", "/catalog/gpus", query={
+        "include": "AVAILABILITY",
+        "product": "POD",
+        "count": args.count,
+        "cloud": args.cloud,
+        "minCudaVersion": "12.8",
+    })
+    response["gpus"] = [gpu for gpu in response["gpus"] if gpu["id"] in BLACKWELL_GPUS]
+elif args.command == "create":
+    if not args.yes:
+        raise SystemExit("create rents billable GPUs; repeat with --yes after checking catalog")
+    body = {
+        "name": "h3-blackwell-distributed",
+        "image": args.image,
+        "gpu": {"id": args.gpu, "count": args.count, "minCudaVersion": "12.8"},
+        "cloud": args.cloud,
+        "disk": args.disk,
+        "ports": ["22/tcp"],
+        "mounts": {"persistent": {"size": args.volume, "path": "/workspace"}},
+        "startSsh": True,
+    }
+    if args.datacenter:
+        body["dataCenterIds"] = [args.datacenter]
+    response = request("POST", "/pods", body=body)
+elif args.command == "get":
+    response = request("GET", f"/pods/{args.pod_id}")
+else:
+    if not args.yes:
+        raise SystemExit("termination is irreversible; repeat with --yes")
+    response = request("POST", f"/pods/{args.pod_id}/actions", body={"action": "terminate"})
+
+json.dump(response, sys.stdout, indent=2)
+sys.stdout.write("\n")
--- a/tools/serve_hot_runtime.py
+++ b/tools/serve_hot_runtime.py
@ -0,0 +1,200 @@
+"""Serve a resident prompt-only H3 runtime over a small JSON HTTP API."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import threading
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from urllib.parse import urlparse
+
+import torch
+from PIL import Image
+
+from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, DEFAULT_ATTENTION_BACKEND, attention_backend_status
+from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig, TURBO_VARIANTS, normalize_upscale
+
+
+def _load_image(value) -> torch.Tensor | None:
+    """Accept a keyframe as on-disk path or base64 JPEG/PNG -> ``[1,3,H,W]`` float ``[0,1]``."""
+    if value in (None, ""):
+        return None
+    if isinstance(value, (list, tuple)):
+        value = value[0]
+    if isinstance(value, dict):
+        value = value.get("url") or value.get("path") or value.get("b64")
+    if isinstance(value, str) and "\n" not in value and len(value) < 2048 and not value.startswith("data:"):
+        path = Path(value)
+        if path.exists():
+            image = Image.open(path).convert("RGB")
+            import numpy as np
+
+            tensor = torch.from_numpy(np.array(image))[None].permute(0, 3, 1, 2).float() / 255.0
+            return tensor
+    data = value
+    if isinstance(value, str) and value.startswith("data:"):
+        data = value.split(",", 1)[1]
+    if isinstance(data, str) and len(data) >= 1024:
+        import base64
+        import io
+
+        raw = base64.b64decode(data)
+        image = Image.open(io.BytesIO(raw)).convert("RGB")
+        import numpy as np
+
+        return torch.from_numpy(np.array(image))[None].permute(0, 3, 1, 2).float() / 255.0
+    raise ValueError("first_frame/last_frame must be a path or a base64/data-URL image")
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--host", default="0.0.0.0")
+parser.add_argument("--port", type=int, default=8000)
+parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default=DEFAULT_ATTENTION_BACKEND, help="Initial attention backend. Requests can switch with the JSON attention field.")
+parser.add_argument("--vae-dtype", choices=("float32", "float16", "bfloat16"), default="float16")
+parser.add_argument("--vae-tile-size", type=int, default=256)
+parser.add_argument("--mlp-chunks", type=int, default=1)
+parser.add_argument("--mlp-chunk-threshold", type=int, default=4096)
+parser.add_argument("--turbo-lora", action="append", default=[], metavar="NAME=PATH", help="Load a resident 4step or 8step Turbo adapter.")
+parser.add_argument("--latent-upscaler", help="Load the optional H3 3D latent upscaler for request-level spatial upscaling.")
+parser.add_argument("--warmup", action="store_true", help="Run a tiny generation before accepting traffic.")
+parser.add_argument("--warmup-output", type=Path, default=Path("/output/h3-blackwell-runtime/hot-runtime-warmup.mp4"))
+args = parser.parse_args()
+
+turbo_loras = []
+for value in args.turbo_lora:
+    if "=" not in value:
+        parser.error("--turbo-lora must use NAME=PATH")
+    name, path = value.split("=", 1)
+    turbo_loras.append((name, path))
+
+runtime = H3HotRuntime(RuntimeConfig(attention=args.attention, vae_dtype=args.vae_dtype, vae_tile_size=args.vae_tile_size, mlp_chunks=args.mlp_chunks, mlp_chunk_threshold=args.mlp_chunk_threshold, turbo_loras=tuple(turbo_loras), latent_upscaler_path=args.latent_upscaler))
+runtime_lock = threading.Lock()
+warmup_result = None
+if args.warmup:
+    warmup_result = runtime.generate(
+        prompt="A small warmup cat blinks in soft light.",
+        output=args.warmup_output,
+        width=320,
+        height=192,
+        frames=22,
+        steps=2,
+        seed=440501,
+        mux_audio=True,
+    )
+
+
+def service_status() -> dict:
+    return {
+        "ready": True,
+        "attention_backends": list(AVAILABLE_BACKENDS),
+        "attention_backend_status": attention_backend_status(),
+        "runtime": runtime.status(),
+    }
+
+
+def write_json(handler: BaseHTTPRequestHandler, status: int, payload: dict) -> None:
+    body = json.dumps(payload, indent=2).encode("utf-8")
+    handler.send_response(status)
+    handler.send_header("Content-Type", "application/json")
+    handler.send_header("Content-Length", str(len(body)))
+    handler.end_headers()
+    handler.wfile.write(body)
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, format: str, *args) -> None:
+        return
+
+    def do_GET(self) -> None:
+        path = urlparse(self.path).path
+        if path in {"/health", "/ready"}:
+            write_json(self, 200, {**service_status(), "warmup_result": warmup_result})
+            return
+        write_json(self, 404, {"error": "not found"})
+
+    def do_POST(self) -> None:
+        path = urlparse(self.path).path
+        if path != "/generate":
+            write_json(self, 404, {"error": "not found"})
+            return
+        try:
+            length = int(self.headers.get("Content-Length", "0"))
+            payload = json.loads(self.rfile.read(length).decode("utf-8")) if length else {}
+            prompt = payload["prompt"]
+            output = payload["output"]
+            width = int(payload.get("width", 960))
+            height = int(payload.get("height", 544))
+            frames = int(payload.get("frames", 124))
+            turbo = payload.get("turbo")
+            if turbo in {"", "none"}:
+                turbo = None
+            if turbo is not None and turbo not in runtime.turbo_loras:
+                write_json(self, 400, {"error": "unsupported turbo", "turbo": turbo, "available": sorted(runtime.turbo_loras)})
+                return
+            steps = int(payload.get("steps", TURBO_VARIANTS[turbo]["steps"] if turbo else 12))
+            if turbo is not None and steps != TURBO_VARIANTS[turbo]["steps"]:
+                write_json(self, 400, {"error": "invalid turbo steps", "turbo": turbo, "required_steps": TURBO_VARIANTS[turbo]["steps"]})
+                return
+            seed = int(payload.get("seed", 440407))
+            attention = payload.get("attention")
+            if attention is not None and attention not in AVAILABLE_BACKENDS:
+                write_json(self, 400, {"error": "unsupported attention", "attention": attention, "available": list(AVAILABLE_BACKENDS)})
+                return
+            mux_audio = bool(payload.get("mux_audio", True))
+            keep_intermediates = bool(payload.get("keep_intermediates", False))
+            try:
+                upscale = normalize_upscale(payload.get("upscale"))
+            except ValueError as exc:
+                write_json(self, 400, {"error": "invalid upscale", "message": str(exc)})
+                return
+            if upscale is not None and runtime.latent_upscaler is None:
+                write_json(self, 400, {"error": "H3 latent upscaler is not loaded"})
+                return
+            ffmpeg_loglevel = payload.get("ffmpeg_loglevel", "error")
+            first_frame = _load_image(payload.get("first_frame"))
+            last_frame = _load_image(payload.get("last_frame"))
+            save_latent = payload.get("save_latent")
+            cache_mode = payload.get("cache_mode")
+            if turbo is not None and cache_mode not in {None, "", "disabled", "none"}:
+                write_json(self, 400, {"error": "turbo does not support denoiser caching", "turbo": turbo})
+                return
+            cache_threshold = float(payload.get("cache_threshold", 0.0))
+            cache_start_percent = float(payload.get("cache_start_percent", 0.0))
+            cache_end_percent = float(payload.get("cache_end_percent", 1.0))
+            cache_subsample_factor = int(payload.get("cache_subsample_factor", 2))
+            started = time.perf_counter()
+            with runtime_lock:
+                result = runtime.generate(
+                    prompt=prompt,
+                    output=output,
+                    width=width,
+                    height=height,
+                    frames=frames,
+                    steps=steps,
+                    seed=seed,
+                    attention=attention,
+                    turbo=turbo,
+                    upscale=upscale,
+                    first_frame=first_frame,
+                    last_frame=last_frame,
+                    mux_audio=mux_audio,
+                    keep_intermediates=keep_intermediates,
+                    ffmpeg_loglevel=ffmpeg_loglevel,
+                    save_latent=save_latent,
+                    cache_mode=cache_mode,
+                    cache_threshold=cache_threshold,
+                    cache_start_percent=cache_start_percent,
+                    cache_end_percent=cache_end_percent,
+                    cache_subsample_factor=cache_subsample_factor,
+                )
+            result["wall_seconds"] = time.perf_counter() - started
+            write_json(self, 200, result)
+        except Exception as exc:
+            write_json(self, 500, {"error": type(exc).__name__, "message": str(exc)})
+
+
+server = ThreadingHTTPServer((args.host, args.port), Handler)
+print(json.dumps({"serving": True, "host": args.host, "port": args.port, **service_status(), "warmup_result": warmup_result}, indent=2), flush=True)
+server.serve_forever()
--- a/tools/smoke_attention_backends.py
+++ b/tools/smoke_attention_backends.py
@ -0,0 +1,58 @@
+"""Verify and time request-selectable H3 attention kernels on the active GPU."""
+
+import argparse
+import json
+import time
+
+import torch
+
+from h3_blackwell_runtime.attention import run_attention
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--backends", nargs="+", default=("sage2", "cudnn_sdpa", "ck_int8"))
+parser.add_argument("--sequence", type=int, default=512)
+parser.add_argument("--heads", type=int, default=56)
+parser.add_argument("--head-dim", type=int, default=128)
+parser.add_argument("--warmup", type=int, default=2)
+parser.add_argument("--iterations", type=int, default=5)
+parser.add_argument("--seed", type=int, default=440407)
+args = parser.parse_args()
+
+torch.manual_seed(args.seed)
+q = torch.randn(1, args.heads, args.sequence, args.head_dim, device="cuda", dtype=torch.bfloat16)
+results = {}
+reference = None
+
+with torch.inference_mode():
+    for backend in args.backends:
+        for _ in range(args.warmup):
+            output = run_attention(q, q, q, backend=backend, is_causal=False)
+        torch.cuda.synchronize()
+        elapsed = []
+        for _ in range(args.iterations):
+            started = time.perf_counter()
+            output = run_attention(q, q, q, backend=backend, is_causal=False)
+            torch.cuda.synchronize()
+            elapsed.append(time.perf_counter() - started)
+        if reference is None:
+            reference = output
+        delta = (output.float() - reference.float()).abs()
+        results[backend] = {
+            "mean_seconds": sum(elapsed) / len(elapsed),
+            "min_seconds": min(elapsed),
+            "finite": bool(torch.isfinite(output).all()),
+            "shape": list(output.shape),
+            "dtype": str(output.dtype),
+            "max_abs_vs_reference": delta.max().item(),
+            "mean_abs_vs_reference": delta.mean().item(),
+        }
+
+print(json.dumps({
+    "gpu": torch.cuda.get_device_name(),
+    "torch": torch.__version__,
+    "cuda": torch.version.cuda,
+    "shape": list(q.shape),
+    "reference": args.backends[0],
+    "results": results,
+}, indent=2))
--- a/tools/smoke_flash4.py
+++ b/tools/smoke_flash4.py
@ -0,0 +1,50 @@
+"""Compare FlashAttention-4 against PyTorch SDPA on an H3-shaped operation."""
+
+import argparse
+import json
+
+import torch
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--sequence", type=int, default=257)
+parser.add_argument("--heads", type=int, default=8)
+parser.add_argument("--head-dim", type=int, default=128)
+parser.add_argument("--iterations", type=int, default=5)
+args = parser.parse_args()
+
+from flash_attn.cute import flash_attn_func
+
+torch.manual_seed(440411)
+shape = (1, args.sequence, args.heads, args.head_dim)
+q = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
+k = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
+v = torch.randn(shape, device="cuda", dtype=torch.bfloat16)
+
+with torch.inference_mode():
+    result = flash_attn_func(q, k, v, causal=False)
+    actual = result[0] if isinstance(result, tuple) else result
+    expected = torch.nn.functional.scaled_dot_product_attention(
+        q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=False,
+    ).transpose(1, 2)
+    torch.cuda.synchronize()
+    started = torch.cuda.Event(enable_timing=True)
+    finished = torch.cuda.Event(enable_timing=True)
+    started.record()
+    for _ in range(args.iterations):
+        result = flash_attn_func(q, k, v, causal=False)
+        actual = result[0] if isinstance(result, tuple) else result
+    finished.record()
+    torch.cuda.synchronize()
+
+error = (actual.float() - expected.float()).abs()
+print(json.dumps({
+    "device": torch.cuda.get_device_name(),
+    "capability": torch.cuda.get_device_capability(),
+    "shape": tuple(actual.shape),
+    "dtype": str(actual.dtype),
+    "contiguous": actual.is_contiguous(),
+    "max_abs_error": error.max().item(),
+    "mean_abs_error": error.mean().item(),
+    "milliseconds": started.elapsed_time(finished) / args.iterations,
+}, indent=2))
--- a/tools/smoke_qwen38_vllm_spark.sh
+++ b/tools/smoke_qwen38_vllm_spark.sh
@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+curl -fsS --max-time 120 http://127.0.0.1:8002/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  --data-binary @- <<'JSON'
+{"model":"qwen38","messages":[{"role":"user","content":"Reply with exactly: ready"}],"max_tokens":64,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}
+JSON
--- a/tools/spark_stack_spark.sh
+++ b/tools/spark_stack_spark.sh
@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT=/home/daniel/aeon-spark-test/h3
+STACK_DIR="$ROOT/spark-stack"
+COMPOSE_FILE="$STACK_DIR/compose.spark-stack.yml"
+
+usage() {
+  cat <<'EOF'
+Usage:
+  spark_stack_spark.sh start h3|qwen|flux
+  spark_stack_spark.sh stop
+  spark_stack_spark.sh status
+
+Ports:
+  h3   -> http://192.168.1.162:8001
+  qwen -> http://192.168.1.162:8002
+  flux -> http://192.168.1.162:8192
+EOF
+}
+
+managed_containers=(
+  h3-hot-runtime
+  h3-blackwell-runtime-h3-hot-runtime-1
+  qwen38-vllm
+  flux2-klein-comfy
+  flux-nvfp4-test
+)
+
+stop_managed() {
+  for name in "${managed_containers[@]}"; do
+    if docker ps -a --format '{{.Names}}' | grep -qx "$name"; then
+      docker stop "$name" >/dev/null 2>&1 || true
+      docker rm "$name" >/dev/null 2>&1 || true
+    fi
+  done
+}
+
+case "${1:-}" in
+  start)
+    target="${2:-}"
+    case "$target" in
+      h3|qwen|flux) ;;
+      *) usage; exit 2 ;;
+    esac
+    mkdir -p "$ROOT/vllm-cache"
+    stop_managed
+    docker compose -f "$COMPOSE_FILE" --profile "$target" up -d
+    ;;
+  stop)
+    stop_managed
+    ;;
+  status)
+    docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' | grep -E 'NAMES|h3-hot-runtime|qwen38-vllm|flux2-klein-comfy|flux-nvfp4-test' || true
+    nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits || true
+    ;;
+  *)
+    usage
+    exit 2
+    ;;
+esac
--- a/tools/start_qwen38_vllm_spark.sh
+++ b/tools/start_qwen38_vllm_spark.sh
@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT=/home/daniel/aeon-spark-test/h3
+MODEL_DIR="$ROOT/models/Qwen3.8-27B-NVFP4"
+COMPOSE_FILE="$ROOT/qwen38-vllm/compose.qwen38-vllm.yml"
+
+if [[ ! -f "$MODEL_DIR/model.safetensors" ]]; then
+  echo "model.safetensors is not present yet: $MODEL_DIR/model.safetensors" >&2
+  echo "Wait for qwen38-27b-nvfp4-download to finish before starting vLLM." >&2
+  exit 1
+fi
+
+python3 - <<'PY'
+import json
+from pathlib import Path
+
+tokenizer = Path('/home/daniel/aeon-spark-test/h3/models/Qwen3.8-27B-NVFP4/tokenizer.json')
+truncation = json.loads(tokenizer.read_text(encoding='utf-8')).get('truncation')
+if truncation is not None:
+    raise SystemExit(f'tokenizer truncation must be null, got: {truncation!r}')
+print('tokenizer truncation: null')
+PY
+
+mkdir -p "$ROOT/vllm-cache"
+docker compose -f "$COMPOSE_FILE" up -d
+docker logs -f qwen38-vllm
--- a/tools/submit_fl2va_keyframes.py
+++ b/tools/submit_fl2va_keyframes.py
@ -0,0 +1,46 @@
+"""Submit a matched Comfy FL2VA first/last-frame reference workflow."""
+
+import argparse
+import json
+from urllib.request import Request, urlopen
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--url", default="http://localhost:8188")
+parser.add_argument("--first", default="fl2va_key_first.png")
+parser.add_argument("--last", default="fl2va_key_last.png")
+parser.add_argument("--prefix", default="fl2va-comfy-keyframes-seed440207")
+args = parser.parse_args()
+
+prompt = {
+    "1": {"class_type": "UNETLoader", "inputs": {"unet_name": "minimax_h3_fl2va_pruned_nvfp4.safetensors", "weight_dtype": "default"}},
+    "3": {"class_type": "CLIPLoader", "inputs": {"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", "type": "minimax"}},
+    "4": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_video_vae_fp16.safetensors"}},
+    "5": {"class_type": "LoadImage", "inputs": {"image": args.first}},
+    "6": {"class_type": "LoadImage", "inputs": {"image": args.last}},
+    "8": {"class_type": "MiniMaxH3ImageToVideo", "inputs": {
+        "clip": ["3", 0],
+        "vae": ["4", 0],
+        "prompt": "A studio time-lapse of the same pink peony bud opening into the same fully bloomed pink peony, fixed camera, cream background.",
+        "width": 384,
+        "height": 384,
+        "length": 22,
+        "first_frame": ["5", 0],
+        "last_frame": ["6", 0],
+    }},
+    "9": {"class_type": "BasicGuider", "inputs": {"model": ["1", 0], "conditioning": ["8", 0]}},
+    "10": {"class_type": "RandomNoise", "inputs": {"noise_seed": 440207}},
+    "11": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "res_multistep"}},
+    "12": {"class_type": "BasicScheduler", "inputs": {"model": ["1", 0], "scheduler": "beta", "steps": 12, "denoise": 1.0}},
+    "13": {"class_type": "SamplerCustomAdvanced", "inputs": {"noise": ["10", 0], "guider": ["9", 0], "sampler": ["11", 0], "sigmas": ["12", 0], "latent_image": ["8", 1]}},
+    "14": {"class_type": "VAEDecode", "inputs": {"samples": ["13", 0], "vae": ["4", 0]}},
+    "15": {"class_type": "SaveImage", "inputs": {"images": ["14", 0], "filename_prefix": args.prefix}},
+}
+
+request = Request(
+    args.url.rstrip("/") + "/prompt",
+    data=json.dumps({"prompt": prompt}).encode(),
+    headers={"Content-Type": "application/json"},
+)
+with urlopen(request) as response:
+    print(response.read().decode())
--- a/tools/submit_h3_comfy_t2v.py
+++ b/tools/submit_h3_comfy_t2v.py
@ -0,0 +1,91 @@
+"""Submit a prompt-only MiniMax H3 T2V graph to ComfyUI."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+from urllib.request import Request, urlopen
+
+
+def post_json(url: str, payload: dict) -> dict:
+    request = Request(url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
+    with urlopen(request, timeout=30) as response:
+        return json.loads(response.read().decode())
+
+
+def get_json(url: str) -> dict:
+    with urlopen(url, timeout=30) as response:
+        return json.loads(response.read().decode())
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--server", default="http://127.0.0.1:8188")
+    parser.add_argument("--prompt", required=True)
+    parser.add_argument("--filename-prefix", required=True)
+    parser.add_argument("--seed", type=int, default=440407)
+    parser.add_argument("--width", type=int, default=960)
+    parser.add_argument("--height", type=int, default=544)
+    parser.add_argument("--frames", type=int, default=124)
+    parser.add_argument("--steps", type=int, default=12)
+    parser.add_argument("--upscale", type=float)
+    args = parser.parse_args()
+
+    graph = {
+        "1": {"class_type": "UNETLoader", "inputs": {"unet_name": "minimax_h3_fl2va_pruned_nvfp4.safetensors", "weight_dtype": "default"}},
+        "3": {"class_type": "CLIPLoader", "inputs": {"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors", "type": "minimax"}},
+        "4": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_video_vae_fp16.safetensors"}},
+        "5": {"class_type": "VAELoader", "inputs": {"vae_name": "minimax_h3_audio_vae_fp32.safetensors"}},
+        "8": {"class_type": "MiniMaxH3ImageToVideo", "inputs": {"clip": ["3", 0], "vae": ["4", 0], "prompt": args.prompt, "width": args.width, "height": args.height, "length": args.frames}},
+        "9": {"class_type": "BasicGuider", "inputs": {"model": ["1", 0], "conditioning": ["8", 0]}},
+        "10": {"class_type": "RandomNoise", "inputs": {"noise_seed": args.seed}},
+        "11": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "res_multistep"}},
+        "12": {"class_type": "BasicScheduler", "inputs": {"model": ["1", 0], "scheduler": "beta", "steps": args.steps, "denoise": 1.0}},
+        "13": {"class_type": "SamplerCustomAdvanced", "inputs": {"noise": ["10", 0], "guider": ["9", 0], "sampler": ["11", 0], "sigmas": ["12", 0], "latent_image": ["8", 1]}},
+    }
+    decoded_latent = ["13", 0]
+    if args.upscale is not None:
+        graph.update({
+            "18": {"class_type": "LTXVSeparateAVLatent", "inputs": {"av_latent": ["13", 0]}},
+            "19": {"class_type": "MinimaxH3LatentUpscaler3D", "inputs": {
+                "latent": ["18", 0],
+                "model_name": "minimax_h3_latent_upscaler_3d_fp16.safetensors",
+                "mode": "scale by multiplier",
+                "mode.scale": args.upscale,
+                "align": 32,
+                "keep_proportion": True,
+                "device": "cuda",
+                "precision": "fp16",
+            }},
+            "20": {"class_type": "LTXVConcatAVLatent", "inputs": {"video_latent": ["19", 0], "audio_latent": ["18", 1]}},
+        })
+        decoded_latent = ["20", 0]
+    graph.update({
+        "14": {"class_type": "VAEDecode", "inputs": {"samples": decoded_latent, "vae": ["4", 0]}},
+        "15": {"class_type": "VAEDecodeAudio", "inputs": {"samples": decoded_latent, "vae": ["5", 0]}},
+        "16": {"class_type": "CreateVideo", "inputs": {"images": ["14", 0], "audio": ["15", 0], "bit_depth": 8, "fps": 24.0}},
+        "17": {"class_type": "SaveVideo", "inputs": {"video": ["16", 0], "filename_prefix": args.filename_prefix, "format": "mp4", "codec": "auto"}},
+    })
+
+    started = time.perf_counter()
+    response = post_json(f"{args.server}/prompt", {"prompt": graph})
+    prompt_id = response["prompt_id"]
+    while True:
+        time.sleep(1)
+        history = get_json(f"{args.server}/history/{prompt_id}").get(prompt_id)
+        if history is None:
+            continue
+        status = history.get("status", {})
+        if status.get("completed") or status.get("status_str") in {"success", "error"}:
+            break
+    print(json.dumps({
+        "prompt_id": prompt_id,
+        "wall_seconds": time.perf_counter() - started,
+        "status": history.get("status", {}),
+        "outputs": history.get("outputs", {}),
+    }, indent=2))
+
+
+if __name__ == "__main__":
+    main()
--- a/tools/sweep_dialogue_audio_boundary.py
+++ b/tools/sweep_dialogue_audio_boundary.py
@ -0,0 +1,186 @@
+"""Run a paired tagged-versus-quoted H3 dialogue audio sweep."""
+
+import argparse
+import json
+import math
+import subprocess
+import time
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig
+from h3_blackwell_runtime.sampler import sample_video_res_multistep
+from h3_blackwell_runtime.t2v import random_av_latents
+
+
+SAMPLE_RATE = 32000
+
+
+def dbfs(value: float) -> float:
+    return 20.0 * math.log10(max(value, 1e-20))
+
+
+def waveform_metrics(waveform: torch.Tensor) -> dict:
+    waveform = waveform.float()
+    first_100ms = waveform[..., :3200]
+    next_400ms = waveform[..., 3200:16000]
+    first_500ms = waveform[..., :16000]
+    derivatives = (first_500ms[..., 1:] - first_500ms[..., :-1]).abs()
+    windows = waveform.unfold(-1, 320, 320)
+    window_rms = windows.square().mean(dim=(0, 2)).sqrt()
+    active = (20.0 * torch.log10(window_rms.clamp_min(1e-20)) > -40.0).nonzero()
+    first_active_ms = None if active.numel() == 0 else int(active[0, 0]) * 10
+
+    first_rms = dbfs(float(first_100ms.square().mean().sqrt()))
+    next_rms = dbfs(float(next_400ms.square().mean().sqrt()))
+    return {
+        "first_sample": waveform[..., 0].flatten().tolist(),
+        "first_100ms_peak_dbfs": dbfs(float(first_100ms.abs().max())),
+        "first_100ms_rms_dbfs": first_rms,
+        "next_400ms_peak_dbfs": dbfs(float(next_400ms.abs().max())),
+        "next_400ms_rms_dbfs": next_rms,
+        "boundary_decay_db": first_rms - next_rms,
+        "first_500ms_peak_dbfs": dbfs(float(first_500ms.abs().max())),
+        "first_500ms_rms_dbfs": dbfs(float(first_500ms.square().mean().sqrt())),
+        "full_peak_dbfs": dbfs(float(waveform.abs().max())),
+        "full_rms_dbfs": dbfs(float(waveform.square().mean().sqrt())),
+        "largest_first_500ms_derivative": float(derivatives.max()),
+        "first_10ms_window_above_minus_40_dbfs_ms": first_active_ms,
+    }
+
+
+def latent_metrics(latent: torch.Tensor) -> dict:
+    frames = latent.float().movedim(-1, 0).flatten(1)
+    return {
+        "shape": list(latent.shape),
+        "first_4_rms": float(frames[:4].square().mean().sqrt()),
+        "frames_4_20_rms": float(frames[4:20].square().mean().sqrt()),
+        "first_frame_rms": float(frames[0].square().mean().sqrt()),
+        "frame_0_to_1_delta_rms": float((frames[1] - frames[0]).square().mean().sqrt()),
+    }
+
+
+def write_waveform(path: Path, waveform: torch.Tensor) -> None:
+    raw = path.with_suffix(".f32le")
+    waveform.transpose(0, 1).contiguous().numpy().tofile(raw)
+    subprocess.run([
+        "ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
+        "-f", "f32le", "-ar", str(SAMPLE_RATE), "-ac", "2", "-i", str(raw),
+        "-c:a", "pcm_f32le", str(path),
+    ], check=True)
+    raw.unlink()
+
+
+def save_report(path: Path, report: dict) -> None:
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+    temporary.replace(path)
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--tagged-benchmark", type=Path, required=True)
+parser.add_argument("--quoted-benchmark", type=Path, required=True)
+parser.add_argument("--seed-start", type=int, default=440420)
+parser.add_argument("--seed-count", type=int, default=10)
+parser.add_argument("--output-dir", type=Path, required=True)
+parser.add_argument("--report", type=Path, required=True)
+parser.add_argument("--attention", default="sage2")
+args = parser.parse_args()
+
+tagged = json.loads(args.tagged_benchmark.read_text(encoding="utf-8"))
+quoted = json.loads(args.quoted_benchmark.read_text(encoding="utf-8"))
+for field in ("resolution", "frames", "steps"):
+    if tagged[field] != quoted[field]:
+        raise ValueError(f"benchmark {field} differs: {tagged[field]} != {quoted[field]}")
+
+args.output_dir.mkdir(parents=True, exist_ok=True)
+args.report.parent.mkdir(parents=True, exist_ok=True)
+if args.report.exists():
+    report = json.loads(args.report.read_text(encoding="utf-8"))
+else:
+    report = {
+        "tagged_benchmark": str(args.tagged_benchmark),
+        "quoted_benchmark": str(args.quoted_benchmark),
+        "attention": args.attention,
+        "seed_start": args.seed_start,
+        "seed_count": args.seed_count,
+        "cases": {},
+        "pairs": {},
+    }
+
+runtime = H3HotRuntime(RuntimeConfig(attention=args.attention))
+conditioned = {
+    "tagged": runtime.refiner(runtime.conditioner(tagged["prompt"])),
+    "quoted": runtime.refiner(runtime.conditioner(quoted["prompt"])),
+}
+
+width, height = tagged["resolution"]
+for seed in range(args.seed_start, args.seed_start + args.seed_count):
+    for prompt_format, benchmark in (("tagged", tagged), ("quoted", quoted)):
+        key = f"{seed}:{prompt_format}"
+        if key in report["cases"]:
+            print(f"skip completed {key}", flush=True)
+            continue
+
+        started = time.perf_counter()
+        video, audio, aligned_frames = random_av_latents(
+            width, height, benchmark["frames"], seed, device=runtime.config.device,
+        )
+        sampled_video, audio_latent = sample_video_res_multistep(
+            runtime.model,
+            runtime.packer,
+            conditioned[prompt_format],
+            video,
+            audio,
+            steps=benchmark["steps"],
+            seed=seed,
+            return_audio=True,
+        )
+        with torch.inference_mode():
+            waveform = runtime.audio_vae.decode(
+                audio_latent.to("cuda", dtype=next(runtime.audio_vae.parameters()).dtype),
+            ).cpu()[0]
+
+        stem = f"dialogue-{prompt_format}-base12-sage2-seed{seed}"
+        wav_path = args.output_dir / f"{stem}.wav"
+        latent_path = args.output_dir / f"{stem}.audio-latent.pt"
+        write_waveform(wav_path, waveform)
+        torch.save({
+            "audio_latent": audio_latent.detach().cpu(),
+            "prompt_format": prompt_format,
+            "prompt": benchmark["prompt"],
+            "seed": seed,
+        }, latent_path)
+
+        report["cases"][key] = {
+            "seed": seed,
+            "prompt_format": prompt_format,
+            "wav": str(wav_path),
+            "audio_latent": str(latent_path),
+            "frames": aligned_frames,
+            "seconds": time.perf_counter() - started,
+            "waveform": waveform_metrics(waveform),
+            "latent": latent_metrics(audio_latent.cpu()),
+        }
+        del sampled_video, audio_latent, waveform, video, audio
+        save_report(args.report, report)
+        print(json.dumps(report["cases"][key]), flush=True)
+
+    tagged_case = report["cases"][f"{seed}:tagged"]
+    quoted_case = report["cases"][f"{seed}:quoted"]
+    report["pairs"][str(seed)] = {
+        "quoted_peak_reduction_db": (
+            tagged_case["waveform"]["first_100ms_peak_dbfs"]
+            - quoted_case["waveform"]["first_100ms_peak_dbfs"]
+        ),
+        "quoted_rms_reduction_db": (
+            tagged_case["waveform"]["first_100ms_rms_dbfs"]
+            - quoted_case["waveform"]["first_100ms_rms_dbfs"]
+        ),
+        "tagged_boundary_decay_db": tagged_case["waveform"]["boundary_decay_db"],
+        "quoted_boundary_decay_db": quoted_case["waveform"]["boundary_decay_db"],
+    }
+    save_report(args.report, report)
+
+print(json.dumps(report["pairs"], indent=2))
--- a/tools/trace_audio_denoising.py
+++ b/tools/trace_audio_denoising.py
@ -0,0 +1,106 @@
+"""Trace when an H3 audio-boundary artifact emerges during base sampling."""
+
+import argparse
+import json
+import math
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.runtime import H3HotRuntime, RuntimeConfig
+from h3_blackwell_runtime.sampler import _decode_audio_latent, sample_video_res_multistep
+from h3_blackwell_runtime.t2v import random_av_latents
+
+
+def dbfs(value: float) -> float:
+    return 20.0 * math.log10(max(value, 1e-20))
+
+
+def latent_metrics(latent: torch.Tensor) -> dict:
+    frames = latent.float().movedim(-1, 0).flatten(1)
+    return {
+        "first_4_rms": float(frames[:4].square().mean().sqrt()),
+        "frames_4_20_rms": float(frames[4:20].square().mean().sqrt()),
+        "first_frame_rms": float(frames[0].square().mean().sqrt()),
+        "frame_0_to_1_delta_rms": float((frames[1] - frames[0]).square().mean().sqrt()),
+    }
+
+
+def waveform_metrics(waveform: torch.Tensor) -> dict:
+    waveform = waveform.float()
+    first_100ms = waveform[..., :3200]
+    first_500ms = waveform[..., :16000]
+    derivative = (first_500ms[..., 1:] - first_500ms[..., :-1]).abs()
+    return {
+        "first_sample": waveform[..., 0].flatten().tolist(),
+        "first_100ms_peak_dbfs": dbfs(float(first_100ms.abs().max())),
+        "first_100ms_rms_dbfs": dbfs(float(first_100ms.square().mean().sqrt())),
+        "first_500ms_peak_dbfs": dbfs(float(first_500ms.abs().max())),
+        "first_500ms_rms_dbfs": dbfs(float(first_500ms.square().mean().sqrt())),
+        "largest_derivative": float(derivative.max()),
+    }
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--benchmark", type=Path, required=True)
+parser.add_argument("--trace", type=Path, required=True)
+parser.add_argument("--report", type=Path, required=True)
+parser.add_argument("--attention", default="sage2")
+args = parser.parse_args()
+
+benchmark = json.loads(args.benchmark.read_text(encoding="utf-8"))
+runtime = H3HotRuntime(RuntimeConfig(attention=args.attention))
+video, initial_audio, aligned_frames = random_av_latents(
+    benchmark["resolution"][0],
+    benchmark["resolution"][1],
+    benchmark["frames"],
+    benchmark["seed"],
+    device=runtime.config.device,
+)
+text = runtime.refiner(runtime.conditioner(benchmark["prompt"]))
+trace = []
+video, final_audio = sample_video_res_multistep(
+    runtime.model,
+    runtime.packer,
+    text,
+    video,
+    initial_audio,
+    steps=benchmark["steps"],
+    seed=benchmark["seed"],
+    return_audio=True,
+    audio_step_trace=trace,
+)
+
+report = {
+    "benchmark": str(args.benchmark),
+    "attention": args.attention,
+    "seed": benchmark["seed"],
+    "frames": aligned_frames,
+    "steps": [],
+}
+with torch.inference_mode():
+    for entry in trace:
+        denoised = _decode_audio_latent(entry["audio_denoised"]).to(
+            "cuda", dtype=next(runtime.audio_vae.parameters()).dtype,
+        )
+        waveform = runtime.audio_vae.decode(denoised).cpu()[0]
+        report["steps"].append({
+            "step": entry["step"],
+            "video_sigma": entry["video_sigma"],
+            "audio_sigma": entry["audio_sigma"],
+            "latent": latent_metrics(denoised.cpu()),
+            "denoised_waveform": waveform_metrics(waveform),
+        })
+
+args.trace.parent.mkdir(parents=True, exist_ok=True)
+torch.save({
+    "initial_audio": initial_audio.detach().cpu(),
+    "final_audio": final_audio.detach().cpu(),
+    "steps": trace,
+    "prompt": benchmark["prompt"],
+    "seed": benchmark["seed"],
+}, args.trace)
+args.report.parent.mkdir(parents=True, exist_ok=True)
+serialized = json.dumps(report, indent=2)
+args.report.write_text(serialized + "\n", encoding="utf-8")
+print(serialized)
--- a/tools/upscale_video_latent.py
+++ b/tools/upscale_video_latent.py
@ -0,0 +1,55 @@
+"""Upscale a saved direct-runtime H3 latent with the learned 3D model."""
+
+import argparse
+import json
+import time
+from pathlib import Path
+
+import torch
+
+from h3_blackwell_runtime.latent_upscaler import load_h3_latent_upscaler, upscale_h3_latent
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--latent", type=Path, required=True)
+parser.add_argument("--model", type=Path, required=True)
+parser.add_argument("--output", type=Path, required=True)
+parser.add_argument("--scale", type=float, default=2.0)
+parser.add_argument("--precision", choices=("float16", "bfloat16", "float32"), default="float16")
+args = parser.parse_args()
+
+dtype = getattr(torch, args.precision)
+state = torch.load(args.latent, map_location="cpu", weights_only=False)
+if not isinstance(state, dict) or "latent" not in state:
+    state = {"latent": state}
+source = state["latent"]
+
+torch.cuda.synchronize()
+started = time.perf_counter()
+model = load_h3_latent_upscaler(args.model, dtype=dtype)
+torch.cuda.synchronize()
+loaded = time.perf_counter()
+upscaled = upscale_h3_latent(model, source, scale=args.scale).cpu()
+torch.cuda.synchronize()
+finished = time.perf_counter()
+
+result = dict(state)
+result["latent"] = upscaled
+result["width"] = upscaled.shape[-1] * 16
+result["height"] = upscaled.shape[-2] * 16
+result["upscale"] = {
+    "model": args.model.name,
+    "scale": args.scale,
+    "precision": args.precision,
+    "source_shape": tuple(source.shape),
+    "output_shape": tuple(upscaled.shape),
+}
+args.output.parent.mkdir(parents=True, exist_ok=True)
+torch.save(result, args.output)
+print(json.dumps({
+    "output": str(args.output),
+    "source_shape": tuple(source.shape),
+    "output_shape": tuple(upscaled.shape),
+    "model_load_seconds": loaded - started,
+    "upscale_seconds": finished - loaded,
+}, indent=2))
Author	SHA1	Message	Date
Daniel Maddern	bd92baeb46	Add distributed H3 execution and validation	2026-08-22 14:09:45 +07:00
Daniel Maddern	1d5faa8f16	Expand H3 runtime acceleration and validation	2026-08-20 22:08:52 +07:00
Daniel Maddern	837c16871d	Add resident FL2VA Turbo variants	2026-08-20 19:13:23 +07:00
Daniel Maddern	6d8c9ca4cf	Add selectable hot attention backends	2026-08-20 17:39:44 +07:00
Daniel Maddern	9bb96a26e8	Fix FL2VA conditioning parity end to end	2026-08-20 16:43:22 +07:00
Daniel Maddern	9f62f6ea83	Remove debug prints from vision tower (fl2va passes e2e smoke)	2026-08-19 23:03:58 +07:00
Daniel Maddern	45d3e7bc64	Fix mrope_freqs_cis call (no device kwarg)	2026-08-19 22:59:26 +07:00
Daniel Maddern	9fbd6fe77e	Batch-dim on input_ids for _embed_rows ([1, seq])	2026-08-19 22:57:03 +07:00
Daniel Maddern	0903b849b2	Flatten _text_run_ids output (input_ids is [1,seq])	2026-08-19 22:54:32 +07:00
Daniel Maddern	428c83d1d8	Coerce token ids to int in build_fl2va_presentation	2026-08-19 22:51:48 +07:00
Daniel Maddern	4add435bf9	Fix vision merger: main=per-patch-norm+2x2-interleave, deepstack=merged-norm	2026-08-19 22:48:26 +07:00
Daniel Maddern	68c49fdc95	Use merge_dim as normalized_shape in _VisionPatchMerger (4*1152 for 2x2 merge)	2026-08-19 22:39:05 +07:00
Daniel Maddern	b085ef02e8	Merge tokens expands C by merge^2 (merger takes pre-expanded)	2026-08-19 22:36:32 +07:00
Daniel Maddern	571d3541e8	Use explicit C (not x.shape[-1]) in _merge_tokens	2026-08-19 22:34:16 +07:00
Daniel Maddern	99a7550a51	Debug _merge_tokens prod	2026-08-19 22:31:52 +07:00
Daniel Maddern	ece7b2497e	Use explicit C instead of -1 in _merge_tokens view	2026-08-19 22:29:26 +07:00
Daniel Maddern	834bd96111	Trace x shape at deepstack layers	2026-08-19 22:27:16 +07:00
Daniel Maddern	b9dcdda116	Debug _merge_tokens shapes	2026-08-19 22:21:30 +07:00
Daniel Maddern	ffb3c9e729	Debug print in vision tower forward	2026-08-19 22:18:33 +07:00
Daniel Maddern	e4d9a6f8ff	Fix position_embeddings unpack (3-tuple of cos/sin/-sin)	2026-08-19 22:12:48 +07:00
Daniel Maddern	a19f051800	Cast qkv to common dtype before SDPA (q/k/v match)	2026-08-19 22:10:39 +07:00
Daniel Maddern	efc5fd9bc1	Fix col_idx expand/reshape precedence in rot_pos_emb	2026-08-19 22:08:21 +07:00
Daniel Maddern	67e4b12246	Make process_image return 2D grid_thw [[t,h,w]] for multi-image support	2026-08-19 22:04:12 +07:00
Daniel Maddern	6866263631	Fix F.conv3d call: remove invalid kernel_size kwarg (uses weight.shape)	2026-08-19 22:01:12 +07:00
Daniel Maddern	453aa86328	Use PIL Lanczos resize in resize_keyframe (F.interpolate no-4D-lanczos)	2026-08-19 21:56:32 +07:00
Daniel Maddern	0b7217485c	nin_shortcut is a plain 1x1x1 conv (no causal padding)	2026-08-19 21:49:40 +07:00
Daniel Maddern	ea6ab87a34	Decouple spatial_padding from temporal_causal in causal conv	2026-08-19 21:47:00 +07:00
Daniel Maddern	b557cba173	Fix Downsample3D spatial reflect pad to (W,H) dims only	2026-08-19 21:41:16 +07:00
Daniel Maddern	136297f4ae	Fix F.pad dim order (spatial reflect + no T pad) for 1-frame path	2026-08-19 21:36:40 +07:00
Daniel Maddern	c2d507691b	Restore front-zero temporal pad for multi-frame causal conv; two-step F.pad (reflect spatial + constant T)	2026-08-19 21:35:07 +07:00
Daniel Maddern	f7c48b067a	Remove permute bug: rely on F.conv3d spatial padding	2026-08-19 21:33:08 +07:00
Daniel Maddern	6fc7cf0ad6	Fix causal conv: correct F.pad spatial dim order + 1-frame kernel truncation	2026-08-19 21:31:05 +07:00
Daniel Maddern	3386975326	Fix VAE: use same multi-frame causal path for keyframes, add frame_pre_padding so 1 keyframe -> 1 latent	2026-08-19 21:20:32 +07:00
Daniel Maddern	a0274a9868	Fix tiled_encode latent x-overlap off-by-one (use [j-1] for left neighbor)	2026-08-19 21:13:40 +07:00
Daniel Maddern	65e80be1cf	Thread single_frame flag so keyframe truncation does not apply to 1-frame tiles	2026-08-19 21:11:43 +07:00
Daniel Maddern	06fd79a8fd	Fix causal temporal padding to match reference (2k-1 front zeros when spatial_padding>0)	2026-08-19 21:09:03 +07:00
Daniel Maddern	390135fca6	Fix VAE pixel normalization in-place bug	2026-08-19 21:07:34 +07:00
Daniel Maddern	9bec53ac36	Fix VAE encoder to load canonical checkpoint key names	2026-08-19 21:04:30 +07:00
Daniel Maddern	807bd64a82	Add direct first/last-frame (fl2va) keyframe conditioning Wire full fl2va into the direct H3 runtime so first/last keyframes flow through VAE encode -> Qwen vision tokens -> DiT cond segments: - vae_encoder.py: direct encoder-only H3 video VAE (causal 3D convs, reflect spatial padding, causal temporal padding, single-frame tap truncation, tiling, FP32 moments + mean/std normalization). - qwen3vl_vision.py: Qwen3-VL-32B visual tower (27 blocks, 2D rotary, deepstack mergers) ported to match the Comfy reference exactly (head_dim=72, no-bias proj, LayerNorm blocks, split-half apply_rope), plus Qwen image preprocess, mrope ids/freqs, DiT token tags, keyframe resize (first=stretch / last=center cover-crop matching Comfy common_upscale), and build_fl2va_presentation. - qwen3vl_text.py: split-half apply_rope, _embed_rows/_run_layers, optional mrope position_ids + DeepStack injection at the first three decoder layers at visual positions. - packing.py: H3PromptPacker builds [text \| cond \| audio \| video] with tag-run text spans, cond rows (first/last cond_t anchors, VISUAL_COND_TIMESTEP=0.999 noise augmentation via CPU-seeded RNG), three-timestep row table (t_row*3 + modality_tag), and rope positions. - runtime.py: load VAE encoder + vision tower; generate() accepts first_frame/last_frame, builds the fl2va presentation, encodes keyframes, and passes text_token_tags/cond_latents/frame_count/seed to the sampler. - sampler.py: thread pack kwargs + seed. - serve_hot_runtime.py / direct_t2v_preview.py: /generate and --first-frame/--last-frame accept image paths or base64.	2026-08-19 20:17:41 +07:00
Daniel Maddern	8730920634	Default Spark runs to Sol attention	2026-08-15 03:37:50 +07:00
Daniel Maddern	731a2813fd	Add Sol native attention baseline	2026-08-15 03:35:59 +07:00
Daniel Maddern	c2fb5c1e3f	Document NVFP4 native pack status	2026-08-15 03:00:06 +07:00
Daniel Maddern	8d90cec133	Fix pack profiler imports	2026-08-15 02:57:21 +07:00
Daniel Maddern	ddc0c5a2b7	Add NVFP4 pack geometry profiler	2026-08-15 02:56:19 +07:00
Daniel Maddern	d4c243c9ef	Match NVFP4 tie-to-even rounding	2026-08-15 02:48:06 +07:00
Daniel Maddern	11d3c4164e	Match NVFP4 midpoint rounding	2026-08-15 02:45:16 +07:00
Daniel Maddern	9c0992350f	Match NVFP4 signed zero encoding	2026-08-15 02:43:03 +07:00
Daniel Maddern	ef84a2786c	Match H3 NVFP4 row scale swizzle	2026-08-15 02:40:09 +07:00
Daniel Maddern	fbfb3453de	Zero initialize native NVFP4 block scales	2026-08-15 02:38:08 +07:00
Daniel Maddern	7f9b1560e4	Add H3 NVFP4 scale column swizzles	2026-08-15 02:34:52 +07:00
Daniel Maddern	1200152d79	Match wide NVFP4 block scale swizzle	2026-08-15 02:30:41 +07:00
Daniel Maddern	b3a2544936	Match NVFP4 block scale swizzle	2026-08-15 02:26:48 +07:00
Daniel Maddern	57b84b8d6b	Match NVFP4 block scale padding	2026-08-15 02:24:09 +07:00
Daniel Maddern	0e855b8e55	Prototype Vortex native NVFP4 quantizer	2026-08-15 02:22:37 +07:00
Daniel Maddern	1fec77d2e9	Tune Vortex NVFP4 scale geometry	2026-08-15 02:15:19 +07:00
Daniel Maddern	1cc0ffdd00	Add Vortex NVFP4 scale v1	2026-08-15 02:11:47 +07:00
Daniel Maddern	684b645546	Match Torch NVFP4 scale precision	2026-08-15 01:57:25 +07:00
Daniel Maddern	0fda84502a	Add Vortex NVFP4 scale kernel	2026-08-15 01:54:02 +07:00
Daniel Maddern	03dbe1456d	Prototype Vortex NVFP4 quantizer seam	2026-08-15 01:47:03 +07:00
Daniel Maddern	d6eabb150e	Add NVFP4 linear profiler	2026-08-15 01:37:42 +07:00
Daniel Maddern	7cc03f3e57	Add H3 block profiler plan	2026-08-15 01:32:18 +07:00
Daniel Maddern	3d0c093168	Add Sol-Attn backend	2026-08-14 21:04:32 +07:00
Daniel Maddern	75ee9ba4ca	Document backend implementation status	2026-08-14 20:39:20 +07:00
Daniel Maddern	c3cc04e98d	Add approximate H3 cache modes	2026-08-14 20:38:48 +07:00
Daniel Maddern	fb257ef982	Add exact memory backend options	2026-08-14 20:34:41 +07:00
Daniel Maddern	eaf9324145	Add KJ Sage attention backends	2026-08-14 20:28:56 +07:00
Daniel Maddern	1c0883a54b	Add persistent hot runtime service	2026-08-14 20:13:48 +07:00
Daniel Maddern	54dd649ccf	Plan persistent hot runtime	2026-08-14 20:02:47 +07:00
Daniel Maddern	d1cdd4b6b3	Record Sage3 cat benchmark	2026-08-14 20:02:14 +07:00
Daniel Maddern	a388aa64ff	Default video VAE decode to FP16	2026-08-14 19:50:36 +07:00
Daniel Maddern	ea7712b4b2	Use paired kitchen RoPE in VAE fast path	2026-08-14 14:58:11 +07:00
Daniel Maddern	8595dd875e	Add optional kitchen VAE fast ops	2026-08-14 14:54:38 +07:00
Daniel Maddern	f6166ede68	Add VAE decode benchmark tool	2026-08-14 14:36:42 +07:00