daniel/h3-blackwell-runtime
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
h3-blackwell-runtime/research/REPRODUCIBILITY.md
Daniel Maddern ea14b98544 Preserve Vortex performance research history
2026-08-25 20:30:22 +07:00

8.7 KiB
Raw Blame History

Reproducibility Record

Audit date: 2026-08-25.

Scope And Revision

The recoverable Git baseline is branch fl2va-direct-runtime at bee998d8403aea18f9636b3773a8e7719ff3817c. At audit time both origin/fl2va-direct-runtime and origin/master resolve to that commit. The research results added after that commit depend on a dirty working tree and are not reproducible from the commit alone. Do not describe the current tree as a committed revision.

The local checkout is S:\PycharmProjects\AuthorCompanion\tmp\h3-blackwell-runtime. The Spark checkout inspected over ssh spark is /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime. It is also dirty, is at the older commit 9f62f6ea838150967422fe648ee25b8b2491356d, and is not a mirror of the local working tree.

Evidenced Environment

  • Spark hostname: Inceptal.
  • Spark host architecture: aarch64; observed kernel: Linux 6.17.0-1026-nvidia #26-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 25 00:57:17 UTC 2026.
  • GPU: NVIDIA GB10, compute capability 12.1 (SM121).
  • NVIDIA driver: 580.173.02.
  • Docker server: 29.2.1.
  • Active research image recorded in the deployment evidence and still present locally on Spark: sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83, created 2026-08-23T21:05:05.748554319+07:00 for linux/arm64.
  • The inspected image environment reports CUDA 13.0.2; benchmark JSONs and pyproject.toml identify Torch 2.9.1+cu130.
  • Dockerfile.spark pins Comfy Kitchen 0.2.31, CUTLASS DSL 4.6.2, FlashAttention 4 4.0.0b27, Quack Kernels 0.6.4, and Sol-Attn commit 930a4d6e432ff8b8ed5e30ff2f72519b92d69bdf.
  • The SageAttention3 ARM64 wheel is wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl (1,680,352 bytes).
  • The exact digest of the base image ghcr.io/aeon-7/comfyui-aeon-spark:slim, Python patch version, OS package set, firmware, clocks, power state, and model/checkpoint hashes were not recorded. They are unavailable historical inputs unless recovered outside this repository.

Do not generalize GB10 facts to the RTX PRO 6000 or B200 runs. Their result JSONs record device-specific measurements, but this audit did not recover complete host, image, driver, and command manifests for those historical runs.

Artifact Locations

  • Local benchmark JSON and profiler evidence: benchmarks/.
  • Local isolated P3 source, patch, validator, and exact launcher: research/sage2_temporal_pair/.
  • Spark top-level benchmark output: /home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks.
  • Spark checkout artifacts mounted read-only by Compose: /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts.
  • Immutable parity locations and their status are catalogued in PARITY.md; notably the FL2VA sampler reference is under /home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference.
  • P3 run work roots and build logs default to /home/daniel/aeon-spark-test/h3/sage2-temporal-pair/<run-id>.
  • Runtime media output is mounted from /home/daniel/StoryStudioAssets/H3-output to /output.

Summary JSONs cite their source artifacts. Examples include benchmarks/gb10-fully-fused-fresh-nsight-summary.json, benchmarks/gb10-sage2-p0-register-scheduler-analysis.json, benchmarks/gb10-sage2-p1-entry-fusion-analysis.json, benchmarks/gb10-sage2-p2-vprep-analysis.json, benchmarks/gb10-sage2-p3-temporal-pair-analysis.json, and benchmarks/gb10-four-gemm-nvfp4-roofline.json.

Recovered Commands

The documented Spark service build, startup, and readiness sequence is:

docker compose -f compose.spark.yml build h3-hot-runtime
docker compose -f compose.spark.yml up -d h3-hot-runtime
curl http://127.0.0.1:8001/ready

The documented contract-suite command is:

python -m unittest discover -s tests -v

The exact P3 temporal-pair experiment launcher is preserved:

bash research/sage2_temporal_pair/run_spark_experiment.sh

Its preserved controls are:

RUN_SANITIZER=0 bash research/sage2_temporal_pair/run_spark_experiment.sh
RUN_NCU=1 bash research/sage2_temporal_pair/run_spark_experiment.sh
SAGE2_TEMPORAL_RUN_ID=manual-01 bash research/sage2_temporal_pair/run_spark_experiment.sh
MAX_JOBS=4 bash research/sage2_temporal_pair/run_spark_experiment.sh

research/sage2_temporal_pair/run_spark_experiment.sh pins SageAttention commit d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5, applies the preserved patch, builds isolated baseline and candidate extensions, runs Compute Sanitizer, checks 13 short cases and the real block-24 SHA, and runs the rotating timing gate. This is the only audited recent experiment for which the complete orchestration command and source patch were recovered.

Exact historical invocation commands for most JSON, Nsight Compute, and Nsight Systems artifacts were not embedded in the artifacts and could not be recovered from Git or current documentation. Tool filenames and summary metadata are not substitutes for the original argv, environment, warmup order, or profiler options. In particular, do not invent commands for the P0/P1/P2 studies, fused producer trajectories, fully fused profiles, or four-GEMM roofline capture.

Production-Default Checks

compose.spark.yml defines the validated Spark service with Sage2 and these environment values:

H3_DISABLE_MMAP=1
H3_NVFP4_SCALE_BACKEND=vortex
H3_NVFP4_SCALE_VERSION=1
H3_FUSED_ELEMENTWISE=1
H3_NVFP4_MODULATE_FUSION=1
H3_NVFP4_SWIGLU_FUSION=1
H3_SAGE_QKV_LAYOUT=strided_nhd
H3_SOL_QKV_LAYOUT=native

The current container metadata independently contains all eight values and the expected image ID. Check a deployment without changing it with:

docker inspect h3-blackwell-runtime-h3-hot-runtime-1 --format '{{.Image}}|{{.State.Status}}|{{json .Config.Env}}'
curl -fsS http://127.0.0.1:8001/ready

At initial audit time the container was Exited (137). Final validation temporarily started the same image, confirmed /ready with Sage2 as both the initial and current backend, and produced the configured 320x192, 22-frame warmup output in 6.293 seconds. The container was stopped again after the check. benchmarks/gb10-post-lora-revert-deployment-smoke.json independently records the same image, both NVFP4 producer fusions enabled, readiness true, three CUDA tests passing, and a valid 320x192, 22-frame H.264 smoke output.

Production must retain the rejected paths as rejected: the bounded-ring QKV runtime gate regressed, Sage2 P1 entry fusion missed its complete-block gate, P2 V preparation missed its absolute saving gate, and P3 temporal pairing missed 220 ms. Their summary artifacts state that production dispatch remained unchanged or integration was removed/skipped.

Tests And Readiness Procedure

  1. Verify the intended revision and record the dirty state with git rev-parse HEAD, git branch --show-current, and git status --short --branch.
  2. Verify the image ID and all production-default environment values with the non-mutating docker inspect command above.
  3. Run python -m unittest discover -s tests -v in the built environment. Final validation ran the Spark checkout through image 1d340e14cb6f; all 41 tests exposed by that checkout passed in 53.079 seconds. The local Windows checkout was not run because its Python environment does not contain Torch, and the independently dirty Spark checkout is not byte-identical to the local tree.
  4. Start or recreate the service only when a deployment action is intended, using the recovered Compose commands above. Wait for startup warmup.
  5. Require curl -fsS http://127.0.0.1:8001/ready to succeed and inspect the returned backend, loaded options, and warmup state. /health alone is not a substitute for readiness.
  6. Run a real resident generation smoke and validate media dimensions, codec, frame count, and duration. The exact historical request command for the final deployment smoke is unavailable; its request fields are preserved in benchmarks/gb10-post-lora-revert-deployment-smoke.json.
  7. For a candidate numerical path, follow the applicable validation ladder in NVFP4_STREAMING_DESIGN.md: adversarial/randomized checks, real projections, complete blocks, all 50 blocks, two-step trajectories, canonical 12-step checksums, then profiler and peak-memory recapture. Skipped gates must remain explicitly marked skipped.

The dependency-free registry validator also passed all 62 records and reported the expected seven-status distribution. git diff --check passed before the preservation commit.

Passing historical summaries establish prior evidence only. A fresh run must record revision or tree hash, full argv, environment, dependency and image digests, input/checkpoint hashes, warmup order, raw outputs, and checksums to be independently reproducible.