h3-blackwell-runtime/research/REPRODUCIBILITY.md

# Reproducibility Record

Audit date: 2026-08-25.

## Scope And Revision

The recoverable Git baseline is branch `fl2va-direct-runtime` at
`bee998d8403aea18f9636b3773a8e7719ff3817c`. At audit time both
`origin/fl2va-direct-runtime` and `origin/master` resolve to that commit. The
research results added after that commit depend on a dirty working tree and are
not reproducible from the commit alone. Do not describe the current tree as a
committed revision.

The local checkout is
`S:\PycharmProjects\AuthorCompanion\tmp\h3-blackwell-runtime`. The Spark
checkout inspected over `ssh spark` is
`/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime`. It is also dirty, is at
the older commit `9f62f6ea838150967422fe648ee25b8b2491356d`, and is not a mirror of
the local working tree.

## Evidenced Environment

- Spark hostname: `Inceptal`.
- Spark host architecture: `aarch64`; observed kernel:
  `Linux 6.17.0-1026-nvidia #26-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 25 00:57:17 UTC 2026`.
- GPU: `NVIDIA GB10`, compute capability `12.1` (`SM121`).
- NVIDIA driver: `580.173.02`.
- Docker server: `29.2.1`.
- Active research image recorded in the deployment evidence and still present
  locally on Spark:
  `sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`,
  created `2026-08-23T21:05:05.748554319+07:00` for `linux/arm64`.
- The inspected image environment reports CUDA `13.0.2`; benchmark JSONs and
  `pyproject.toml` identify Torch `2.9.1+cu130`.
- `Dockerfile.spark` pins Comfy Kitchen `0.2.31`, CUTLASS DSL `4.6.2`,
  FlashAttention 4 `4.0.0b27`, Quack Kernels `0.6.4`, and Sol-Attn commit
  `930a4d6e432ff8b8ed5e30ff2f72519b92d69bdf`.
- The SageAttention3 ARM64 wheel is
  `wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl` (1,680,352 bytes).
- The exact digest of the base image
  `ghcr.io/aeon-7/comfyui-aeon-spark:slim`, Python patch version, OS package
  set, firmware, clocks, power state, and model/checkpoint hashes were not
  recorded. They are unavailable historical inputs unless recovered outside
  this repository.

Do not generalize GB10 facts to the RTX PRO 6000 or B200 runs. Their result JSONs
record device-specific measurements, but this audit did not recover complete
host, image, driver, and command manifests for those historical runs.

## Artifact Locations

- Local benchmark JSON and profiler evidence: `benchmarks/`.
- Local isolated P3 source, patch, validator, and exact launcher:
  `research/sage2_temporal_pair/`.
- Spark top-level benchmark output:
  `/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks`.
- Spark checkout artifacts mounted read-only by Compose:
  `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts`.
- Immutable parity locations and their status are catalogued in `PARITY.md`;
  notably the FL2VA sampler reference is under
  `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference`.
- P3 run work roots and build logs default to
  `/home/daniel/aeon-spark-test/h3/sage2-temporal-pair/<run-id>`.
- Runtime media output is mounted from
  `/home/daniel/StoryStudioAssets/H3-output` to `/output`.

Summary JSONs cite their source artifacts. Examples include
`benchmarks/gb10-fully-fused-fresh-nsight-summary.json`,
`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`,
`benchmarks/gb10-sage2-p1-entry-fusion-analysis.json`,
`benchmarks/gb10-sage2-p2-vprep-analysis.json`,
`benchmarks/gb10-sage2-p3-temporal-pair-analysis.json`, and
`benchmarks/gb10-four-gemm-nvfp4-roofline.json`.

## Recovered Commands

The documented Spark service build, startup, and readiness sequence is:

```bash
docker compose -f compose.spark.yml build h3-hot-runtime
docker compose -f compose.spark.yml up -d h3-hot-runtime
curl http://127.0.0.1:8001/ready
```

The documented contract-suite command is:

```bash
python -m unittest discover -s tests -v
```

The exact P3 temporal-pair experiment launcher is preserved:

```bash
bash research/sage2_temporal_pair/run_spark_experiment.sh
```

Its preserved controls are:

```bash
RUN_SANITIZER=0 bash research/sage2_temporal_pair/run_spark_experiment.sh
RUN_NCU=1 bash research/sage2_temporal_pair/run_spark_experiment.sh
SAGE2_TEMPORAL_RUN_ID=manual-01 bash research/sage2_temporal_pair/run_spark_experiment.sh
MAX_JOBS=4 bash research/sage2_temporal_pair/run_spark_experiment.sh
```

`research/sage2_temporal_pair/run_spark_experiment.sh` pins SageAttention commit
`d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5`, applies the preserved patch, builds
isolated baseline and candidate extensions, runs Compute Sanitizer, checks 13
short cases and the real block-24 SHA, and runs the rotating timing gate. This
is the only audited recent experiment for which the complete orchestration
command and source patch were recovered.

Exact historical invocation commands for most JSON, Nsight Compute, and Nsight
Systems artifacts were not embedded in the artifacts and could not be recovered
from Git or current documentation. Tool filenames and summary metadata are not
substitutes for the original argv, environment, warmup order, or profiler
options. In particular, do not invent commands for the P0/P1/P2 studies, fused
producer trajectories, fully fused profiles, or four-GEMM roofline capture.

## Production-Default Checks

`compose.spark.yml` defines the validated Spark service with Sage2 and these
environment values:

```text
H3_DISABLE_MMAP=1
H3_NVFP4_SCALE_BACKEND=vortex
H3_NVFP4_SCALE_VERSION=1
H3_FUSED_ELEMENTWISE=1
H3_NVFP4_MODULATE_FUSION=1
H3_NVFP4_SWIGLU_FUSION=1
H3_SAGE_QKV_LAYOUT=strided_nhd
H3_SOL_QKV_LAYOUT=native
```

The current container metadata independently contains all eight values and the
expected image ID. Check a deployment without changing it with:

```bash
docker inspect h3-blackwell-runtime-h3-hot-runtime-1 --format '{{.Image}}|{{.State.Status}}|{{json .Config.Env}}'
curl -fsS http://127.0.0.1:8001/ready
```

At initial audit time the container was `Exited (137)`. Final validation
temporarily started the same image, confirmed `/ready` with Sage2 as both the
initial and current backend, and produced the configured 320x192, 22-frame
warmup output in 6.293 seconds. The container was stopped again after the check.
`benchmarks/gb10-post-lora-revert-deployment-smoke.json` independently records
the same image, both NVFP4 producer fusions enabled, readiness true, three CUDA
tests passing, and a valid 320x192, 22-frame H.264 smoke output.

Production must retain the rejected paths as rejected: the bounded-ring QKV
runtime gate regressed, Sage2 P1 entry fusion missed its complete-block gate,
P2 V preparation missed its absolute saving gate, and P3 temporal pairing
missed 220 ms. Their summary artifacts state that production dispatch remained
unchanged or integration was removed/skipped.

## Tests And Readiness Procedure

1. Verify the intended revision and record the dirty state with `git rev-parse HEAD`,
   `git branch --show-current`, and `git status --short --branch`.
2. Verify the image ID and all production-default environment values with the
   non-mutating `docker inspect` command above.
3. Run `python -m unittest discover -s tests -v` in the built environment. Final
   validation ran the Spark checkout through image `1d340e14cb6f`; all 41 tests
   exposed by that checkout passed in 53.079 seconds. The local Windows checkout
   was not run because its Python environment does not contain Torch, and the
   independently dirty Spark checkout is not byte-identical to the local tree.
4. Start or recreate the service only when a deployment action is intended,
   using the recovered Compose commands above. Wait for startup warmup.
5. Require `curl -fsS http://127.0.0.1:8001/ready` to succeed and inspect the
   returned backend, loaded options, and warmup state. `/health` alone is not a
   substitute for readiness.
6. Run a real resident generation smoke and validate media dimensions, codec,
   frame count, and duration. The exact historical request command for the final
   deployment smoke is unavailable; its request fields are preserved in
   `benchmarks/gb10-post-lora-revert-deployment-smoke.json`.
7. For a candidate numerical path, follow the applicable validation ladder in
   `NVFP4_STREAMING_DESIGN.md`: adversarial/randomized checks, real projections,
   complete blocks, all 50 blocks, two-step trajectories, canonical 12-step
   checksums, then profiler and peak-memory recapture. Skipped gates must remain
   explicitly marked skipped.

The dependency-free registry validator also passed all 62 records and reported
the expected seven-status distribution. `git diff --check` passed before the
preservation commit.

Passing historical summaries establish prior evidence only. A fresh run must
record revision or tree hash, full argv, environment, dependency and image
digests, input/checkpoint hashes, warmup order, raw outputs, and checksums to be
independently reproducible.