# Preservation Audit

Audit date: 2026-08-25.

## Preservation Scope

This audit covers the local repository metadata and working tree, the configured
Git remote, the Spark checkout inspected through `ssh spark`, local top-level
benchmark/profiler artifacts plus the packaged SageAttention wheel, Spark's
selected physical research artifacts, and SHA-256 reconciliation of top-level
benchmark JSON filenames. It does not claim to preserve model checkpoints,
mounted parity tensors, generated media, container layers, external package
repositories, or every file under the broader Spark output tree.

No commit, tag, branch, archive, upload, or copy was created by this audit. The
audit records are this document, `research/REPRODUCIBILITY.md`, and the physical
file inventory in `research/artifact_manifest.json`.

## Git Topology And Commit Context

- Current local branch: `fl2va-direct-runtime`.
- Local `HEAD`: `bee998d8403aea18f9636b3773a8e7719ff3817c` (`Record RTX Pro 6000 multi-GPU scaling`,
  committed 2026-08-22 16:42:09 +0700).
- `origin/fl2va-direct-runtime` and `origin/master` both resolve to
  `bee998d8403aea18f9636b3773a8e7719ff3817c`; `git ls-remote` confirms the same
  two server heads.
- Local `master` remains at `ea4f4a2dadbd9754bcadfcfbe9a789f773cf838a`,
  which is the merge base with `fl2va-direct-runtime`; the feature branch is 76
  commits ahead and zero behind that local branch.
- `fl2va-direct-runtime` was created from
  `8730920634c3f4572353222e4ad307f01862fa16` on 2026-08-19 and then advanced
  linearly to `bee998d`.
- The other advertised head is `vae-decode-optimization` at
  `8730920634c3f4572353222e4ad307f01862fa16`.
- There are no tags locally or on the configured remote.
- Local reflogs contain the visible branch history but no deleted branch tip.
  Local `git fsck --full --no-reflogs --unreachable` finds only the empty blob
  `e69de29bb2d1d6434b8b29ae775ad8c2e48c5391`; it finds no useful dangling or
  unreachable commit. There is therefore no recoverable deleted branch history
  in the audited local object database.
- The Spark checkout is older: branch `fl2va-direct-runtime` and its tracking ref
  are at `9f62f6ea838150967422fe648ee25b8b2491356d`. Spark also has a local preservation
  branch `preserve/spark-parity-diagnostics-2026-08-13` at
  `f1ddc416f732c3de3a9f9de92eec3724d7f65ebc`, based on
  `a8df9ef1730b9cf30f9341fd1298214a3b406dce`. It is historical parity context,
  not a preservation point for the current optimization tree.

The exact former P1 dispatch source and some other removed experimental source
cannot be recovered from Git. They were never present in the reachable commit
history, reflogs do not expose a deleted tip containing them, and `fsck` exposes
no useful dangling commit. Current summaries can preserve decisions and measured
results, but they cannot reconstruct those exact source states.

## Dirty-Worktree Risk

The local tree has extensive tracked modifications and untracked source,
documentation, tests, tools, benchmark JSONs, and profiler reports relative to
`bee998d`. The post-commit research narrative and implementation are therefore
not protected by the remote heads. A clone of the remote reproduces only the
committed baseline.

The Spark checkout is independently dirty and differs from both local `HEAD` and
the local working tree. It contains modified runtime source and many untracked
research artifacts while remaining at `9f62f6e`. Neither dirty tree is a backup
of the other. Editing, cleaning, container rebuilds, output rotation, checkout
replacement, disk loss, or an accidental `git clean` can permanently remove
unique evidence. Git cannot restore untracked profiler binaries or source that
was never committed.

The risk is immediate rather than theoretical: the expected Spark production
container exists, but container state and Spark's mutable output directories
must not be treated as durable preservation. Final validation temporarily
started that container, confirmed `/ready` with Sage2 as both the initial and
current backend, and then stopped it again to leave the research environment
quiescent.

## Local Inventory

The local inventory is byte-exact as observed:

| Class | Location | Count | Bytes |
| --- | --- | ---: | ---: |
| Benchmark JSON | top-level `benchmarks/*.json` | 148 | 1,175,484 |
| Nsight Compute reports | top-level `benchmarks/*.ncu-rep` | 14 | 199,677,223 |
| Nsight Systems reports | top-level `benchmarks/*.nsys-rep` | 2 | 7,686,256 |
| Nsight CSV exports | top-level `benchmarks/*.csv` | 2 | 169,373 |
| SageAttention wheel | `wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl` | 1 | 1,680,352 |
| Selected local physical artifacts | all rows above | 167 | 210,388,688 |

All 167 selected files have one path, byte size, and SHA-256 record in
`research/artifact_manifest.json`; the manifest totals reproduce this table.

The two CSV exports and wheel explain why the selected total is larger than the
JSON plus `.ncu-rep` plus `.nsys-rep` subtotal. The unrelated
`benchmarks/rtxpro6000-server-2gpu-topology.txt` and research source files are
outside this physical-artifact total, although they remain within the broader
preservation scope.

## Spark Inventory

The audit originally retained a selected Spark aggregate of 279 physical
artifacts totaling 665,950,155 bytes. Within it, the top-level output JSON set
contains exactly 215 files totaling 2,259,709 bytes at
`/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks`.

The selected 279-file figure is a scoped preservation aggregate, not an
unqualified recursive count of the output directory. Re-enumeration found 280
current top-level files totaling 547,708,583 bytes, including the exact 215-file
JSON set, while the recursive directory has 291 JSONs because older/nested result
collections are also present. The original 279 paths were not retained, so the
665,950,155-byte aggregate cannot be reproduced or honestly assigned to physical
files. `research/artifact_manifest.json` therefore inventories the complete
reproducible 280-file top-level set and records the discrepancy explicitly: one
additional observed file and 118,241,572 fewer bytes than the retained aggregate.
It must not be presented as a reconstruction of the lost 279-path selection.

Spark also has the same 1,680,352-byte SageAttention ARM64 wheel in the checkout.
The current top-level output includes large `.ncu-rep`, `.nsys-rep`, `.sqlite`,
`.qdstrm`, CSV, tensor, media, log, and report files. Presence on Spark is not a
second durable copy unless the artifact is also independently stored and
checksum-verified.

## JSON Reconciliation

Reconciliation uses basename identity and SHA-256 over local top-level
`benchmarks/*.json` and Spark top-level output `*.json`:

| Result | Count |
| --- | ---: |
| Identical filename and SHA-256 | 106 |
| Same filename, different SHA-256 | 2 |
| Local only | 40 |
| Spark only | 107 |
| Local total | 148 |
| Spark total | 215 |

The two hash mismatches are:

- `gb10-cute-p0-attn-qkv-128rows.json`
- `gb10-fully-fused-one-step-fresh.json`

The arithmetic reconciles exactly: 106 identical plus 2 mismatches plus 40
local-only equals 148 local files; 106 plus 2 plus 107 Spark-only equals 215
Spark files. Same-name mismatches must be preserved as two versions, not resolved
by timestamp or overwritten. Local-only and Spark-only files are unique evidence
until copied into an immutable, checksummed collection.

## Large Artifact Policy

Git is appropriate for source, patches, small JSON summaries, exact commands,
manifests, and checksums. Raw `.ncu-rep`, `.nsys-rep`, `.sqlite`, `.qdstrm`,
tensor captures, media, and wheels should not be added casually to ordinary Git
history. They are large, tool/version-sensitive, and can make every clone carry
research payload indefinitely.

For each large-artifact preservation set:

1. Freeze an explicit relative-path manifest with byte size and SHA-256 for every
   file, including both versions of hash-mismatched JSONs.
2. Record producer revision or working-tree archive hash, exact command and
   environment where known, host/GPU facts, profiler version, and linked summary
   JSON.
3. Store the payload in immutable or versioned object storage, an archival share,
   or a dedicated artifact release. Use Git LFS only if its remote retention and
   clone policy are intentionally managed.
4. Keep at least two independently administered copies and verify restoration by
   checksum. A mutable Spark output directory and its checkout count as one
   failure domain, not two.
5. Do not delete raw profiler databases after exporting CSV until the export and
   summary are verified and the raw file is archived.
6. Never replace same-name mismatches in place. Preserve origin-qualified paths
   such as `local/` and `spark/` in the archive.

## Recoverability Conclusions

- `bee998d` and all reachable committed ancestors are recoverable from the
  configured remote.
- The present local optimization source, tests, documents, and many artifacts
  are not recoverable from that commit or remote because they are dirty or
  untracked.
- Spark provides additional unique evidence but is older, dirty, mutable, and
  not an exact working-tree replica.
- The 106 identical JSONs have two observed copies; the 40 local-only and 107
  Spark-only JSONs do not. The two mismatches represent four distinct payloads.
- Raw local profiler evidence is concentrated in 16 reports totaling
  207,363,479 bytes; Git cannot recreate it.
- No tag identifies the research state, no deleted branch tip or useful dangling
  commit was found, and the exact former P1 dispatch source plus other removed
  source cannot be recovered from Git.
- The preserved P3 patch and launcher are a positive exception: they retain an
  exact external source commit, patch, and orchestration path.
- The production image still exists locally on Spark, but the container is
  stopped and an image present in one Docker store is not archival recovery.

## Missing Evidence

- The original path list and class-by-class composition behind the retained
  279-file, 665,950,155-byte Spark aggregate; the complete current 280-file
  top-level inventory does not resolve that historical discrepancy.
- Exact commands and complete environments for most historical benchmarks and
  profiler captures.
- A content-addressed archive of each dirty working tree.
- Base-image digest, Python patch version, full installed-package lock, firmware,
  clock/power state, and model/checkpoint hashes.
- Checksums and retention status for mounted parity captures, model files,
  generated media, build logs, and P3 run worktrees.
- A Git-recoverable copy of the former P1 dispatch and other removed experimental
  source.
- A currently passing `/ready` response and a fresh complete test-suite result.

Until those gaps are closed, the evidence supports the recorded conclusions but
does not provide complete independent replay of every historical experiment.