Record Guard curated dataset v02

docs/handoff_current_state_20260817.md

@@ -226,3 +226,52 @@ Latest verification after Model Studio work:
 - `python manage.py check`: passed
 - `python -m pytest -p no:django -q tests/test_model_studio_v01.py`: `6 passed`
 - Full suite: `163 passed, 1 skipped`
+
+## Guard Dataset Factory Update
+
+Commit `e27df77 Add Guard dataset curation pipeline` added Spark-backed dataset curation.
+
+The source policy is now:
+
+- All existing Guard data, including legacy holdouts, is authorized as a curation source by user mandate.
+- Former holdouts can enter a new training version only after source-hash re-splitting produces new validation and regression manifests.
+- Existing benchmark suites must be replaced/versioned before comparisons if their old holdout sources are used for training.
+- Source data is never overwritten; every output is a new immutable `DatasetVersion`.
+
+Spark Guard location:
+
+- `/home/daniel/forgeguard`
+- Active data root inspected: `/home/daniel/forgeguard/project/data`
+
+The first bad recursive Spark inventory created `227,137` null-record-count DB rows from individual evidence JSON objects. Those malformed local DB rows were purged. The importer now accepts only explicit manifest paths; it does not recurse through evidence stores.
+
+Qwen Dataset Curator proposal:
+
+- Proposal ID: `0afcb3f5-a097-4461-b890-14d350e9d8f6`
+- Qwen identified heterogeneous schemas, weak/contradictory labels, provenance risks, and exact-holdout handling requirements.
+- Its proposal is persisted as `DatasetCurationProposal` and a Model Studio artifact.
+
+Spark materialized versions:
+
+1. `guard_curated_v01`
+   - `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v01/`
+   - Train `4,684`; validation `657`; regression `541`.
+
+2. `guard_curated_v02` (current recommended corpus)
+   - Train manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/train.json`
+   - Validation manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/validation.json`
+   - Regression manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/regression.json`
+   - Artifex DatasetVersion ID: `416fc9dd-c922-4581-8ed5-ca10a488d224`
+   - Train: `3,030`
+   - Validation: `437`
+   - Regression: `332`
+   - Train SHA-256: `d4c1a6accee0fac511641a24287cfdb469819fdd6d34af720c7349dc66b4803c`
+   - Validation SHA-256: `3e3d5ad0d306bdd4a530fd2cf1d32bea030980af8d964d95c34b28237f6d5f`
+   - Regression SHA-256: `a868c4221f291953a91cf5e03f8163b2fa4e134db5f21e369fbf7c75c2387f1c`
+   - Source records read: `10,057`
+   - Accepted after strict repair: `3,799`
+   - Exact duplicates removed: `4,099`
+   - Malformed/weak-schema records removed: `2,159`
+   - Strict repair requires Solidity pragma, parseable JSON output with `findings`, and excludes `c4_invalid` records.
+
+No Guard model training was started. The next required milestone is to deploy/sync Artifex state to Spark, register a new evaluation suite version using the v02 validation/regression manifests, run the fresh base-model baseline on Spark, then create the first actual training canary.