From bc0c5214ab4dc362605921a83b87ed0f0e10eae9 Mon Sep 17 00:00:00 2001 From: Daniel Maddern Date: Mon, 17 Aug 2026 02:05:27 +0700 Subject: [PATCH] Record Guard curated dataset v02 --- docs/handoff_current_state_20260817.md | 49 ++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/handoff_current_state_20260817.md b/docs/handoff_current_state_20260817.md index cb92f37..c6b4fc5 100644 --- a/docs/handoff_current_state_20260817.md +++ b/docs/handoff_current_state_20260817.md @@ -226,3 +226,52 @@ Latest verification after Model Studio work: - `python manage.py check`: passed - `python -m pytest -p no:django -q tests/test_model_studio_v01.py`: `6 passed` - Full suite: `163 passed, 1 skipped` + +## Guard Dataset Factory Update + +Commit `e27df77 Add Guard dataset curation pipeline` added Spark-backed dataset curation. + +The source policy is now: + +- All existing Guard data, including legacy holdouts, is authorized as a curation source by user mandate. +- Former holdouts can enter a new training version only after source-hash re-splitting produces new validation and regression manifests. +- Existing benchmark suites must be replaced/versioned before comparisons if their old holdout sources are used for training. +- Source data is never overwritten; every output is a new immutable `DatasetVersion`. + +Spark Guard location: + +- `/home/daniel/forgeguard` +- Active data root inspected: `/home/daniel/forgeguard/project/data` + +The first bad recursive Spark inventory created `227,137` null-record-count DB rows from individual evidence JSON objects. Those malformed local DB rows were purged. The importer now accepts only explicit manifest paths; it does not recurse through evidence stores. + +Qwen Dataset Curator proposal: + +- Proposal ID: `0afcb3f5-a097-4461-b890-14d350e9d8f6` +- Qwen identified heterogeneous schemas, weak/contradictory labels, provenance risks, and exact-holdout handling requirements. +- Its proposal is persisted as `DatasetCurationProposal` and a Model Studio artifact. + +Spark materialized versions: + +1. `guard_curated_v01` + - `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v01/` + - Train `4,684`; validation `657`; regression `541`. + +2. `guard_curated_v02` (current recommended corpus) + - Train manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/train.json` + - Validation manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/validation.json` + - Regression manifest: `/home/daniel/forgeguard/artifex_dataset_versions/guard_curated_v02/regression.json` + - Artifex DatasetVersion ID: `416fc9dd-c922-4581-8ed5-ca10a488d224` + - Train: `3,030` + - Validation: `437` + - Regression: `332` + - Train SHA-256: `d4c1a6accee0fac511641a24287cfdb469819fdd6d34af720c7349dc66b4803c` + - Validation SHA-256: `3e3d5ad0d306bdd4a530fd2cf1d32bea030980af8d964d95c34b28237f6d5f` + - Regression SHA-256: `a868c4221f291953a91cf5e03f8163b2fa4e134db5f21e369fbf7c75c2387f1c` + - Source records read: `10,057` + - Accepted after strict repair: `3,799` + - Exact duplicates removed: `4,099` + - Malformed/weak-schema records removed: `2,159` + - Strict repair requires Solidity pragma, parseable JSON output with `findings`, and excludes `c4_invalid` records. + +No Guard model training was started. The next required milestone is to deploy/sync Artifex state to Spark, register a new evaluation suite version using the v02 validation/regression manifests, run the fresh base-model baseline on Spark, then create the first actual training canary.