diff --git a/docs/execution_graph_architecture.md b/docs/execution_graph_architecture.md new file mode 100644 index 0000000..6c7816c --- /dev/null +++ b/docs/execution_graph_architecture.md @@ -0,0 +1,204 @@ +# Execution Graph Architecture + +Date: 2026-08-15 + +## Implemented Architecture + +Artifex now has two intentionally separate graph concepts. + +Project DAG remains the canonical representation of what work exists: + +- `Project` +- `Milestone` +- `Feature` +- `Task` +- `TaskDependency` + +Execution Graph is now the canonical representation of how autonomous work is performed. It is runtime-neutral and persisted in Django/Postgres. + +## Graph Domain Models + +Implemented graph domain models: + +- `ExecutionGraphDefinition` +- `ExecutionGraphVersion` +- `GraphRun` +- `GraphNodeRun` +- `GraphEdgeTraversal` +- `GraphApproval` + +Execution graph version statuses: + +- `DRAFT` +- `CHALLENGER` +- `CHAMPION` +- `RETIRED` + +Graph run statuses: + +- `PENDING` +- `RUNNING` +- `PAUSED` +- `COMPLETE` +- `FAILED` +- `CANCELLED` + +Node runs persist status, timing, input/output metadata, failure evidence, telemetry, optional agent version, and optional model request reference. + +Edge traversals persist selected conditional transitions. + +## Graph Specification + +Graph specs are serializable dictionaries with: + +- name +- version +- graph type +- entry node +- nodes +- edges +- conditional edge labels +- terminal nodes +- metadata + +Specs do not execute arbitrary Python. Node execution resolves through `NodeHandlerRegistry`. + +## TaskExecutionGraph V1 + +`TaskExecutionGraph v1` mirrors the prior `AutonomousTaskLoop` behavior. + +Nodes: + +- `claim_task` +- `prepare_worktree` +- `build_context` +- `coder` +- `run_tests` +- `review` +- `judge` +- `commit` +- `retry_or_fail` +- `cleanup` +- `complete` +- `fail` + +Important semantics preserved: + +- deterministic test failure still reaches Reviewer +- Reviewer failure goes to retry/fail +- Judge failure goes to retry/fail +- maximum semantic task attempts remains three total attempts +- commit is guarded against duplicate commits on resume +- CoderToolLoop remains inside the `coder` node + +## NativeGraphRuntime + +`NativeGraphRuntime` is the reference implementation. + +It supports: + +- conditional edges +- loops +- persisted graph runs +- persisted node runs +- edge traversal history +- checkpoint/resume at major node boundaries +- terminal success/failure +- pause state +- cancellation +- graph events +- bounded metadata + +Django/Postgres remains canonical for task, attempt, worktree, test, review, judge, commit, event, agent, and Progeny state. + +## AutonomousTaskLoop Delegation + +`AutonomousTaskLoop` still uses `TaskScheduler` to claim Project DAG work. + +After a task is claimed, it now creates a `GraphRun` for the champion `TaskExecutionGraph v1` and delegates task lifecycle execution to `NativeGraphRuntime`. + +The scheduler remains Project-DAG-oriented. The execution runtime handles task-attempt workflow. + +## LangGraphRuntime + +`LangGraphRuntime` is no longer a pure placeholder. It builds a LangGraph `StateGraph` from Artifex graph definitions when the `langgraph` package is installed. + +Artifex remains runtime-neutral: + +- Artifex owns graph definitions +- Artifex owns persisted graph state +- Artifex owns node contracts +- LangGraph is an execution backend + +Current limitation: the local environment used during implementation did not have `langgraph` installed, so deterministic tests validate the adapter boundary and missing-dependency behavior. Full LangGraph execution parity requires installing the declared `langgraph>=0.2,<0.3` dependency in local/Spark environments. + +## Checkpoint And Resume + +Coarse checkpointing is implemented at graph node boundaries. + +Resume behavior avoids repeating a completed interrupted node. Loop re-entry creates a new `GraphNodeRun` visit via `visit_index`. + +Irreversible commit behavior is guarded by checking for an existing `CommitRecord` for the task before creating a new commit. + +## Human Approval + +Graph approval support is implemented with `GraphApproval`. + +A node can pause the graph with `AWAITING_APPROVAL`. A signal can approve pending graph approvals and resume execution. + +This is intentionally minimal and prepares future gates for risky migrations, deployment, Progeny promotion, destructive infrastructure changes, and project plan approvals. + +## Subgraph Support + +The graph spec supports runtime-neutral subgraph representation through node metadata. + +CoderToolLoop is not moved into a subgraph yet. It remains inside the `coder` node. + +## Graph Inspection + +`graph_run_inspection()` exposes UI-ready JSON containing: + +- graph/version +- node list +- edges +- node statuses +- current node +- durations +- failures +- selected edge traversals + +This is sufficient for a future visual execution graph UI. + +## Progeny Integration + +`ProgenySignal` can now reference: + +- `graph_run` +- `graph_node_run` +- `execution_graph_version` + +This enables future investigations such as: + +- failures by graph version +- failures by node type +- patch success rate by workflow version +- Reviewer rework changes after inserting a verification node + +## Migrations + +New migrations: + +- `graph/0001_initial.py` +- `graph/0002_graphnoderun_visit_index.py` +- `graph/0003_graphapproval.py` +- `projects/0003_commitrecord_graph_run.py` +- `agents/0005_progenysignal_graph_lineage.py` + +## Known Limitations + +- Full LangGraph parity execution is implemented but not exercised in this environment because `langgraph` is not installed. +- Parallel branch execution is represented by the graph model but not executed concurrently by `NativeGraphRuntime`. +- Subgraph support is represented in the spec but not yet expanded into nested `GraphRun` execution. +- CoderToolLoop remains attempt-granularity for checkpointing. +- Graph telemetry is persisted at node level, but model request references are not yet automatically linked to individual node runs. +- Event Bus records graph facts but is not an orchestration engine.