Skip to content

Agent Queue Structured Store v1

Purpose: - define the next queue storage model before implementation starts, - remove fragile mutable-state editing from Agent_Work_Queue.yaml, - preserve current operator commands while making queue state/history more reliable.

Use this together with: - Agent_Orchestrator_v1.md - Multi_Agent_Execution_Playbook.md - Multi_Agent_Lane_Worktrees_v1.md - archive/README.md

1. Problem Statement

The current queue file mixes two very different concerns: 1. task definition and planning metadata, 2. fast-changing execution state.

That causes repeated failure modes: 1. hand-edited state drift (owner, completed_at, commit, missing fields), 2. validator breakage from queue-format mistakes, 3. archive sweeps as a recurring operational burden, 4. weak historical trace for claims, blockers, and handoffs, 5. rising merge friction as multiple planning updates touch one YAML file.

The existing governance docs already treat archive sweeps as an interim defense and anticipate a future structured queue/orchestrator store. This document makes that future model explicit.

2. Decision Summary

Use a split model: 1. YAML remains the source of truth for durable task definitions. 2. SQLite becomes the source of truth for mutable execution state and history. 3. Operator-facing commands remain the same unless a later V2 decision changes them.

Short version: - YAML defines what the work is. - SQLite defines where the work stands.

3. Source-of-Truth Split

3.1 YAML owns task definition

doc/governance/Agent_Work_Queue.yaml remains git-reviewed and human-readable.

YAML-owned fields: 1. id 2. title 3. role 4. priority 5. depends_on 6. acceptance_checks 7. notes

Definition fields should change only when planning or task shape changes.

3.2 SQLite owns mutable execution state

SQLite holds fast-changing coordination data.

State-owned fields: 1. status 2. owner 3. branch 4. completed_at 5. commit 6. optional lane 7. optional review metadata 8. append-only transition history

State changes happen through queue/orchestrator utilities, not direct YAML edits.

4. Operator Compatibility Rule

The first implementation must preserve the existing operator workflow.

Expected commands remain: 1. make queue-ready 2. make queue-claim 3. make queue-set-status 4. make queue-git-check 5. make orchestrator-status 6. make orchestrator-tick

Implementation rule: - change storage behind these commands, - do not force operators to learn a new command set in the first slice.

5.1 task_state

One current-state row per task id.

Minimum columns: 1. task_id (PK) 2. status 3. owner 4. branch 5. completed_at 6. commit 7. updated_at 8. updated_by 9. lane nullable

5.2 task_state_history

Append-only state transition log.

Minimum columns: 1. id (PK) 2. task_id 3. changed_at 4. changed_by 5. command_source 6. from_status 7. to_status 8. from_owner 9. to_owner 10. from_branch 11. to_branch 12. from_commit 13. to_commit 14. reason nullable 15. lane nullable

This history is required because git diffs on YAML are not a reliable operational history for claim/block/done transitions.

5.3 Optional future tables

Not required in the first slice, but the schema should leave room for: 1. review records, 2. handoff records, 3. parent/child task decomposition, 4. lane-level ownership, 5. exported snapshot metadata.

The orchestrator v2 implementation uses these reserved concepts as concrete tables: 1. task_handoffs for changed files, commands, results, risks, blockers, and next slice. 2. task_evidence for command-level pass/fail/skipped/blocked records. 3. task_reviews for D/E findings and approval state. 4. agent_sessions for session adapter status and transcript pointers.

6. Review Metadata Direction

Review should be schema-ready even if not enforced in the first implementation.

Recommended nullable state fields: 1. review_required 2. review_status 3. reviewer_role 4. reviewer 5. reviewed_at 6. review_note

V1 rule: - preserve current writer flow, - allow optional reviewer metadata, - do not enforce no-self-approval until the workflow is exercised further.

7. Lane-Ready Design (Deferred V2)

V1 remains one role -> one active writer.

But the schema should be lane-ready now: 1. lane nullable in current-state and history tables, 2. optional future fields: - parent_task_id - slice_key - merge_order

Why: - lane-based scaling is a known follow-on need, - nullable reserved fields are cheaper than redesigning the store later.

Do not expose lanes in operator commands yet unless V2 is activated.

8. Validation Model

Queue validation should split into three layers:

8.1 Definition validation

Validate YAML only: 1. schema shape, 2. required definition fields, 3. role/priority/dependency declaration sanity.

8.2 State validation

Validate SQLite only: 1. legal status values, 2. branch rules, 3. done-state completeness, 4. role/lane invariants, 5. review-state consistency if present.

8.3 Cross-reference validation

Validate YAML <-> SQLite join: 1. every state row maps to a known task id, 2. dependencies reference known definition ids, 3. in-progress/done tasks satisfy dependency rules, 4. exported snapshots match the effective joined view.

9. Git Storage Rule

Do not commit the live SQLite database file.

Git should store: 1. YAML task definitions, 2. schema/migrations for the queue DB, 3. deterministic exported snapshots when needed, 4. archive snapshots, 5. tooling/scripts.

Git must not store: 1. the live SQLite DB, 2. WAL/SHM files, 3. opaque mutable binary queue state.

10. Snapshot Rule

For now, generate readable snapshots only at: 1. archive sweeps, 2. handoff boundaries, 3. major planning transitions, 4. explicit governance evidence capture.

Do not require a git snapshot for every queue state mutation.

That keeps the DB operational and the repo readable without turning every claim or status flip into a commit.

11. Migration Plan

Phase 1

  1. Keep current YAML active queue intact.
  2. Introduce SQLite schema and import current execution state from YAML.
  3. Update queue/orchestrator utilities to read YAML definitions + SQLite state.
  4. Keep current make targets unchanged.

Phase 2

  1. Stop treating YAML execution fields as writable authority.
  2. Make YAML definition-focused.
  3. Export snapshots at archive/handoff only.

Phase 3

  1. Add optional reviewer metadata.
  2. Add optional lane fields when V2 is activated.
  3. Revisit archive policy once the DB-backed flow is proven.

12. Non-Goals

This slice does not attempt to: 1. redesign the entire agent orchestration model, 2. replace git-reviewable planning docs with a DB-only system, 3. introduce lane-based concurrency immediately, 4. make review enforcement mandatory in the first cut.

13. Implementation Standard

Any implementation of this model must satisfy: 1. existing operator commands remain stable in v1, 2. YAML task definition review remains easy in git, 3. queue history becomes more reliable than git-diff-on-YAML alone, 4. archive/handoff snapshots remain readable without opening SQLite directly, 5. rollback to YAML-only operation is documented before cutover.

Rollback guidance: - the live SQLite database is local mutable state and may be moved aside when corrupt or stale; - make queue-prune-stale removes state rows that reference deleted YAML tasks; - if the DB cannot be opened, move it to a timestamped .broken.* path and let the queue store rebuild from Agent_Work_Queue.yaml; - never commit the live SQLite DB, WAL, or SHM files.