fix(reaper): Tier-2 finalization grace + claim-before-act (no dup advance)
Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes agent_runs.exit_code FIRST, then does git push / PR / Plane comments before _finalize_job, and the agent pid is already dead in that window — so the old "exit_code recorded -> reap now" had no grace and could race a healthy job. Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job) BEFORE the atomic claim, so a reaper that lost the race had already enqueued the next stage (dup advance / dup enqueue), violating ADR-001 Р-1. Fix: - Tier-2 grace: reap only once agent_runs.exit_code has been recorded for >= reaper_finalize_grace_s (new setting, default 300s; > max finalization window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New finished_age_s column computed in get_running_jobs. - claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the reconciler pattern) to choose the terminal status, then atomically claim 'done' FIRST; only the claim winner runs the advance. A loser performs no side effects -> no dup advance / dup enqueue. Docs (golden source) updated in the same change: ADR-001, global adr-0011, README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011 link). New tests cover the grace window, lost-claim no-side-effects, and the already-advanced idempotent path. Refs: ORCH-065 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
12
src/db.py
12
src/db.py
@@ -601,11 +601,15 @@ def requeue_running_jobs() -> int:
|
||||
def get_running_jobs() -> list[dict]:
|
||||
"""ORCH-065: snapshot of every 'running' job for the job-reaper scan.
|
||||
|
||||
Each row carries the job columns plus three reaper inputs:
|
||||
Each row carries the job columns plus four reaper inputs:
|
||||
* ``running_age_s`` — seconds since ``started_at`` (Tier-3 backstop);
|
||||
* ``exit_code`` — the linked ``agent_runs.exit_code`` (Tier-2: process
|
||||
finished but the job is still 'running' -> monitor died mid-finalize);
|
||||
* ``finished_at_run`` — the linked ``agent_runs.finished_at`` (debug only).
|
||||
* ``finished_at_run`` — the linked ``agent_runs.finished_at``;
|
||||
* ``finished_age_s`` — seconds since ``agent_runs.finished_at`` (Tier-2
|
||||
finalization grace: a LIVE monitor writes exit_code, THEN does git
|
||||
push / PR / Plane comments before _finalize_job, so a freshly-finished
|
||||
run is NOT yet a zombie — the reaper waits ``reaper_finalize_grace_s``).
|
||||
|
||||
A LEFT JOIN on ``run_id`` keeps jobs with no agent_runs row (exit_code NULL).
|
||||
Read-only; never mutates. The reaper applies liveness/streak/backstop on top.
|
||||
@@ -616,7 +620,9 @@ def get_running_jobs() -> list[dict]:
|
||||
"SELECT j.*, "
|
||||
"CAST(strftime('%s','now') - strftime('%s', j.started_at) AS INTEGER) "
|
||||
" AS running_age_s, "
|
||||
"r.exit_code AS exit_code, r.finished_at AS finished_at_run "
|
||||
"r.exit_code AS exit_code, r.finished_at AS finished_at_run, "
|
||||
"CAST(strftime('%s','now') - strftime('%s', r.finished_at) AS INTEGER) "
|
||||
" AS finished_age_s "
|
||||
"FROM jobs j LEFT JOIN agent_runs r ON r.id = j.run_id "
|
||||
"WHERE j.status='running'"
|
||||
).fetchall()
|
||||
|
||||
Reference in New Issue
Block a user