fix(reaper): Tier-2 finalization grace + claim-before-act (no dup advance)

Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes
agent_runs.exit_code FIRST, then does git push / PR / Plane comments before
_finalize_job, and the agent pid is already dead in that window — so the old
"exit_code recorded -> reap now" had no grace and could race a healthy job.
Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job)
BEFORE the atomic claim, so a reaper that lost the race had already enqueued
the next stage (dup advance / dup enqueue), violating ADR-001 Р-1.

Fix:
- Tier-2 grace: reap only once agent_runs.exit_code has been recorded for
  >= reaper_finalize_grace_s (new setting, default 300s; > max finalization
  window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New
  finished_age_s column computed in get_running_jobs.
- claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the
  reconciler pattern) to choose the terminal status, then atomically claim
  'done' FIRST; only the claim winner runs the advance. A loser performs no
  side effects -> no dup advance / dup enqueue.

Docs (golden source) updated in the same change: ADR-001, global adr-0011,
README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011
link). New tests cover the grace window, lost-claim no-side-effects, and the
already-advanced idempotent path.

Refs: ORCH-065

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-07 16:06:27 +00:00
committed by Dev Agent
parent 9b7c855df3
commit 720c31393a
10 changed files with 354 additions and 99 deletions

View File

@@ -601,11 +601,15 @@ def requeue_running_jobs() -> int:
def get_running_jobs() -> list[dict]:
"""ORCH-065: snapshot of every 'running' job for the job-reaper scan.
Each row carries the job columns plus three reaper inputs:
Each row carries the job columns plus four reaper inputs:
* ``running_age_s`` — seconds since ``started_at`` (Tier-3 backstop);
* ``exit_code`` — the linked ``agent_runs.exit_code`` (Tier-2: process
finished but the job is still 'running' -> monitor died mid-finalize);
* ``finished_at_run`` — the linked ``agent_runs.finished_at`` (debug only).
* ``finished_at_run`` — the linked ``agent_runs.finished_at``;
* ``finished_age_s`` — seconds since ``agent_runs.finished_at`` (Tier-2
finalization grace: a LIVE monitor writes exit_code, THEN does git
push / PR / Plane comments before _finalize_job, so a freshly-finished
run is NOT yet a zombie — the reaper waits ``reaper_finalize_grace_s``).
A LEFT JOIN on ``run_id`` keeps jobs with no agent_runs row (exit_code NULL).
Read-only; never mutates. The reaper applies liveness/streak/backstop on top.
@@ -616,7 +620,9 @@ def get_running_jobs() -> list[dict]:
"SELECT j.*, "
"CAST(strftime('%s','now') - strftime('%s', j.started_at) AS INTEGER) "
" AS running_age_s, "
"r.exit_code AS exit_code, r.finished_at AS finished_at_run "
"r.exit_code AS exit_code, r.finished_at AS finished_at_run, "
"CAST(strftime('%s','now') - strftime('%s', r.finished_at) AS INTEGER) "
" AS finished_age_s "
"FROM jobs j LEFT JOIN agent_runs r ON r.id = j.run_id "
"WHERE j.status='running'"
).fetchall()