fix(reaper): Tier-2 finalization grace + claim-before-act (no dup advance)

Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes
agent_runs.exit_code FIRST, then does git push / PR / Plane comments before
_finalize_job, and the agent pid is already dead in that window — so the old
"exit_code recorded -> reap now" had no grace and could race a healthy job.
Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job)
BEFORE the atomic claim, so a reaper that lost the race had already enqueued
the next stage (dup advance / dup enqueue), violating ADR-001 Р-1.

Fix:
- Tier-2 grace: reap only once agent_runs.exit_code has been recorded for
  >= reaper_finalize_grace_s (new setting, default 300s; > max finalization
  window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New
  finished_age_s column computed in get_running_jobs.
- claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the
  reconciler pattern) to choose the terminal status, then atomically claim
  'done' FIRST; only the claim winner runs the advance. A loser performs no
  side effects -> no dup advance / dup enqueue.

Docs (golden source) updated in the same change: ADR-001, global adr-0011,
README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011
link). New tests cover the grace window, lost-claim no-side-effects, and the
already-advanced idempotent path.

Refs: ORCH-065

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-07 16:06:27 +00:00
committed by Dev Agent
parent 9b7c855df3
commit 720c31393a
10 changed files with 354 additions and 99 deletions

View File

@@ -314,6 +314,14 @@ class Settings(BaseSettings):
# reaper_max_running_s -> Tier-3 backstop ceiling: a job 'running' longer than
# this is reaped even when liveness is unknowable. MUST be
# > max agent_timeout + grace so a legit agent is safe.
# reaper_finalize_grace_s -> Tier-2 anti-false-positive: a LIVE monitor writes
# agent_runs.exit_code FIRST, THEN does git commit/push +
# PR + Plane usage comments (seconds..minutes) and only
# then _finalize_job. The agent pid is already dead in
# that window, so pid cannot tell "monitor died" from
# "monitor still finalizing". A job is reaped via Tier-2
# only once exit_code has been recorded for at least this
# many seconds (MUST be > the max finalization window).
# lease_reclaim_enabled -> kill-switch for the proactive stale/dead lease reclaim
# (false -> only the legacy lazy TTL reclaim in acquire).
# (reuse) merge_lock_timeout_s -> lease TTL; merge_gate_repos -> reclaim scope.
@@ -321,6 +329,7 @@ class Settings(BaseSettings):
reaper_interval_s: int = 60
reaper_dead_ticks: int = 2
reaper_max_running_s: int = 3600
reaper_finalize_grace_s: int = 300
lease_reclaim_enabled: bool = True
# Telegram notifications