admin/orchestrator
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix(reaper): Tier-2 finalization grace + claim-before-act (no dup advance)

Browse Source 
Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes
agent_runs.exit_code FIRST, then does git push / PR / Plane comments before
_finalize_job, and the agent pid is already dead in that window — so the old
"exit_code recorded -> reap now" had no grace and could race a healthy job.
Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job)
BEFORE the atomic claim, so a reaper that lost the race had already enqueued
the next stage (dup advance / dup enqueue), violating ADR-001 Р-1.

Fix:
- Tier-2 grace: reap only once agent_runs.exit_code has been recorded for
  >= reaper_finalize_grace_s (new setting, default 300s; > max finalization
  window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New
  finished_age_s column computed in get_running_jobs.
- claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the
  reconciler pattern) to choose the terminal status, then atomically claim
  'done' FIRST; only the claim winner runs the advance. A loser performs no
  side effects -> no dup advance / dup enqueue.

Docs (golden source) updated in the same change: ADR-001, global adr-0011,
README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011
link). New tests cover the grace window, lost-claim no-side-effects, and the
already-advanced idempotent path.

Refs: ORCH-065

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
claude-bot
2026-06-07 16:06:27 +00:00
committed by Dev Agent
parent 9b7c855df3
commit 720c31393a
10 changed files with 354 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.env.example
View File

@@ -123,20 +123,27 @@ ORCH_RECONCILE_SKIP_BLOCKED_ENABLED=true
# (one zombie at max_concurrency=1 blocks the whole shared queue) and periodically
# reclaims dead/stale merge-leases. Liveness is three-tier: Tier-1 dead jobs.pid
# (os.kill(pid,0)) after REAPER_DEAD_TICKS consecutive dead ticks (anti-false-positive
# for a live agent); Tier-2 agent_runs.exit_code recorded but job still 'running';
# Tier-3 backstop after REAPER_MAX_RUNNING_S. The terminal flip carries an atomic
# status='running' guard so it never double-processes a row racing requeue_running_jobs.
# REAPER_ENABLED -> global kill-switch (false -> strictly prior behaviour).
# REAPER_INTERVAL_S -> background scan period (seconds).
# REAPER_DEAD_TICKS -> consecutive dead-pid ticks before reaping (Tier-1, >=2).
# REAPER_MAX_RUNNING_S -> Tier-3 backstop ceiling; must exceed max agent_timeout+grace.
# LEASE_RECLAIM_ENABLED -> kill-switch for the proactive stale/dead lease reclaim
# (false -> only the legacy lazy TTL reclaim in acquire_merge_lease).
# for a live agent); Tier-2 agent_runs.exit_code recorded but job still 'running'
# (only after a REAPER_FINALIZE_GRACE_S finalization grace, so a live monitor still
# doing git push / PR / Plane comments is never reaped); Tier-3 backstop after
# REAPER_MAX_RUNNING_S. The terminal flip carries an atomic status='running' guard and
# precedes any advance/enqueue (claim-before-act) so it never double-processes/-advances
# a row racing a late monitor or requeue_running_jobs.
# REAPER_ENABLED -> global kill-switch (false -> strictly prior behaviour).
# REAPER_INTERVAL_S -> background scan period (seconds).
# REAPER_DEAD_TICKS -> consecutive dead-pid ticks before reaping (Tier-1, >=2).
# REAPER_MAX_RUNNING_S -> Tier-3 backstop ceiling; must exceed max agent_timeout+grace.
# REAPER_FINALIZE_GRACE_S -> Tier-2 grace: how long agent_runs.exit_code must have been
# recorded before a still-'running' job is reaped; MUST exceed
# the max finalization window (git push + PR + Plane comments).
# LEASE_RECLAIM_ENABLED -> kill-switch for the proactive stale/dead lease reclaim
# (false -> only the legacy lazy TTL reclaim in acquire_merge_lease).
# (reuse) ORCH_MERGE_LOCK_TIMEOUT_S -> lease TTL; ORCH_MERGE_GATE_REPOS -> reclaim scope.
ORCH_REAPER_ENABLED=true
ORCH_REAPER_INTERVAL_S=60
ORCH_REAPER_DEAD_TICKS=2
ORCH_REAPER_MAX_RUNNING_S=3600
ORCH_REAPER_FINALIZE_GRACE_S=300
ORCH_LEASE_RECLAIM_ENABLED=true
# ORCH-021: post-deploy production monitoring + degradation reaction. After the