claude-bot
7cb1f83f6c
On the deploy-staging -> deploy edge the live monitor stamps agent_runs.finished_at FIRST, then runs the heavy edge sub-gates (security/merge-gate re-test/coverage/image-freshness) in-thread for MINUTES and only THEN _finalize_job. Reaper Tier-2 measures finished_age_s from finished_at, so past reaper_finalize_grace_s it treated the live, long finalizer as dead and independently re-ran the advance -> a second re-test went red -> false rollback deploy-staging -> development while the original finalizer concurrently merged the PR (incident ORCH-111, job 1914). Add a process-local finalizer-ownership registry (src/finalizer_liveness.py, never-raise): the monitor mark()s ownership right after the exit_code stamp and clear()s it in a try/finally around the (verbatim-extracted) finalization tail, so an exception in the monitor thread still releases ownership and a genuinely dead finalizer is reaped. The reaper Tier-2 consults the marker only when the kill-switch is on AND the task stage == deploy-staging AND ownership is active -> DEFER (no second advance) and fall through to the Tier-3 backstop, which ignores the marker (a stuck/dead finalizer is still reaped in bounded time). In-memory is authoritative (monitor + reaper are daemon threads of one uvicorn process); restart is covered by the startup requeue_running_jobs. Additive, global kill-switch reaper_finalizer_liveness_enabled (default True; false -> reaper byte-for-byte prior). STAGE_TRANSITIONS / QG_CHECKS / every check_* / machine-verdict keys / DB schema unchanged; grace/ceiling and the ORCH-065/109/110 budget invariant untouched; never restarts prod, never pushes main. Observability: finalizer_defers_total + finalizer_owned in GET /queue. Tests: tests/test_orch113_reaper_finalizer_liveness.py (TC-01..TC-08, incl. the mandatory ORCH-111 regression: red before the fix, green after). Refs: ORCH-113 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
62 KiB
62 KiB