feat(reaper): job-reaper + stale merge-lease reclaim + idempotent merge finalization

Closes the "zombie jobs" incident class: job status was set only inside
the live launcher process, so a process death left jobs.status='running'
forever; at max_concurrency=1 one zombie blocked ALL projects' queue
(self-hosting risk). Adds a background daemon (src/job_reaper.py) with
three-tier liveness (dead-pid streak / known exit_code / max-running
backstop) whose only mutating write is an atomic terminal flip guarded by
WHERE status='running' (no double-process). For exit0 the canonical QG is
the source of truth via gate-driven advance, not "exit0".

Also proactively reclaims stale merge-lease (dead pid OR TTL) via file
delete only (no git ops), and makes merge finalization idempotent
(pr_already_merged guard + up-to-date short-circuit on re-drive).

New jobs.pid column via idempotent _ensure_column (no migration); pid
stamped in launcher._spawn after Popen. Reaper start/stop in lifespan;
"reaper" snapshot in GET /queue. Kill-switches: ORCH_REAPER_ENABLED,
ORCH_REAPER_INTERVAL_S, ORCH_REAPER_DEAD_TICKS, ORCH_REAPER_MAX_RUNNING_S,
ORCH_LEASE_RECLAIM_ENABLED.

Invariants unchanged (AC-13): STAGE_TRANSITIONS, QG_CHECKS registry,
check_branch_mergeable signature/behaviour, BUG-8 rollback, hook exit
codes. restart-safe, never-raise per unit of background work.

Docs: docs/architecture/README.md, CHANGELOG.md, .env.example.
Tests: tests/test_job_reaper.py, tests/test_merge_lease_reclaim.py,
tests/test_merge_gate.py (TC-16), tests/test_merge_gate_race.py (TC-17),
tests/test_queue.py, tests/test_config.py (TC-19/TC-20). 742 passed.

Refs: ORCH-065

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

src/main.py

@@ -60,6 +60,19 @@ async def lifespan(app: FastAPI):
     if requeued:
         log.warning(f"Queue-recovery: requeued {requeued} running job(s) after restart")
 
+    # ORCH-065: proactive startup reclaim of dead/stale merge-leases, next to the
+    # queue-recovery above. A lease held by the previous (now dead) process pid is
+    # released at once instead of waiting for the TTL / a foreign acquire so the
+    # next merge is not blocked. Conditional (merge_gate_repos / self-hosting) and
+    # gated by ORCH_LEASE_RECLAIM_ENABLED; never raises.
+    try:
+        from .job_reaper import reclaim_all_stale_leases
+        reclaimed = reclaim_all_stale_leases()
+        if reclaimed:
+            log.warning(f"Startup lease-reclaim: reclaimed {reclaimed} stale merge-lease(s)")
+    except Exception as e:
+        log.warning(f"Startup lease-reclaim skipped: {e}")
+
     # L-2: rotate old per-run logs at startup (best-effort; never fatal).
     try:
         import os as _os
@@ -85,13 +98,22 @@ async def lifespan(app: FastAPI):
     from .reconciler import reconciler
     reconciler.start()
 
+    # ORCH-065: start the job-reaper LAST (after requeue_running_jobs + the worker
+    # + the reconciler) so its atomic status='running' guard never races the
+    # startup requeue. It reaps zombie jobs and periodically reclaims stale
+    # merge-leases. Kill-switch: ORCH_REAPER_ENABLED.
+    from .job_reaper import reaper
+    reaper.start()
+
     try:
         yield
     finally:
-        # Graceful shutdown order mirrors startup in reverse: stop the reconciler
-        # first (it must not enqueue new work while the worker is winding down),
-        # then the worker. Running agents keep going; their jobs are requeued on
-        # next start via queue-recovery if the process dies.
+        # Graceful shutdown order mirrors startup in reverse: stop the reaper
+        # first, then the reconciler (it must not enqueue new work while the
+        # worker is winding down), then the worker. Running agents keep going;
+        # their jobs are requeued on next start via queue-recovery if the
+        # process dies.
+        reaper.stop()
         reconciler.stop()
         worker.stop()
 
@@ -123,6 +145,7 @@ async def queue():
     from .db import job_status_counts, recent_jobs
     from .queue_worker import worker
     from .reconciler import reconciler
+    from .job_reaper import reaper
     from . import post_deploy
     return {
         "counts": job_status_counts(),
@@ -130,6 +153,7 @@ async def queue():
         "poll_interval": worker.poll_interval,
         "resilience": worker.status(),
         "reconcile": reconciler.status(),
+        "reaper": reaper.status(),
         "post_deploy": post_deploy.status(),
         "recent": recent_jobs(10),
     }