feat(reaper): job-reaper + stale merge-lease reclaim + idempotent merge finalization

Closes the "zombie jobs" incident class: job status was set only inside
the live launcher process, so a process death left jobs.status='running'
forever; at max_concurrency=1 one zombie blocked ALL projects' queue
(self-hosting risk). Adds a background daemon (src/job_reaper.py) with
three-tier liveness (dead-pid streak / known exit_code / max-running
backstop) whose only mutating write is an atomic terminal flip guarded by
WHERE status='running' (no double-process). For exit0 the canonical QG is
the source of truth via gate-driven advance, not "exit0".

Also proactively reclaims stale merge-lease (dead pid OR TTL) via file
delete only (no git ops), and makes merge finalization idempotent
(pr_already_merged guard + up-to-date short-circuit on re-drive).

New jobs.pid column via idempotent _ensure_column (no migration); pid
stamped in launcher._spawn after Popen. Reaper start/stop in lifespan;
"reaper" snapshot in GET /queue. Kill-switches: ORCH_REAPER_ENABLED,
ORCH_REAPER_INTERVAL_S, ORCH_REAPER_DEAD_TICKS, ORCH_REAPER_MAX_RUNNING_S,
ORCH_LEASE_RECLAIM_ENABLED.

Invariants unchanged (AC-13): STAGE_TRANSITIONS, QG_CHECKS registry,
check_branch_mergeable signature/behaviour, BUG-8 rollback, hook exit
codes. restart-safe, never-raise per unit of background work.

Docs: docs/architecture/README.md, CHANGELOG.md, .env.example.
Tests: tests/test_job_reaper.py, tests/test_merge_lease_reclaim.py,
tests/test_merge_gate.py (TC-16), tests/test_merge_gate_race.py (TC-17),
tests/test_queue.py, tests/test_config.py (TC-19/TC-20). 742 passed.

Refs: ORCH-065

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

src/db.py

@@ -76,6 +76,11 @@ def init_db():
     # (CREATE TABLE IF NOT EXISTS won't add columns to an already-created table).
     _ensure_column(conn, "jobs", "transient_attempts", "INTEGER NOT NULL DEFAULT 0")
     _ensure_column(conn, "jobs", "available_at", "TEXT")
+    # ORCH-065: pid of the spawned agent process, stamped in launcher._spawn next to
+    # run_id/started_at. The job-reaper uses it for Tier-1 liveness (os.kill(pid, 0))
+    # to detect a 'running' job whose process died before _finalize_job. Idempotent
+    # ALTER (no-op once present) -> safe on the live prod DB.
+    _ensure_column(conn, "jobs", "pid", "INTEGER")
     # ORCH-5 (M-7): webhook delivery de-dup. Add events.delivery_id and a PARTIAL
     # unique index. Partial (WHERE delivery_id IS NOT NULL) so pre-existing rows
     # (which have NULL delivery_id) never collide with each other. Restart-safe:
@@ -593,6 +598,76 @@ def requeue_running_jobs() -> int:
     return int(n)
 
 
+def get_running_jobs() -> list[dict]:
+    """ORCH-065: snapshot of every 'running' job for the job-reaper scan.
+
+    Each row carries the job columns plus three reaper inputs:
+      * ``running_age_s`` — seconds since ``started_at`` (Tier-3 backstop);
+      * ``exit_code``     — the linked ``agent_runs.exit_code`` (Tier-2: process
+        finished but the job is still 'running' -> monitor died mid-finalize);
+      * ``finished_at_run`` — the linked ``agent_runs.finished_at`` (debug only).
+
+    A LEFT JOIN on ``run_id`` keeps jobs with no agent_runs row (exit_code NULL).
+    Read-only; never mutates. The reaper applies liveness/streak/backstop on top.
+    """
+    conn = get_db()
+    try:
+        rows = conn.execute(
+            "SELECT j.*, "
+            "CAST(strftime('%s','now') - strftime('%s', j.started_at) AS INTEGER) "
+            "  AS running_age_s, "
+            "r.exit_code AS exit_code, r.finished_at AS finished_at_run "
+            "FROM jobs j LEFT JOIN agent_runs r ON r.id = j.run_id "
+            "WHERE j.status='running'"
+        ).fetchall()
+    finally:
+        conn.close()
+    return [dict(r) for r in rows]
+
+
+def reap_running_job(
+    job_id: int,
+    status: str,
+    run_id: int | None = None,
+    error: str | None = None,
+) -> bool:
+    """ORCH-065: atomic terminal flip of a RUNNING job by the job-reaper.
+
+    Mirrors ``mark_job`` but carries the ``status='running'`` guard in the WHERE
+    clause and reports ``rowcount`` so a late-arriving monitor / the startup
+    ``requeue_running_jobs`` / a second reaper tick can never double-process the
+    same row (AC-5, restart-safe). Returns True iff THIS call won the flip
+    (rowcount == 1); False -> someone else already moved the row.
+
+    Status semantics match ``mark_job``: done/failed stamp ``finished_at``; queued
+    clears ``started_at``/``finished_at`` so the next claim treats it as fresh.
+    """
+    conn = get_db()
+    try:
+        sets = ["status = ?"]
+        params: list = [status]
+        if run_id is not None:
+            sets.append("run_id = ?")
+            params.append(run_id)
+        if error is not None:
+            sets.append("error = ?")
+            params.append(error)
+        if status in ("done", "failed"):
+            sets.append("finished_at = datetime('now')")
+        elif status == "queued":
+            sets.append("started_at = NULL")
+            sets.append("finished_at = NULL")
+        params.append(job_id)
+        cur = conn.execute(
+            f"UPDATE jobs SET {', '.join(sets)} WHERE id = ? AND status='running'",
+            params,
+        )
+        conn.commit()
+        return cur.rowcount == 1
+    finally:
+        conn.close()
+
+
 def get_job(job_id: int) -> dict | None:
     """Fetch a single job by id."""
     conn = get_db()