fix(stage-engine): durable transition-ownership lease + expected-stage CAS (ORCH-114)

Close the root class of the ORCH-110/111/112/113 incident chain: side-effectful
stage transitions had no single ownership. `advance_stage` is re-enterable and wrote
the stage with a bare `UPDATE ... WHERE id=?` (no compare-and-swap), while >=5 actors
(monitor / Plane-webhook / reconciler F-1 / job-reaper / deploy-finalizer) enter the
same transition independently. A concurrent or post-restart re-entry therefore
re-applied irreversible effects (merge_pr / coverage-ratchet / image-rebuild /
prod-deploy initiation) and produced a contradictory rollback<->done (incident
ORCH-111, job 1914 / PR #130).

Two complementary layers, both additive, under one kill-switch, never-raise:
  1. Durable transition-lease (new table `transition_lease`) — owner-exclusion on
     ENTRY to the side-effectful region: a second actor that sees a LIVE owner does
     not start the heavy sub-gates at all (prevention, not post-hoc repair).
  2. Expected-stage CAS (`db.update_task_stage_cas`) — atomicity on the stage WRITE:
     a lost race aborts with NO side effect. Also closes the 6 paths that write the
     stage in bypass of advance_stage (gitea x5 + plane rollback).

Owner liveness = owner_pid + owner_boot_id (NOT a heartbeat — a blocking 900s merge
re-test cannot beat one; ADR-001 D3), making restart recovery free (a fresh boot_id
renders every prior lease stale -> reclaimed by recover_on_startup). The lease has no
own TTL: its hard age ceiling is the reaper Tier-3 backstop reaper_max_running_s, so
the cross-cutting budget invariant ORCH-065/109/110/113 is untouched.

Generalises ORCH-113 finalizer-liveness (process-local, Tier-2, deploy-staging) to a
durable cross-path lease: the reaper consults it on all relevant paths (defer live,
reclaim dead; Tier-3 ignores the marker -> bounded; a reap force-releases the lease);
reconciler F-1 and the Plane webhook defer on an active lease; main.lifespan calls
recover_on_startup() after requeue_running_jobs. finalizer_liveness.py is unchanged
(it remains the kill-switch-off fallback).

Scope self-hosting (transition_lease_repos="" -> orchestrator only; enduro untouched).
Kill-switch ORCH_TRANSITION_LEASE_ENABLED=false -> CAS degenerates to the prior
unconditional update_task_stage, lease inert, reaper -> ORCH-113 fallback (byte-for-
byte pre-ORCH-114). STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys /
existing table schemas — byte-for-byte (one additive table, no epoch column on tasks).

Observability: read-only `transition_lease` block in GET /queue + a Telegram alert on
forced/stale reclaim + optional POST /transition-lease/release?work_item=<id>.

Coverage: tests/test_orch114_transition_ownership.py (TC-01 mandatory regression of
the ORCH-111 class — red before fix, green after; TC-02..TC-14). Full suite green
(2048 passed); the 4 webhook tests that spied on the removed gitea.update_task_stage
were updated to spy on the new commit_stage_cas write path.

ADR: docs/work-items/ORCH-114/06-adr/ADR-001-transition-ownership-lease-and-stage-cas.md
Cross-cutting: docs/architecture/adr/adr-0045-transition-ownership-lease-and-stage-cas.md

Refs: ORCH-114
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/db.py

@@ -263,6 +263,28 @@ def init_db():
     _ensure_column(conn, "lessons", "attribution", "TEXT")
     _ensure_column(conn, "lessons", "target_repo", "TEXT")
     _ensure_column(conn, "lessons", "target_domain", "TEXT")
+    # ORCH-114 (adr-0045 / 08-data-requirements.md): durable transition-ownership
+    # lease. ONE additive object (CREATE TABLE IF NOT EXISTS, pattern repo_freeze/
+    # coverage_baseline/lessons) -> idempotent, restart-safe on the shared prod DB;
+    # existing tables (tasks/jobs/agent_runs/...) untouched byte-for-byte (NFR-3,
+    # AC-11). One row per task = at most one active owner of a side-effectful
+    # transition. Liveness of the holder = owner_boot_id (this process's start nonce)
+    # + owner_pid (os.getpid of the holding process); a row from a previous boot is
+    # instantly stale on restart -> reclaimed (ADR-001 D3). No index needed (access by
+    # PK task_id; snapshot() is a full-scan over a tiny table). The src/transition_lease.py
+    # leaf wraps all access in its never-raise contract. NO epoch/version column (D2:
+    # for the one-process model the stage IS the CAS version).
+    conn.executescript("""
+        CREATE TABLE IF NOT EXISTS transition_lease (
+            task_id       INTEGER PRIMARY KEY,
+            owner         TEXT NOT NULL,
+            owner_pid     INTEGER,
+            owner_boot_id TEXT,
+            run_id        INTEGER,
+            stage         TEXT,
+            acquired_at   TEXT NOT NULL DEFAULT (datetime('now'))
+        );
+    """)
     conn.commit()
     conn.close()
 
@@ -679,6 +701,39 @@ def update_task_stage(task_id: int, stage: str):
     conn.close()
 
 
+def update_task_stage_cas(task_id: int, expected_stage: str, new_stage: str) -> bool:
+    """ORCH-114 (adr-0045 / FR-2): compare-and-swap variant of update_task_stage.
+
+    Writes the stage ONLY when the task is still at ``expected_stage`` (the value the
+    caller read before running the side-effectful region) — ``UPDATE … SET stage=?
+    WHERE id=? AND stage=?`` — and reports whether THIS writer won. Returns:
+
+      * ``True``  -> ``rowcount == 1``: the CAS succeeded, the stage moved exactly once.
+      * ``False`` -> ``rowcount == 0``: the task is no longer at ``expected_stage``
+        (another actor already advanced/rolled it back, or the row is gone) -> the
+        caller MUST abort WITHOUT applying any side effect (merge_pr / ratchet /
+        rebuild / deploy-init / enqueue) — it lost the race.
+
+    In the current one-process model each side-effectful edge leads to a DISTINCT
+    next stage, so the stage itself is a complete version for the compare-and-swap;
+    no separate epoch/version column is needed (ADR-001 D2). The plain
+    ``update_task_stage`` above is kept unchanged for the kill-switch-off path and
+    for non-side-effectful writes. Mirrors the atomic rowcount-guard idiom of
+    ``claim_next_job`` / ``reap_running_job``.
+    """
+    conn = get_db()
+    try:
+        cur = conn.execute(
+            "UPDATE tasks SET stage = ?, updated_at = datetime('now') "
+            "WHERE id = ? AND stage = ?",
+            (new_stage, task_id, expected_stage),
+        )
+        conn.commit()
+        return cur.rowcount == 1
+    finally:
+        conn.close()
+
+
 # ---------------------------------------------------------------------------
 # ORCH-019: bug-fast-track task type (tasks.track) helpers
 # ---------------------------------------------------------------------------