fix(stage-engine): durable transition-ownership lease + expected-stage CAS (ORCH-114)
Close the root class of the ORCH-110/111/112/113 incident chain: side-effectful stage transitions had no single ownership. `advance_stage` is re-enterable and wrote the stage with a bare `UPDATE ... WHERE id=?` (no compare-and-swap), while >=5 actors (monitor / Plane-webhook / reconciler F-1 / job-reaper / deploy-finalizer) enter the same transition independently. A concurrent or post-restart re-entry therefore re-applied irreversible effects (merge_pr / coverage-ratchet / image-rebuild / prod-deploy initiation) and produced a contradictory rollback<->done (incident ORCH-111, job 1914 / PR #130). Two complementary layers, both additive, under one kill-switch, never-raise: 1. Durable transition-lease (new table `transition_lease`) — owner-exclusion on ENTRY to the side-effectful region: a second actor that sees a LIVE owner does not start the heavy sub-gates at all (prevention, not post-hoc repair). 2. Expected-stage CAS (`db.update_task_stage_cas`) — atomicity on the stage WRITE: a lost race aborts with NO side effect. Also closes the 6 paths that write the stage in bypass of advance_stage (gitea x5 + plane rollback). Owner liveness = owner_pid + owner_boot_id (NOT a heartbeat — a blocking 900s merge re-test cannot beat one; ADR-001 D3), making restart recovery free (a fresh boot_id renders every prior lease stale -> reclaimed by recover_on_startup). The lease has no own TTL: its hard age ceiling is the reaper Tier-3 backstop reaper_max_running_s, so the cross-cutting budget invariant ORCH-065/109/110/113 is untouched. Generalises ORCH-113 finalizer-liveness (process-local, Tier-2, deploy-staging) to a durable cross-path lease: the reaper consults it on all relevant paths (defer live, reclaim dead; Tier-3 ignores the marker -> bounded; a reap force-releases the lease); reconciler F-1 and the Plane webhook defer on an active lease; main.lifespan calls recover_on_startup() after requeue_running_jobs. finalizer_liveness.py is unchanged (it remains the kill-switch-off fallback). Scope self-hosting (transition_lease_repos="" -> orchestrator only; enduro untouched). Kill-switch ORCH_TRANSITION_LEASE_ENABLED=false -> CAS degenerates to the prior unconditional update_task_stage, lease inert, reaper -> ORCH-113 fallback (byte-for- byte pre-ORCH-114). STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys / existing table schemas — byte-for-byte (one additive table, no epoch column on tasks). Observability: read-only `transition_lease` block in GET /queue + a Telegram alert on forced/stale reclaim + optional POST /transition-lease/release?work_item=<id>. Coverage: tests/test_orch114_transition_ownership.py (TC-01 mandatory regression of the ORCH-111 class — red before fix, green after; TC-02..TC-14). Full suite green (2048 passed); the 4 webhook tests that spied on the removed gitea.update_task_stage were updated to spy on the new commit_stage_cas write path. ADR: docs/work-items/ORCH-114/06-adr/ADR-001-transition-ownership-lease-and-stage-cas.md Cross-cutting: docs/architecture/adr/adr-0045-transition-ownership-lease-and-stage-cas.md Refs: ORCH-114 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -590,6 +590,43 @@ class Settings(BaseSettings):
|
||||
lease_reclaim_enabled: bool = True
|
||||
reaper_finalizer_liveness_enabled: bool = True
|
||||
|
||||
# ORCH-114 (adr-0045): durable transition-ownership lease + expected-stage CAS for
|
||||
# side-effectful stage transitions. Generalises the process-local ORCH-113
|
||||
# finalizer-liveness to a DURABLE, cross-path owner-exclusion (additive table
|
||||
# `transition_lease`) so a concurrent OR post-restart re-entry into a side-effectful
|
||||
# transition (reaper / reconciler / webhook / startup-requeue) is deferred or a
|
||||
# no-op instead of re-applying an irreversible effect (merge_pr / coverage-ratchet /
|
||||
# image-rebuild / prod-deploy initiation / contradictory rollback↔done). Two
|
||||
# complementary layers, both gated by the SINGLE kill-switch below:
|
||||
# (1) durable lease on ENTRY to the side-effectful region (a second actor seeing a
|
||||
# live owner does not start the heavy sub-gates at all — prevention, not repair);
|
||||
# (2) expected-stage CAS on the stage WRITE (update_task_stage_cas: a lost race ->
|
||||
# abort with NO side effect), which also closes the 6 paths that write the
|
||||
# stage in bypass of advance_stage (gitea/plane direct update_task_stage).
|
||||
# Liveness of the owner = owner_pid + owner_boot_id (NOT a heartbeat — a blocking
|
||||
# 900s merge re-test cannot beat a heartbeat; ADR-001 D3), which makes restart
|
||||
# recovery free (a new process -> new boot_id -> all prior leases are instantly
|
||||
# stale -> reclaimed). The lease has NO own TTL: its hard age ceiling IS the reaper
|
||||
# Tier-3 backstop reaper_max_running_s (5400), so the cross-cutting budget invariant
|
||||
# ORCH-065/109/110/113 is untouched. STAGE_TRANSITIONS / QG_CHECKS / check_* /
|
||||
# machine-verdict keys / existing table schemas — byte-for-byte. never-raise:
|
||||
# hot-path guard fail-open (never wedge the shared queue), prod-safety fail-closed.
|
||||
# See docs/work-items/ORCH-114/06-adr/ADR-001-transition-ownership-lease-and-stage-cas.md
|
||||
# and the cross-cutting docs/architecture/adr/adr-0045-…md.
|
||||
# transition_lease_enabled -> SINGLE kill-switch (env ORCH_TRANSITION_LEASE_ENABLED).
|
||||
# False -> the lease is neither written nor read AND the
|
||||
# CAS degenerates to the prior unconditional
|
||||
# update_task_stage -> behaviour byte-for-byte as before
|
||||
# ORCH-114 (reaper -> ORCH-113 in-memory fallback,
|
||||
# reconciler/webhook skip-guard inert). Default True.
|
||||
# transition_lease_repos -> CSV scope (env ORCH_TRANSITION_LEASE_REPOS). Empty ->
|
||||
# applies ONLY to the self-hosting repo (orchestrator),
|
||||
# where the irreversible side-effectful edges live;
|
||||
# non-empty -> only the listed repos. Mirrors
|
||||
# coverage_gate_repos -> enduro untouched at the default.
|
||||
transition_lease_enabled: bool = True
|
||||
transition_lease_repos: str = ""
|
||||
|
||||
# ORCH-063: disk-watchdog — background heartbeat that measures host-FS fill via
|
||||
# the mounted bind-paths and Telegram-alerts the operator at >= threshold. On
|
||||
# 07.06.2026 the mva154 host disk silently hit 100% and stalled the WHOLE
|
||||
|
||||
Reference in New Issue
Block a user