fix(staging): host-side ssh execution + env classification for staging-runner (ORCH-123)
All checks were successful
CI / test (push) Successful in 1m8s
CI / test (pull_request) Successful in 1m8s

The ORCH-115 deterministic staging-runner ran `docker exec` FROM INSIDE the prod
`orchestrator` container, which ships only `openssh-client git curl` — no `docker`
CLI (Dockerfile:11). `Popen(["docker", ...])` hit FileNotFoundError -> a PERMANENT
environment defect that was mis-routed as a code-fail rollback
`deploy-staging -> development` (burning developer-retries). Incident ORCH-116:
every self-hosting task reaching deploy-staging was doomed to a false rollback.

Fix (adr-0049, additive, flag-gated, never-raise, self-hosting scope; the gate /
artifact contract / STAGE_TRANSITIONS / DB schema are byte-for-byte unchanged):

- D1: build_staging_command() wraps the SAME `docker exec ... staging_check.py
  ... --mode stub` in `ssh <user@host> '<...>'` so it runs HOST-SIDE over the
  existing trusted ssh channel (mirror self_deploy / image_freshness). New flag
  staging_runner_exec_host_side (default True). No docker CLI/SDK added to the
  image, docker.sock not used in-container (D2 security).
- D3: three-way classify_staging_outcome (suite-ran / permanent-env /
  transient-infra), disambiguating the exit=1 collision by scanning stderr.
- D4: invariant "infra != code-fail" — permanent-env / exhausted transient-infra
  end in an infra-HOLD (no rollback, no developer-retry), NOT a false FAILED
  rollback (supersedes ORCH-115 D5). A really-executed failing suite still rolls
  back (anti-over-tolerance). R-2 verified: a held deploy-staging task is not
  rolled back by the reconciler.
- D5: prod-like preflight() of the host-side channel at startup (main.lifespan,
  best-effort, never blocks).
- D8: snapshot adds permanent_env / exec_host_side / preflight.

Docs (golden source, same PR): INFRA.md execution-boundary section,
architecture/README.md, CLAUDE.md, CHANGELOG.md, .env.example.

Tests: tests/test_orch123_staging_runner_exec.py (TC-01 mandatory regression
red->green; TC-02..TC-14 + R-2). ORCH-115 anti-drift green (3 tests updated for
the D1/D4/D8 supersession). Full suite: 2131 passed.

Refs: ORCH-123

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-16 08:42:36 +03:00
parent e1872e3d94
commit cc41dd849c
9 changed files with 917 additions and 52 deletions

View File

@@ -79,6 +79,21 @@ async def lifespan(app: FastAPI):
except Exception as e:
log.warning(f"Transition-lease recovery skipped: {e}")
# ORCH-123 (adr-0049 / D5 / FR-4): prod-like preflight of the host-side staging
# execution channel. The deploy-staging staging-runner (ORCH-115) runs the suite
# HOST-SIDE over ssh because the prod container ships no docker CLI (Dockerfile:11);
# probe the channel at startup so a broken environment (no docker on host / staging
# down / ssh unreachable / no ssh target) surfaces HERE — not postfactum as a false
# rollback of a real task (incident ORCH-116). Purely observational: it never blocks
# the start / gates the pipeline. never raises (runs after requeue + lease-recovery).
try:
from . import staging_runner
ok, reason = staging_runner.preflight()
if not ok:
log.warning(f"Staging-runner preflight: {reason}")
except Exception as e:
log.warning(f"Staging-runner preflight skipped: {e}")
# ORCH-065: proactive startup reclaim of dead/stale merge-leases, next to the
# queue-recovery above. A lease held by the previous (now dead) process pid is
# released at once instead of waiting for the TTL / a foreign acquire so the