From a1f3b7588af211f9fc70756a4c54c29f36608e08 Mon Sep 17 00:00:00 2001 From: claude-bot Date: Mon, 15 Jun 2026 14:50:43 +0300 Subject: [PATCH] fix(deploy): resilient-pull hygiene for dirty shared deploy-base (ORCH-112) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-deploy git pull blocked on a dirty shared main checkout (manual/abandoned WIP from a failed/cancelled task) — incident ORCH-111: "Your local changes to src/config.py would be overwritten by merge" wedged the prod deploy and required manual intervention (a group risk on self-hosting). The deploy hook (--deploy) now converges the deploy-base to a clean, current origin/main BEFORE the pull (git fetch + reset --hard origin/main + a SCOPED `git clean -fd`, NEVER -x), strictly preserving the rollback/log artefacts (.deploy-prev-image-* / deploy-hook.log via -e), gitignored .env/data/*.db/build (no -x), and sibling/.git state (out of clean scope). Gated by CHECKOUT_HYGIENE env injected by self_deploy.build_deploy_command only when the new pure never-raise leaf src/checkout_hygiene.py says applies(repo) (kill-switch + self-hosting scope). Convergence after failed/cancelled is this same deploy-time self-heal — cancel_task is NOT extended and no background janitor is introduced. Observability: the hook writes a `hygiene` sentinel, the Phase-C finalizer reads it and sends a best-effort Telegram alert. Additive, under kill-switch (ORCH_CHECKOUT_HYGIENE_ENABLED, default true; off -> bare `git pull origin main` 1:1 before ORCH-112), never-raise, self-hosting scope. STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys / DB schema / the hook exit-code contract (0/1/2, ORCH-036) are byte-for-byte untouched. Coverage: tests/test_deploy_checkout_hygiene.py (TC-01..TC-10; real-hook shell simulation in a temp git repo, no network/prod/ssh, + unit). TC-01 is the mandatory ORCH-111 regression (RED before the fix, GREEN after). Docs golden source updated in the same PR (CLAUDE.md, CHANGELOG.md, .env.example; INFRA.md / architecture/README.md / adr-0044 written at the architecture stage). Refs: ORCH-112 Co-Authored-By: Claude Opus 4.8 --- .env.example | 9 + .task-dev.md | 4 +- CHANGELOG.md | 5 + CLAUDE.md | 40 +++ scripts/orchestrator-deploy-hook.sh | 29 ++ src/checkout_hygiene.py | 214 ++++++++++++ src/config.py | 19 + src/main.py | 4 + src/self_deploy.py | 8 +- src/stage_engine.py | 11 + tests/test_deploy_checkout_hygiene.py | 485 ++++++++++++++++++++++++++ 11 files changed, 825 insertions(+), 3 deletions(-) create mode 100644 src/checkout_hygiene.py create mode 100644 tests/test_deploy_checkout_hygiene.py diff --git a/.env.example b/.env.example index 866d0df..caea51a 100644 --- a/.env.example +++ b/.env.example @@ -340,6 +340,15 @@ ORCH_DEPLOY_PROD_TARGET_IMAGE=orchestrator-orchestrator ORCH_DEPLOY_PROD_COMPOSE_PROFILE= ORCH_DEPLOY_PROD_PREV_IMAGE_FILE=.deploy-prev-image-prod +# ORCH-112: deploy-base checkout-hygiene (resilient-pull). The self-deploy hook +# converges a DIRTY shared deploy-base to a clean, current origin/main BEFORE the +# `git pull` (git fetch + reset --hard + a SCOPED `git clean -fd`, NEVER `-x`), so +# manual/abandoned WIP left by a failed/cancelled task never blocks the deploy +# (incident ORCH-111). False -> bare `git pull origin main` 1:1 as before ORCH-112. +# Empty REPOS -> only the self-hosting repo (orchestrator). +ORCH_CHECKOUT_HYGIENE_ENABLED=true +ORCH_CHECKOUT_HYGIENE_REPOS= + # ORCH-058: staging-image provenance before the BUILD-ONCE prod retag (INV-FRESH). # Guarantees the staging image promoted to prod is the EXACT artefact rebuilt from the # validated commit — two layers, self-hosting only: diff --git a/.task-dev.md b/.task-dev.md index 9753f7f..c218318 100644 --- a/.task-dev.md +++ b/.task-dev.md @@ -1,4 +1,4 @@ -Work item: ORCH-110 +Work item: ORCH-112 Repo: orchestrator -Branch: feature/ORCH-110-bug-merge-gate-local-re-test-t +Branch: feature/ORCH-112-bug-failed-cancelled-task-arti Stage: development \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index a9cd9bd..ccc2783 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,11 @@ Формат: [Keep a Changelog](https://keepachangelog.com/). Записи — на смысловой PR/задачу. ## [Unreleased] +- **Гигиена shared deploy-базы: устойчивый self-deploy `git pull` к грязному дереву** (ORCH-112, `fix`, bug→escalate full-cycle): устранён инцидент ORCH-111 — self-deploy падал на шаге `git pull origin main` хост-хука с `error: Your local changes to the following files would be overwritten by merge: src/config.py` (грязь от неуспешной/отменённой/брошенной задачи ORCH-104 в общем main checkout) → деплой вставал → ручное вмешательство (на self-hosting — групповой риск). Решение — **resilient-pull, встроенный в прод-deploy-хук** (`--deploy`): перед `git pull` хук при обнаружении грязи приводит deploy-базу к чистому актуальному `origin/main` (`git fetch` + `git reset --hard origin/main` + **скоупленный** `git clean -fd`). Аддитивно, под kill-switch, never-raise, скоуп self-hosting; `STAGE_TRANSITIONS` / реестр `QG_CHECKS` / семантика и имена `check_*` / machine-verdict-ключи / схема БД / exit-code-контракт хука (0/1/2, ORCH-036) — **байт-в-байт не тронуты** (это устойчивость deploy-пути, **не** Quality Gate и **не** стадия). ADR: `docs/work-items/ORCH-112/06-adr/ADR-001-deploy-base-checkout-hygiene.md`, сквозной `docs/architecture/adr/adr-0044-deploy-base-checkout-hygiene.md`. + - **Leaf `src/checkout_hygiene.py` (новый, чистый never-raise):** по образцу `serial_gate`/`cancel`/`self_deploy` (импортирует только `config`, лениво `self_deploy`/`qg.checks`/`notifications`) — `applies(repo)` (kill-switch `checkout_hygiene_enabled` + скоуп `checkout_hygiene_repos`, пусто → self-hosting only, локально и ПЕРВЫМ), `hook_env(repo, work_item_id)` (env-префикс `CHECKOUT_HYGIENE=1 HYGIENE_REPORT=`, инжектится в detached-команду хука только при `applies==True`, иначе `""` → голый pull 1:1), `read_report`/`alert_dirty` (наблюдаемость), `snapshot()` (read-only блок `GET /queue`). + - **Хук-блок «2a. Resilient pull» (`scripts/orchestrator-deploy-hook.sh`):** между шагом «1. Capture PREV_IMG» и «2. Pull», под `if [[ "${CHECKOUT_HYGIENE:-0}" == "1" ]]`. **Сохранность (NFR-2, жёсткий контракт):** `git clean` — **только `-fd`, НИКОГДА `-x`** (иначе удалил бы gitignored `.env`/прод-секреты, `data/*.db`/БД, `build/`); явные `-e '.deploy-prev-image-*'` и `-e 'deploy-hook.log'` (untracked-но-НЕ-ignored — иначе сломался бы rollback `do_rollback`); sibling `/.deploy-state-*`/`.merge-lease-*.json` (под родителем `$REPO`) и `.git/worktrees/*` (внутри `.git/`) — вне области `git clean` в `$REPO`. Каждый git-шаг — `|| log "...continuing"` (never-break): сбой гигиены не ухудшает исход относительно текущего голого pull; на чистой базе блок — no-op (happy-path и exit-коды байт-в-байт). `--build-staging` (build из worktree, без pull) не затронут. + - **Сходимость после failed/cancelled (FR-2)** — этим же deploy-time self-heal (база сходится на следующем же self-deploy); `cancel_task` (ORCH-090) **не расширяется**, фоновый janitor **не вводится**. **Наблюдаемость (FR-4)** — хук пишет sentinel `hygiene` в deploy-state каталог; Phase-C finalizer (`stage_engine.run_deploy_finalizer`) читает (`read_report`) и шлёт Telegram-алерт (`alert_dirty`, кликабельный номер, best-effort, never-raise — сбой алерта не валит деплой). + - **Флаги** (`config.py`, дефолт = боевое): `checkout_hygiene_enabled` (env `ORCH_CHECKOUT_HYGIENE_ENABLED`), `checkout_hygiene_repos` (env `ORCH_CHECKOUT_HYGIENE_REPOS`). Откат = `ORCH_CHECKOUT_HYGIENE_ENABLED=false` → деплой байт-в-байт до ORCH-112. Покрытие — `tests/test_deploy_checkout_hygiene.py` (TC-01…TC-10: шелл-симуляция реального хука во временном git-репо без сети/прода/ssh + unit; TC-01 — обязательный регресс ORCH-111: КРАСНЫЙ до фикса, ЗЕЛЁНЫЙ после). - **Job-reaper не реапит живой долго финализирующий монитор `deploy-staging`** (ORCH-113, `fix`, bug→escalate full-cycle): устранено расхождение состояния из инцидента ORCH-111 (deployer job 1914 / run_id 683). На ребре `deploy-staging → deploy` живой монитор (`launcher._monitor_agent`) штампит `agent_runs.finished_at`/`exit_code` **первым**, затем синхронно в своём потоке прогоняет тяжёлые edge-под-гейты (`security → merge-gate re-test → coverage → image-freshness`) — **минуты** — и лишь потом `_finalize_job`. Reaper Tier-2 меряет `finished_age_s` от `finished_at` (= начала финализации), поэтому по истечении `reaper_finalize_grace_s=300` трактовал живого долго финализирующего монитора как мёртвого и **независимо** повторял тот же тяжёлый advance: повторный re-test стал красным → ложный откат `deploy-staging → development` (+ ложный developer-retry) **параллельно** с тем, что исходный finalizer довёл deploy до SUCCESS и смержил PR — состояние раздвоилось. Аддитивно, под глобальным kill-switch, never-raise; `STAGE_TRANSITIONS`/`QG_CHECKS`/каждый `check_*`/machine-verdict ключи/схема БД — **байт-в-байт не тронуты**; `reaper_finalize_grace_s`/`reaper_max_running_s` и сквозной бюджет ORCH-065/109/110 (`5400 > Σ(gate-work)+grace`) сохранены; фикс не рестартит прод и не пушит `main`. ADR: `docs/work-items/ORCH-113/06-adr/ADR-001-reaper-finalizer-liveness-ownership.md`, сквозной `docs/architecture/adr/adr-0043-reaper-finalizer-liveness-ownership.md`. - **Leaf `src/finalizer_liveness.py` (новый, процесс-локальный реестр владения):** чистый never-raise модуль (паттерн `serial_gate`/`coverage_gate`, без сети/БД) — `mark(job_id, run_id, stage)` / `clear(job_id)` / `is_active(job_id)` / `snapshot()`; состояние `{job_id: {...}}` под `threading.Lock`. Авторитетно in-memory, т.к. монитор и reaper — daemon-**потоки одного** uvicorn-процесса (CMD без `--workers`) с общей SQLite-БД. Собственного TTL нет — ограничение по времени даёт Tier-3 backstop. `is_active` при ошибке → `False` (консервативно: не блокировать добивание). - **Эмиссия владения (`launcher._monitor_agent`):** `mark()` вызывается **сразу после** штампа `exit_code` (самый ранний момент Tier-2), хвост финализации вынесен в `_run_monitor_finalization` и обёрнут в `try/finally` с `clear()` в `finally` → исключение в потоке монитора гарантированно снимает владение, и реально мёртвый finalizer добивается. Маркер пишется **безусловно** (kill-switch гейтит только консультацию reaper, поэтому выключенный путь — байт-в-байт прежний). Хвост перенесён **дословно** (проверяется `git diff -w`: +49/−0, нулевое изменение логики). diff --git a/CLAUDE.md b/CLAUDE.md index 15edf53..ff45b52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -283,6 +283,46 @@ INV-4 (никогда push/force-push `main`) и запрет рестарта `docs/work-items/ORCH-110/06-adr/ADR-001-merge-gate-retest-infra-tolerance-and-tree-kill.md`, сквозной `docs/architecture/adr/adr-0042-merge-gate-retest-infra-tolerance-and-tree-kill.md`. +## Гигиена shared deploy-базы: устойчивый self-deploy `git pull` (ORCH-112) +Багфикс инцидента **ORCH-111** (bug → escalate full-cycle): прод-self-deploy падал на шаге +`git pull origin main` хост-хука (`scripts/orchestrator-deploy-hook.sh`) с `error: Your local changes +to the following files would be overwritten by merge: src/config.py` — грязь, оставленная +неуспешной/отменённой/брошенной задачей ORCH-104 в **общем** main checkout (`settings.deploy_host_repo_path`). +Деплой вставал → ручное вмешательство; на self-hosting (один прод-инстанс на все проекты) — групповой +риск. **Инвариант (нормативно):** shared main checkout `/` — **deploy/worktree-management +база, НЕ редактируемый workspace** (агенты — worktree `git_worktree`, build — worktree-контекст, fallback'и +гейтов — read-only `git show origin/main`); локальных правок там быть не должно. Решение — **resilient-pull, +встроенный в хук** (`--deploy`): перед `git pull` хук при грязи приводит базу к чистому актуальному +`origin/main` (`git fetch` + `git reset --hard origin/main` + **скоупленный** `git clean -fd`). Аддитивно, +под kill-switch, never-raise, скоуп self-hosting; `STAGE_TRANSITIONS` / реестр `QG_CHECKS` / семантика и +имена `check_*` / machine-verdict-ключи / схема БД / exit-code-контракт хука (0/1/2, ORCH-036) — **байт-в-байт +не тронуты** (это устойчивость deploy-пути, **не** Quality Gate и **не** стадия). +- **Leaf `src/checkout_hygiene.py` (чистый never-raise):** по образцу `serial_gate`/`cancel`/`self_deploy` + (импортирует только `config`, лениво `self_deploy`/`qg.checks`/`notifications`) — `applies(repo)` + (kill-switch `checkout_hygiene_enabled` + скоуп `checkout_hygiene_repos`, **пусто → self-hosting only**, + локально и ПЕРВЫМ), `hook_env(repo, work_item_id)` (env-префикс `CHECKOUT_HYGIENE=1 HYGIENE_REPORT=`, + инжектится в detached-команду `self_deploy.build_deploy_command` только при `applies==True`, иначе `""` → + хук видит `CHECKOUT_HYGIENE` неустановленным → голый `git pull` 1:1 до ORCH-112), `read_report`/`alert_dirty` + (наблюдаемость), `snapshot()` (read-only блок `GET /queue`). +- **Хук-блок «2a. Resilient pull»:** между шагом «1. Capture PREV_IMG» и «2. Pull», под + `if [[ "${CHECKOUT_HYGIENE:-0}" == "1" ]]`. **Сохранность (NFR-2, жёсткий контракт):** `git clean` — + **только `-fd`, НИКОГДА `-x`** (иначе удалил бы gitignored `.env`/прод-секреты, `data/*.db`/БД, `build/`); + явные `-e '.deploy-prev-image-*'` и `-e 'deploy-hook.log'` (untracked-но-НЕ-ignored — иначе сломался бы + rollback `do_rollback`); sibling `/.deploy-state-*`/`.merge-lease-*.json` и `.git/worktrees/*` — + вне области `git clean` в `$REPO`. Каждый git-шаг — `|| log "...continuing"` (never-break): сбой гигиены не + ухудшает исход относительно голого pull; чистая база → no-op (happy-path/exit-коды байт-в-байт); + `--build-staging` (build из worktree, без pull) не затронут. +- **Сходимость после failed/cancelled (FR-2)** — этим же deploy-time self-heal (база сходится на следующем же + self-deploy); `cancel_task` (ORCH-090) **не расширяется**, фоновый janitor **не вводится**. + **Наблюдаемость (FR-4)** — хук пишет sentinel `hygiene`; Phase-C finalizer (`stage_engine.run_deploy_finalizer`) + читает (`read_report`) и шлёт Telegram-алерт (`alert_dirty`, кликабельный номер, best-effort, never-raise). +- **Флаги** (`config.py`, дефолт = боевое): `checkout_hygiene_enabled` (env `ORCH_CHECKOUT_HYGIENE_ENABLED`), + `checkout_hygiene_repos` (env `ORCH_CHECKOUT_HYGIENE_REPOS`). Откат = `ORCH_CHECKOUT_HYGIENE_ENABLED=false` → + деплой байт-в-байт до ORCH-112. Покрытие — `tests/test_deploy_checkout_hygiene.py` (шелл-симуляция реального + хука во временном git-репо без сети/прода/ssh + unit; TC-01 — обязательный регресс ORCH-111: красный до + фикса, зелёный после). Детали — `docs/work-items/ORCH-112/06-adr/ADR-001-deploy-base-checkout-hygiene.md`, + сквозной `docs/architecture/adr/adr-0044-deploy-base-checkout-hygiene.md`. + ## Машинный журнал уроков (ORCH-098) Шаг 1 («Фундамент», F2) эпика саморазвития: формализует свободнотекстовые «уроки» из `memory/` в **машинную структурированную таблицу отклонений конвейера** `lessons`, фундамент для будущих diff --git a/scripts/orchestrator-deploy-hook.sh b/scripts/orchestrator-deploy-hook.sh index ef14f8d..7f7a0bd 100755 --- a/scripts/orchestrator-deploy-hook.sh +++ b/scripts/orchestrator-deploy-hook.sh @@ -220,6 +220,35 @@ else log "No previous image captured (first deploy or service not running?)" fi +# 2a. ORCH-112: resilient pull — converge the shared deploy-base to a clean, current +# origin/main BEFORE the pull, so a dirty working tree (manual/abandoned WIP left +# by a failed/cancelled task) never blocks the deploy (incident ORCH-111, dirt from +# ORCH-104). Gated by CHECKOUT_HYGIENE (Python kill-switch + self-hosting scope, +# injected by self_deploy.build_deploy_command). NEVER `-x` (would delete gitignored +# .env / data/*.db / build/); EXCLUDES the untracked-but-not-ignored rollback/log +# artefacts .deploy-prev-image-* and deploy-hook.log (NFR-2). Best-effort: every git +# step is `|| log "...continuing"` and the bare `git pull` below still runs +# (never-break). On a CLEAN base the whole block is a no-op -> the happy-path +# behaviour and exit-codes (0/1/2, ORCH-036) are byte-for-byte unchanged. +if [[ "${CHECKOUT_HYGIENE:-0}" == "1" ]]; then + dirty="$(git status --porcelain 2>/dev/null || true)" + if [[ -n "$dirty" ]]; then + log "HYGIENE: dirty deploy-base detected, converging to origin/main:" + log "$dirty" + git fetch origin main >> "$LOG" 2>&1 || log "HYGIENE: fetch failed (continuing)" + git reset --hard origin/main >> "$LOG" 2>&1 || log "HYGIENE: reset failed (continuing)" + git clean -fd \ + -e '.deploy-prev-image-*' \ + -e 'deploy-hook.log' \ + >> "$LOG" 2>&1 || log "HYGIENE: clean failed (continuing)" + if [[ -n "${HYGIENE_REPORT:-}" ]]; then + { printf 'dirty=1\n'; printf '%s\n' "$dirty"; } > "$HYGIENE_REPORT" 2>/dev/null || true + fi + else + log "HYGIENE: deploy-base already clean (no-op)" + fi +fi + # 2. Pull latest code (keeps the host working tree current for future builds; # the DEPLOYED artefact is the retagged SOURCE_IMAGE below when build-once). log "git pull origin main" diff --git a/src/checkout_hygiene.py b/src/checkout_hygiene.py new file mode 100644 index 0000000..ecde2b8 --- /dev/null +++ b/src/checkout_hygiene.py @@ -0,0 +1,214 @@ +"""ORCH-112 (ADR-001 / adr-0044): deploy-base checkout-hygiene leaf — pure policy. + +Leaf module mirroring ``src/serial_gate.py`` / ``src/cancel.py`` / ``src/self_deploy.py``: +pure, unit-testable, never-raise functions over ``config`` + the deploy-state sentinels. +Module-level imports are limited to ``config`` (and stdlib); ``self_deploy``, +``qg.checks.is_self_hosting_repo`` and ``notifications`` are imported LAZILY so this +stays a leaf and an import cycle can never form. + +What it answers / does (the MECHANISM — git fetch/reset/clean — lives in the host +deploy hook ``scripts/orchestrator-deploy-hook.sh`` block "2a. Resilient pull"; this +leaf only decides conditionality, builds the env gate, reads the report and alerts): + + * ``applies(repo)`` — is resilient-pull hygiene REAL here? + * ``hook_env(repo, work_item_id)`` — the ``CHECKOUT_HYGIENE=1 HYGIENE_REPORT=…`` + env prefix injected into the detached + deploy-hook command ("" when not applies). + * ``read_report(repo, work_item_id)`` — read the ``hygiene`` sentinel the hook wrote. + * ``alert_dirty(repo, work_item_id, report)``— best-effort Telegram + structured log. + * ``snapshot()`` — read-only block for ``GET /queue``. + +never-raise contract (self-hosting safety): every public function degrades +conservatively. ``applies`` -> False on error (hygiene inert == kill-switch off, the +safe default that keeps the bare ``git pull`` 1:1 as before ORCH-112). ``hook_env`` -> +"" on error (no env -> the hook's ``${CHECKOUT_HYGIENE:-0}`` guard stays 0). The report +reader / alert swallow every error so a deploy is NEVER crashed by an observability +hiccup (D5 / AC-8). +""" +from __future__ import annotations + +import logging +import os +import re +import shlex + +from .config import settings + +logger = logging.getLogger("orchestrator.checkout_hygiene") + +# Sentinel filename the hook writes (HYGIENE_REPORT points at it) and read_report +# reads back. Lives in the SAME deploy-state dir as self_deploy's ``result`` (shared +# mount visible to both host and container). +REPORT_NAME = "hygiene" + +# Repo tokens in the CSV scope must match this (mirrors serial_gate._REPO_TOKEN). The +# CSV is operator config, not user input, but the guard is mandatory; an invalid token +# is dropped. +_REPO_TOKEN = re.compile(r"^[A-Za-z0-9._-]+$") + + +# --------------------------------------------------------------------------- +# Conditionality (mirrors self_deploy_applies / serial_gate_applies) +# --------------------------------------------------------------------------- +def _scope_repos() -> set[str]: + """Sanitised set of in-scope repo tokens from ``checkout_hygiene_repos`` (CSV). + + Empty/blank CSV -> empty set, meaning "self-hosting only" (resolved in ``applies``). + Invalid tokens (regex miss) are dropped. Never raises. + """ + try: + raw = (settings.checkout_hygiene_repos or "").strip() + except Exception: # noqa: BLE001 + return set() + if not raw: + return set() + out: set[str] = set() + for tok in raw.split(","): + t = tok.strip() + if t and _REPO_TOKEN.match(t): + out.add(t) + elif t: + logger.warning("checkout_hygiene: dropping invalid repo token %r from CSV", t) + return out + + +def applies(repo: str) -> bool: + """Whether resilient-pull hygiene is REAL for this repo (D3 / AC-6). + + * ``checkout_hygiene_enabled=False`` -> always False (kill-switch; the hook sees + no CHECKOUT_HYGIENE env -> bare ``git pull origin main`` 1:1 as before ORCH-112). + * ``checkout_hygiene_repos`` (CSV) non-empty -> real only for listed repos. + * empty CSV -> real ONLY for the self-hosting repo (``orchestrator``), mirroring + ``self_deploy_repos`` — this is a self-hosting prod-deploy-path feature, so it + must NOT touch enduro / other repos' synchronous deploy. + Local-only (no network), meant to be checked FIRST. Never raises -> False on error. + """ + try: + if not getattr(settings, "checkout_hygiene_enabled", False): + return False + scope = _scope_repos() + if scope: + return (repo or "").strip() in scope + # Lazy import keeps this module a leaf (no qg import at module load). + from .qg.checks import is_self_hosting_repo + return is_self_hosting_repo(repo) + except Exception as e: # noqa: BLE001 - never-raise + logger.warning("checkout_hygiene.applies error for %s: %s", repo, e) + return False + + +# --------------------------------------------------------------------------- +# Env gate injected into the detached deploy-hook command (Phase B wiring) +# --------------------------------------------------------------------------- +def report_path_host(repo: str, work_item_id: str | None) -> str: + """HOST view of the ``hygiene`` sentinel path (the wrapper writes it there).""" + from . import self_deploy + return os.path.join(self_deploy.host_state_dir(repo, work_item_id), REPORT_NAME) + + +def hook_env(repo: str, work_item_id: str | None) -> str: + """Build the env-assignment prefix injected into the detached deploy-hook command. + + Returns ``CHECKOUT_HYGIENE=1 HYGIENE_REPORT=`` (shlex-quoted) ONLY when + ``applies(repo)`` is True; otherwise ``""`` so the hook's ``${CHECKOUT_HYGIENE:-0}`` + guard stays 0 and the bare ``git pull`` runs (1:1 before ORCH-112). The + ``HYGIENE_REPORT`` path is the HOST view of the deploy-state dir (the host wrapper + writes the sentinel there; the container reads it back via ``read_report``). Never + raises -> "" (no hygiene env, the safe default). + """ + try: + if not applies(repo): + return "" + report = report_path_host(repo, work_item_id) + return f"CHECKOUT_HYGIENE=1 HYGIENE_REPORT={shlex.quote(report)}" + except Exception as e: # noqa: BLE001 - never-raise -> no hygiene env + logger.warning("checkout_hygiene.hook_env error for %s/%s: %s", repo, work_item_id, e) + return "" + + +# --------------------------------------------------------------------------- +# Report sentinel reader (Phase C observability) +# --------------------------------------------------------------------------- +def read_report(repo: str, work_item_id: str | None) -> dict | None: + """Read the ``hygiene`` sentinel the hook wrote (container view of deploy-state). + + The hook writes the sentinel ONLY when it detected a dirty base, body:: + + dirty=1 + + + Returns ``{"dirty": True, "paths": [...]}`` when the sentinel exists and reports a + dirty base; ``None`` when there is no sentinel (clean base / hygiene disabled / not + written yet). Never raises -> None on error. + """ + try: + from . import self_deploy + p = os.path.join(self_deploy.container_state_dir(repo, work_item_id), REPORT_NAME) + with open(p, "r", encoding="utf-8") as f: + raw = f.read() + except FileNotFoundError: + return None + except Exception as e: # noqa: BLE001 - never-raise + logger.warning("checkout_hygiene.read_report error for %s/%s: %s", repo, work_item_id, e) + return None + lines = raw.splitlines() + if not any(ln.strip() == "dirty=1" for ln in lines): + return None + paths = [ + ln.strip() for ln in lines + if ln.strip() and not ln.strip().startswith("dirty=") + ] + return {"dirty": True, "paths": paths} + + +# --------------------------------------------------------------------------- +# Best-effort Telegram alert (Phase C observability) — D5 / AC-8 +# --------------------------------------------------------------------------- +def alert_dirty(repo: str, work_item_id: str | None, report: dict | None) -> bool: + """Structured log + best-effort Telegram that the deploy-base was dirty and was + converged to ``origin/main`` before the pull (D5 / AC-8). Returns True iff an alert + was sent. Its failure NEVER crashes the finalizer (never-raise) — observability is + best-effort and must not block the conveyor (AC-8 FAIL is "alert crashes deploy"). + """ + try: + if not report or not report.get("dirty"): + return False + paths = report.get("paths") or [] + n = len(paths) + logger.warning( + "checkout_hygiene: dirty deploy-base converged to origin/main for %s/%s " + "(%d path(s)): %s", repo, work_item_id, n, paths[:20], + ) + from .notifications import link_for, send_telegram + send_telegram( + f"\U0001f9f9 {link_for(work_item_id)}: грязная deploy-база сведена к " + f"origin/main перед прод-деплоем ({n} путь(ей) сброшено)." + ) + return True + except Exception as e: # noqa: BLE001 - never-raise: alert is best-effort + logger.warning("checkout_hygiene.alert_dirty error for %s/%s: %s", repo, work_item_id, e) + return False + + +# --------------------------------------------------------------------------- +# Observability snapshot for GET /queue (D3, optional) +# --------------------------------------------------------------------------- +def snapshot() -> dict: + """Read-only checkout-hygiene summary for GET /queue. + + Additive block; existing /queue keys are untouched. never-raise -> a minimal dict + with the flags on error. + """ + try: + enabled = bool(getattr(settings, "checkout_hygiene_enabled", False)) + except Exception: # noqa: BLE001 + enabled = False + try: + repos_cfg = getattr(settings, "checkout_hygiene_repos", "") or "" + except Exception: # noqa: BLE001 + repos_cfg = "" + return { + "enabled": enabled, + "repos": repos_cfg, + "scope": "csv" if (repos_cfg or "").strip() else "self-hosting-only", + } diff --git a/src/config.py b/src/config.py index 33f1bf4..d142ba1 100644 --- a/src/config.py +++ b/src/config.py @@ -290,6 +290,25 @@ class Settings(BaseSettings): deploy_prod_compose_profile: str = "" deploy_prod_prev_image_file: str = ".deploy-prev-image-prod" + # ORCH-112: deploy-base checkout-hygiene (resilient-pull). The self-deploy hook's + # bare `git pull origin main` in the shared main clone blocked on a dirty working + # tree (manual/abandoned WIP left by a failed/cancelled task — incident ORCH-111 + # from ORCH-104). The fix converges the deploy-base to a clean, current origin/main + # (git fetch + reset --hard + a SCOPED `git clean -fd`, NEVER `-x`) BEFORE the pull, + # gated by the CHECKOUT_HYGIENE env injected by self_deploy.build_deploy_command. + # Pure leaf: src/checkout_hygiene.py (never-raise). Not a Quality Gate / not a stage + # — STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict / DB schema / the + # hook's exit-code contract (0/1/2, ORCH-036) are byte-for-byte untouched. + # + # checkout_hygiene_enabled -> kill-switch (env ORCH_CHECKOUT_HYGIENE_ENABLED). + # False -> the hook gets no CHECKOUT_HYGIENE env -> + # bare `git pull origin main` 1:1 as before ORCH-112. + # checkout_hygiene_repos -> CSV scope (env ORCH_CHECKOUT_HYGIENE_REPOS). Empty + # -> only the self-hosting repo (orchestrator). Mirrors + # self_deploy_repos (a self-hosting prod-deploy feature). + checkout_hygiene_enabled: bool = True + checkout_hygiene_repos: str = "" + # ORCH-058: staging-image provenance before the BUILD-ONCE retag to prod. # Closes the INV-FRESH gap (ADR-001): the BUILD-ONCE retag (ORCH-36) promotes # the staging image to prod WITHOUT a rebuild, assuming the staging image is diff --git a/src/main.py b/src/main.py index f83cb75..f47fdad 100644 --- a/src/main.py +++ b/src/main.py @@ -214,6 +214,7 @@ async def queue(): from . import cancel from . import bug_fast_track from . import lessons + from . import checkout_hygiene from .disk_watchdog import disk_watchdog from .build_cache_pruner import build_cache_pruner return { @@ -254,6 +255,9 @@ async def queue(): # kill-switch, label, scope, bug-task counts + the structural savings metric # (architecture stages skipped). Additive block; never-raise. "bug_fast_track": bug_fast_track.snapshot(), + # ORCH-112 (D3): deploy-base checkout-hygiene observability (read-only) — + # kill-switch + scope. Additive block; never-raise. + "checkout_hygiene": checkout_hygiene.snapshot(), # ORCH-098 (FR-4 / AC-4): lessons-journal observability (read-only) — # kill-switch + counts by type/status + last N lessons. Additive block; # never-raise (snapshot() returns {"enabled": ...} minimum on error). diff --git a/src/self_deploy.py b/src/self_deploy.py index 8b23b1b..eae9e01 100644 --- a/src/self_deploy.py +++ b/src/self_deploy.py @@ -239,7 +239,7 @@ def build_deploy_command(repo: str, work_item_id: str | None, branch: str) -> li ``expected_revision`` returns ``""`` and the env is omitted, keeping the hook's backward-compatible "no provenance check" behaviour (AC-5 / AC-7). """ - from . import image_freshness + from . import checkout_hygiene, image_freshness host_dir = host_state_dir(repo, work_item_id) result_sentinel = os.path.join(host_dir, RESULT) @@ -262,6 +262,12 @@ def build_deploy_command(repo: str, work_item_id: str | None, branch: str) -> li expected_rev = image_freshness.expected_revision(repo, branch) if expected_rev: env_assignments += f" EXPECTED_REVISION={shlex.quote(expected_rev)}" + # ORCH-112: inject CHECKOUT_HYGIENE=1 HYGIENE_REPORT= only when the leaf says + # hygiene applies (kill-switch + self-hosting scope). Empty -> the hook's + # ${CHECKOUT_HYGIENE:-0} guard stays 0 -> bare `git pull` 1:1 as before ORCH-112. + hygiene_env = checkout_hygiene.hook_env(repo, work_item_id) + if hygiene_env: + env_assignments += f" {hygiene_env}" inner = ( f"cd {shlex.quote(settings.deploy_host_repo_path)} && " f"{env_assignments} " diff --git a/src/stage_engine.py b/src/stage_engine.py index b7627d7..ae6c8ea 100644 --- a/src/stage_engine.py +++ b/src/stage_engine.py @@ -1957,6 +1957,17 @@ def run_deploy_finalizer(job: dict): logger.info( f"Task {task_id}: deploy finalized, hook exit={code} -> deploy_status={status}" ) + + # ORCH-112 (D5 / AC-8): if the host hook converged a DIRTY deploy-base to + # origin/main before the pull, surface it (structured log + best-effort Telegram). + # never-raise — observability must never crash the finalizer. + try: + from . import checkout_hygiene + report = checkout_hygiene.read_report(repo, work_item_id) + if report: + checkout_hygiene.alert_dirty(repo, work_item_id, report) + except Exception as e: # noqa: BLE001 - never break the finalizer + logger.warning("Task %s: checkout-hygiene report read failed: %s", task_id, e) if status == "SUCCESS" and work_item_id: plane_add_comment( work_item_id, diff --git a/tests/test_deploy_checkout_hygiene.py b/tests/test_deploy_checkout_hygiene.py new file mode 100644 index 0000000..53fe253 --- /dev/null +++ b/tests/test_deploy_checkout_hygiene.py @@ -0,0 +1,485 @@ +"""ORCH-112: deploy-base checkout-hygiene (resilient-pull) — TC-01…TC-10. + +Two test layers: + + * SHELL simulation (TC-01..TC-04, TC-07) — drives the REAL + ``scripts/orchestrator-deploy-hook.sh`` in a hermetic sandbox. ``git`` is REAL + (against a LOCAL bare "origin" — no network), while ``docker`` / ``curl`` / + ``sleep`` are PATH-shimmed stubs so no real infra is touched and prod is never + restarted (INFRA safety). Models tests/test_deploy_hook_rollback_sim.py. + + * UNIT (TC-05, TC-06, TC-08, TC-09, TC-10) — the ``checkout_hygiene`` leaf, the + static safety contract of the hook (never ``-x`` / explicit excludes), the + pipeline-invariant guard and the documentation invariant. + +TC-01 is the MANDATORY incident-reproduction regression (ORCH-111): a dirty tracked +edit to ``src/config.py`` over an advanced ``origin/main`` makes the bare ``git pull`` +abort with "local changes would be overwritten by merge" (RED before the fix); the +resilient-pull hygiene converges the base and the deploy proceeds (GREEN after). +""" + +import os +import shutil +import stat +import subprocess + +import pytest + +from src import checkout_hygiene + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +HOOK = os.path.join(ROOT, "scripts", "orchestrator-deploy-hook.sh") + +pytestmark = pytest.mark.skipif( + shutil.which("bash") is None or shutil.which("git") is None, + reason="bash + git required for the deploy-hook hygiene simulation", +) + +# Distinctive file bodies so assertions prove WHICH version won. +_V1 = "ORIGIN-V1\n" +_V2 = "ORIGIN-V2-ADVANCED\n" +_DIRTY = "DIRTY-LOCAL-WIP\n" + +# Isolate git from any host/global config (hermetic). +_GIT_ENV = { + "GIT_AUTHOR_NAME": "t", + "GIT_AUTHOR_EMAIL": "t@example.com", + "GIT_COMMITTER_NAME": "t", + "GIT_COMMITTER_EMAIL": "t@example.com", + "GIT_CONFIG_GLOBAL": os.devnull, + "GIT_CONFIG_SYSTEM": os.devnull, +} + + +def _git(cwd, *args): + r = subprocess.run( + ["git", "-C", str(cwd), *args], + capture_output=True, text=True, env={**os.environ, **_GIT_ENV}, + ) + assert r.returncode == 0, f"git {args} failed: {r.stdout}\n{r.stderr}" + return r + + +def _write(path, content): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(content) + + +def _write_exec(path, content): + _write(path, content) + os.chmod(path, os.stat(path).st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + +def _make_stubs(binx, prev_running=True): + """Healthy docker/curl/sleep stubs (no real infra; deploy always succeeds). + + ``prev_running`` controls whether ``docker compose ps -q`` returns a container id: + True -> step 1 captures a previous image and writes PREV_IMAGE_FILE (the normal + case); False -> no previous image is recorded (first-deploy / service-down), so the + deploy-base stays genuinely clean at the hygiene step (exercises the no-op branch). + """ + ps_id = "fakecid" if prev_running else "" + _write_exec(str(binx / "docker"), f"""#!/bin/bash +case "$1" in + compose) + for a in "$@"; do [ "$a" = "ps" ] && {{ echo "{ps_id}"; exit 0; }}; done + exit 0;; + inspect) echo "sha256:previmage"; exit 0;; + image) exit 0;; + tag) exit 0;; + *) exit 0;; +esac +""") + # curl: ALWAYS healthy -> deploy health-check passes immediately -> exit 0. + _write_exec(str(binx / "curl"), """#!/bin/bash +iscode="" +for a in "$@"; do [ "$a" = "-w" ] && iscode=1; done +[ -n "$iscode" ] && echo "200" || echo '{"status":"ok"}' +exit 0 +""") + _write_exec(str(binx / "sleep"), "#!/bin/bash\nexit 0\n") + + +def _seed_origin_and_clone(tmp_path): + """Build a local bare origin (at V2) + a deploy-base clone (at V1). + + Returns ``repo`` (the deploy-base path). The clone is one commit BEHIND origin so + that, with a conflicting dirty edit to src/config.py, a bare ``git pull`` would + abort (the exact ORCH-111 incident), while hygiene's reset --hard converges it. + """ + work = tmp_path / "work" + work.mkdir() + _write(str(work / "src" / "config.py"), _V1) + _write(str(work / ".gitignore"), ".env\ndata/\n*.db\nbuild/\n") + _git(work, "init", "-q") + _git(work, "add", "-A") + _git(work, "commit", "-q", "-m", "v1") + _git(work, "branch", "-M", "main") + + origin = tmp_path / "origin.git" + _git(tmp_path, "init", "-q", "--bare", str(origin)) + _git(work, "remote", "add", "origin", str(origin)) + _git(work, "push", "-q", "-u", "origin", "main") + + repo = tmp_path / "repo" + _git(tmp_path, "clone", "-q", str(origin), str(repo)) + + # Advance origin/main to V2 (touches the SAME file we will dirty locally). + _write(str(work / "src" / "config.py"), _V2) + _git(work, "commit", "-q", "-am", "v2") + _git(work, "push", "-q", "origin", "main") + # Make repo's remote-tracking ref aware of V2's existence is the hook's job + # (it runs `git fetch`); leave repo at V1 deliberately. + return repo + + +def _run_hook(repo, tmp_path, hygiene="1", extra_env=None, prev_running=True): + """Run the real hook in --deploy mode against ``repo`` with stubbed infra.""" + binx = tmp_path / "bin" + if not binx.exists(): + binx.mkdir() + _make_stubs(binx, prev_running=prev_running) + state = tmp_path / "state" + state.mkdir(exist_ok=True) + env = { + **os.environ, + **_GIT_ENV, + "PATH": f"{binx}:{os.environ['PATH']}", + "REPO": str(repo), + "LOG": str(state / "hook.log"), + "PREV_IMAGE_FILE": str(repo / ".deploy-prev-image-prod"), + "COMPOSE_PROFILE": "", + "TARGET_SERVICE": "orchestrator", + "TARGET_PORT": "8500", + } + if hygiene is not None: + env["CHECKOUT_HYGIENE"] = hygiene + env["HYGIENE_REPORT"] = str(state / "hygiene") + if extra_env: + env.update(extra_env) + return subprocess.run( + ["bash", HOOK, "--deploy"], env=env, capture_output=True, text=True, timeout=60, + ) + + +def _porcelain(repo): + r = subprocess.run( + ["git", "-C", str(repo), "status", "--porcelain"], + capture_output=True, text=True, env={**os.environ, **_GIT_ENV}, + ) + return r.stdout.strip() + + +def _wip_dirt(repo): + """Porcelain output MINUS the intentionally-preserved deploy artefacts. + + After hygiene, the deploy-base is converged to origin/main but the rollback/log + artefacts (.deploy-prev-image-* / deploy-hook.log) are legitimately untracked-and- + preserved (NFR-2). This returns ONLY the *real* residual dirt (a non-empty result + means a tracked edit survived or WIP was not cleaned).""" + lines = [] + for ln in _porcelain(repo).splitlines(): + name = ln[3:] if len(ln) > 3 else "" + if name.startswith(".deploy-prev-image-") or name == "deploy-hook.log": + continue + lines.append(ln) + return "\n".join(lines).strip() + + +def _head_config(repo): + with open(repo / "src" / "config.py", encoding="utf-8") as f: + return f.read() + + +# =========================================================================== +# TC-01 — MANDATORY regression (red->green): dirty tracked edit + advanced origin +# =========================================================================== +def test_tc01_dirty_tracked_edit_converges_and_deploys(tmp_path): + repo = _seed_origin_and_clone(tmp_path) + # Dirty the SAME tracked file origin advanced -> a bare `git pull` would abort. + _write(str(repo / "src" / "config.py"), _DIRTY) + # Untracked WIP left behind too (failed/cancelled task residue). + _write(str(repo / "scripts" / "install_lite.py"), "# wip\n") + + proc = _run_hook(repo, tmp_path, hygiene="1") + + assert proc.returncode == 0, ( + "resilient-pull must converge a dirty base and let the deploy proceed " + f"(stdout={proc.stdout}\nstderr={proc.stderr})" + ) + # Base converged to the ADVANCED origin/main (dirty local edit discarded). + assert _head_config(repo) == _V2 + # No real WIP remains (tracked edit reset, untracked WIP cleaned); the rollback + # snapshot .deploy-prev-image-prod is legitimately preserved (NFR-2), so we check + # the residual dirt MINUS the preserved artefacts. + assert _wip_dirt(repo) == "" + assert not (repo / "scripts" / "install_lite.py").exists() + out = proc.stdout + proc.stderr + assert "HYGIENE" in out + + +def test_tc01b_bare_pull_aborts_without_hygiene_documents_incident(tmp_path): + """ORCH-111 reproduction: WITHOUT hygiene the same dirty base aborts the pull.""" + repo = _seed_origin_and_clone(tmp_path) + _write(str(repo / "src" / "config.py"), _DIRTY) + + proc = _run_hook(repo, tmp_path, hygiene=None) # CHECKOUT_HYGIENE unset + + assert proc.returncode != 0, "bare `git pull` must abort on the conflicting dirty edit" + log = (tmp_path / "state" / "hook.log").read_text(encoding="utf-8") + assert "would be overwritten by merge" in (log + proc.stdout + proc.stderr) + + +# =========================================================================== +# TC-02 — untracked WIP files do not block and do not leak into the deploy +# =========================================================================== +def test_tc02_untracked_wip_does_not_block(tmp_path): + repo = _seed_origin_and_clone(tmp_path) + for rel in ( + "scripts/install_lite.py", + "tests/test_install_lite.py", + "docs/deployment/lite-install.example.yaml", + ): + _write(str(repo / rel), "# abandoned WIP\n") + + proc = _run_hook(repo, tmp_path, hygiene="1") + + assert proc.returncode == 0, f"{proc.stdout}\n{proc.stderr}" + assert _wip_dirt(repo) == "" + for rel in ( + "scripts/install_lite.py", + "tests/test_install_lite.py", + "docs/deployment/lite-install.example.yaml", + ): + assert not (repo / rel).exists(), f"{rel} must be cleaned, not leaked into deploy" + + +# =========================================================================== +# TC-03 — preservation of rollback/log/gitignored/sibling artefacts (NFR-2) +# =========================================================================== +def test_tc03_preserves_rollback_and_sibling_artifacts(tmp_path): + repo = _seed_origin_and_clone(tmp_path) + _write(str(repo / "src" / "config.py"), _DIRTY) # force the hygiene path + + # In-$REPO artefacts that MUST survive (untracked, NOT gitignored). + _write(str(repo / ".deploy-prev-image-staging"), "sha256:stagingprev\n") + _write(str(repo / "deploy-hook.log"), "audit line\n") + # gitignored prod secrets / DB — must survive `git clean -fd` (NO -x). + _write(str(repo / ".env"), "ORCH_SECRET=keepme\n") + _write(str(repo / "data" / "orchestrator.db"), "sqlite-bytes\n") + # .git internal worktree admin record — git clean never touches .git/. + _write(str(repo / ".git" / "worktrees" / "wt1" / "HEAD"), "ref: refs/heads/x\n") + # Sibling state under the PARENT of $REPO — outside the clean scope. + _write(str(tmp_path / ".deploy-state-orchestrator" / "ORCH-112" / "result"), "0\n") + _write(str(tmp_path / ".merge-lease-orchestrator.json"), '{"branch":"x"}\n') + + proc = _run_hook(repo, tmp_path, hygiene="1") + assert proc.returncode == 0, f"{proc.stdout}\n{proc.stderr}" + + # Rollback snapshot freshly written by step 1 (PREV_IMAGE_FILE) survived hygiene. + assert (repo / ".deploy-prev-image-prod").is_file() + assert (repo / ".deploy-prev-image-prod").read_text().strip() != "" + # Wildcard-excluded sibling prev-image + log survived. + assert (repo / ".deploy-prev-image-staging").read_text() == "sha256:stagingprev\n" + assert (repo / "deploy-hook.log").read_text() == "audit line\n" + # gitignored secrets/DB survived (proves NO -x at runtime). + assert (repo / ".env").read_text() == "ORCH_SECRET=keepme\n" + assert (repo / "data" / "orchestrator.db").read_text() == "sqlite-bytes\n" + # .git internal + sibling state untouched. + assert (repo / ".git" / "worktrees" / "wt1" / "HEAD").is_file() + assert (tmp_path / ".deploy-state-orchestrator" / "ORCH-112" / "result").is_file() + assert (tmp_path / ".merge-lease-orchestrator.json").is_file() + + +# =========================================================================== +# TC-04 — happy-path: genuinely clean base -> hygiene no-op + plain fast-forward. +# Uses prev_running=False so step 1 records NO prev-image, leaving the base clean at +# the hygiene step (no untracked artefact) — the no-op `else` branch is exercised and +# the deploy reduces to the plain `git pull` fast-forward (exit-codes byte-for-byte). +# =========================================================================== +def test_tc04_clean_base_fast_forwards_no_op_hygiene(tmp_path): + repo = _seed_origin_and_clone(tmp_path) # repo is CLEAN, just behind origin + + proc = _run_hook(repo, tmp_path, hygiene="1", prev_running=False) + + assert proc.returncode == 0, f"{proc.stdout}\n{proc.stderr}" + log = (tmp_path / "state" / "hook.log").read_text(encoding="utf-8") + assert "deploy-base already clean (no-op)" in log + assert "dirty deploy-base detected" not in log + # Plain fast-forward brought origin/main's V2. + assert _head_config(repo) == _V2 + + +# =========================================================================== +# TC-07 — convergence after cancel/failed: leftovers cleared, next deploy clean +# =========================================================================== +def test_tc07_convergence_then_next_deploy_is_clean(tmp_path): + repo = _seed_origin_and_clone(tmp_path) + # Leftovers from a failed/cancelled task: dirty tracked + untracked WIP. + _write(str(repo / "src" / "config.py"), _DIRTY) + _write(str(repo / "tests" / "test_install_lite.py"), "# wip\n") + + first = _run_hook(repo, tmp_path, hygiene="1") + assert first.returncode == 0, f"{first.stdout}\n{first.stderr}" + assert _wip_dirt(repo) == "" # base converged, no WIP residue + assert _head_config(repo) == _V2 + + # A subsequent self-deploy proceeds without manual intervention (no WIP to block it). + second = _run_hook(repo, tmp_path, hygiene="1") + assert second.returncode == 0, f"{second.stdout}\n{second.stderr}" + assert _wip_dirt(repo) == "" + + +# =========================================================================== +# TC-05 — self-hosting safety + static hook safety contract (never -x / excludes) +# =========================================================================== +def _hook_code_lines(): + """Non-comment, non-blank lines of the hook (so a comment mentioning `-x` or + `exit` for documentation does not trip the static safety asserts).""" + out = [] + for ln in open(HOOK, encoding="utf-8").read().splitlines(): + s = ln.strip() + if not s or s.startswith("#"): + continue + out.append(ln) + return out + + +def test_tc05_hook_clean_is_never_destructive(): + text = open(HOOK, encoding="utf-8").read() + code = "\n".join(_hook_code_lines()) + assert "CHECKOUT_HYGIENE" in text, "hygiene block must exist in the hook" + # INV-HYGIENE-1: the hook's only `git clean` is `-fd`, NEVER `-x` (which would + # delete gitignored .env / data/*.db / build/). Checked against CODE only. + assert "git clean -fd" in code + assert "-x" not in code # no -x / -xfd / -fdx in any executable line + # INV-HYGIENE-2: explicit excludes for the untracked-but-not-ignored artefacts. + assert "-e '.deploy-prev-image-*'" in code + assert "-e 'deploy-hook.log'" in code + # Converge to the authoritative remote, never a local guess. + assert "git reset --hard origin/main" in code + # Self-hosting safety: the hygiene path never pushes/force-pushes the remote. + assert "push --force" not in code and "push -f " not in code + + +def test_tc05_leaf_is_a_pure_leaf(): + """checkout_hygiene must not import stage_engine / launcher at module load.""" + src = open(os.path.join(ROOT, "src", "checkout_hygiene.py"), encoding="utf-8").read() + import_lines = [ + ln for ln in src.splitlines() + if ln.startswith("import ") or ln.startswith("from ") + ] + joined = "\n".join(import_lines) + assert "stage_engine" not in joined + assert "launcher" not in joined + + +# =========================================================================== +# TC-06 — kill-switch + repo scope (applies / hook_env) +# =========================================================================== +def test_tc06_kill_switch_off_is_inert(monkeypatch): + from src.config import settings + monkeypatch.setattr(settings, "checkout_hygiene_enabled", False) + assert checkout_hygiene.applies("orchestrator") is False + assert checkout_hygiene.hook_env("orchestrator", "ORCH-112") == "" + + +def test_tc06_empty_csv_is_self_hosting_only(monkeypatch): + from src.config import settings + monkeypatch.setattr(settings, "checkout_hygiene_enabled", True) + monkeypatch.setattr(settings, "checkout_hygiene_repos", "") + assert checkout_hygiene.applies("orchestrator") is True + assert checkout_hygiene.applies("enduro-trails") is False + env = checkout_hygiene.hook_env("orchestrator", "ORCH-112") + assert env.startswith("CHECKOUT_HYGIENE=1 ") + assert "HYGIENE_REPORT=" in env + # A non-self repo gets no hygiene env (other repos unaffected). + assert checkout_hygiene.hook_env("enduro-trails", "ET-1") == "" + + +def test_tc06_csv_scope_limits_repos(monkeypatch): + from src.config import settings + monkeypatch.setattr(settings, "checkout_hygiene_enabled", True) + monkeypatch.setattr(settings, "checkout_hygiene_repos", "alpha, beta") + assert checkout_hygiene.applies("alpha") is True + assert checkout_hygiene.applies("beta") is True + assert checkout_hygiene.applies("orchestrator") is False + + +# =========================================================================== +# TC-08 — observability: read_report / alert_dirty never-raise +# =========================================================================== +def test_tc08_read_report_none_when_absent(monkeypatch, tmp_path): + from src.config import settings + monkeypatch.setattr(settings, "repos_dir", str(tmp_path)) + assert checkout_hygiene.read_report("orchestrator", "ORCH-112") is None + + +def test_tc08_read_report_parses_dirty_sentinel(monkeypatch, tmp_path): + from src import self_deploy + from src.config import settings + monkeypatch.setattr(settings, "repos_dir", str(tmp_path)) + d = self_deploy.container_state_dir("orchestrator", "ORCH-112") + os.makedirs(d, exist_ok=True) + _write(os.path.join(d, "hygiene"), "dirty=1\n M src/config.py\n?? scripts/x.py\n") + rep = checkout_hygiene.read_report("orchestrator", "ORCH-112") + assert rep == {"dirty": True, "paths": ["M src/config.py", "?? scripts/x.py"]} + + +def test_tc08_alert_dirty_never_raises_on_send_failure(monkeypatch): + import src.notifications as notifications + + def boom(*a, **k): + raise RuntimeError("telegram down") + + monkeypatch.setattr(notifications, "send_telegram", boom) + # Must swallow the error (best-effort) and NOT crash the finalizer. + assert checkout_hygiene.alert_dirty( + "orchestrator", "ORCH-112", {"dirty": True, "paths": ["x"]} + ) is False + # No report / not dirty -> no alert, no raise. + assert checkout_hygiene.alert_dirty("orchestrator", "ORCH-112", None) is False + + +# =========================================================================== +# TC-09 — pipeline invariant: STAGE_TRANSITIONS / QG_CHECKS / exit-codes untouched +# =========================================================================== +def test_tc09_pipeline_contracts_untouched(): + from src import stages + from src.qg import checks + # The hygiene feature is NOT a stage and NOT a QG check. + assert "checkout_hygiene" not in { + k for tr in stages.STAGE_TRANSITIONS.values() for k in (tr if isinstance(tr, dict) else {}) + } + assert not any("hygiene" in name for name in checks.QG_CHECKS) + + +def test_tc09_hook_exit_code_contract_intact(): + text = open(HOOK, encoding="utf-8").read() + # The hook still maps to the 0/1/2 contract (ORCH-036). + assert "exit 0" in text and "exit 1" in text and "exit 2" in text + # The hygiene block itself never emits an `exit` statement (best-effort, + # never-break). Inspect only the CODE lines of the 2a block (a comment that + # mentions "exit-codes" must not trip this). + block = text.split("# 2a.", 1)[1].split("# 2.", 1)[0] + code_lines = [ + ln for ln in block.splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + for ln in code_lines: + assert not ln.strip().startswith("exit"), ( + f"hygiene block must never change the hook exit-code: {ln!r}" + ) + + +# =========================================================================== +# TC-10 — documentation invariant (golden source) +# =========================================================================== +def test_tc10_docs_state_deploy_base_invariant(): + infra = open(os.path.join(ROOT, "docs", "operations", "INFRA.md"), encoding="utf-8").read() + readme = open(os.path.join(ROOT, "docs", "architecture", "README.md"), encoding="utf-8").read() + for doc in (infra, readme): + assert "ORCH-112" in doc + assert "deploy/worktree-management база" in doc + assert "workspace" in doc