From 90704899688689d6069d367157bd9f8d19467c1d Mon Sep 17 00:00:00 2001 From: claude-bot Date: Sun, 7 Jun 2026 12:39:00 +0000 Subject: [PATCH] fix(staging): tolerate sandbox-infra-only FAILs (C9a/C9b) in deploy-staging verdict MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The self-hosting orchestrator looped on deploy-staging -> development because scripts/staging_check.py exited 1 on ANY failed check, so two infra-only checks (C9a sandbox branch / C9b analyst-job — caused by SANDBOX bot accounts not being members of the sandbox Plane project, NOT a pipeline regress) forced staging_status: FAILED -> rollback -> loop, burning developer retries and tokens. Direction (б) per ADR-001: classify staging checks as REAL (all pipeline checks, fail-closed) vs SANDBOX_INFRA (narrow allowlist {C9a, C9b}, waivable). New leaf module src/staging_verdict.py (stdlib-only, never-raise): classify_check + compute_staging_verdict fold per-check results into a tolerant-but-fail-closed verdict — any REAL failure -> FAILED/exit1 (safety net holds under any flag); only C9a/C9b failed & tolerant -> SUCCESS/exit0 with waived list; only infra & strict -> FAILED/exit1; any internal error -> FAILED/exit1 (never a false green). staging_check.py now auto-classifies each check (public 3-tuple _items shape kept as an ORCH-048 b6 regression guard), exposes categorized_items(), prints INFRA-WAIVED/VERDICT lines, and exits via the verdict; new --strict flag forces legacy strictness per-run. Kill-switch ORCH_STAGING_INFRA_TOLERANCE_ENABLED (default true) restores legacy strict mode globally. launcher gains action_stage_no_changes_note so "no changes to commit" on action stages is logged as expected, not treated as under-delivery. Contracts unchanged: STAGE_TRANSITIONS, QG_CHECKS registry, staging_status:/ deploy_status: frontmatter, hook exit-code (0/1/2), check_staging_status; no DB migration. Docs: README, STAGING_CHECK.md, deployer.md, .env.example, CHANGELOG. Refs: ORCH-061 Co-Authored-By: Claude Opus 4.7 --- .env.example | 10 ++ .openclaw/agents/deployer.md | 15 ++- CHANGELOG.md | 1 + docs/architecture/README.md | 2 +- docs/operations/STAGING_CHECK.md | 54 +++++++++- scripts/staging_check.py | 116 ++++++++++++++++++++- src/agents/launcher.py | 43 ++++++++ src/config.py | 16 +++ src/staging_verdict.py | 173 +++++++++++++++++++++++++++++++ tests/test_config.py | 23 ++++ tests/test_launcher.py | 45 ++++++++ tests/test_qg.py | 21 ++++ tests/test_qg_checks.py | 97 +++++++++++++++++ tests/test_stage_engine.py | 155 +++++++++++++++++++++++++++ tests/test_staging_check_b6.py | 67 ++++++++++++ 15 files changed, 831 insertions(+), 7 deletions(-) create mode 100644 src/staging_verdict.py diff --git a/.env.example b/.env.example index 40dd85f..eb9fbfa 100644 --- a/.env.example +++ b/.env.example @@ -85,6 +85,16 @@ ORCH_DEPLOY_PROD_PREV_IMAGE_FILE=.deploy-prev-image-prod ORCH_IMAGE_FRESHNESS_ENABLED=true ORCH_IMAGE_FRESHNESS_REPOS= +# ORCH-061: staging-verdict tolerance to sandbox-infra-only FAILs. The self-hosting +# orchestrator looped on deploy-staging because staging_check.py exited 1 on ANY FAIL, +# so two infra-only checks (C9a sandbox branch / C9b analyst-job — caused by SANDBOX +# bot accounts not being members of the sandbox Plane project, NOT a pipeline regress) +# forced staging_status: FAILED -> rollback -> loop. With this ON, C9a/C9b are WAIVED +# to SUCCESS when every REAL check is green; any REAL failure still fails closed. +# true (default) -> tolerant; false -> legacy strict (1:1 pre-ORCH-061, any FAIL rolls back). +# Lives in .env.staging (the staging instance). CLI --strict overrides this per-run. +ORCH_STAGING_INFRA_TOLERANCE_ENABLED=true + # ORCH-053: stuck-task reconciler (sweeper for lost webhooks). A background daemon # replays a missed stage transition through the SAME gates/handlers a webhook would, # fixing tasks that got stuck on a dropped event (502 on rebuild, no Plane/Gitea diff --git a/.openclaw/agents/deployer.md b/.openclaw/agents/deployer.md index 6126307..3e74082 100644 --- a/.openclaw/agents/deployer.md +++ b/.openclaw/agents/deployer.md @@ -37,8 +37,19 @@ On stage `deploy-staging` your job is to run the staging test suite and write a not exist. Details: `docs/operations/STAGING_CHECK.md`. 2. Check the exit code: - - Exit code **0** = all tests PASS → `staging_status: SUCCESS` - - Exit code **non-zero** = tests FAILED → `staging_status: FAILED` + - Exit code **0** = advance → `staging_status: SUCCESS` + - Exit code **non-zero** = rollback → `staging_status: FAILED` + + > **ORCH-061**: exit 0 may now include *waived* sandbox-infra failures. The two + > infra-only checks **C9a/C9b** (sandbox branch / analyst-job, which depend on + > SANDBOX bot accounts being project members — not on the pipeline) are tolerated + > when every REAL check is green; the script prints an `INFRA-WAIVED:` line and a + > `VERDICT:` line, and still exits 0. Any REAL check failing still yields exit 1 + > (fail-closed). If you see `INFRA-WAIVED:` in the output, copy that line into the + > `15-staging-log.md` body for observability. The exit-code → `staging_status` + > mapping above is unchanged: trust the exit code, do NOT re-judge waived checks. + > Kill-switch: `ORCH_STAGING_INFRA_TOLERANCE_ENABLED=false` (or `--strict`) restores + > legacy strictness. Details: `docs/operations/STAGING_CHECK.md`. 3. Write the verdict to `docs/work-items//15-staging-log.md` with YAML frontmatter: ```markdown diff --git a/CHANGELOG.md b/CHANGELOG.md index a3670e8..655c084 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,7 @@ - Цепочка стадий: `... testing → deploy-staging → deploy → done` (была без `deploy-staging`). ### Fixed +- **`deploy-staging` больше не зацикливается на infra-only FAIL песочницы (C9a/C9b)** (ORCH-061): self-hosting `orchestrator` крутился в петле `deploy-staging → development` — `scripts/staging_check.py` давал `exit 1` при ЛЮБОМ упавшем чеке, поэтому две чисто инфраструктурные проверки **C9a** (ветка не появилась в `orchestrator-sandbox`) и **C9b** (job аналитика не встал в очередь staging) — вызванные тем, что SANDBOX-бот-аккаунты не состоят в sandbox-проекте Plane (шаги 6+ конвейера в песочнице недостижимы, это НЕ регресс конвейера) — приводили к `staging_status: FAILED` → откат → цикл (выжигание developer-ретраев, токенов, паразитная нагрузка общего инстанса). Решение (Direction «б», ADR-001): чеки классифицируются на `REAL` (все проверки конвейера A*/B*/C7/C8 — fail-closed) и `SANDBOX_INFRA` (строго allowlist `{C9a, C9b}` — waivable). Новый leaf-модуль `src/staging_verdict.py` (stdlib-only, контракт «never raise», по образцу `merge_gate`/`image_freshness`): `classify_check(label)` (allowlist по ведущему токену, всё неизвестное/малформенное → `REAL` fail-closed) и `compute_staging_verdict(items, infra_tolerant) -> StagingVerdict`: любой REAL-FAIL → `FAILED`/exit 1 (страховка при ЛЮБОМ значении флага); упали ТОЛЬКО C9a/C9b и толерантность включена → `SUCCESS`/exit 0 + упавшие метки в `waived` (наблюдаемость); только C9a/C9b и толерантность выключена → `FAILED`/exit 1 (legacy-строгий); любая внутренняя ошибка вердикта → `FAILED`/exit 1 (никогда не ложный green). `scripts/staging_check.py`: `Results` авто-классифицирует каждый чек (публичная 3-tuple форма `_items` сохранена — регрессия-гард ORCH-048 b6), `categorized_items()` отдаёт категорию, `summary()` печатает разбивку REAL/SANDBOX_INFRA; `main()` сворачивает прогон через `_verdict(...)`, печатает строки `INFRA-WAIVED:`/`VERDICT:` и делает `sys.exit(verdict.exit_code)`; новый флаг `--strict` форсит строгий режим для одного запуска. Глобальный kill-switch `ORCH_STAGING_INFRA_TOLERANCE_ENABLED` (`Settings.staging_infra_tolerance_enabled`, default `true`; `false` → строгий 1:1 до ORCH-061), живёт в `.env.staging`; `--strict` имеет приоритет над env. Наблюдаемость на стороне конвейера: `src/agents/launcher.py` получил `action_stage_no_changes_note(stage, repo)` — на action-стадиях (`deploy-staging`/`deploy`) self-hosting-репо «нет изменений для коммита» логируется как ожидаемое, а не трактуется как недопоставка. Контракты НЕ менялись: `STAGE_TRANSITIONS`, реестр `QG_CHECKS`, frontmatter `staging_status: SUCCESS|FAILED` / `deploy_status:` (толерантность применяется в скрипте ДО записи артефакта деплоером), exit-code-контракт хука (0/1/2), `check_staging_status`/`_parse_staging_status`; схема БД — без миграций. ADR `docs/work-items/ORCH-061/06-adr/ADR-001-staging-infra-tolerance.md`. Документация: `docs/architecture/README.md`, `docs/operations/STAGING_CHECK.md`, `.openclaw/agents/deployer.md`. Тесты: `tests/test_staging_check_b6.py`, `tests/test_qg_checks.py`, `tests/test_config.py`, `tests/test_launcher.py`, `tests/test_qg.py`, `tests/test_stage_engine.py::TestStagingInfraTolerance`. - **Reconciler (F-1) больше не разблокирует escalated / Blocked / Needs-Input задачи** (ORCH-060): sweeper потерянных webhook (ORCH-053) не отличал «застряла из-за потерянного события» от «исчерпала лимит developer-ретраев и ждёт человека» — если CI зелёный, а reviewer слал REQUEST_CHANGES до `MAX_DEVELOPER_RETRIES`, каждый тик F-1 видел зелёный `check_ci_green` и доигрывал `development → review` → reviewer снова REQUEST_CHANGES → откат (стадия не меняется, escalated в `gitea.py` лишь шлёт `notify_error`) → следующий тик снова разблокировал. Бесконечная петля (инцидент ET-013: 10 разблокировок за ночь, лишние запуски агентов/токены, спам в Telegram, паразитная нагрузка общего self-hosting-инстанса). В `Reconciler._reconcile_gate_task` (`src/reconciler.py`) ПОСЛЕ существующих гардов (`analysis` carve-out, нет гейта, активный job, grace) и ДО пред-оценки гейта добавлены два пред-гарда с ранним `return` (молчаливый skip — без `advance`, без инкремента `unblocked_total`, без нотификаций): **Guard 1 (escalated, детерминированный, без сети, проверяется первым)** — `developer_retry_count(task_id) >= MAX_DEVELOPER_RETRIES`; приватный `stage_engine._developer_retry_count` повышен до публичного `developer_retry_count` (единый источник истины по подсчёту ретраев `agent_runs`, приватное имя сохранено как алиас), граница берётся из `stage_engine.MAX_DEVELOPER_RETRIES` (не хардкод `3`). **Guard 2 (явный человеческий Plane-статус, Вариант A — без миграции БД)** — новый never-raise хелпер `plane_sync.fetch_issue_state(issue_id, project_id) -> str|None` (тот же endpoint/headers, что `fetch_issue_sequence_id`) + `Reconciler._is_blocked_or_needs_input(task)`: резолв проекта (`projects.get_project_by_repo`) → `get_project_states(pid)` → сверка текущего state issue с `blocked`/`needs_input`; любая ошибка/`None`/нерезолвленный проект → консервативный skip (`True`: не-разблокировать безопаснее). F-2 по существу не менялся: Blocked/Needs Input не входят в опрашиваемый набор `{in_progress, approved, rejected}` → не доигрываются (зафиксировано регресс-тестом). Новый под-флаг `ORCH_RECONCILE_SKIP_BLOCKED_ENABLED` (true) гасит ТОЛЬКО сетевой Guard 2 (escape hatch при Plane-outage); Guard 1 всегда активен. Схема БД, `STAGE_TRANSITIONS`, `QG_CHECKS`, never-raise на единицу работы, `analysis` carve-out и kill-switch'и (`reconcile_enabled`/`reconcile_plane_enabled`) не менялись. ADR `docs/work-items/ORCH-060/06-adr/ADR-001-reconciler-skip-escalated.md`. Тесты: `tests/test_reconciler.py` (TC-01…TC-11 + регресс ORCH-053). - **Re-deploy после отката больше не зависает на `deploy`; `.env.example` дополнен** (ORCH-036, review-fix): sentinel-маркеры самодеплоя (`approve-requested`/`initiated`/`result`) ключуются по стабильному `work_item_id`, поэтому при FAILED-деплое и откате БАГ-8 (`deploy → development`) они оставались на диске — после фикса developer-ом и повторного захода задачи на `deploy` Фаза B по idempotency-guard видела STALE `initiated` и становилась no-op: detached-хук не перезапускался, finalizer не ставился, задача висела на `deploy` навсегда (нарушался retry-контракт стадии, AC-4/AC-10; устаревший `result` к тому же был бы перечитан новым finalizer'ом). Добавлен `self_deploy.clear_state(repo, work_item_id)` (never-raise, idempotent, рекурсивное удаление `/.deploy-state-//`), вызывается в ветке БАГ-8-отката `check_deploy_status` FAILED (`src/stage_engine.py`) и дополнительно в начале Фазы A (`_handle_self_deploy_phase_a`) — каждый новый прод-деплой-проход стартует с чистого состояния. Отдельно: канонический `.env.example` (CLAUDE.md правило №8, ТЗ §2.6) дополнен полным блоком новых дескрипторов `ORCH_SELF_DEPLOY_*` / `ORCH_DEPLOY_*` (плейсхолдеры, секреты не коммитятся) по образцу merge-gate ORCH-043. Контракты `STAGE_TRANSITIONS` / `QG_CHECKS` / `_parse_deploy_status` / БАГ-8 / merge-gate не тронуты. Тесты: `tests/test_deploy_rollback.py::test_tc11_re_deploy_after_rollback_not_wedged`, `tests/test_deploy_hook_mapping.py::test_clear_state_removes_all_markers_and_is_idempotent`. - **Контейнер и агенты бегут под uid хоста (1000:1000), не root** (ORCH-040): оба сервиса в `docker-compose.yml` (`orchestrator`, `orchestrator-staging`) получили `user: "1000:1000"` (slin) — устраняет корень проблемы, при которой Claude-CLI агенты, запускаемые через `subprocess.Popen` внутри root-контейнера, создавали все артефакты конвейера (git worktree `/repos/_wt/...`, коммиты в `docs/work-items/...`) с владельцем `root:root` на хосте, из-за чего `git pull`/`git reset` под slin падали с `insufficient permission for adding an object` и каждый деплой требовал ручного `chown`. Теперь файлы сразу `slin:slin`. Доступ к docker.sock сохранён через `group_add: ["999"]` (МИНА 1 — НЕ удалена). SSH-маунт приведён к единому HOME агента: target `/root/.ssh` → `/home/slin/.ssh` (`/home/slin/.orchestrator-ssh:/home/slin/.ssh:ro`), синхронно с `HOME=/home/slin`, который launcher форсит в env Popen и git_env — устранён скрытый рассинхрон SSH-маунта с форсимым HOME. `src/agents/launcher.py` и `Dockerfile` НЕ менялись (numeric uid работает без записи в `/etc/passwd`; `safe.directory '*'` уже покрывает git над bind-mount). Требует host-prerequisites Owner (P-1…P-4, вне кода): блокер P-1 — `chown -R 1000:1000 /home/slin/.claude` для доступа uid 1000 к claude creds (иначе preflight заворачивает конвейер); прод-рестарт self — только в окно тишины (общий инстанс с enduro-trails), страховка — staging-гейт (adr-0003). ADR `docs/work-items/ORCH-040/06-adr/ADR-001-run-agents-as-host-uid.md`, глобальный `docs/architecture/adr/adr-0005-container-runs-as-host-uid.md`; INFRA.md обновлён (рантайм-uid, volumes/SSH target, host-prerequisites). Тесты: `tests/test_orch040_compose.py`. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index c578e55..c8b2de5 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -211,4 +211,4 @@ never-raise на единицу работы; тишина при синхрон Схема БД, потоки данных, resilience-слой, детали Dockerfile — [internals.md](internals.md). --- -*Актуально на 2026-06-07. Обновлять при изменении src/stages.py, src/qg/checks.py, src/main.py. Статусы доработок: ORCH-036 (исполняемый самодеплой `deploy`, adr-0007) — реализовано; ORCH-043 (merge-gate, adr-0006) — design, ветка feature/ORCH-043; ORCH-053 (reconciler, adr-0007, src/reconciler.py) — реализовано; ORCH-060 (F-1 skip escalated/Blocked/Needs-Input, `docs/work-items/ORCH-060/06-adr/ADR-001`) — реализовано в ветке feature/ORCH-060 (Guard 1 `developer_retry_count>=MAX_DEVELOPER_RETRIES` + Guard 2 `plane_sync.fetch_issue_state` Blocked/Needs-Input, флаг `ORCH_RECONCILE_SKIP_BLOCKED_ENABLED`); ORCH-058 (провенанс staging-образа: check_staging_image_fresh + staging_check свежего образа + хук-guard, adr-0008) — реализовано в ветке feature/ORCH-058 (обновлять также при изменении src/image_freshness.py, scripts/orchestrator-deploy-hook.sh, Dockerfile); ORCH-061 (толерантность staging-вердикта к инфра-FAIL C9a/C9b, adr-0009, `docs/work-items/ORCH-061/06-adr/ADR-001`) — design, ветка feature/ORCH-061 (обновлять также при изменении src/staging_verdict.py, scripts/staging_check.py, флаг staging_infra_tolerance_enabled).* +*Актуально на 2026-06-07. Обновлять при изменении src/stages.py, src/qg/checks.py, src/main.py. Статусы доработок: ORCH-036 (исполняемый самодеплой `deploy`, adr-0007) — реализовано; ORCH-043 (merge-gate, adr-0006) — design, ветка feature/ORCH-043; ORCH-053 (reconciler, adr-0007, src/reconciler.py) — реализовано; ORCH-060 (F-1 skip escalated/Blocked/Needs-Input, `docs/work-items/ORCH-060/06-adr/ADR-001`) — реализовано в ветке feature/ORCH-060 (Guard 1 `developer_retry_count>=MAX_DEVELOPER_RETRIES` + Guard 2 `plane_sync.fetch_issue_state` Blocked/Needs-Input, флаг `ORCH_RECONCILE_SKIP_BLOCKED_ENABLED`); ORCH-058 (провенанс staging-образа: check_staging_image_fresh + staging_check свежего образа + хук-guard, adr-0008) — реализовано в ветке feature/ORCH-058 (обновлять также при изменении src/image_freshness.py, scripts/orchestrator-deploy-hook.sh, Dockerfile); ORCH-061 (толерантность staging-вердикта к инфра-FAIL C9a/C9b, adr-0009, `docs/work-items/ORCH-061/06-adr/ADR-001`) — реализовано в ветке feature/ORCH-061 (обновлять также при изменении src/staging_verdict.py, scripts/staging_check.py, флаг staging_infra_tolerance_enabled).* diff --git a/docs/operations/STAGING_CHECK.md b/docs/operations/STAGING_CHECK.md index f3e275e..fd4920e 100644 --- a/docs/operations/STAGING_CHECK.md +++ b/docs/operations/STAGING_CHECK.md @@ -12,7 +12,9 @@ | B | ACCESS | Plane sandbox (R), Gitea sandbox (R+push), реестр проектов | | C | E2E | Создать задачу → триггер конвейера → ветка + коммент → cleanup | -Exit code: **0** = все PASS, **non-zero** = есть FAIL. +Exit code: **0** = advance (все REAL-проверки PASS), **1** = rollback (есть REAL-FAIL). +С ORCH-061 exit 0 может включать *waived* sandbox-infra FAIL (C9a/C9b) — см. +[«Толерантность к sandbox-infra (ORCH-061)»](#толерантность-к-sandbox-infra-orch-061). --- @@ -85,6 +87,56 @@ B6 «Registry: sandbox present, prod ET/ORCH absent» подтверждает --- +## Толерантность к sandbox-infra (ORCH-061) + +**Проблема.** Self-hosting `orchestrator` зацикливался на `deploy-staging → development`: +прежде скрипт давал exit 1 при **любом** FAIL, поэтому две чисто инфраструктурные +проверки — **C9a** (ветка не появилась в `orchestrator-sandbox`) и **C9b** (job +аналитика не встал в очередь staging) — приводили к `staging_status: FAILED` → +откат → цикл. Корень: SANDBOX-бот-аккаунты не состоят в sandbox-проекте Plane, +поэтому шаги 6+ конвейера в песочнице недостижимы. Это **не** регресс конвейера. + +**Решение.** Проверки классифицируются на две категории (`src/staging_verdict.py`): + +| Категория | Что входит | Поведение | +|-----------|-----------|-----------| +| `REAL` | все проверки конвейера (A*, B*, C7, C8) | **fail-closed** — любой FAIL = rollback | +| `SANDBOX_INFRA` | строго allowlist `{C9a, C9b}` | **waivable** — FAIL терпится, если все REAL зелёные | + +Вердикт сворачивается в `compute_staging_verdict(items, infra_tolerant)`: + +- любой REAL-FAIL → `FAILED` / exit 1 (страховка сохраняется при ЛЮБОМ значении флага); +- упали **только** C9a/C9b и толерантность включена → `SUCCESS` / exit 0, + упавшие метки попадают в `waived` (наблюдаемость, печатается строкой `INFRA-WAIVED:`); +- упали только C9a/C9b, толерантность выключена → `FAILED` / exit 1 (legacy-строгий); +- любая внутренняя ошибка вердикта → `FAILED` / exit 1 (никогда не ложный green). + +Blast-radius waiver-а ровно две allowlist-метки; всё неизвестное классифицируется +как `REAL` (fail-closed). + +### Kill-switch и `--strict` + +| Управление | Эффект | +|-----------|--------| +| env `ORCH_STAGING_INFRA_TOLERANCE_ENABLED` (default `true`) | глобальный флаг; `false` → строгий режим (1:1 до ORCH-061) | +| CLI `--strict` | форсит строгий режим для одного запуска, игнорируя env | + +Флаг живёт в `.env.staging` (staging-инстанс). `--strict` имеет приоритет над env. + +### Что печатает скрипт + +В конце прогона `summary()` показывает разбивку REAL/SANDBOX_INFRA, затем: + +``` +INFRA-WAIVED: C9a Branch appears in orchestrator-sandbox; C9b Analyst job enqueued ... +VERDICT: SUCCESS (infra-waived): ['C9a …', 'C9b …'] are known sandbox-infra checks; all real checks green +``` + +Контракт `staging_status: SUCCESS|FAILED` во frontmatter **не меняется** — +толерантность применяется в скрипте ДО записи артефакта деплоером. + +--- + ## Режимы (`--mode`) | Режим | Описание | Скорость | diff --git a/scripts/staging_check.py b/scripts/staging_check.py index 75ba892..6cbac67 100644 --- a/scripts/staging_check.py +++ b/scripts/staging_check.py @@ -51,6 +51,46 @@ import datetime import urllib.request import urllib.error import urllib.parse +from collections import namedtuple + +# --------------------------------------------------------------------------- +# ORCH-061: pure staging-verdict logic (classification + infra-tolerant verdict). +# Imported from src.staging_verdict — a stdlib-only leaf, safe to import inside +# the orchestrator-staging container (PYTHONPATH=/app, pattern B6 / ORCH-048). +# Guarded so the suite still runs (in strict mode) if src is somehow unimportable +# from a host invocation; the fallback NEVER yields a silent green (fail-closed). +# --------------------------------------------------------------------------- +try: + from src.staging_verdict import ( # type: ignore + classify_check as _classify_check, + compute_staging_verdict as _compute_staging_verdict, + REAL as _REAL, + SANDBOX_INFRA as _SANDBOX_INFRA, + ) +except Exception: # pragma: no cover - exercised only on a broken host import + _classify_check = None + _compute_staging_verdict = None + _REAL = "real" + _SANDBOX_INFRA = "sandbox_infra" + +_FallbackVerdict = namedtuple("StagingVerdict", "status exit_code waived summary") + + +def _classify(label: str) -> str: + """Classify a check label via staging_verdict; fail-closed to REAL if absent.""" + if _classify_check is not None: + return _classify_check(label) + return _REAL + + +def _verdict(items, infra_tolerant: bool): + """Compute the suite verdict via staging_verdict; strict fail-closed fallback.""" + if _compute_staging_verdict is not None: + return _compute_staging_verdict(items, infra_tolerant) + failed = [lbl for (lbl, ok, _cat) in items if not ok] + if failed: + return _FallbackVerdict("FAILED", 1, [], f"FAILED (strict fallback): {failed}") + return _FallbackVerdict("SUCCESS", 0, [], "SUCCESS (strict fallback): all green") # --------------------------------------------------------------------------- # Colour helpers @@ -152,23 +192,47 @@ def _sign_payload(secret: str, body: bytes) -> str: class Results: def __init__(self): + # _items keeps the (label, passed, detail) 3-tuple shape that existing + # ORCH-048 B6 tests unpack — categories live in a PARALLEL list so the + # public tuple contract is unchanged. self._items: list[tuple[str, bool, str]] = [] # (label, passed, detail) + self._categories: list[str] = [] # ORCH-061: REAL | SANDBOX_INFRA - def add(self, label: str, passed: bool, detail: str = ""): + def add(self, label: str, passed: bool, detail: str = "", category: str | None = None): + # ORCH-061: every check carries a category. None -> auto-classify by label + # (C9a/C9b -> SANDBOX_INFRA, everything else -> REAL). Fail-closed: an + # unknown label is REAL, so it still counts toward the safety net. + if category is None: + category = _classify(label) self._items.append((label, passed, detail)) + self._categories.append(category) line = _ok(label) if passed else _fail(label) if detail: line += f" [{detail}]" print(line) + def categorized_items(self) -> list[tuple[str, bool, str]]: + """Rows as ``(label, passed, category)`` for ``compute_staging_verdict``.""" + return [ + (label, passed, cat) + for (label, passed, _detail), cat in zip(self._items, self._categories) + ] + def summary(self) -> bool: passed = sum(1 for _, ok, _ in self._items if ok) total = len(self._items) all_ok = passed == total colour = _GREEN if all_ok else _RED + # ORCH-061: per-category breakdown so an operator can tell a REAL failure + # (regression — fail-closed) from a SANDBOX_INFRA one (waivable). + rows = self.categorized_items() + real_fail = [lbl for lbl, ok, cat in rows if not ok and cat == _REAL] + infra_fail = [lbl for lbl, ok, cat in rows if not ok and cat == _SANDBOX_INFRA] print() print(f"{_BOLD}{'='*60}{_RESET}") print(f"{colour}{_BOLD} RESULT: {passed}/{total} checks PASS{_RESET}") + print(f" REAL failed : {real_fail or 'none'}") + print(f" SANDBOX_INFRA failed: {infra_fail or 'none'}") print(f"{_BOLD}{'='*60}{_RESET}") return all_ok @@ -637,6 +701,28 @@ def _cleanup(plane_base, workspace, gitea_base, plane_headers, gitea_headers, # Main # --------------------------------------------------------------------------- +def _resolve_tolerance(cli_strict: bool) -> bool: + """Resolve whether the infra-FAIL waiver is active (ORCH-061). + + Precedence: an explicit ``--strict`` CLI flag forces it OFF (for honest manual + runs). Otherwise read ``settings.staging_infra_tolerance_enabled`` from the + running instance's own config (same pattern as B6's src.* import inside the + container). On ANY import/read error -> STRICT (False): we never waive when the + config is unreadable (fail-safe), and we say so. + """ + if cli_strict: + print(_info("tolerance: DISABLED via --strict (honest run)")) + return False + try: + from src.config import settings # noqa: WPS433 - lazy, mirrors B6 + enabled = bool(settings.staging_infra_tolerance_enabled) + print(_info(f"tolerance: staging_infra_tolerance_enabled={enabled}")) + return enabled + except Exception as e: + print(_info(f"tolerance: config unavailable, defaulting to STRICT: {e}")) + return False + + def main(): parser = argparse.ArgumentParser( description="Live staging-stand check suite (ORCH-33)" @@ -656,6 +742,15 @@ def main(): "full-real: also wait for the analyst agent (slow, costs credits)." ), ) + parser.add_argument( + "--strict", + action="store_true", + help=( + "ORCH-061: force strict suite — disable the sandbox-infra (C9a/C9b) " + "FAIL waiver even if staging_infra_tolerance_enabled=True. Use for an " + "honest 10/10 run once the sandbox bot accounts are provisioned." + ), + ) args = parser.parse_args() base = args.base_url.rstrip("/") @@ -673,8 +768,23 @@ def main(): block_b(results) block_c(base, results, args.mode) - all_ok = results.summary() - sys.exit(0 if all_ok else 1) + results.summary() + + # ORCH-061: the EXIT CODE (which drives the deployer's staging_status verdict) + # comes from the infra-tolerant verdict, NOT a raw passed==total count. A run + # whose only failures are known sandbox-infra checks (C9a/C9b) is waived to + # exit 0 when tolerance is on; ANY real check failure still exits 1 (FR-4). + infra_tolerant = _resolve_tolerance(args.strict) + verdict = _verdict(results.categorized_items(), infra_tolerant) + if verdict.waived: + # FR-7 observability: make "green with an allowance" distinguishable from + # an honest green in the logs / captured deployer output. + print(f"{_YELLOW}{_BOLD}INFRA-WAIVED:{_RESET} " + f"{', '.join(verdict.waived)} " + f"(known sandbox-infra; real checks green)") + print(f"{_BOLD}VERDICT:{_RESET} {verdict.status} " + f"(exit {verdict.exit_code}) — {verdict.summary}") + sys.exit(verdict.exit_code) if __name__ == "__main__": diff --git a/src/agents/launcher.py b/src/agents/launcher.py index 9d7598b..31454ef 100644 --- a/src/agents/launcher.py +++ b/src/agents/launcher.py @@ -20,6 +20,33 @@ logger = logging.getLogger("orchestrator.launcher") # never passed through to the CLI. VALID_EFFORTS = frozenset({"low", "medium", "high", "xhigh", "max"}) +# ORCH-061: action stages whose success is an ACTION (restart/retag), not a src +# edit — so "no changes to commit" is EXPECTED there, not under-delivery (FR-3). +_ACTION_STAGES = frozenset({"deploy-staging", "deploy"}) + + +def action_stage_no_changes_note(stage, repo) -> str | None: + """ORCH-061 (FR-3 / FR-7): observability for an empty diff on an action stage. + + The ``deploy-staging`` / ``deploy`` stages are actions (restart / retag), not + code edits, so the post-run "no changes to commit" is the NORMAL case there — + advancement is decided by the agent exit-code + the staging/deploy gate verdict, + NEVER by the presence of a commit (FR-3 / AC-4). This is a PURE decision used + only to emit an explicit log line distinguishing an expected action-stage no-op + from a code-stage no-op; it has no effect on stage advancement. + + Returns an explicit note string when the empty diff is expected (an action + stage of a self-deploy repo), else ``None``. Never raises. + """ + try: + if stage in _ACTION_STAGES: + from ..self_deploy import self_deploy_applies + if self_deploy_applies(repo): + return f"{stage}: no code changes (expected on action stage)" + return None + except Exception: # noqa: BLE001 - observability only, never raise + return None + def _resolve_agent_attr(agent, project_id, project_map_attr, env_attr_prefix, default_attr): @@ -582,6 +609,22 @@ class AgentLauncher: logger.warning(f"Agent run_id={run_id}: commit failed: {commit_result.stderr}") else: logger.info(f"Agent run_id={run_id}: no changes to commit") + # ORCH-061: on a self-deploy action stage (deploy-staging/deploy) + # an empty diff is EXPECTED (action, not a src edit). Emit an + # explicit observability line so an operator can tell this apart + # from a code-stage no-op. Does NOT affect advancement (decided by + # exit-code + gate verdict, never by a commit existing). + try: + _t = get_task_by_repo_branch(repo, branch) + _stage = _t["stage"] if _t else None + _note = action_stage_no_changes_note(_stage, repo) + if _note: + logger.info(f"Agent run_id={run_id}: {_note}") + except Exception as _e: + logger.debug( + f"Agent run_id={run_id}: action-stage no-changes note " + f"skipped: {_e}" + ) except Exception as e: logger.error(f"Agent run_id={run_id}: post-run git failed: {e}") diff --git a/src/config.py b/src/config.py index c2781b2..1a0612b 100644 --- a/src/config.py +++ b/src/config.py @@ -219,6 +219,22 @@ class Settings(BaseSettings): image_freshness_enabled: bool = True image_freshness_repos: str = "" + # ORCH-061: tolerate KNOWN sandbox-infra FAILs (C9a/C9b) in the staging suite. + # The self-hosting deploy-staging stage looped because scripts/staging_check.py + # exited non-zero on ANY failed check, so two infra-only failures (sandbox bot + # accounts not members of the sandbox Plane project) produced staging_status: + # FAILED -> rollback deploy-staging -> development -> loop. + # True -> a run whose ONLY failures are allowlisted sandbox-infra checks + # (C9a/C9b) is waived to SUCCESS; ANY real pipeline check that fails + # still fails closed -> FAILED -> rollback (safety net intact, FR-4). + # False -> 1:1 pre-ORCH-061 strict behaviour: any FAIL -> FAILED -> rollback. + # Default True (mirrors merge_gate_enabled / image_freshness_enabled / + # self_deploy_enabled): the safety net holds regardless of the flag; the flag + # exists to instantly restore legacy strictness without a code redeploy. Lives + # in .env.staging (ORCH_ prefix) so it is reachable inside orchestrator-staging. + # Env ORCH_STAGING_INFRA_TOLERANCE_ENABLED. + staging_infra_tolerance_enabled: bool = True + # ORCH-053: stuck-task reconciler (sweeper for lost webhooks). A background # daemon thread reconciles the "source of truth (gate / Plane) != task stage" # drift left behind by a dropped webhook (502 on rebuild, no Plane/Gitea diff --git a/src/staging_verdict.py b/src/staging_verdict.py new file mode 100644 index 0000000..1766248 --- /dev/null +++ b/src/staging_verdict.py @@ -0,0 +1,173 @@ +"""ORCH-061: pure staging-verdict logic (classification + tolerant verdict). + +The self-hosting ``orchestrator`` looped on ``deploy-staging`` because +``scripts/staging_check.py`` summed ``all_ok = passed == total`` and exited +non-zero on ANY failed check — so two *infrastructure-only* failures (C9a branch +not found / C9b analyst-job not in queue, both caused by the SANDBOX bot accounts +not being members of the sandbox Plane project) produced ``staging_status: +FAILED`` → rollback ``deploy-staging → development`` → loop (ADR-001 §Context). + +This module isolates the **pure verdict logic** so both outcomes are unit-testable +without a live staging stand or docker (TRZ §9): + + * ``classify_check(label)`` — label → ``REAL`` | ``SANDBOX_INFRA`` (narrow, + allowlist-driven, fail-closed to ``REAL`` on anything unrecognised); + * ``compute_staging_verdict(items, infra_tolerant)`` — fold the per-check + pass/fail + category into a single ``StagingVerdict``. + +It is a **leaf**: stdlib only, no I/O, no project imports — so it is safe to import +both from the orchestrator process and from ``scripts/staging_check.py`` (which +runs inside the ``orchestrator-staging`` container, pattern B6 / ORCH-048). Every +public function honours a **never-raise** contract: on any malformed input it +returns the *conservative* (fail-closed) result, never an exception. + +Safety invariant (FR-4 / AC-3): a failed REAL check ALWAYS yields ``FAILED`` / +exit 1 regardless of ``infra_tolerant``. The waiver applies ONLY to the named +``SANDBOX_INFRA`` checks and ONLY when every REAL check (incl. C7/C8) is green — +so the blast-radius of the tolerance is exactly the two allowlisted checks. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +# Category constants --------------------------------------------------------- +REAL = "real" # a real pipeline check — fail-closed, always counts +SANDBOX_INFRA = "sandbox_infra" # known to depend on sandbox infra (waivable) + +# Narrow allowlist of checks known to depend on sandbox infrastructure rather +# than the pipeline itself (ADR-001 §1). Matched by the check's leading label +# token, e.g. "C9a Branch appears in orchestrator-sandbox" -> token "C9a". +# Keep this set MINIMAL — every entry is a hole in the staging safety-net. +SANDBOX_INFRA_CHECKS = frozenset({"C9a", "C9b"}) + + +def classify_check(label) -> str: + """Classify a staging-check label as ``REAL`` or ``SANDBOX_INFRA``. + + A label is ``SANDBOX_INFRA`` iff its leading whitespace-delimited token is one + of :data:`SANDBOX_INFRA_CHECKS` (exact match or prefix, e.g. ``"C9a"`` from + ``"C9a Branch appears…"``). Everything else — and anything unrecognised / + malformed — is ``REAL`` (conservative / fail-closed: an unknown check counts + toward the safety net). Never raises. + """ + try: + text = str(label).strip() + if not text: + return REAL + token = text.split()[0] + for prefix in SANDBOX_INFRA_CHECKS: + if token == prefix or token.startswith(prefix): + return SANDBOX_INFRA + return REAL + except Exception: + return REAL + + +@dataclass +class StagingVerdict: + """Outcome of folding the staging-check suite into a single verdict. + + ``status`` — ``"SUCCESS"`` | ``"FAILED"`` (mirrors the ``staging_status:`` + frontmatter contract the deployer writes; unchanged). + ``exit_code`` — ``0`` (advance) | ``1`` (rollback). Drives ``sys.exit`` in + ``staging_check.py``. + ``waived`` — labels of SANDBOX_INFRA checks that failed but were tolerated + (empty unless the waiver actually fired — observability, FR-7). + ``summary`` — human-readable one-liner for logs. + """ + + status: str + exit_code: int + waived: list = field(default_factory=list) + summary: str = "" + + +def _coerce_item(item) -> tuple[str, bool, str]: + """Normalise an input row into ``(label, passed, category)``. + + Accepts ``(label, passed)`` or ``(label, passed, category)``. A missing/None + category is resolved via :func:`classify_check`. Never raises — a malformed + row degrades to a failed REAL check (fail-closed) so it cannot silently pass. + """ + try: + label = str(item[0]) + passed = bool(item[1]) + category = item[2] if len(item) > 2 and item[2] else None + except Exception: + return ("", False, REAL) + if category not in (REAL, SANDBOX_INFRA): + category = classify_check(label) + return (label, passed, category) + + +def compute_staging_verdict(items, infra_tolerant: bool) -> StagingVerdict: + """Fold per-check results into a tolerant-but-fail-closed staging verdict. + + ``items`` — iterable of ``(label, passed: bool[, category: str])``. + + Decision table (ADR-001 §1): + * any REAL check failed -> FAILED / exit 1 (safety net) + * only SANDBOX_INFRA failed & infra_tolerant -> SUCCESS / exit 0 (waived) + * only SANDBOX_INFRA failed & !infra_tolerant -> FAILED / exit 1 (legacy strict) + * nothing failed -> SUCCESS / exit 0 + + Never raises: on any internal error the verdict degrades to a conservative + ``FAILED`` / exit 1 (never a false green) — AC-10. + """ + try: + real_failed: list[str] = [] + infra_failed: list[str] = [] + for raw in items: + label, passed, category = _coerce_item(raw) + if passed: + continue + if category == SANDBOX_INFRA: + infra_failed.append(label) + else: + real_failed.append(label) + + if real_failed: + # Safety net (FR-4): a real pipeline regression always fails closed, + # regardless of tolerance. Infra failures (if any) are noted but the + # verdict is dominated by the real failure. + extra = f"; infra-fail {infra_failed}" if infra_failed else "" + return StagingVerdict( + status="FAILED", + exit_code=1, + waived=[], + summary=f"FAILED: real checks failed {real_failed}{extra}", + ) + if infra_failed and infra_tolerant: + # Waiver fires ONLY here: every REAL check is green and the only + # failures are allowlisted sandbox-infra checks (FR-2). + return StagingVerdict( + status="SUCCESS", + exit_code=0, + waived=list(infra_failed), + summary=( + f"SUCCESS (infra-waived): {infra_failed} are known sandbox-infra " + "checks; all real checks green" + ), + ) + if infra_failed and not infra_tolerant: + # Legacy strict (kill-switch off): any failure fails closed (1:1 pre-061). + return StagingVerdict( + status="FAILED", + exit_code=1, + waived=[], + summary=f"FAILED (strict): {infra_failed} failed and tolerance disabled", + ) + return StagingVerdict( + status="SUCCESS", + exit_code=0, + waived=[], + summary="SUCCESS: all checks green", + ) + except Exception as e: # noqa: BLE001 - never-raise; fail closed on doubt + return StagingVerdict( + status="FAILED", + exit_code=1, + waived=[], + summary=f"FAILED (verdict error, fail-closed): {e}", + ) diff --git a/tests/test_config.py b/tests/test_config.py index b751be4..ea44e0c 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -142,3 +142,26 @@ def test_image_freshness_settings_env_override(monkeypatch): s = Settings() assert s.image_freshness_enabled is False assert s.image_freshness_repos == "orchestrator,enduro-trails" + + +# --------------------------------------------------------------------------- +# ORCH-061 / TC-09: staging_infra_tolerance_enabled kill-switch (AC-7). +# --------------------------------------------------------------------------- +def test_staging_infra_tolerance_defaults_true(monkeypatch): + """TC-09 / AC-7: the kill-switch defaults ON (safe default — the safety net + holds regardless; the flag exists to restore legacy strictness instantly).""" + monkeypatch.delenv("ORCH_STAGING_INFRA_TOLERANCE_ENABLED", raising=False) + assert Settings().staging_infra_tolerance_enabled is True + + +def test_staging_infra_tolerance_env_override_false(monkeypatch): + """TC-09 / AC-7: ORCH_STAGING_INFRA_TOLERANCE_ENABLED=false -> strict (1:1 + pre-ORCH-061: infra-only FAIL again rolls back).""" + monkeypatch.setenv("ORCH_STAGING_INFRA_TOLERANCE_ENABLED", "false") + assert Settings().staging_infra_tolerance_enabled is False + + +def test_staging_infra_tolerance_env_override_true(monkeypatch): + """The field is read verbatim from its ORCH_* env var.""" + monkeypatch.setenv("ORCH_STAGING_INFRA_TOLERANCE_ENABLED", "true") + assert Settings().staging_infra_tolerance_enabled is True diff --git a/tests/test_launcher.py b/tests/test_launcher.py index e2ec215..970f226 100644 --- a/tests/test_launcher.py +++ b/tests/test_launcher.py @@ -278,3 +278,48 @@ class TestWatchdogGracefulKill: assert signal.SIGKILL not in sent assert recorded["called"] is False + + +# --------------------------------------------------------------------------- +# ORCH-061 / TC-06 + TC-07: "no changes to commit" on an action stage is EXPECTED, +# not under-delivery (FR-3 / AC-4). action_stage_no_changes_note is the PURE +# observability decision used by the post-run no-changes branch: it returns an +# explicit note for self-deploy action stages (deploy-staging/deploy) and None +# everywhere else. It NEVER signals a rollback — advancement is decided by the +# exit-code + gate verdict, never by a commit existing. +# --------------------------------------------------------------------------- +from src.agents.launcher import action_stage_no_changes_note # noqa: E402 + + +class TestActionStageNoChangesNote: + def test_tc06_deploy_staging_self_deploy_returns_note(self): + """TC-06 / AC-4: on deploy-staging for the self-hosting repo, an empty diff + yields an explicit "expected on action stage" note (no rollback signal).""" + note = action_stage_no_changes_note("deploy-staging", "orchestrator") + assert note is not None + assert "deploy-staging" in note + assert "expected on action stage" in note + + def test_tc06_deploy_self_deploy_returns_note(self): + """The `deploy` stage is equally an action stage for self-deploy.""" + note = action_stage_no_changes_note("deploy", "orchestrator") + assert note is not None + assert "deploy: no code changes" in note + + def test_tc07_development_stage_returns_none(self): + """TC-07 / AC-4 regression-guard: on a CODE stage (development) the new + action-stage allowance does NOT apply — no note, behaviour unchanged.""" + assert action_stage_no_changes_note("development", "orchestrator") is None + + def test_tc06_non_self_repo_returns_none(self): + """Conditionality (FR-5): the action-stage allowance is self-deploy only; + a non-self repo on deploy-staging gets no special note.""" + assert action_stage_no_changes_note("deploy-staging", "enduro-trails") is None + + def test_review_stage_returns_none(self): + """Any non-action stage -> None (defensive: only deploy stages qualify).""" + assert action_stage_no_changes_note("review", "orchestrator") is None + + def test_never_raises_on_bad_input(self): + """never-raise: odd inputs (None stage / None repo) degrade to None.""" + assert action_stage_no_changes_note(None, None) is None diff --git a/tests/test_qg.py b/tests/test_qg.py index eb41680..b318c85 100644 --- a/tests/test_qg.py +++ b/tests/test_qg.py @@ -689,6 +689,27 @@ class TestCheckStagingStatus: assert passed is True assert "N/A" in reason + # ------------------------------------------------------------------ + # ORCH-061 / TC-08: the conditional staging gate is unchanged for + # non-self-hosting repos AND independent of the new tolerance flag (FR-5/AC-6). + # ------------------------------------------------------------------ + + def test_tc08_non_self_na_independent_of_tolerance_flag(self, tmp_path, monkeypatch): + """TC-08 / AC-6: for a non-self-hosting repo check_staging_status is the + byte-identical (True, "Staging gate N/A …") regardless of whether the + ORCH-061 infra-tolerance flag is on or off — the new behaviour never + activates off the self-hosting path.""" + monkeypatch.setattr("src.qg.checks.settings.repos_dir", str(tmp_path)) + from src.qg.checks import check_staging_status + for flag in (True, False): + monkeypatch.setattr( + "src.config.settings.staging_infra_tolerance_enabled", flag, + raising=False, + ) + passed, reason = check_staging_status("enduro-trails", "ET-035") + assert passed is True + assert reason == "Staging gate N/A for enduro-trails" + # ------------------------------------------------------------------ # is_self_hosting_repo helper # ------------------------------------------------------------------ diff --git a/tests/test_qg_checks.py b/tests/test_qg_checks.py index c6bfb01..2ab3ea2 100644 --- a/tests/test_qg_checks.py +++ b/tests/test_qg_checks.py @@ -51,3 +51,100 @@ def test_tc15_finalizer_log_roundtrips_through_parser(): ok_f, _ = _parse_deploy_status(build_deploy_log("ORCH-036", 2, "FAILED")) assert ok_s is True assert ok_f is False + + +# --------------------------------------------------------------------------- +# ORCH-061 / TC-04 + TC-05: infra-tolerant staging verdict (pure logic, AC-2/AC-3). +# +# compute_staging_verdict folds the staging-check suite into a single +# SUCCESS/FAILED verdict that is TOLERANT to known sandbox-infra failures +# (C9a/C9b) but stays fail-closed for any REAL pipeline check. These tests +# exercise the verdict directly — no live staging stand / docker (02-trz §9). +# --------------------------------------------------------------------------- +from src.staging_verdict import ( # noqa: E402 + REAL, + SANDBOX_INFRA, + compute_staging_verdict, +) + + +def _rows(*specs): + """Helper: build (label, passed, category) rows.""" + return [(label, passed, cat) for label, passed, cat in specs] + + +def test_tc04_only_infra_failures_waived_to_success(): + """TC-04 / AC-2: every REAL check PASS, only known sandbox-infra checks + (C9a/C9b) FAIL, tolerance ON -> SUCCESS / exit 0 (no false rollback).""" + rows = _rows( + ("C7 Create issue in Plane SANDBOX", True, REAL), + ("C8 Trigger pipeline via /webhook/plane", True, REAL), + ("C9a Branch appears in orchestrator-sandbox", False, SANDBOX_INFRA), + ("C9b Analyst job enqueued in staging queue", False, SANDBOX_INFRA), + ) + v = compute_staging_verdict(rows, infra_tolerant=True) + assert v.status == "SUCCESS" + assert v.exit_code == 0 + # Both infra checks are surfaced as waived (observability, FR-7). + assert set(v.waived) == { + "C9a Branch appears in orchestrator-sandbox", + "C9b Analyst job enqueued in staging queue", + } + + +def test_tc05_any_real_failure_fails_closed(): + """TC-05 / AC-3: at least one REAL pipeline check FAILS (alongside the infra + ones) -> FAILED / exit 1 even with tolerance ON (safety net not weakened).""" + rows = _rows( + ("C7 Create issue in Plane SANDBOX", False, REAL), # real regression + ("C8 Trigger pipeline via /webhook/plane", True, REAL), + ("C9a Branch appears in orchestrator-sandbox", False, SANDBOX_INFRA), + ) + v = compute_staging_verdict(rows, infra_tolerant=True) + assert v.status == "FAILED" + assert v.exit_code == 1 + assert v.waived == [] # nothing waived when a real check failed + + +def test_tc05_real_failure_fails_closed_even_alone(): + """A single REAL failure (no infra failures) is still FAILED (fail-closed).""" + rows = _rows(("C7 Create issue in Plane SANDBOX", False, REAL)) + v = compute_staging_verdict(rows, infra_tolerant=True) + assert v.status == "FAILED" + assert v.exit_code == 1 + + +def test_tc09_infra_failure_strict_mode_fails_closed(): + """TC-09 / AC-7: with tolerance OFF, an infra-only FAIL again -> FAILED + (1:1 pre-ORCH-061 strict behaviour).""" + rows = _rows( + ("C7 Create issue in Plane SANDBOX", True, REAL), + ("C9a Branch appears in orchestrator-sandbox", False, SANDBOX_INFRA), + ) + v = compute_staging_verdict(rows, infra_tolerant=False) + assert v.status == "FAILED" + assert v.exit_code == 1 + + +def test_all_green_is_success_regardless_of_tolerance(): + rows = _rows( + ("C7 Create issue in Plane SANDBOX", True, REAL), + ("C9a Branch appears in orchestrator-sandbox", True, SANDBOX_INFRA), + ) + for tol in (True, False): + v = compute_staging_verdict(rows, infra_tolerant=tol) + assert v.status == "SUCCESS" + assert v.exit_code == 0 + assert v.waived == [] + + +def test_tc12_compute_verdict_never_raises_on_garbage(): + """AC-10 never-raise: malformed rows degrade to a conservative FAILED, never + an exception.""" + v = compute_staging_verdict([("only-one-element",)], infra_tolerant=True) + assert v.status == "FAILED" + assert v.exit_code == 1 + # A completely broken iterable also fails closed without raising. + v2 = compute_staging_verdict(None, infra_tolerant=True) + assert v2.status == "FAILED" + assert v2.exit_code == 1 diff --git a/tests/test_stage_engine.py b/tests/test_stage_engine.py index f229141..ca3dab6 100644 --- a/tests/test_stage_engine.py +++ b/tests/test_stage_engine.py @@ -1132,3 +1132,158 @@ class TestDelegation: assert args[0] == 5 assert args[1] == "analysis" assert args[-1] is None + + +# --------------------------------------------------------------------------- +# ORCH-061: no deploy-staging loop on a healthy self-deploy; the ORCH-35 safety +# net (real staging FAIL -> rollback) stays intact; the new logic never raises +# into advance_stage; and "green with an infra allowance" is distinguishable from +# an honest green (observability). +# --------------------------------------------------------------------------- +class TestStagingInfraTolerance: + """The verdict that produces ``staging_status:`` is computed in the suite + BEFORE the gate (ORCH-061 ADR-001 §4: check_staging_status is unchanged). At + the engine level we therefore assert the REACTION to the resulting verdict: + SUCCESS advances (no loop), a REAL FAILED rolls back (safety net).""" + + def _patch_self_deploy_state(self, monkeypatch, tmp_path): + # Phase A writes restart-safe markers under repos_dir — keep them in tmp. + monkeypatch.setattr(stage_engine.self_deploy.settings, "repos_dir", str(tmp_path)) + monkeypatch.setattr(stage_engine.self_deploy.settings, "host_repos_dir", str(tmp_path)) + + def test_tc01_healthy_self_deploy_advances_no_rollback(self, monkeypatch, tmp_path): + """TC-01 / AC-1: staging SUCCESS (infra-FAIL already waived in the suite) + + green merge/freshness sub-gates -> deploy-staging advances to `deploy` + (Phase A approval-pending). NO rollback to development (loop is gone).""" + self._patch_self_deploy_state(monkeypatch, tmp_path) + monkeypatch.setattr( + stage_engine, "QG_CHECKS", + {**stage_engine.QG_CHECKS, + "check_staging_status": _pass, + "check_branch_mergeable": _pass, + "check_staging_image_fresh": _pass}, + ) + task_id = _make_task("deploy-staging", repo="orchestrator", wi="ORCH-061", + branch="feature/ORCH-061-x") + res = advance_stage( + task_id, "deploy-staging", "orchestrator", "ORCH-061", + "feature/ORCH-061-x", finished_agent="deployer", + ) + assert res.advanced is True + assert res.to_stage == "deploy" + assert _stage(task_id) == "deploy" # Phase A advanced the stage + assert res.rolled_back_to is None # NO loop back to development + assert res.note == "self-deploy-approval-pending" + + def test_tc02_real_staging_failed_rolls_back(self, monkeypatch, tmp_path): + """TC-02 / AC-3: a REAL staging failure (verdict FAILED) still rolls + deploy-staging back to development + set_issue_blocked + alert — the + ORCH-35 safety net is NOT weakened by the infra tolerance (FR-4).""" + self._patch_self_deploy_state(monkeypatch, tmp_path) + monkeypatch.setattr( + stage_engine, "QG_CHECKS", + {**stage_engine.QG_CHECKS, + "check_staging_status": _fail("Staging status: FAILED")}, + ) + task_id = _make_task("deploy-staging", repo="orchestrator", wi="ORCH-061", + branch="feature/ORCH-061-x") + res = advance_stage( + task_id, "deploy-staging", "orchestrator", "ORCH-061", + "feature/ORCH-061-x", finished_agent="deployer", + ) + assert res.advanced is False + assert res.rolled_back_to == "development" + assert _stage(task_id) == "development" + assert res.alerted is True + assert stage_engine.set_issue_blocked.called + assert stage_engine.send_telegram.called + + def test_tc12_gate_exception_never_crashes_advance(self, monkeypatch, tmp_path): + """TC-12 / AC-10 never-raise: if the staging gate raises (io/parse/docker + hiccup), advance_stage catches it deterministically — no exception escapes, + the task does NOT advance and is NOT falsely rolled back to development.""" + self._patch_self_deploy_state(monkeypatch, tmp_path) + + def _boom(*a, **k): + raise RuntimeError("staging gate blew up") + + monkeypatch.setattr( + stage_engine, "QG_CHECKS", + {**stage_engine.QG_CHECKS, "check_staging_status": _boom}, + ) + task_id = _make_task("deploy-staging", repo="orchestrator", wi="ORCH-061", + branch="feature/ORCH-061-x") + # Must NOT raise. + res = advance_stage( + task_id, "deploy-staging", "orchestrator", "ORCH-061", + "feature/ORCH-061-x", finished_agent="deployer", + ) + assert res.advanced is False + assert res.rolled_back_to is None # exception != gate FAILED + assert _stage(task_id) == "deploy-staging" # stays put, no loop + assert res.note and "error" in res.note + + def test_tc13_end_to_end_self_deploy_no_single_rollback(self, monkeypatch, tmp_path): + """TC-13 / AC-1+AC-4 integration: a healthy self-deploy goes + deploy-staging -> deploy (Phase A) -> (approve/finalize SUCCESS) -> done + WITHOUT a single rollback to development in the transition log.""" + self._patch_self_deploy_state(monkeypatch, tmp_path) + monkeypatch.setattr( + stage_engine, "QG_CHECKS", + {**stage_engine.QG_CHECKS, + "check_staging_status": _pass, + "check_branch_mergeable": _pass, + "check_staging_image_fresh": _pass, + "check_deploy_status": _pass}, + ) + task_id = _make_task("deploy-staging", repo="orchestrator", wi="ORCH-061", + branch="feature/ORCH-061-x") + + seen_stages = [] + + # 1) deploy-staging -> deploy (Phase A approval-pending). + r1 = advance_stage( + task_id, "deploy-staging", "orchestrator", "ORCH-061", + "feature/ORCH-061-x", finished_agent="deployer", + ) + seen_stages.append(_stage(task_id)) + assert r1.advanced is True + assert _stage(task_id) == "deploy" + + # 2) finalizer (Phase C): deploy verdict SUCCESS -> done. + r2 = advance_stage( + task_id, "deploy", "orchestrator", "ORCH-061", + "feature/ORCH-061-x", finished_agent="deployer", + ) + seen_stages.append(_stage(task_id)) + assert r2.advanced is True + assert _stage(task_id) == "done" + + # Not a single rollback to development anywhere in the path. + assert "development" not in seen_stages + assert r1.rolled_back_to is None and r2.rolled_back_to is None + + def test_tc14_waived_green_distinguishable_from_honest_green(self): + """TC-14 / AC-11 observability: the staging verdict makes "green with an + infra allowance" distinguishable from an honest green — the waived list is + populated and the summary says so, vs an empty waived list + plain summary + for an all-green run.""" + from src.staging_verdict import REAL, SANDBOX_INFRA, compute_staging_verdict + + waived = compute_staging_verdict( + [("C7", True, REAL), + ("C9a", False, SANDBOX_INFRA)], + infra_tolerant=True, + ) + honest = compute_staging_verdict( + [("C7", True, REAL), + ("C9a", True, SANDBOX_INFRA)], + infra_tolerant=True, + ) + # Both advance... + assert waived.status == honest.status == "SUCCESS" + # ...but only the waived one carries the explicit allowance marker. + assert waived.waived == ["C9a"] + assert "infra-waived" in waived.summary.lower() + assert honest.waived == [] + assert "infra-waived" not in honest.summary.lower() diff --git a/tests/test_staging_check_b6.py b/tests/test_staging_check_b6.py index 0eb8940..a2668f8 100644 --- a/tests/test_staging_check_b6.py +++ b/tests/test_staging_check_b6.py @@ -149,3 +149,70 @@ def test_run_b6_records_pass_for_clean_registry(monkeypatch): _label, passed, detail = results._items[0] assert passed is True assert "sandbox=YES" in detail + + +# --------------------------------------------------------------------------- +# ORCH-061 / TC-03: the suite classifies checks as REAL vs SANDBOX_INFRA so the +# verdict (and exit-code) can tolerate KNOWN sandbox-infra FAILs (C9a/C9b) while +# staying fail-closed for real pipeline checks. Tested without a live stand. +# --------------------------------------------------------------------------- +from src.staging_verdict import REAL, SANDBOX_INFRA # noqa: E402 + + +def test_tc03_classify_infra_checks(): + """C9a/C9b classify as SANDBOX_INFRA; pipeline checks (A/B/C7/C8) as REAL.""" + assert sc._classify("C9a Branch appears in orchestrator-sandbox") == SANDBOX_INFRA + assert sc._classify("C9b Analyst job enqueued in staging queue") == SANDBOX_INFRA + assert sc._classify("C7 Create issue in Plane SANDBOX") == REAL + assert sc._classify("C8 Trigger pipeline via /webhook/plane") == REAL + assert sc._classify("A1 GET /health") == REAL + assert sc._classify("B6 Registry: sandbox present") == REAL + + +def test_tc03_results_records_categories_and_keeps_tuple_shape(): + """Results.add auto-classifies each check; categorized_items() exposes the + category WITHOUT changing the public 3-tuple shape of _items (ORCH-048 b6 + tests still unpack (label, passed, detail)).""" + results = sc.Results() + results.add("C7 Create issue in Plane SANDBOX", True) + results.add("C9a Branch appears in orchestrator-sandbox", False) + + # Public _items shape unchanged (regression guard for ORCH-048 tests). + for item in results._items: + assert len(item) == 3 + + cats = {label: cat for label, _passed, cat in results.categorized_items()} + assert cats["C7 Create issue in Plane SANDBOX"] == REAL + assert cats["C9a Branch appears in orchestrator-sandbox"] == SANDBOX_INFRA + + +def test_tc03_explicit_category_overrides_autoclassify(): + """An explicit category arg is honoured (caller can force REAL).""" + results = sc.Results() + results.add("C9a Branch appears in orchestrator-sandbox", False, category=REAL) + label, _passed, cat = results.categorized_items()[0] + assert cat == REAL + + +def test_tc03_suite_verdict_waives_infra_only_failure(): + """End-to-end through the suite helpers: a run whose only failures are C9a/C9b + -> exit 0 (waived) under tolerance; the waiver is surfaced for observability.""" + results = sc.Results() + results.add("C7 Create issue in Plane SANDBOX", True) + results.add("C8 Trigger pipeline via /webhook/plane", True) + results.add("C9a Branch appears in orchestrator-sandbox", False) + results.add("C9b Analyst job enqueued in staging queue", False) + + verdict = sc._verdict(results.categorized_items(), infra_tolerant=True) + assert verdict.status == "SUCCESS" + assert verdict.exit_code == 0 + assert len(verdict.waived) == 2 + + # Strict mode (kill-switch off) re-fails the same run. + strict = sc._verdict(results.categorized_items(), infra_tolerant=False) + assert strict.exit_code == 1 + + +def test_tc03_resolve_tolerance_strict_flag_forces_off(): + """--strict forces tolerance OFF regardless of the config default.""" + assert sc._resolve_tolerance(cli_strict=True) is False