fix(merge-gate): tolerate re-test infra-timeout + tree-kill spawned pytest
Eliminate the false `deploy-staging -> development` rollback that fired when the merge-gate local re-test timed out (infra/resource) on a green CI + tester + staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget under CPU starvation from orphaned pytest processes -> timeout misrouted as a code fault -> developer-retry loop -> manual gate). Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push main) and the no-prod-restart rule are preserved. - D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage pytest in its own process group (start_new_session) and tree-kills the WHOLE group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak. Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the prior subprocess.run. - D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/ other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded re-queue, task stays on deploy-staging, no rollback / no developer-retry); a red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert. - D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated); fail-safe runs the re-test on any uncertainty. Flag merge_retest_skip_when_current_enabled. - D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation; reaper_max_running_s invariant preserved without change. - D6: in-process counters + read-only merge_gate block in GET /queue; appended ("ORCH-110","classify_retest_failure","src/merge_gate.py") to MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/ .env.example) updated in the same PR. Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after incident regression). Full suite green (1988 passed). Refs: ORCH-110 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
23
.env.example
23
.env.example
@@ -164,11 +164,32 @@ ORCH_TRACKER_LIVE_STATUS_TIMEOUT_S=3
|
||||
# DEFER_MAX_ATTEMPTS -> defer retries before escalation (avoids livelock).
|
||||
ORCH_MERGE_GATE_ENABLED=true
|
||||
ORCH_MERGE_GATE_REPOS=
|
||||
ORCH_MERGE_RETEST_TIMEOUT_S=600
|
||||
# ORCH-110 (D5): re-test budget raised 600 -> 900 (74% headroom over the observed
|
||||
# 516.7s suite). Cross-invariant (ORCH-065/109): keep ORCH_REAPER_MAX_RUNNING_S
|
||||
# (5400) > Σ(deploy-staging gate-work) + grace if you raise this — see
|
||||
# docs/work-items/ORCH-110/07-infra-requirements.md.
|
||||
ORCH_MERGE_RETEST_TIMEOUT_S=900
|
||||
ORCH_MERGE_RETEST_TARGET=tests/
|
||||
ORCH_MERGE_LOCK_TIMEOUT_S=300
|
||||
ORCH_MERGE_DEFER_DELAY_S=60
|
||||
ORCH_MERGE_DEFER_MAX_ATTEMPTS=5
|
||||
# ORCH-110: merge-gate re-test infra-timeout tolerance + tree-kill of the
|
||||
# orchestrator-spawned pytest subprocess (re-test + coverage). Each default = the
|
||||
# desired prod behaviour; each flag is an independent kill-switch (off ->
|
||||
# byte-for-byte pre-ORCH-110). The tree-kill grace reuses ORCH_AGENT_KILL_GRACE_SECONDS.
|
||||
# SUBPROCESS_TREE_KILL_ENABLED -> D1: spawn re-test/coverage pytest in its
|
||||
# own process group; tree-kill the WHOLE group on timeout (no orphan grandchildren).
|
||||
# MERGE_RETEST_INFRA_TOLERANCE_ENABLED -> D3: a re-test TIMEOUT is a transient
|
||||
# (bounded infra-retry, NOT a code-fault rollback to development).
|
||||
# MERGE_RETEST_INFRA_MAX_RETRIES -> D3: infra-retry budget before an infra-alert.
|
||||
# MERGE_RETEST_INFRA_RETRY_DELAY_S -> D3: delay before the staging-deployer re-run.
|
||||
# MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED-> D4: skip the local re-test when the
|
||||
# pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated).
|
||||
ORCH_SUBPROCESS_TREE_KILL_ENABLED=true
|
||||
ORCH_MERGE_RETEST_INFRA_TOLERANCE_ENABLED=true
|
||||
ORCH_MERGE_RETEST_INFRA_MAX_RETRIES=2
|
||||
ORCH_MERGE_RETEST_INFRA_RETRY_DELAY_S=120
|
||||
ORCH_MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED=true
|
||||
# ORCH-026 Level A: unconditional pre-merge rebase. With the flag ON (default),
|
||||
# check_branch_mergeable ALWAYS rebases the branch onto origin/main under the held
|
||||
# merge-lease (not only when behind) — a deterministic structural anti-phantom on
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Work item: ORCH-111
|
||||
Work item: ORCH-110
|
||||
Repo: orchestrator
|
||||
Branch: feature/ORCH-111-bug-watchdog-must-alert-on-lon
|
||||
Branch: feature/ORCH-110-bug-merge-gate-local-re-test-t
|
||||
Stage: development
|
||||
@@ -3,6 +3,13 @@
|
||||
Формат: [Keep a Changelog](https://keepachangelog.com/). Записи — на смысловой PR/задачу.
|
||||
|
||||
## [Unreleased]
|
||||
- **Merge-gate re-test: толерантность к инфра-таймауту + tree-kill спавненных pytest + контракт необходимости re-test** (ORCH-110, `fix`, bug→escalate full-cycle): устранён ложный откат `deploy-staging → development`, возникавший когда локальный re-test merge-gate падал по **таймауту** (инфра/ресурс) при зелёных CI + tester + staging (инцидент ORCH-109/PR #129: сюит 516.7s упёрся в бюджет 600s под CPU-голоданием от осиротевших pytest-процессов → `(False, "re-test timeout after 600s")` → `_handle_merge_gate_rollback` → каждый из 3 developer-retry падал так же → «Merge-gate still failing after 3 developer retries» → ручное вмешательство). Аддитивно, под 5 независимыми kill-switch, never-raise, скоуп self-hosting; `STAGE_TRANSITIONS`/реестр `QG_CHECKS`/семантика `check_*`/machine-verdict-ключи/схема БД — **байт-в-байт не тронуты**; INV-4 (никогда push/force-push `main`) и запрет рестарта прод-контейнера — соблюдены. ADR: `docs/work-items/ORCH-110/06-adr/ADR-001-merge-gate-retest-infra-tolerance-and-tree-kill.md`, сквозной `docs/architecture/adr/adr-0042-merge-gate-retest-infra-tolerance-and-tree-kill.md`.
|
||||
- **D1 — process-group tree-kill (`src/proc_group.py`, новый stdlib-only leaf):** `merge_gate.retest_branch` и `coverage_gate.measure_coverage` теперь спавнят pytest в **отдельной группе процессов** (`start_new_session`) и при таймауте убивают **всё дерево** (`os.killpg`, каскад SIGTERM→grace→SIGKILL по образцу `launcher.stop_process`), а не только прямого потомка — осиротевшие внуки-pytest больше не переживают бюджет и не грузят CPU. Контракты возврата сохранены (меняется лишь побочный эффект — нет утечки). Грейс реюзит `agent_kill_grace_seconds`. Fallback never-break: `subprocess_tree_kill_enabled=False` или не-POSIX → прежний `subprocess.run(timeout=)`.
|
||||
- **D2/D3 — классификация + маршрутизация инфра-таймаута:** чистый предикат `merge_gate.classify_retest_failure(reason)` различает `timeout`/`red`/`lock-busy`/`other` (scope-guard: `auto_rebase_onto_main`'s «rebase timeout» — НЕ инфра-таймаут re-test, остаётся на rollback-пути). Инфра-таймаут → новый `_handle_merge_gate_infra_retry` (ограниченный повтор по образцу `_handle_merge_gate_defer`: задача **остаётся на deploy-staging**, staging-deployer перезапускается с задержкой, **БЕЗ** отката на `development` и **БЕЗ** расхода developer-retry). Анти-над-толерантность (BR-6): детерминированно **красный** re-test / конфликт по-прежнему → `_handle_merge_gate_rollback`. Anti-loop: исчерпание бюджета → один **инфра-alert** (явно инфраструктурная формулировка «НЕ дефект кода» с кликабельным номером), задача НЕ уходит в `development`.
|
||||
- **D4 — контракт необходимости re-test:** при `premerge_rebase_always=True` re-test теперь **пропускается**, когда rebase оказался доказанным no-op (HEAD не сдвинулся = ветка уже содержит свежий `origin/main`, тот же коммит уже подтвердили CI + tester + staging) — distribute той же оптимизации, что путь `premerge_rebase_always=False` уже имеет для не-behind ветки. Fail-safe: при любой неопределённости (`head_sha` пуст / git-ошибка) re-test **выполняется** (BR-6/AC-3 не ослаблен).
|
||||
- **D5 — бюджет:** `merge_retest_timeout_s` 600 → **900** (запас 74% над 516.7s) + валидация `_resolve_retest_timeout` (малформ/непозитив → дефолт 900 + WARNING). Сквозной инвариант ORCH-065/109 `reaper_max_running_s (5400) > Σ(deploy-staging gate-work)+grace (≈4460)` соблюдён **без** правки `reaper_max_running_s`.
|
||||
- **D6 — наблюдаемость:** in-process счётчики (`retest_timeout_total`/`retest_infra_retry_total`/`retest_infra_exhausted_total`/`retest_skipped_current_total`/`last_infra_timeout_wi`) + read-only блок `merge_gate` в `GET /queue` (отличим от код-фейл-отката); координация с ORCH-111 (`proc_blocking`) без дубля (ORCH-110 предотвращает/толерирует, ORCH-111 наблюдает). Append-only regression-guard: добавлен `("ORCH-110", "classify_retest_failure", "src/merge_gate.py")` в `MAIN_REGRESSION_MARKERS`.
|
||||
- **Конфиг (5 новых ключей, дефолт = боевое):** `ORCH_SUBPROCESS_TREE_KILL_ENABLED`/`ORCH_MERGE_RETEST_INFRA_TOLERANCE_ENABLED`/`ORCH_MERGE_RETEST_INFRA_MAX_RETRIES=2`/`ORCH_MERGE_RETEST_INFRA_RETRY_DELAY_S=120`/`ORCH_MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED` + бамп `ORCH_MERGE_RETEST_TIMEOUT_S=900`. Покрытие — `tests/test_orch110_*.py` (TC-01…TC-12, включая регресс инцидента red-before/green-after).
|
||||
- **Watchdog-сигнал `proc_blocking`: алерт на долго живущий осиротевший тест-процесс** (ORCH-111, `feat`): закрыта слепая зона наблюдаемости между `agent_hung` (видит только треканые джобы по `jobs.pid`) и осиротевшими субпроцессами `pytest`, которые орк запускает сам (`merge_gate.retest_branch`/`coverage_gate.measure_coverage`) и которые при timeout-kill агента (`-9`, ORCH-109) репарентируются на tini и живут сутками, грузя CPU и валя merge-gate re-test (инцидент: процессы `test_install_lite_script.py` жили >2 суток без единого алерта). Изменения **строго внутри наблюдателя** (`watchdog/**` + сервис watchdog в compose); `src/**`/`/metrics`/`schema_version`/`STAGE_TRANSITIONS`/`QG_CHECKS`/`check_*`/machine-verdict/схема БД — **байт-в-байт не тронуты**; выкат пересобирает **только** `orchestrator-watchdog`, прод `orchestrator` не рестартится (NFR-3). ADR: `docs/work-items/ORCH-111/06-adr/ADR-001-watchdog-orphan-test-process-alert.md`, сквозной `docs/architecture/adr/adr-0041-watchdog-orphan-test-process-alert.md`.
|
||||
- **Коллектор `watchdog/collectors/proc.py` (D3):** новый stdlib-only `/proc`-скан (под `pid: host` контейнерный `/proc` отражает хост-namespace) — читает `/proc/stat` (`btime`) + `os.sysconf("SC_CLK_TCK")`, итерирует числовые `/proc/<pid>`, матчит `/proc/<pid>/cmdline` по паттерну тест-класса, парсит `/proc/<pid>/stat` (поле 22 `starttime` → `age_s`, поля 14+15 `utime+stime` → `cpu_s` информационно). Строго **read-only** (никаких `os.kill`/сигналов/`subprocess`; **никогда** не читает `/proc/<pid>/environ` — секреты); **never-raise** (per-pid гонка «процесс умер между listdir и read» пропускается, top-level → `[]`); чистый разбор отделён от I/O (тестируется на фейковом `/proc`-дереве).
|
||||
- **Чистый builder `proc_signals` + синтез RECOVERY (D4):** per-entity `Signal("proc_blocking", pid)` active ⇔ `age_s > cfg.proc_age_s` (cmdline уже отфильтрована коллектором); действенный RU-`detail` (PID + возраст + усечённый фрагмент cmdline + CPU-время). Исчезновение процесса не оставляет «висящего» алерта: в `core.tick()` для каждого alerting-ключа без свежего сигнала **синтезируется** `Signal(active=False)` → существующая `decision.decide()`/`AlertState` даёт **однократный** RECOVERY и чистит состояние (никакой новой анти-спам-логики — FR-5).
|
||||
|
||||
48
CLAUDE.md
48
CLAUDE.md
@@ -235,6 +235,54 @@ kill-switch, never-raise, fail-safe → полный цикл.
|
||||
`docs/work-items/ORCH-027/06-adr/ADR-001-coverage-gate.md`,
|
||||
`docs/architecture/adr/adr-0029-coverage-gate.md`.
|
||||
|
||||
## Merge-gate re-test: инфра-толерантность + tree-kill + контракт re-test (ORCH-110)
|
||||
Багфикс инцидента **ORCH-109/PR #129** (bug → escalate full-cycle): локальный re-test merge-gate
|
||||
(`check_branch_mergeable`, ребро `deploy-staging → deploy`) падал по **таймауту** (516.7s-сюит упёрся
|
||||
в бюджет 600s под CPU-голоданием от **осиротевших** pytest-процессов) при зелёных CI+tester+staging →
|
||||
маршрутизировался как **код-фейл** в `_handle_merge_gate_rollback` (откат на `development` + расход
|
||||
developer-retry) → каждый retry падал так же → «Merge-gate still failing after 3 developer retries» →
|
||||
ручное вмешательство. Аддитивно, под **5 независимыми kill-switch**, never-raise, скоуп self-hosting;
|
||||
`STAGE_TRANSITIONS`/реестр `QG_CHECKS`/семантика/имя `check_branch_mergeable`/machine-verdict-ключи/
|
||||
схема БД — **байт-в-байт не тронуты** (merge-gate остаётся под-гейтом-врезкой, не новой стадией/QG);
|
||||
INV-4 (никогда push/force-push `main`) и запрет рестарта прод-контейнера — соблюдены.
|
||||
- **D1 (tree-kill, корень утечки):** новый stdlib-only leaf `src/proc_group.py`
|
||||
(`run_in_process_group`, паттерн чистоты `serial_gate`) спавнит спавненный pytest в **отдельной
|
||||
группе процессов** (`start_new_session`) и при таймауте убивает **всё дерево** (`os.killpg`, каскад
|
||||
SIGTERM→grace→SIGKILL, зеркало `launcher.stop_process`, грейс = `agent_kill_grace_seconds`).
|
||||
Используют его `merge_gate.retest_branch` И `coverage_gate.measure_coverage` (сиблинг-источник
|
||||
утечки). Контракты возврата сохранены — меняется лишь отсутствие сирот. Fallback never-break:
|
||||
`subprocess_tree_kill_enabled=False`/не-POSIX → прежний `subprocess.run`.
|
||||
- **D2/D3 (классификация + маршрутизация):** чистый предикат `merge_gate.classify_retest_failure`
|
||||
различает `timeout`/`red`/`lock-busy`/`other` (scope-guard: «rebase timeout» git'а — НЕ инфра-таймаут
|
||||
re-test). Инфра-таймаут → `_handle_merge_gate_infra_retry` (зеркало `_handle_merge_gate_defer`:
|
||||
задача **остаётся на `deploy-staging`**, staging-deployer перезапускается с задержкой, **БЕЗ** отката
|
||||
на `development` и **БЕЗ** developer-retry; restart-safe счётчик `_merge_infra_retry_count` по маркеру
|
||||
`merge-gate infra-timeout retry` в `task_content`). Анти-над-толерантность (BR-6): детерминированно
|
||||
**красный** re-test / конфликт → прежний `_handle_merge_gate_rollback`. Anti-loop: исчерпание
|
||||
`merge_retest_infra_max_retries` (дефолт 2) → один **инфра-alert** (явно «НЕ дефект кода»,
|
||||
кликабельный номер), задача НЕ уходит в `development`. Kill-switch `merge_retest_infra_tolerance_enabled`
|
||||
off → таймаут = прежний rollback (байт-в-байт).
|
||||
- **D4 (контракт необходимости re-test):** при `premerge_rebase_always=True` re-test **пропускается**,
|
||||
когда rebase — доказанный no-op (`merge_gate.head_sha` до==после, обе непусты = ветка уже содержит
|
||||
свежий `origin/main`, тот же коммит уже прошёл CI+tester+staging) → лиз HELD, без re-test. Распространяет
|
||||
на путь `=True` ту же оптимизацию, что путь `=False` уже имеет для не-behind ветки. Fail-safe: любая
|
||||
неопределённость (`head_sha` пуст / git-ошибка) → re-test **выполняется** (BR-6/AC-3 не ослаблен).
|
||||
Kill-switch `merge_retest_skip_when_current_enabled`.
|
||||
- **D5 (бюджет):** `merge_retest_timeout_s` 600 → **900** (запас 74% над 516.7s) + валидация
|
||||
`_resolve_retest_timeout` (малформ/непозитив → дефолт 900 + WARNING). Сквозной инвариант ORCH-065/109
|
||||
`reaper_max_running_s (5400) > Σ(deploy-staging gate-work)+grace (≈4460)` соблюдён **без** правки
|
||||
`reaper_max_running_s`.
|
||||
- **D6 (наблюдаемость):** in-process счётчики (`_MERGE_GATE_COUNTERS`) + read-only блок `merge_gate` в
|
||||
`GET /queue` (отличим от код-фейл-отката); координация с ORCH-111 (`proc_blocking`) без дубля (ORCH-110
|
||||
предотвращает/толерирует у источника, ORCH-111 наблюдает). Append-only regression-guard:
|
||||
`("ORCH-110", "classify_retest_failure", "src/merge_gate.py")` в `MAIN_REGRESSION_MARKERS`.
|
||||
- **Флаги** (`config.py`, дефолт = боевое): `subprocess_tree_kill_enabled`/
|
||||
`merge_retest_infra_tolerance_enabled`/`merge_retest_infra_max_retries`/`merge_retest_infra_retry_delay_s`/
|
||||
`merge_retest_skip_when_current_enabled` (env `ORCH_*`). Откат = выставить 4 kill-switch в `False` +
|
||||
`ORCH_MERGE_RETEST_TIMEOUT_S=600` → байт-в-байт до-ORCH-110. Детали —
|
||||
`docs/work-items/ORCH-110/06-adr/ADR-001-merge-gate-retest-infra-tolerance-and-tree-kill.md`, сквозной
|
||||
`docs/architecture/adr/adr-0042-merge-gate-retest-infra-tolerance-and-tree-kill.md`.
|
||||
|
||||
## Машинный журнал уроков (ORCH-098)
|
||||
Шаг 1 («Фундамент», F2) эпика саморазвития: формализует свободнотекстовые «уроки» из `memory/` в
|
||||
**машинную структурированную таблицу отклонений конвейера** `lessons`, фундамент для будущих
|
||||
|
||||
@@ -447,7 +447,10 @@ Self-hosting зацикливался на `deploy-staging`: `scripts/staging_ch
|
||||
Назначение: ветка валидируется относительно того `main`, из которого создана; параллельная задача могла уйти вперёд → семантический конфликт слияния (зелёная ветка ломает обновлённый `main`). Merge-gate гарантирует проверку против **актуального** `origin/main` перед слиянием:
|
||||
- **Догон:** ветка отстаёт (⇔ `origin/main` не предок HEAD) → `rebase origin/main` в worktree + `push --force-with-lease` (ТОЛЬКО ветка задачи; `main` — никогда). Текстовый конфликт → `rebase --abort` → откат на `development`.
|
||||
- **Безусловный pre-merge rebase (ORCH-026, A-2):** при `premerge_rebase_always` (дефолт `True`, скоуп `merge_gate_repos`) short-circuit `branch_is_behind_main` пропускается — `auto_rebase_onto_main` вызывается **всегда** под лизом. На актуальной ветке это no-op (`rebase` не меняет HEAD, `push --force-with-lease` → «Everything up-to-date», CI не триггерится); на отстающей — реальный догон. Детерминированный структурный анти-фантом на уровне планировщика (дополняет рубежи ORCH-073, не заменяет). Kill-switch `premerge_rebase_always=False` → прежнее поведение (ребейз только при behind).
|
||||
- **Re-test:** `python -m pytest` (`merge_retest_target`, дефолт `tests/`) в worktree догнанной ветки, тайм-аут `merge_retest_timeout_s`. **Красный** → откат на `development`. **ORCH-110 (design):** (1) re-test исполняется лишь когда rebase реально сдвинул HEAD (`main` уехал); доказанный no-op rebase (ветка уже актуальна, HEAD уже прошёл CI+tester+staging) **пропускает** локальный re-test; (2) **таймаут** re-test — инфра-транзиент (ограниченный повтор + отдельный инфра-alert), а НЕ код-фейл (раньше шёл в откат на `development` + расход developer-retry); (3) спавненный pytest бежит в отдельной группе процессов и при таймауте убивается **деревом** (`os.killpg`) — нет осиротевших процессов (см. ниже).
|
||||
- **Re-test:** `python -m pytest` (`merge_retest_target`, дефолт `tests/`) в worktree догнанной ветки, тайм-аут `merge_retest_timeout_s` (ORCH-110 D5: 600 → **900**, валидация `_resolve_retest_timeout` — малформ/непозитив → дефолт 900 + WARNING). **Красный** (детерминированный сбой теста) → откат на `development` + developer-retry (без изменений). **ORCH-110 (реализовано, [adr-0042](adr/adr-0042-merge-gate-retest-infra-tolerance-and-tree-kill.md)):**
|
||||
- **(D4) Контракт необходимости re-test:** локальный re-test исполняется лишь когда rebase реально сдвинул HEAD (`main` уехал). Доказанный **no-op** rebase (`head_sha` до==после, обе непусты — ветка уже содержит свежий `origin/main`, HEAD уже прошёл CI+tester+staging) **пропускает** re-test (`(True, "branch up-to-date (re-test skipped: rebase no-op, HEAD CI-validated)")`, лиз HELD) — распространяет на путь `premerge_rebase_always=True` ту же оптимизацию, что путь `=False` уже имеет для не-behind ветки. Fail-safe: любая неопределённость (`head_sha` пуст / git-ошибка) → re-test **выполняется**. Kill-switch `merge_retest_skip_when_current_enabled`.
|
||||
- **(D2/D3) Толерантность к инфра-таймауту:** предикат `classify_retest_failure(reason)` различает `timeout`/`red`/`lock-busy`/`other`. **Таймаут** re-test = инфра-транзиент → `_handle_merge_gate_infra_retry` (ограниченный повтор по образцу defer: задача **остаётся на `deploy-staging`**, staging-deployer перезапускается с задержкой `merge_retest_infra_retry_delay_s`, **БЕЗ** отката на `development` и **БЕЗ** расхода developer-retry), а НЕ прежний rollback. Исчерпание `merge_retest_infra_max_retries` → один **инфра-alert** (явно «НЕ дефект кода», кликабельный номер), задача НЕ уходит в `development`. Анти-над-толерантность (BR-6): **красный** re-test / конфликт по-прежнему откатывают. Scope-guard: «rebase timeout» (git завис) — НЕ инфра-таймаут re-test, остаётся на rollback-пути. Kill-switch `merge_retest_infra_tolerance_enabled` (off → таймаут=прежний rollback).
|
||||
- **(D1) Tree-kill спавненных pytest:** re-test (и coverage `measure_coverage`) бегут через `proc_group.run_in_process_group` в **отдельной группе процессов** (`start_new_session`); при таймауте убивается **всё дерево** (`os.killpg`, каскад SIGTERM→grace→SIGKILL, грейс = `agent_kill_grace_seconds`) — осиротевшие внуки-pytest не переживают бюджет (корень CPU-голодания инцидента ORCH-109). Контракты возврата сохранены. Fallback never-break: `subprocess_tree_kill_enabled=False`/не-POSIX → прежний `subprocess.run`. Наблюдаемость — блок `merge_gate` в `GET /queue`; координация с ORCH-111 (`proc_blocking`) без дубля.
|
||||
- **Сериализация (merge-lock):** файловый **merge-lease** на репо (`<repos_dir>/.merge-lease-<repo>.json`), живёт от гейта до фактического merge. Acquire **неблокирующий** (anti-deadlock при `max_concurrency=1`): busy → **defer** (повторная постановка deployer'а на `deploy-staging` с задержкой через `available_at`), а не откат. Release — на PR-merged вебхуке / `deploy→done` / откате / по возрасту (crash-реклейм). Restart-safe; без изменения схемы БД. **ORCH-026 (A-1):** это окно = «merge → main-updated» (для self `done` ⇔ SHA-in-main, ORCH-073) — пока A не в `main`, B того же репо получает `merge-lock busy` → defer. Окно сериализации per-repo НЕ переписывается; кросс-репо параллелизм сохранён (лиз — per-repo файл).
|
||||
- **Условность (как ORCH-35):** реален для `orchestrator`; прочие репо — no-op. Флаги `merge_gate_enabled` / `merge_gate_repos` — поэтапный раскат. Контракт **never-raise**.
|
||||
|
||||
|
||||
@@ -46,7 +46,10 @@ created → analysis → architecture → development → review → testing →
|
||||
4. `check_staging_image_fresh` — staging-образ собран из актуального кода.
|
||||
|
||||
Это **врезки в переход, а не стадии**: они не появляются в карте `STAGE_TRANSITIONS`, а
|
||||
исполняются stage engine'ом внутри ребра. Провал любого из них — откат на доработку. На ребре
|
||||
исполняются stage engine'ом внутри ребра. Провал любого из них — откат на доработку. Исключение
|
||||
(ORCH-110): **инфра-таймаут** локального re-test merge-gate (а не детерминированный красный тест) —
|
||||
это транзиент, а не дефект кода → ограниченный повтор + отдельный инфра-alert, без отката на
|
||||
доработку и без расхода developer-retry (красный re-test/конфликт по-прежнему откатывают). На ребре
|
||||
`deploy → done` аналогичная врезка merge-verify подтверждает, что код задачи реально слит в
|
||||
`main` (слияние — только через PR-API Gitea, см. [интеграции](tech-integrations.md)).
|
||||
|
||||
|
||||
@@ -202,18 +202,50 @@ class Settings(BaseSettings):
|
||||
# only the self-hosting repo (orchestrator). Other
|
||||
# repos -> conditional no-op (mirrors ORCH-35 staging).
|
||||
# merge_retest_timeout_s -> wall-clock budget for the post-rebase re-test.
|
||||
# ORCH-110 (D5): raised 600 -> 900 (74% headroom over the observed 516.7s
|
||||
# suite vs the prior ~16%). Cross-invariant (ORCH-065/109): keep
|
||||
# reaper_max_running_s (5400) > Σ(deploy-staging gate-work) + grace — see
|
||||
# docs/work-items/ORCH-110/07-infra-requirements.md.
|
||||
# merge_retest_target -> pytest target for the re-test (portability across repos).
|
||||
# merge_lock_timeout_s -> max lease age; an older lease is reclaimed (crash backstop).
|
||||
# merge_defer_delay_s -> delay before re-running the gate when the lock is busy.
|
||||
# merge_defer_max_attempts -> defer retries before escalation (avoids livelock).
|
||||
merge_gate_enabled: bool = True
|
||||
merge_gate_repos: str = ""
|
||||
merge_retest_timeout_s: int = 600
|
||||
merge_retest_timeout_s: int = 900
|
||||
merge_retest_target: str = "tests/"
|
||||
merge_lock_timeout_s: int = 300
|
||||
merge_defer_delay_s: int = 60
|
||||
merge_defer_max_attempts: int = 5
|
||||
|
||||
# ORCH-110: merge-gate re-test infra-timeout tolerance + tree-kill of the
|
||||
# orchestrator-spawned pytest subprocess (re-test + coverage). Each default =
|
||||
# the desired prod behaviour (ORCH-101 canon); each flag is an INDEPENDENT
|
||||
# kill-switch (off -> byte-for-byte pre-ORCH-110). Detailed ADR:
|
||||
# docs/work-items/ORCH-110/06-adr/ADR-001-merge-gate-retest-infra-tolerance-and-tree-kill.md.
|
||||
# subprocess_tree_kill_enabled -> D1: spawn the re-test / coverage
|
||||
# pytest in its own process group and tree-kill the WHOLE group on timeout
|
||||
# (no orphan grandchildren grinding the host CPU). off -> the prior
|
||||
# subprocess.run(timeout=) (ORCH_SUBPROCESS_TREE_KILL_ENABLED).
|
||||
# merge_retest_infra_tolerance_enabled -> D3: a re-test TIMEOUT is a transient
|
||||
# (bounded infra-retry, NOT a code-fault rollback to development burning a
|
||||
# developer retry). off -> timeout = the prior rollback
|
||||
# (ORCH_MERGE_RETEST_INFRA_TOLERANCE_ENABLED).
|
||||
# merge_retest_infra_max_retries -> D3: infra-retry budget before an
|
||||
# infra-alert (anti-loop). (ORCH_MERGE_RETEST_INFRA_MAX_RETRIES)
|
||||
# merge_retest_infra_retry_delay_s -> D3: delay before the staging-deployer
|
||||
# re-run. (ORCH_MERGE_RETEST_INFRA_RETRY_DELAY_S)
|
||||
# merge_retest_skip_when_current_enabled -> D4: skip the local re-test when the
|
||||
# pre-merge rebase was a PROVEN no-op (branch already at origin/main; HEAD is
|
||||
# exactly the CI/tester/staging-validated commit). off -> always re-test
|
||||
# after rebase (ORCH_MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED).
|
||||
# The tree-kill grace reuses the existing agent_kill_grace_seconds (no new key).
|
||||
subprocess_tree_kill_enabled: bool = True
|
||||
merge_retest_infra_tolerance_enabled: bool = True
|
||||
merge_retest_infra_max_retries: int = 2
|
||||
merge_retest_infra_retry_delay_s: int = 120
|
||||
merge_retest_skip_when_current_enabled: bool = True
|
||||
|
||||
# ORCH-036: executable self-deploy (deploy stage drives the host hook).
|
||||
# The `deploy` stage for the self-hosting repo is turned into a REAL prod
|
||||
# restart via a detached host process, gated by a manual approve. Three-phase
|
||||
|
||||
@@ -44,7 +44,8 @@ Invariants (ADR-001 §7, never broken):
|
||||
artefact / decides. It never calls the deploy hook, never restarts the prod
|
||||
container, never pushes / force-pushes ``main``.
|
||||
|
||||
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and lazily
|
||||
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` /
|
||||
``proc_group`` (the ORCH-110 stdlib-only process-group runner) and lazily
|
||||
``qg.checks.is_self_hosting_repo`` / ``db`` / ``notifications``; it never imports
|
||||
``stage_engine``.
|
||||
"""
|
||||
@@ -52,11 +53,11 @@ This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and laz
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from .config import settings
|
||||
from .git_worktree import ensure_worktree, get_worktree_path
|
||||
from .proc_group import run_in_process_group # ORCH-110 D1: tree-kill on timeout
|
||||
|
||||
logger = logging.getLogger("orchestrator.coverage_gate")
|
||||
|
||||
@@ -152,22 +153,25 @@ def measure_coverage(repo: str, branch: str) -> float | None:
|
||||
"-q",
|
||||
]
|
||||
timeout = settings.coverage_run_timeout_s
|
||||
try:
|
||||
subprocess.run(cmd, cwd=wt, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
# ORCH-110 (D1 / FR-2 / BR-3): run the coverage suite in its OWN process group so
|
||||
# a timeout tree-kills the WHOLE subtree (the sibling orphan-leak source of the
|
||||
# ORCH-109 incident), not just the direct child. The metric is read from the JSON
|
||||
# file regardless of the exit code, so a non-timeout spawn/OS error (returncode
|
||||
# None) just falls through to "no coverage json produced" -> None, byte-for-byte
|
||||
# the prior fail-open contract.
|
||||
res = run_in_process_group(
|
||||
cmd,
|
||||
cwd=wt,
|
||||
timeout=timeout,
|
||||
tree_kill=bool(getattr(settings, "subprocess_tree_kill_enabled", True)),
|
||||
grace_s=settings.agent_kill_grace_seconds,
|
||||
)
|
||||
if res.timed_out:
|
||||
logger.warning(
|
||||
"measure_coverage: pytest --cov timed out after %ss for %s/%s",
|
||||
timeout, repo, branch,
|
||||
)
|
||||
return None
|
||||
except FileNotFoundError:
|
||||
logger.warning(
|
||||
"measure_coverage: pytest / pytest-cov not available for %s/%s", repo, branch
|
||||
)
|
||||
return None
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("measure_coverage: pytest --cov error for %s/%s: %s", repo, branch, e)
|
||||
return None
|
||||
|
||||
data = None
|
||||
try:
|
||||
|
||||
@@ -225,6 +225,11 @@ async def queue():
|
||||
"reaper": reaper.status(),
|
||||
"post_deploy": post_deploy.status(),
|
||||
"merge_verify": merge_gate.merge_verify_status(),
|
||||
# ORCH-110 (D6): merge-gate re-test infra-timeout observability (read-only) —
|
||||
# tolerance/skip/tree-kill flags + timeout/infra-retry/skip counters, so an
|
||||
# infra-timeout is distinguishable from a code-fault rollback. Additive block;
|
||||
# never-raise.
|
||||
"merge_gate": merge_gate.merge_gate_status(),
|
||||
# ORCH-026 (G-2): declarative task-dependency observability (read-only,
|
||||
# NOT a source of truth) — declared edges, blocked tasks, detected cycle.
|
||||
"task_deps": task_deps.snapshot(),
|
||||
|
||||
@@ -36,6 +36,7 @@ import time
|
||||
|
||||
from .config import settings
|
||||
from .git_worktree import ensure_worktree, get_worktree_path
|
||||
from .proc_group import run_in_process_group # ORCH-110 D1: tree-kill on timeout
|
||||
|
||||
logger = logging.getLogger("orchestrator.merge_gate")
|
||||
|
||||
@@ -175,17 +176,103 @@ def auto_rebase_onto_main(repo: str, branch: str) -> tuple[bool, str]:
|
||||
# ---------------------------------------------------------------------------
|
||||
# re-test in the caught-up worktree
|
||||
# ---------------------------------------------------------------------------
|
||||
def _resolve_retest_timeout() -> int:
|
||||
"""Resolve the re-test wall-clock budget (ORCH-110 D5), never-break.
|
||||
|
||||
``int(settings.merge_retest_timeout_s)`` when it is a positive int; otherwise
|
||||
the safe default (900) + WARNING (mirrors ``launcher._resolve_timeout``). A
|
||||
malformed / non-positive config never reaches ``subprocess`` (FR-3 / AC-5).
|
||||
"""
|
||||
default = 900
|
||||
try:
|
||||
v = int(settings.merge_retest_timeout_s)
|
||||
if v > 0:
|
||||
return v
|
||||
logger.warning(
|
||||
"Non-positive merge_retest_timeout_s=%r; using default %ss",
|
||||
settings.merge_retest_timeout_s, default,
|
||||
)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"Invalid merge_retest_timeout_s=%r; using default %ss",
|
||||
settings.merge_retest_timeout_s, default,
|
||||
)
|
||||
return default
|
||||
|
||||
|
||||
def head_sha(repo: str, branch: str) -> str:
|
||||
"""Return ``git rev-parse HEAD`` of the branch worktree, or ``""`` (ORCH-110 D4).
|
||||
|
||||
Used by ``check_branch_mergeable`` to detect a no-op rebase (HEAD unchanged ->
|
||||
branch already contained the latest origin/main). Reads ONLY the existing
|
||||
per-branch worktree (``get_worktree_path``; the worktree exists by the
|
||||
deploy-staging->deploy edge) — it never CREATES one, so on an absent worktree it
|
||||
safely returns ``""`` (the no-op-skip then never triggers; re-test runs —
|
||||
fail-safe). Never-raise: any git/OS error -> ``""``.
|
||||
"""
|
||||
try:
|
||||
wt = get_worktree_path(repo, branch)
|
||||
if not os.path.isdir(wt):
|
||||
return ""
|
||||
r = subprocess.run(
|
||||
["git", "-C", wt, "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, timeout=_SHORT_TIMEOUT,
|
||||
)
|
||||
if r.returncode != 0:
|
||||
return ""
|
||||
return (r.stdout or "").strip()
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("head_sha error for %s/%s: %s", repo, branch, e)
|
||||
return ""
|
||||
|
||||
|
||||
def classify_retest_failure(reason: str) -> str:
|
||||
"""Classify a merge-gate FAIL ``reason`` (ORCH-110 D2). Pure, never-raise.
|
||||
|
||||
The single point for the 'magic string' of the re-test outcome so the routing
|
||||
in ``stage_engine._handle_merge_gate`` can distinguish an INFRA timeout from a
|
||||
real RED re-test without scattering ``"timeout" in reason`` checks:
|
||||
|
||||
* ``"timeout"`` — the re-test hit its wall-clock budget (infra/resource);
|
||||
``check_branch_mergeable`` returns ``"re-test timeout after <T>s"``.
|
||||
* ``"red"`` — a deterministically red re-test (a real code defect):
|
||||
``"re-test failed after rebase: ..."``.
|
||||
* ``"lock-busy"`` — merge-lock contention (``"merge-lock busy"``).
|
||||
* ``"other"`` — rebase conflict / setup error / anything else.
|
||||
|
||||
SCOPE (ADR D2): ONLY the re-test timeout is an infra-tolerated transient.
|
||||
``auto_rebase_onto_main``'s own ``"rebase timeout"`` is a DIFFERENT timeout (git
|
||||
hung) and stays on the rollback path — without a successful rebase the branch
|
||||
cannot be caught up to ``main``, so a merge is impossible by construction.
|
||||
"""
|
||||
try:
|
||||
r = (reason or "").strip().lower()
|
||||
if r == "merge-lock busy":
|
||||
return "lock-busy"
|
||||
if "re-test timeout" in r:
|
||||
return "timeout"
|
||||
if r.startswith("re-test failed"):
|
||||
return "red"
|
||||
return "other"
|
||||
except Exception: # noqa: BLE001 - never-raise; unknown -> other (safe: rollback)
|
||||
return "other"
|
||||
|
||||
|
||||
def retest_branch(repo: str, branch: str) -> tuple[bool, str]:
|
||||
"""Run the project test-suite in the (already caught-up) branch worktree.
|
||||
|
||||
Command: ``python -m pytest <merge_retest_target>`` (default ``tests/``),
|
||||
matching the orchestrator CI / check_tests_local pattern. Bounded by
|
||||
``settings.merge_retest_timeout_s``.
|
||||
matching the orchestrator CI / check_tests_local pattern. Bounded by the
|
||||
validated ``_resolve_retest_timeout()`` budget (ORCH-110 D5).
|
||||
|
||||
Returns:
|
||||
ORCH-110 (D1 / FR-2): the suite is spawned in its OWN process group via
|
||||
``run_in_process_group`` so a timeout tree-kills the WHOLE subtree (no orphan
|
||||
grandchildren grinding the host CPU). The RETURN contract is byte-for-byte the
|
||||
prior one — only the side effect (no leak) changes:
|
||||
* ``(True, "re-test green")`` — pytest rc == 0
|
||||
* ``(False, "re-test timeout after <T>s")`` — exceeded the timeout (AC-6)
|
||||
* ``(False, "re-test timeout after <T>s")`` — exceeded the timeout (AC-4/AC-6)
|
||||
* ``(False, "re-test failed: ...<tail>")`` — non-zero rc, with output tail
|
||||
* ``(False, "re-test error: ...")`` — spawn / OS error (not a timeout)
|
||||
Never-raise (AC-9): any setup/OS error -> ``(False, "<reason>")``.
|
||||
"""
|
||||
wt = get_worktree_path(repo, branch)
|
||||
@@ -197,21 +284,25 @@ def retest_branch(repo: str, branch: str) -> tuple[bool, str]:
|
||||
return False, f"re-test setup error: {e}"
|
||||
|
||||
target = settings.merge_retest_target or "tests/"
|
||||
timeout = settings.merge_retest_timeout_s
|
||||
try:
|
||||
r = subprocess.run(
|
||||
["python", "-m", "pytest", target, "-q"],
|
||||
cwd=wt, capture_output=True, text=True, timeout=timeout,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
timeout = _resolve_retest_timeout()
|
||||
res = run_in_process_group(
|
||||
["python", "-m", "pytest", target, "-q"],
|
||||
cwd=wt,
|
||||
timeout=timeout,
|
||||
tree_kill=bool(getattr(settings, "subprocess_tree_kill_enabled", True)),
|
||||
grace_s=settings.agent_kill_grace_seconds,
|
||||
)
|
||||
if res.timed_out:
|
||||
logger.warning("retest_branch: timeout (%ss) on %s/%s", timeout, repo, branch)
|
||||
return False, f"re-test timeout after {timeout}s"
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
return False, f"re-test error: {e}"
|
||||
|
||||
if r.returncode == 0:
|
||||
if res.returncode is None:
|
||||
# Spawn / OS error that was NOT a timeout -> preserve the prior error reason.
|
||||
detail = (res.stderr or "").strip()[:200]
|
||||
logger.warning("retest_branch: spawn error on %s/%s: %s", repo, branch, detail)
|
||||
return False, f"re-test error: {detail}"
|
||||
if res.returncode == 0:
|
||||
return True, "re-test green"
|
||||
tail = ((r.stdout or "") + (r.stderr or ""))[-500:]
|
||||
tail = ((res.stdout or "") + (res.stderr or ""))[-500:]
|
||||
logger.warning("retest_branch: red on %s/%s", repo, branch)
|
||||
return False, f"re-test failed: ...{tail}"
|
||||
|
||||
@@ -576,6 +667,82 @@ def merge_verify_status() -> dict:
|
||||
return {"enabled": False}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-110 (D6): re-test infra-timeout observability counters (read-only).
|
||||
#
|
||||
# In-process counters (mirror _MERGE_VERIFY_COUNTERS): reset only on process start,
|
||||
# surfaced read-only via merge_gate_status() in GET /queue. NEVER the source of any
|
||||
# decision — purely informational, so an infra timeout is distinguishable from a
|
||||
# code-fault rollback in the snapshot. never-raise.
|
||||
# ---------------------------------------------------------------------------
|
||||
_MERGE_GATE_COUNTERS: dict = {
|
||||
"retest_timeout_total": 0, # re-test hit its wall-clock budget
|
||||
"retest_infra_retry_total": 0, # infra-timeout -> bounded re-queue (D3)
|
||||
"retest_infra_exhausted_total": 0, # infra-retry budget exhausted -> infra-alert
|
||||
"retest_skipped_current_total": 0, # re-test skipped on a no-op rebase (D4)
|
||||
"last_infra_timeout_wi": None,
|
||||
}
|
||||
|
||||
|
||||
def note_retest_timeout(work_item_id: str | None = None) -> None:
|
||||
"""Bump the 're-test timeout' counter (observability only). Never raises."""
|
||||
try:
|
||||
_MERGE_GATE_COUNTERS["retest_timeout_total"] += 1
|
||||
_MERGE_GATE_COUNTERS["last_infra_timeout_wi"] = work_item_id
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def note_retest_infra_retry() -> None:
|
||||
"""Bump the 'infra-timeout re-queue' counter (observability only). Never raises."""
|
||||
try:
|
||||
_MERGE_GATE_COUNTERS["retest_infra_retry_total"] += 1
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def note_retest_infra_exhausted() -> None:
|
||||
"""Bump the 'infra-retry exhausted' counter (observability only). Never raises."""
|
||||
try:
|
||||
_MERGE_GATE_COUNTERS["retest_infra_exhausted_total"] += 1
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def note_retest_skipped_current() -> None:
|
||||
"""Bump the 're-test skipped (no-op rebase)' counter (D4). Never raises."""
|
||||
try:
|
||||
_MERGE_GATE_COUNTERS["retest_skipped_current_total"] += 1
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def merge_gate_status() -> dict:
|
||||
"""Read-only snapshot of the ORCH-110 merge-gate re-test path for GET /queue.
|
||||
|
||||
Additive block; existing /queue keys are untouched. Surfaces the flags +
|
||||
counters so an infra-timeout (retry/exhaustion) is distinguishable from a
|
||||
code-fault rollback. Never the source of any decision. never-raise.
|
||||
"""
|
||||
try:
|
||||
return {
|
||||
"infra_tolerance_enabled": bool(settings.merge_retest_infra_tolerance_enabled),
|
||||
"infra_max_retries": settings.merge_retest_infra_max_retries,
|
||||
"infra_retry_delay_s": settings.merge_retest_infra_retry_delay_s,
|
||||
"skip_when_current_enabled": bool(settings.merge_retest_skip_when_current_enabled),
|
||||
"tree_kill_enabled": bool(settings.subprocess_tree_kill_enabled),
|
||||
"retest_timeout_s": settings.merge_retest_timeout_s,
|
||||
"retest_timeout_total": _MERGE_GATE_COUNTERS["retest_timeout_total"],
|
||||
"retest_infra_retry_total": _MERGE_GATE_COUNTERS["retest_infra_retry_total"],
|
||||
"retest_infra_exhausted_total": _MERGE_GATE_COUNTERS["retest_infra_exhausted_total"],
|
||||
"retest_skipped_current_total": _MERGE_GATE_COUNTERS["retest_skipped_current_total"],
|
||||
"last_infra_timeout_wi": _MERGE_GATE_COUNTERS["last_infra_timeout_wi"],
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning("merge_gate_status error: %s", e)
|
||||
return {"infra_tolerance_enabled": False}
|
||||
|
||||
|
||||
def merge_verify_applies(repo: str) -> bool:
|
||||
"""Whether the ORCH-071 merge-verify under-gate is REAL for this repo.
|
||||
|
||||
@@ -1037,6 +1204,7 @@ MAIN_REGRESSION_MARKERS: list[tuple[str, str, str]] = [
|
||||
("ORCH-073", "check_main_regression", "src/merge_gate.py"),
|
||||
("ORCH-082", "ensure_open_pr", "src/merge_gate.py"),
|
||||
("ORCH-093", "_classify_merge_response", "src/merge_gate.py"),
|
||||
("ORCH-110", "classify_retest_failure", "src/merge_gate.py"),
|
||||
]
|
||||
|
||||
|
||||
|
||||
179
src/proc_group.py
Normal file
179
src/proc_group.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""Process-group runner with tree-kill on timeout (ORCH-110 D1).
|
||||
|
||||
Background
|
||||
----------
|
||||
The orchestrator spawns its OWN pytest subprocesses OUTSIDE the LLM-agent path:
|
||||
``merge_gate.retest_branch`` (the merge-gate re-test) and
|
||||
``coverage_gate.measure_coverage`` (the coverage measurement). Both used a plain
|
||||
``subprocess.run(..., timeout=...)``. On ``TimeoutExpired`` CPython kills only the
|
||||
DIRECT child (``proc.kill()``); any grandchildren (pytest-xdist workers, tests that
|
||||
spawn their own processes) are re-parented to PID 1 and keep running — for days,
|
||||
starving the host CPU. That orphan leak is a ROOT cause of the ORCH-109 / PR #129
|
||||
incident (a 516s suite blew its 600s budget under CPU starvation -> a false
|
||||
merge-gate rollback after a green CI).
|
||||
|
||||
This leaf gives those orchestrator-spawned runs the SAME SIGTERM->grace->SIGKILL
|
||||
cascade the LLM agents already get from ``launcher.stop_process`` — but at the
|
||||
process-GROUP level so the WHOLE subtree dies on a timeout:
|
||||
|
||||
* the child becomes the leader of a NEW session / process group
|
||||
(``start_new_session=True`` -> ``setsid``), so every descendant shares
|
||||
``pgid == child.pid``;
|
||||
* on timeout the whole group is killed with ``os.killpg`` (SIGTERM, grace poll,
|
||||
then SIGKILL), then reaped (no zombies).
|
||||
|
||||
Invariants:
|
||||
* **stdlib-only leaf** — imports only ``os`` / ``signal`` / ``subprocess`` /
|
||||
``time`` / ``logging`` / ``dataclasses``; NEVER another ``src/*`` module
|
||||
(mirrors the purity of ``serial_gate`` / ``staging_verdict``).
|
||||
* **never-raise** — any OS error degrades to a safe ``ProcResult`` (the caller maps
|
||||
it to its existing contract); an exception never escapes.
|
||||
* **never-break fallback** — with ``tree_kill=False`` OR on a platform without
|
||||
``os.killpg`` / ``os.setsid`` (non-POSIX) it degrades to the prior
|
||||
``subprocess.run(cmd, ..., timeout=timeout)`` (byte-for-byte the old behaviour;
|
||||
prod is Linux and never hits the fallback).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
logger = logging.getLogger("orchestrator.proc_group")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProcResult:
|
||||
"""Outcome of a process-group run (ORCH-110 D1).
|
||||
|
||||
``returncode`` is the child's exit code, or ``None`` when the process could not
|
||||
be reaped with a real code (timeout-kill / spawn error). ``timed_out`` is True
|
||||
iff the command exceeded its wall-clock budget (and the whole group was killed).
|
||||
"""
|
||||
|
||||
returncode: int | None
|
||||
stdout: str
|
||||
stderr: str
|
||||
timed_out: bool
|
||||
|
||||
|
||||
def _tree_kill_supported() -> bool:
|
||||
"""POSIX process-group ops available? (non-POSIX -> fall back to subprocess.run)."""
|
||||
return hasattr(os, "killpg") and hasattr(os, "getpgid") and hasattr(os, "setsid")
|
||||
|
||||
|
||||
def _kill_group(pid: int, grace_s: float) -> None:
|
||||
"""SIGTERM -> grace -> SIGKILL the whole process GROUP led by ``pid``.
|
||||
|
||||
Mirrors ``launcher.stop_process`` but targets the process GROUP (``os.killpg``)
|
||||
so grandchildren die too. ``ProcessLookupError`` is tolerated at every step (the
|
||||
group may already be gone). Never raises.
|
||||
"""
|
||||
try:
|
||||
pgid = os.getpgid(pid)
|
||||
except (ProcessLookupError, OSError):
|
||||
return
|
||||
|
||||
# Phase 1: SIGTERM the whole group (graceful).
|
||||
try:
|
||||
os.killpg(pgid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
except OSError as e: # noqa: BLE001 - never-raise
|
||||
logger.warning("proc_group: SIGTERM killpg(%s) error: %s", pgid, e)
|
||||
|
||||
# Phase 2: poll for graceful group exit within the grace window.
|
||||
poll = 0.2
|
||||
waited = 0.0
|
||||
while waited < max(0.0, grace_s):
|
||||
time.sleep(poll)
|
||||
waited += poll
|
||||
try:
|
||||
os.killpg(pgid, 0) # signal 0 = liveness probe of the group
|
||||
except ProcessLookupError:
|
||||
return # whole group gone
|
||||
except OSError:
|
||||
break
|
||||
|
||||
# Phase 3: still alive -> hard SIGKILL the group.
|
||||
try:
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
except OSError as e: # noqa: BLE001 - never-raise
|
||||
logger.warning("proc_group: SIGKILL killpg(%s) error: %s", pgid, e)
|
||||
|
||||
|
||||
def _run_plain(cmd, *, cwd, timeout, env=None) -> ProcResult:
|
||||
"""Fallback path: prior ``subprocess.run`` semantics (never-break)."""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
cmd, cwd=cwd, env=env, capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
return ProcResult(
|
||||
returncode=r.returncode, stdout=r.stdout or "", stderr=r.stderr or "",
|
||||
timed_out=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return ProcResult(returncode=None, stdout="", stderr="", timed_out=True)
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("proc_group(_run_plain): error for %s: %s", cmd, e)
|
||||
return ProcResult(returncode=None, stdout="", stderr=str(e), timed_out=False)
|
||||
|
||||
|
||||
def run_in_process_group(
|
||||
cmd, *, cwd, timeout, env=None, grace_s: float = 5.0, tree_kill: bool = True
|
||||
) -> ProcResult:
|
||||
"""Run ``cmd`` in its own process group; tree-kill the whole group on timeout.
|
||||
|
||||
See the module docstring. Returns a ``ProcResult``; ``timed_out=True`` iff the
|
||||
command exceeded ``timeout`` (and the whole group was killed). Never raises.
|
||||
|
||||
Fallback (never-break): ``tree_kill=False`` or a non-POSIX platform -> a plain
|
||||
``subprocess.run(cmd, ..., timeout=timeout)`` (the prior behaviour).
|
||||
"""
|
||||
if not tree_kill or not _tree_kill_supported():
|
||||
return _run_plain(cmd, cwd=cwd, timeout=timeout, env=env)
|
||||
|
||||
try:
|
||||
proc = subprocess.Popen( # noqa: S603 - cmd is an internal, fixed argv
|
||||
cmd,
|
||||
cwd=cwd,
|
||||
env=env,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
start_new_session=True, # setsid -> child leads a new process group
|
||||
)
|
||||
except (OSError, ValueError) as e:
|
||||
logger.warning("proc_group: spawn error for %s: %s", cmd, e)
|
||||
return ProcResult(returncode=None, stdout="", stderr=str(e), timed_out=False)
|
||||
|
||||
try:
|
||||
out, err = proc.communicate(timeout=timeout)
|
||||
return ProcResult(
|
||||
returncode=proc.returncode, stdout=out or "", stderr=err or "",
|
||||
timed_out=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
# Tree-kill the WHOLE group (children + grandchildren), then reap to avoid
|
||||
# leaving a zombie behind.
|
||||
_kill_group(proc.pid, grace_s)
|
||||
out, err = "", ""
|
||||
try:
|
||||
out, err = proc.communicate(timeout=grace_s + 5.0)
|
||||
except (subprocess.TimeoutExpired, OSError, ValueError):
|
||||
try:
|
||||
proc.wait(timeout=grace_s + 5.0)
|
||||
except (subprocess.TimeoutExpired, OSError, ValueError):
|
||||
pass
|
||||
return ProcResult(
|
||||
returncode=proc.returncode, stdout=out or "", stderr=err or "",
|
||||
timed_out=True,
|
||||
)
|
||||
except (OSError, ValueError) as e: # noqa: BLE001 - never-raise
|
||||
logger.warning("proc_group: communicate error for %s: %s", cmd, e)
|
||||
_kill_group(proc.pid, grace_s)
|
||||
return ProcResult(returncode=proc.returncode, stdout="", stderr=str(e), timed_out=False)
|
||||
@@ -708,10 +708,38 @@ def check_branch_mergeable(repo: str, work_item_id: str, branch: str) -> tuple[b
|
||||
logger.info("check_branch_mergeable: %s up-to-date with main", branch)
|
||||
return True, "branch up-to-date with main"
|
||||
|
||||
# ORCH-110 (D4): capture HEAD before/after the rebase to detect a no-op
|
||||
# rebase (branch already contained the latest origin/main).
|
||||
pre_sha = merge_gate.head_sha(repo, branch)
|
||||
ok, rb_reason = merge_gate.auto_rebase_onto_main(repo, branch)
|
||||
if not ok:
|
||||
merge_gate.release_merge_lease(repo, branch)
|
||||
return False, rb_reason # "rebase conflict: ..."
|
||||
post_sha = merge_gate.head_sha(repo, branch)
|
||||
|
||||
# ORCH-110 (D4 / FR-4 / AC-6): re-test contract. ORCH-043 catches a
|
||||
# SEMANTIC merge conflict that can only arise when ``main`` actually moved
|
||||
# and the branch was really rebased onto new commits. When the rebase was
|
||||
# a PROVEN no-op (HEAD unchanged), there is no "moved main" -> the local
|
||||
# re-test re-checks exactly the commit CI + tester + staging already
|
||||
# validated on THIS HEAD -> it is a redundant single point of false
|
||||
# failure (the ORCH-109 timeout incident). Skip it ONLY on a proven no-op
|
||||
# (both SHAs non-empty AND equal); on ANY uncertainty (empty SHA / flag
|
||||
# off) the re-test runs (fail-safe to BR-6/AC-3). This extends to the
|
||||
# premerge_rebase_always=True path the same optimisation the
|
||||
# premerge_rebase_always=False not-behind short-circuit already has.
|
||||
if (
|
||||
bool(getattr(settings, "merge_retest_skip_when_current_enabled", False))
|
||||
and pre_sha and post_sha and pre_sha == post_sha
|
||||
):
|
||||
logger.info(
|
||||
"check_branch_mergeable: %s rebase no-op (HEAD %s unchanged) -> "
|
||||
"re-test skipped (HEAD CI-validated)", branch, pre_sha[:8],
|
||||
)
|
||||
merge_gate.note_retest_skipped_current()
|
||||
return True, (
|
||||
"branch up-to-date (re-test skipped: rebase no-op, HEAD CI-validated)"
|
||||
)
|
||||
|
||||
ok_t, t_reason = merge_gate.retest_branch(repo, branch)
|
||||
if ok_t:
|
||||
|
||||
@@ -995,6 +995,22 @@ def _handle_merge_gate(
|
||||
)
|
||||
return True
|
||||
|
||||
# ORCH-110 (D2/D3): classify the FAIL cause. An INFRA re-test TIMEOUT (the
|
||||
# ORCH-109 incident: green CI + green tester, but the local re-test blew its
|
||||
# wall-clock budget under CPU starvation) is a TRANSIENT, NOT a code fault — it
|
||||
# must NOT take the _handle_merge_gate_rollback path (rollback to development +
|
||||
# burn a developer retry that nobody can fix). Route it to a bounded infra-retry
|
||||
# instead. A deterministically RED re-test / conflict still rolls back (BR-6 /
|
||||
# AC-3). Kill-switch off -> byte-for-byte the prior behaviour (timeout ->
|
||||
# rollback), NFR-2.
|
||||
if merge_gate.classify_retest_failure(reason) == "timeout":
|
||||
merge_gate.note_retest_timeout(work_item_id)
|
||||
if settings.merge_retest_infra_tolerance_enabled:
|
||||
_handle_merge_gate_infra_retry(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result
|
||||
)
|
||||
return True
|
||||
|
||||
_handle_merge_gate_rollback(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result
|
||||
)
|
||||
@@ -1044,6 +1060,110 @@ def _handle_merge_gate_defer(
|
||||
)
|
||||
|
||||
|
||||
def _merge_infra_retry_count(task_id: int) -> int:
|
||||
"""How many times this task was re-queued by the merge-gate infra-timeout path.
|
||||
|
||||
Restart-safe (mirror of ``_merge_defer_count``): counted from the persisted jobs
|
||||
queue by the infra-retry marker in task_content, so a service restart never
|
||||
resets the infra-retry budget (ORCH-110 D3 / AC-9).
|
||||
"""
|
||||
conn = get_db()
|
||||
n = conn.execute(
|
||||
"SELECT COUNT(*) FROM jobs WHERE task_id=? AND "
|
||||
"task_content LIKE '%merge-gate infra-timeout retry%'",
|
||||
(task_id,),
|
||||
).fetchone()[0]
|
||||
conn.close()
|
||||
return n
|
||||
|
||||
|
||||
def _handle_merge_gate_infra_retry(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result: AdvanceResult
|
||||
):
|
||||
"""ORCH-110 (D3): merge-gate re-test INFRA timeout -> bounded infra-retry.
|
||||
|
||||
Mirror of ``_handle_merge_gate_defer`` (NOT ``_handle_merge_gate_rollback``): the
|
||||
task STAYS on deploy-staging, the staging-deployer is re-queued after a delay, and
|
||||
there is NO ``update_task_stage("development")``, NO developer-retry increment and
|
||||
NO code-fault QG-failure semantics — so a green-CI + green-tester task is never
|
||||
falsely sent back to development for an infra hiccup (BR-1 / AC-1). The merge lease
|
||||
was already released by ``check_branch_mergeable`` on the timeout, so the re-run
|
||||
re-acquires it normally (defer-path parity).
|
||||
|
||||
Bounded (anti-loop, NFR-5 / AC-9): after ``merge_retest_infra_max_retries``
|
||||
re-tries the task is blocked with an INFRA-specific alert (a resource problem,
|
||||
explicitly NOT "developer must fix"), so it never bounces forever. The first
|
||||
timeout already tree-killed the orphan processes (D1), so the next re-test usually
|
||||
passes — the retry is recovery, not masking. The task does NOT move to development.
|
||||
|
||||
never-raise (TC-09): any unexpected error in this transient path is logged with a
|
||||
WARNING and swallowed (the task simply stays on deploy-staging for the reconciler /
|
||||
reaper to re-drive) so an exception never escapes into ``advance_stage``.
|
||||
"""
|
||||
try:
|
||||
_merge_gate_infra_retry_impl(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 - never-raise; do not crash advance_stage
|
||||
logger.warning(
|
||||
f"Task {task_id}: merge-gate infra-retry path error: {e} "
|
||||
f"(swallowed; task stays on deploy-staging)"
|
||||
)
|
||||
result.note = "merge-gate-infra-retry-error"
|
||||
|
||||
|
||||
def _merge_gate_infra_retry_impl(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result: AdvanceResult
|
||||
):
|
||||
"""Body of the merge-gate infra-retry (ORCH-110 D3); wrapped by the never-raise
|
||||
``_handle_merge_gate_infra_retry``."""
|
||||
retries = _merge_infra_retry_count(task_id)
|
||||
if retries < settings.merge_retest_infra_max_retries:
|
||||
task_desc = (
|
||||
f"Work item: {work_item_id}\nRepo: {repo}\nBranch: {branch}\n"
|
||||
f"Stage: deploy-staging\nNote: merge-gate infra-timeout retry "
|
||||
f"(attempt {retries + 1}/{settings.merge_retest_infra_max_retries}) — "
|
||||
f"re-test hit the wall-clock budget (infra/resource, NOT a code defect); "
|
||||
f"retrying after {settings.merge_retest_infra_retry_delay_s}s. Reason: {reason}."
|
||||
)
|
||||
new_job = enqueue_job(
|
||||
"deployer", repo, task_desc, task_id=task_id,
|
||||
available_at_delay_s=settings.merge_retest_infra_retry_delay_s,
|
||||
)
|
||||
merge_gate.note_retest_infra_retry()
|
||||
result.enqueued_agent = "deployer"
|
||||
result.enqueued_job_id = new_job
|
||||
result.note = "merge-gate-infra-retry"
|
||||
logger.warning(
|
||||
f"Task {task_id}: merge-gate re-test infra-timeout, re-queued deployer "
|
||||
f"(job_id={new_job}, attempt {retries + 1}/"
|
||||
f"{settings.merge_retest_infra_max_retries}); NOT a code fault ({reason})"
|
||||
)
|
||||
else:
|
||||
merge_gate.note_retest_infra_exhausted()
|
||||
set_issue_blocked(work_item_id)
|
||||
send_telegram(
|
||||
f"⏳ {link_for(work_item_id)}: merge-gate re-test infra-timeout "
|
||||
f"сохраняется после {settings.merge_retest_infra_max_retries} повторов — "
|
||||
f"ресурсная проблема (CPU / осиротевшие процессы), НЕ дефект кода. "
|
||||
f"Проверьте хост / нужно ручное вмешательство."
|
||||
)
|
||||
plane_add_comment(
|
||||
work_item_id,
|
||||
f"⏳ Merge-gate re-test infra-timeout сохраняется после "
|
||||
f"{settings.merge_retest_infra_max_retries} повторов — инфраструктурная "
|
||||
f"проблема (ресурсы / осиротевшие процессы), НЕ дефект кода. Задача "
|
||||
f"остаётся на deploy-staging; нужно ручное вмешательство (НЕ доработка кода).",
|
||||
author="deployer",
|
||||
)
|
||||
result.alerted = True
|
||||
result.note = "merge-gate-infra-retry-exhausted"
|
||||
logger.error(
|
||||
f"Task {task_id}: merge-gate re-test infra-timeout exhausted "
|
||||
f"({settings.merge_retest_infra_max_retries} retries) ({reason})"
|
||||
)
|
||||
|
||||
|
||||
def _handle_merge_gate_rollback(
|
||||
task_id, current_stage, repo, work_item_id, branch, reason, result: AdvanceResult
|
||||
):
|
||||
|
||||
@@ -56,7 +56,7 @@ def test_merge_gate_settings_defaults(monkeypatch):
|
||||
s = Settings()
|
||||
assert s.merge_gate_enabled is True
|
||||
assert s.merge_gate_repos == ""
|
||||
assert s.merge_retest_timeout_s == 600
|
||||
assert s.merge_retest_timeout_s == 900 # ORCH-110 D5: raised 600 -> 900
|
||||
assert s.merge_retest_target == "tests/"
|
||||
assert s.merge_lock_timeout_s == 300
|
||||
assert s.merge_defer_delay_s == 60
|
||||
|
||||
@@ -425,12 +425,14 @@ def test_tc14_real_measurement(tmp_path, monkeypatch):
|
||||
|
||||
|
||||
def test_tc14_measure_timeout_returns_none(monkeypatch):
|
||||
import subprocess
|
||||
# ORCH-110: measure_coverage now runs via proc_group.run_in_process_group
|
||||
# (tree-kill on timeout). A timed_out ProcResult -> None (prior contract).
|
||||
from src.proc_group import ProcResult
|
||||
monkeypatch.setattr(cg, "ensure_worktree", lambda r, b: "/tmp")
|
||||
|
||||
def _timeout(*a, **k):
|
||||
raise subprocess.TimeoutExpired(cmd="pytest", timeout=1)
|
||||
monkeypatch.setattr(cg.subprocess, "run", _timeout)
|
||||
monkeypatch.setattr(
|
||||
cg, "run_in_process_group",
|
||||
lambda *a, **k: ProcResult(returncode=None, stdout="", stderr="", timed_out=True),
|
||||
)
|
||||
assert cg.measure_coverage(_REPO, _BRANCH) is None
|
||||
|
||||
|
||||
|
||||
@@ -203,10 +203,16 @@ def fake_worktree(tmp_path, monkeypatch):
|
||||
return str(wt)
|
||||
|
||||
|
||||
# ORCH-110: retest_branch now runs the suite via proc_group.run_in_process_group
|
||||
# (tree-kill on timeout). The seam is mocked with a ProcResult; the RETURN contract
|
||||
# is byte-for-byte the prior one.
|
||||
from src.proc_group import ProcResult # noqa: E402
|
||||
|
||||
|
||||
def test_tc07_retest_green(fake_worktree, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
merge_gate.subprocess, "run",
|
||||
lambda *a, **k: subprocess.CompletedProcess(a, 0, "1 passed", ""),
|
||||
merge_gate, "run_in_process_group",
|
||||
lambda *a, **k: ProcResult(returncode=0, stdout="1 passed", stderr="", timed_out=False),
|
||||
)
|
||||
ok, reason = merge_gate.retest_branch("orchestrator", "feature/x")
|
||||
assert ok is True
|
||||
@@ -215,9 +221,11 @@ def test_tc07_retest_green(fake_worktree, monkeypatch):
|
||||
|
||||
def test_tc08_retest_red_with_tail(fake_worktree, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
merge_gate.subprocess, "run",
|
||||
lambda *a, **k: subprocess.CompletedProcess(
|
||||
a, 1, "FAILED tests/test_x.py::t - AssertionError\n1 failed", ""
|
||||
merge_gate, "run_in_process_group",
|
||||
lambda *a, **k: ProcResult(
|
||||
returncode=1,
|
||||
stdout="FAILED tests/test_x.py::t - AssertionError\n1 failed",
|
||||
stderr="", timed_out=False,
|
||||
),
|
||||
)
|
||||
ok, reason = merge_gate.retest_branch("orchestrator", "feature/x")
|
||||
@@ -227,11 +235,11 @@ def test_tc08_retest_red_with_tail(fake_worktree, monkeypatch):
|
||||
|
||||
|
||||
def test_tc09_retest_timeout(fake_worktree, monkeypatch):
|
||||
def _boom(*a, **k):
|
||||
raise subprocess.TimeoutExpired(cmd="pytest", timeout=1)
|
||||
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_retest_timeout_s", 1)
|
||||
monkeypatch.setattr(merge_gate.subprocess, "run", _boom)
|
||||
monkeypatch.setattr(
|
||||
merge_gate, "run_in_process_group",
|
||||
lambda *a, **k: ProcResult(returncode=None, stdout="", stderr="", timed_out=True),
|
||||
)
|
||||
ok, reason = merge_gate.retest_branch("orchestrator", "feature/x")
|
||||
assert ok is False
|
||||
assert "re-test timeout" in reason
|
||||
|
||||
82
tests/test_orch110_budget_invariants.py
Normal file
82
tests/test_orch110_budget_invariants.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""ORCH-110 TC-08: re-test budget validation + cross-invariants (D5).
|
||||
|
||||
Covers FR-3 / AC-5 / NFR-6:
|
||||
* ``_resolve_retest_timeout`` validates the config (malformed / non-positive ->
|
||||
safe default 900 + WARNING; never reaches subprocess);
|
||||
* the budget was bumped 600 -> 900;
|
||||
* the cross-invariant ``reaper_max_running_s > Σ(deploy-staging gate-work) + grace``
|
||||
(ORCH-065/109) still holds with the new 900s re-test budget — WITHOUT raising
|
||||
``reaper_max_running_s``.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
os.environ.setdefault("ORCH_DB_PATH", os.path.join(tempfile.gettempdir(), "test_orch110_budget.db"))
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from src import merge_gate # noqa: E402
|
||||
from src.config import Settings, settings # noqa: E402
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _resolve_retest_timeout — validation (never-break).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc08_resolve_uses_positive_config(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_retest_timeout_s", 1234, raising=False)
|
||||
assert merge_gate._resolve_retest_timeout() == 1234
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", [0, -5, "abc", None, 3.0])
|
||||
def test_tc08_resolve_bad_config_falls_back_to_default(monkeypatch, bad):
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_retest_timeout_s", bad, raising=False)
|
||||
# 3.0 is a valid positive int(3) -> stays 3; everything else -> 900 default.
|
||||
out = merge_gate._resolve_retest_timeout()
|
||||
if bad == 3.0:
|
||||
assert out == 3
|
||||
else:
|
||||
assert out == 900
|
||||
|
||||
|
||||
def test_tc08_default_budget_bumped_to_900():
|
||||
"""D5: the shipped default budget is 900 (raised from 600)."""
|
||||
assert Settings().merge_retest_timeout_s == 900
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-invariant: reaper backstop covers the worst-case deploy-staging gate-work.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc08_reaper_covers_deploy_staging_worstcase():
|
||||
"""ORCH-065/109 invariant with the new 900s re-test budget (ADR D5 table)."""
|
||||
s = Settings()
|
||||
# Worst-case sum of work charged to a deploy-staging-deployer job (ADR D5).
|
||||
security = 120
|
||||
rebase = 120
|
||||
image = 600
|
||||
worst = (
|
||||
s.agent_timeout_seconds # deployer agent (1800)
|
||||
+ security
|
||||
+ rebase
|
||||
+ s.merge_retest_timeout_s # re-test (900, new)
|
||||
+ s.coverage_run_timeout_s # coverage (900)
|
||||
+ image
|
||||
+ s.agent_kill_grace_seconds # grace (20)
|
||||
)
|
||||
assert worst <= 4460 # matches the ADR D5 table
|
||||
assert s.reaper_max_running_s > worst, (
|
||||
f"reaper_max_running_s={s.reaper_max_running_s} must exceed "
|
||||
f"deploy-staging worst-case {worst}"
|
||||
)
|
||||
|
||||
|
||||
def test_tc08_reaper_still_covers_max_agent_timeout():
|
||||
"""ORCH-065/109: reaper_max_running_s > max(agent timeout) + grace (unchanged)."""
|
||||
s = Settings()
|
||||
assert s.reaper_max_running_s > s.agent_timeout_developer_s + s.agent_kill_grace_seconds
|
||||
|
||||
|
||||
def test_tc08_reaper_max_running_s_unchanged():
|
||||
"""D5 must NOT change reaper_max_running_s (stays 5400 from ORCH-109)."""
|
||||
assert settings.reaper_max_running_s == 5400
|
||||
213
tests/test_orch110_false_rollback_regression.py
Normal file
213
tests/test_orch110_false_rollback_regression.py
Normal file
@@ -0,0 +1,213 @@
|
||||
"""ORCH-110 TC-10: regression of the ORCH-109 / PR #129 incident.
|
||||
|
||||
Incident: tester PASS + green CI + the branch not-behind, but the merge-gate local
|
||||
re-test blew its wall-clock budget under CPU starvation -> ``check_branch_mergeable``
|
||||
returned ``(False, "re-test timeout ...")`` -> the engine routed it to
|
||||
``_handle_merge_gate_rollback`` (rollback deploy-staging -> development + a developer
|
||||
retry) -> every retry timed out the same way -> "Merge-gate still failing after 3
|
||||
developer retries" -> a stuck task needing manual intervention.
|
||||
|
||||
This drives the REAL ``check_branch_mergeable`` through ``advance_stage`` (only the
|
||||
git/test primitives are mocked) and asserts the incident can no longer happen:
|
||||
|
||||
* Scenario A (D4) — a not-behind branch (no-op rebase) on a green-CI HEAD: the
|
||||
local re-test is SKIPPED entirely -> the gate passes -> advance to deploy.
|
||||
* Scenario B (D3) — a real catch-up whose re-test times out: a bounded infra-retry
|
||||
(task stays on deploy-staging), NEVER a rollback to development and NEVER the
|
||||
"Merge-gate still failing after N developer retries" alert.
|
||||
|
||||
RED-before / GREEN-after: on pre-ORCH-110 code both scenarios would roll the task
|
||||
back to development (Scenario A would even run the doomed re-test), so the
|
||||
``rolled_back_to is None`` / ``stage == deploy-staging`` assertions below would FAIL.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
|
||||
_test_db = os.path.join(tempfile.gettempdir(), "test_orch110_regression.db")
|
||||
os.environ["ORCH_DB_PATH"] = _test_db
|
||||
os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir()
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
from unittest.mock import MagicMock # noqa: E402
|
||||
|
||||
import src.db as _db # noqa: E402
|
||||
from src.db import init_db, get_db # noqa: E402
|
||||
from src import stage_engine # noqa: E402
|
||||
from src import merge_gate # noqa: E402
|
||||
from src.qg import checks as qg # noqa: E402
|
||||
from src.stage_engine import advance_stage # noqa: E402
|
||||
|
||||
_REPO = "orchestrator"
|
||||
_WI = "ORCH-110"
|
||||
_BRANCH = "feature/ORCH-110-x"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_db(monkeypatch):
|
||||
monkeypatch.setattr(_db.settings, "db_path", _test_db)
|
||||
if os.path.exists(_test_db):
|
||||
os.unlink(_test_db)
|
||||
init_db()
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def silence(monkeypatch):
|
||||
for name in (
|
||||
"notify_stage_change", "notify_qg_failure", "send_telegram",
|
||||
"plane_notify_stage", "plane_notify_qg", "plane_add_comment",
|
||||
"set_issue_in_progress", "set_issue_blocked", "notify_approve_requested",
|
||||
"set_issue_in_review", "set_issue_needs_input",
|
||||
):
|
||||
monkeypatch.setattr(stage_engine, name, MagicMock())
|
||||
# Keep the merge-gate the only intervening sub-gate on the edge; no Phase A.
|
||||
monkeypatch.setattr(stage_engine.settings, "deploy_require_manual_approve", False)
|
||||
# Real gate scope: orchestrator self-hosting.
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_enabled", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_repos", "", raising=False)
|
||||
monkeypatch.setattr(qg.settings, "premerge_rebase_always", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_retest_skip_when_current_enabled", True, raising=False)
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_tolerance_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_max_retries", 2)
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_retry_delay_s", 120)
|
||||
|
||||
|
||||
def _pass(*a, **k):
|
||||
return (True, "ok")
|
||||
|
||||
|
||||
def _patch_edge_gates_except_merge(monkeypatch):
|
||||
"""All edge gates pass EXCEPT check_branch_mergeable, which stays the REAL one."""
|
||||
patched = {**stage_engine.QG_CHECKS}
|
||||
patched["check_staging_status"] = _pass
|
||||
patched["check_security_gate"] = _pass
|
||||
patched["check_coverage_gate"] = _pass
|
||||
patched["check_staging_image_fresh"] = _pass
|
||||
monkeypatch.setattr(stage_engine, "QG_CHECKS", patched)
|
||||
|
||||
|
||||
def _mock_merge_primitives(monkeypatch, *, head_shas, retest_result, retest_calls):
|
||||
"""Mock the git/test primitives the real check_branch_mergeable composes."""
|
||||
monkeypatch.setattr(merge_gate, "acquire_merge_lease", lambda *a, **k: (True, "lease acquired"))
|
||||
monkeypatch.setattr(merge_gate, "release_merge_lease", lambda *a, **k: None)
|
||||
monkeypatch.setattr(merge_gate, "branch_is_behind_main", lambda r, b: True)
|
||||
monkeypatch.setattr(merge_gate, "auto_rebase_onto_main", lambda r, b: (True, "rebased onto origin/main"))
|
||||
|
||||
shas = list(head_shas)
|
||||
|
||||
def _head_sha(r, b):
|
||||
return shas.pop(0) if shas else ""
|
||||
|
||||
monkeypatch.setattr(merge_gate, "head_sha", _head_sha)
|
||||
|
||||
def _retest(r, b):
|
||||
retest_calls.append((r, b))
|
||||
return retest_result
|
||||
|
||||
monkeypatch.setattr(merge_gate, "retest_branch", _retest)
|
||||
|
||||
|
||||
def _make_task():
|
||||
conn = get_db()
|
||||
cur = conn.execute(
|
||||
"INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?,?,?,?,?)",
|
||||
(f"plane-{_WI}", _WI, _REPO, _BRANCH, "deploy-staging"),
|
||||
)
|
||||
tid = cur.lastrowid
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return tid
|
||||
|
||||
|
||||
def _stage(task_id):
|
||||
conn = get_db()
|
||||
row = conn.execute("SELECT stage FROM tasks WHERE id=?", (task_id,)).fetchone()
|
||||
conn.close()
|
||||
return row[0]
|
||||
|
||||
|
||||
def _agents():
|
||||
conn = get_db()
|
||||
rows = conn.execute("SELECT agent FROM jobs ORDER BY id").fetchall()
|
||||
conn.close()
|
||||
return [r[0] for r in rows]
|
||||
|
||||
|
||||
def _advance(task_id):
|
||||
return advance_stage(task_id, "deploy-staging", _REPO, _WI, _BRANCH, finished_agent="deployer")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario A (D4): the literal incident — not-behind branch, green CI HEAD.
|
||||
# The re-test is SKIPPED, so the timeout can never happen; advance to deploy.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc10_incident_noop_rebase_skips_retest_and_advances(monkeypatch):
|
||||
_patch_edge_gates_except_merge(monkeypatch)
|
||||
retest_calls = []
|
||||
# Same SHA before/after rebase -> proven no-op. retest is rigged to TIME OUT if
|
||||
# it were (wrongly) called — proving the skip is what avoids the false rollback.
|
||||
_mock_merge_primitives(
|
||||
monkeypatch,
|
||||
head_shas=["abc123", "abc123"],
|
||||
retest_result=(False, "re-test timeout after 900s"),
|
||||
retest_calls=retest_calls,
|
||||
)
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
|
||||
assert retest_calls == [], "re-test must be SKIPPED on a no-op rebase (D4)"
|
||||
assert res.rolled_back_to is None # the incident's false rollback is gone
|
||||
assert res.advanced is True
|
||||
assert res.to_stage == "deploy"
|
||||
assert _stage(task_id) == "deploy"
|
||||
assert "developer" not in _agents() # no developer-retry burned
|
||||
assert stage_engine.set_issue_blocked.called is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scenario B (D3): a real catch-up whose re-test times out -> infra-retry, not
|
||||
# rollback, and never the "still failing after N retries" manual gate.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc10_real_catchup_retest_timeout_infra_retries_not_rollback(monkeypatch):
|
||||
_patch_edge_gates_except_merge(monkeypatch)
|
||||
retest_calls = []
|
||||
# HEAD moved (real catch-up) -> re-test runs -> times out.
|
||||
_mock_merge_primitives(
|
||||
monkeypatch,
|
||||
head_shas=["old111", "new222"],
|
||||
retest_result=(False, "re-test timeout after 900s"),
|
||||
retest_calls=retest_calls,
|
||||
)
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
|
||||
assert retest_calls, "re-test SHOULD run on a real (HEAD-moving) catch-up"
|
||||
assert res.rolled_back_to is None # NOT the code-fault rollback
|
||||
assert res.note == "merge-gate-infra-retry"
|
||||
assert _stage(task_id) == "deploy-staging" # stays put for a bounded retry
|
||||
assert _agents() == ["deployer"] # staging-deployer re-queued, NOT developer
|
||||
assert stage_engine.set_issue_blocked.called is False
|
||||
# The "Merge-gate still failing after N developer retries" manual gate never fires.
|
||||
for call in stage_engine.send_telegram.call_args_list:
|
||||
assert "developer retries" not in call[0][0]
|
||||
|
||||
|
||||
def test_tc10_real_catchup_red_retest_still_rolls_back(monkeypatch):
|
||||
"""Anti-over-tolerance guard inside the incident scenario: a genuinely RED
|
||||
re-test on a real catch-up STILL rolls back (BR-6 / AC-3)."""
|
||||
_patch_edge_gates_except_merge(monkeypatch)
|
||||
retest_calls = []
|
||||
_mock_merge_primitives(
|
||||
monkeypatch,
|
||||
head_shas=["old111", "new222"],
|
||||
retest_result=(False, "re-test failed: ...1 failed"),
|
||||
retest_calls=retest_calls,
|
||||
)
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development"
|
||||
assert _agents() == ["developer"]
|
||||
103
tests/test_orch110_killswitch.py
Normal file
103
tests/test_orch110_killswitch.py
Normal file
@@ -0,0 +1,103 @@
|
||||
"""ORCH-110 TC-07: kill-switches + non-self repo => byte-for-byte pre-ORCH-110.
|
||||
|
||||
Covers NFR-2 / FR-5 / AC-7. Each ORCH-110 behaviour is an INDEPENDENT kill-switch;
|
||||
with a flag off the affected path reverts to the prior behaviour. A non-self-hosting
|
||||
repo (enduro-trails) never reaches the infra-retry / tree-kill paths at all (the
|
||||
merge-gate is N/A for it).
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
os.environ.setdefault("ORCH_DB_PATH", os.path.join(tempfile.gettempdir(), "test_orch110_killswitch.db"))
|
||||
os.environ.setdefault("ORCH_REPOS_DIR", tempfile.gettempdir())
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from src import merge_gate # noqa: E402
|
||||
from src.qg import checks as qg # noqa: E402
|
||||
from src.proc_group import ProcResult # noqa: E402
|
||||
|
||||
_REPO = "orchestrator"
|
||||
_BRANCH = "feature/ORCH-110-x"
|
||||
_WI = "ORCH-110"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D1 kill-switch: subprocess_tree_kill_enabled=False -> the fallback path.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc07_tree_kill_off_passes_tree_kill_false(tmp_path, monkeypatch):
|
||||
wt = tmp_path / "wt"
|
||||
wt.mkdir()
|
||||
monkeypatch.setattr(merge_gate, "get_worktree_path", lambda r, b: str(wt))
|
||||
monkeypatch.setattr(merge_gate.settings, "subprocess_tree_kill_enabled", False, raising=False)
|
||||
captured = {}
|
||||
|
||||
def _fake(cmd, *, cwd, timeout, env=None, grace_s=5.0, tree_kill=True):
|
||||
captured["tree_kill"] = tree_kill
|
||||
return ProcResult(returncode=0, stdout="1 passed", stderr="", timed_out=False)
|
||||
|
||||
monkeypatch.setattr(merge_gate, "run_in_process_group", _fake)
|
||||
ok, reason = merge_gate.retest_branch(_REPO, _BRANCH)
|
||||
assert ok is True and reason == "re-test green"
|
||||
assert captured["tree_kill"] is False # -> run_in_process_group degrades to subprocess.run
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D4 kill-switch: merge_retest_skip_when_current_enabled=False -> always re-test
|
||||
# after a rebase (even a proven no-op).
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.fixture
|
||||
def gate_primitives(monkeypatch):
|
||||
calls = {"retest": 0}
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_enabled", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_repos", "", raising=False)
|
||||
monkeypatch.setattr(qg.settings, "premerge_rebase_always", True, raising=False)
|
||||
monkeypatch.setattr(merge_gate, "acquire_merge_lease", lambda *a, **k: (True, "lease acquired"), raising=False)
|
||||
monkeypatch.setattr(merge_gate, "branch_is_behind_main", lambda r, b: True, raising=False)
|
||||
monkeypatch.setattr(merge_gate, "auto_rebase_onto_main", lambda r, b: (True, "rebased"), raising=False)
|
||||
monkeypatch.setattr(merge_gate, "release_merge_lease", lambda *a, **k: None, raising=False)
|
||||
# A PROVEN no-op rebase: HEAD is identical before/after.
|
||||
monkeypatch.setattr(merge_gate, "head_sha", lambda r, b: "deadbeefcafe", raising=False)
|
||||
|
||||
def _retest(r, b):
|
||||
calls["retest"] += 1
|
||||
return True, "re-test green"
|
||||
|
||||
monkeypatch.setattr(merge_gate, "retest_branch", _retest, raising=False)
|
||||
return calls
|
||||
|
||||
|
||||
def test_tc07_skip_when_current_off_always_retests(gate_primitives, monkeypatch):
|
||||
monkeypatch.setattr(qg.settings, "merge_retest_skip_when_current_enabled", False, raising=False)
|
||||
ok, reason = qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert ok is True
|
||||
assert reason == "rebased onto main, re-test green"
|
||||
assert gate_primitives["retest"] == 1 # re-test STILL runs (D4 off)
|
||||
|
||||
|
||||
def test_tc07_skip_when_current_on_skips_noop_retest(gate_primitives, monkeypatch):
|
||||
"""Mirror sanity: with the flag ON, the proven no-op rebase skips the re-test."""
|
||||
monkeypatch.setattr(qg.settings, "merge_retest_skip_when_current_enabled", True, raising=False)
|
||||
ok, reason = qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert ok is True
|
||||
assert "re-test skipped" in reason
|
||||
assert gate_primitives["retest"] == 0 # re-test NOT run on the no-op
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Non-self repo (enduro-trails): the merge-gate is N/A -> never the new paths.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc07_non_self_repo_is_noop(monkeypatch):
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_enabled", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_repos", "", raising=False)
|
||||
# Guard: if the gate wrongly engaged it would touch the lease -> fail loudly.
|
||||
monkeypatch.setattr(
|
||||
merge_gate, "acquire_merge_lease",
|
||||
lambda *a, **k: pytest.fail("merge-gate must be N/A for enduro-trails"),
|
||||
raising=False,
|
||||
)
|
||||
ok, reason = qg.check_branch_mergeable("enduro-trails", "ET-1", "feature/ET-1-x")
|
||||
assert ok is True
|
||||
assert "N/A" in reason
|
||||
64
tests/test_orch110_merge_gate_classify.py
Normal file
64
tests/test_orch110_merge_gate_classify.py
Normal file
@@ -0,0 +1,64 @@
|
||||
"""ORCH-110 TC-03: classify_retest_failure distinguishes infra-timeout from red.
|
||||
|
||||
Covers D2 (FR-1 / AC-2): the pure predicate that lets the engine route an INFRA
|
||||
re-test timeout differently from a deterministically RED re-test, WITHOUT changing
|
||||
the name / PASS-FAIL semantics of the registered ``check_branch_mergeable``.
|
||||
|
||||
The critical scope guard: an ``auto_rebase_onto_main`` "rebase timeout" is a
|
||||
DIFFERENT timeout (git hung) and must NOT be classified as the infra-tolerated
|
||||
re-test timeout (it stays on the rollback path).
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
os.environ.setdefault("ORCH_DB_PATH", os.path.join(tempfile.gettempdir(), "test_orch110_classify.db"))
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from src import merge_gate # noqa: E402
|
||||
|
||||
classify = merge_gate.classify_retest_failure
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reason,expected",
|
||||
[
|
||||
("re-test timeout after 900s", "timeout"),
|
||||
("re-test timeout after 600s", "timeout"),
|
||||
("re-test failed after rebase: 1 failed, 5 passed", "red"),
|
||||
("re-test failed: ...AssertionError\n1 failed", "red"),
|
||||
("merge-lock busy", "lock-busy"),
|
||||
("rebase conflict: src/db.py", "other"),
|
||||
# SCOPE GUARD: a git "rebase timeout" is NOT the infra-tolerated re-test
|
||||
# timeout — it must stay on the rollback path (ADR D2).
|
||||
("rebase timeout", "other"),
|
||||
("push --force-with-lease failed: ...", "other"),
|
||||
("", "other"),
|
||||
],
|
||||
)
|
||||
def test_tc03_classify_reasons(reason, expected):
|
||||
assert classify(reason) == expected
|
||||
|
||||
|
||||
def test_tc03_classify_never_raises_on_bad_input():
|
||||
# None / non-str must degrade to the safe "other" (-> rollback), never raise.
|
||||
assert classify(None) == "other"
|
||||
assert classify(12345) == "other"
|
||||
|
||||
|
||||
def test_tc03_case_insensitive():
|
||||
assert classify("RE-TEST TIMEOUT AFTER 900S") == "timeout"
|
||||
assert classify("Merge-Lock Busy") == "lock-busy"
|
||||
|
||||
|
||||
def test_tc03_distinct_from_lock_busy_and_conflict():
|
||||
"""timeout is a distinct class from the existing defer (lock-busy) and rollback
|
||||
(conflict) reasons — the three must never collide."""
|
||||
classes = {
|
||||
classify("re-test timeout after 900s"),
|
||||
classify("merge-lock busy"),
|
||||
classify("rebase conflict: x"),
|
||||
}
|
||||
assert classes == {"timeout", "lock-busy", "other"}
|
||||
261
tests/test_orch110_merge_gate_routing.py
Normal file
261
tests/test_orch110_merge_gate_routing.py
Normal file
@@ -0,0 +1,261 @@
|
||||
"""ORCH-110 TC-04 / TC-05 / TC-06 / TC-09: merge-gate infra-timeout routing.
|
||||
|
||||
Drives the engine (``stage_engine.advance_stage``) on the deploy-staging -> deploy
|
||||
edge with ``check_branch_mergeable`` monkeypatched, exactly like the existing
|
||||
``test_stage_engine.TestMergeGate`` suite, and asserts the NEW routing (D3):
|
||||
|
||||
* TC-04 — an INFRA re-test timeout -> bounded infra-retry (re-queue the
|
||||
staging-deployer with a delay, task STAYS on deploy-staging) — NOT a rollback to
|
||||
development and NOT a developer-retry.
|
||||
* TC-05 — a deterministically RED re-test STILL rolls back to development +
|
||||
developer retry (BR-6 / AC-3 anti-over-tolerance).
|
||||
* TC-06 — the infra-retry is bounded (anti-loop): after the budget it blocks with
|
||||
ONE infra-alert, no infinite bounce, no new job, task NOT in development.
|
||||
* TC-09 — never-raise: an error in the transient path is swallowed (a WARNING) and
|
||||
never escapes into advance_stage.
|
||||
|
||||
Offline: isolated sqlite DB; Plane/Telegram/notifications mocked at stage_engine.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
|
||||
_test_db = os.path.join(tempfile.gettempdir(), "test_orch110_routing.db")
|
||||
os.environ["ORCH_DB_PATH"] = _test_db
|
||||
os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir()
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
from unittest.mock import MagicMock # noqa: E402
|
||||
|
||||
import src.db as _db # noqa: E402
|
||||
from src.db import init_db, get_db # noqa: E402
|
||||
from src import stage_engine # noqa: E402
|
||||
from src.stage_engine import advance_stage # noqa: E402
|
||||
|
||||
_REPO = "orchestrator"
|
||||
_WI = "ORCH-110"
|
||||
_BRANCH = "feature/ORCH-110-x"
|
||||
_TIMEOUT_REASON = "re-test timeout after 900s"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_db(monkeypatch):
|
||||
monkeypatch.setattr(_db.settings, "db_path", _test_db)
|
||||
if os.path.exists(_test_db):
|
||||
os.unlink(_test_db)
|
||||
init_db()
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def silence_side_effects(monkeypatch):
|
||||
for name in (
|
||||
"notify_stage_change", "notify_qg_failure", "send_telegram",
|
||||
"plane_notify_stage", "plane_notify_qg", "plane_add_comment",
|
||||
"set_issue_in_progress", "set_issue_blocked",
|
||||
):
|
||||
monkeypatch.setattr(stage_engine, name, MagicMock())
|
||||
# The merge-gate sub-gate runs only AFTER the stage QG + security + coverage pass;
|
||||
# the self-deploy Phase A interception is irrelevant (merge-gate intervenes first).
|
||||
monkeypatch.setattr(stage_engine.settings, "deploy_require_manual_approve", False)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def tolerance_on(monkeypatch):
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_tolerance_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_max_retries", 2)
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_retry_delay_s", 120)
|
||||
|
||||
|
||||
def _pass(*a, **k):
|
||||
return (True, "ok")
|
||||
|
||||
|
||||
def _fail(reason):
|
||||
def _f(*a, **k):
|
||||
return (False, reason)
|
||||
return _f
|
||||
|
||||
|
||||
def _patch_gates(monkeypatch, merge_reason):
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS,
|
||||
"check_staging_status": _pass,
|
||||
"check_security_gate": _pass,
|
||||
"check_coverage_gate": _pass,
|
||||
"check_branch_mergeable": _fail(merge_reason),
|
||||
"check_staging_image_fresh": _pass},
|
||||
)
|
||||
|
||||
|
||||
def _make_task(stage="deploy-staging"):
|
||||
conn = get_db()
|
||||
cur = conn.execute(
|
||||
"INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?,?,?,?,?)",
|
||||
(f"plane-{_WI}", _WI, _REPO, _BRANCH, stage),
|
||||
)
|
||||
tid = cur.lastrowid
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return tid
|
||||
|
||||
|
||||
def _stage(task_id):
|
||||
conn = get_db()
|
||||
row = conn.execute("SELECT stage FROM tasks WHERE id=?", (task_id,)).fetchone()
|
||||
conn.close()
|
||||
return row[0]
|
||||
|
||||
|
||||
def _jobs():
|
||||
conn = get_db()
|
||||
rows = conn.execute(
|
||||
"SELECT agent, task_content, available_at FROM jobs ORDER BY id"
|
||||
).fetchall()
|
||||
conn.close()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def _seed_infra_retry_jobs(task_id, n):
|
||||
conn = get_db()
|
||||
for _ in range(n):
|
||||
conn.execute(
|
||||
"INSERT INTO jobs (agent, repo, task_id, task_content) "
|
||||
"VALUES ('deployer','orchestrator',?, 'Note: merge-gate infra-timeout retry')",
|
||||
(task_id,),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
def _advance(task_id):
|
||||
return advance_stage(
|
||||
task_id, "deploy-staging", _REPO, _WI, _BRANCH, finished_agent="deployer"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-04 — infra-timeout -> bounded infra-retry, NOT a rollback to development.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc04_infra_timeout_reschedules_not_rollback(monkeypatch):
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
|
||||
assert res.advanced is False
|
||||
assert res.rolled_back_to is None # NOT a code-fault rollback
|
||||
assert res.note == "merge-gate-infra-retry"
|
||||
assert _stage(task_id) == "deploy-staging" # stays put
|
||||
jobs = _jobs()
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["agent"] == "deployer" # re-queued staging-deployer, NOT developer
|
||||
assert "merge-gate infra-timeout retry" in jobs[0]["task_content"]
|
||||
assert jobs[0]["available_at"] is not None # delayed re-pickup
|
||||
assert stage_engine.set_issue_blocked.called is False
|
||||
# No developer-retry semantics: the rollback comment / in-progress is never set.
|
||||
assert stage_engine.set_issue_in_progress.called is False
|
||||
|
||||
|
||||
def test_tc04_killswitch_within_routing_is_observed(monkeypatch):
|
||||
"""The infra-timeout always bumps the timeout counter (observability), even when
|
||||
routed to the retry path."""
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
before = stage_engine.merge_gate.merge_gate_status()["retest_timeout_total"]
|
||||
task_id = _make_task()
|
||||
_advance(task_id)
|
||||
after = stage_engine.merge_gate.merge_gate_status()["retest_timeout_total"]
|
||||
assert after == before + 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-05 — a deterministically RED re-test STILL rolls back (BR-6 / AC-3).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc05_red_retest_still_rolls_back(monkeypatch):
|
||||
_patch_gates(monkeypatch, "re-test failed after rebase: 1 failed, 5 passed")
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
|
||||
assert res.advanced is False
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development"
|
||||
jobs = _jobs()
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["agent"] == "developer" # developer re-queued (retry)
|
||||
|
||||
|
||||
def test_tc05_conflict_still_rolls_back(monkeypatch):
|
||||
_patch_gates(monkeypatch, "rebase conflict: src/db.py")
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development"
|
||||
assert _jobs()[0]["agent"] == "developer"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-06 — anti-loop: infra-retry is bounded; exhaustion -> ONE infra-alert.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc06_infra_retry_bounded_then_infra_alert(monkeypatch):
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_max_retries", 2)
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
task_id = _make_task()
|
||||
_seed_infra_retry_jobs(task_id, 2) # budget already spent
|
||||
|
||||
res = _advance(task_id)
|
||||
assert res.advanced is False
|
||||
assert res.rolled_back_to is None # NOT a rollback even at exhaustion
|
||||
assert res.note == "merge-gate-infra-retry-exhausted"
|
||||
assert res.alerted is True
|
||||
assert _stage(task_id) == "deploy-staging" # NOT moved to development
|
||||
assert stage_engine.set_issue_blocked.called
|
||||
assert stage_engine.send_telegram.called
|
||||
# No NEW retry job past the cap (still only the 2 we seeded).
|
||||
assert len(_jobs()) == 2
|
||||
# The alert is INFRA-specific, not "developer must fix".
|
||||
msg = stage_engine.send_telegram.call_args[0][0]
|
||||
assert "infra" in msg.lower() or "ресурс" in msg.lower()
|
||||
assert "НЕ дефект кода" in msg
|
||||
|
||||
|
||||
def test_tc06_below_budget_keeps_retrying(monkeypatch):
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_max_retries", 3)
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
task_id = _make_task()
|
||||
_seed_infra_retry_jobs(task_id, 1) # one retry already done, budget 3
|
||||
|
||||
res = _advance(task_id)
|
||||
assert res.note == "merge-gate-infra-retry"
|
||||
assert res.alerted is not True
|
||||
# The seeded job + the new retry job.
|
||||
assert len(_jobs()) == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-09 — never-raise: an error in the transient path is swallowed.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc09_infra_retry_never_raises(monkeypatch):
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
|
||||
def _boom(*a, **k):
|
||||
raise RuntimeError("enqueue exploded")
|
||||
|
||||
monkeypatch.setattr(stage_engine, "enqueue_job", _boom)
|
||||
task_id = _make_task()
|
||||
# Must NOT raise into advance_stage.
|
||||
res = _advance(task_id)
|
||||
assert res.note == "merge-gate-infra-retry-error"
|
||||
assert _stage(task_id) == "deploy-staging" # left for the reconciler/reaper
|
||||
|
||||
|
||||
def test_tc09_killswitch_off_falls_back_to_rollback(monkeypatch):
|
||||
"""tolerance off -> a timeout takes the prior rollback path byte-for-byte (NFR-2)."""
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_tolerance_enabled", False)
|
||||
_patch_gates(monkeypatch, _TIMEOUT_REASON)
|
||||
task_id = _make_task()
|
||||
res = _advance(task_id)
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development"
|
||||
assert _jobs()[0]["agent"] == "developer"
|
||||
135
tests/test_orch110_observability.py
Normal file
135
tests/test_orch110_observability.py
Normal file
@@ -0,0 +1,135 @@
|
||||
"""ORCH-110 TC-12: observability of the infra-timeout path (D6 / FR-6 / AC-9).
|
||||
|
||||
The infra-timeout state must be (a) reflected in read-only counters, (b) surfaced in
|
||||
the additive ``merge_gate`` block of ``GET /queue``, and (c) distinguishable from a
|
||||
code-fault rollback — with the exhaustion alert carrying the CLICKABLE issue number
|
||||
and an explicitly infrastructural (NOT "developer must fix") wording. No dedup/overlap
|
||||
with ORCH-111 (which only OBSERVES surviving processes; ORCH-110 prevents/tolerates).
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
_test_db = os.path.join(tempfile.gettempdir(), "test_orch110_observability.db")
|
||||
os.environ["ORCH_DB_PATH"] = _test_db
|
||||
os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir()
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
from unittest.mock import MagicMock # noqa: E402
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
import src.db as _db # noqa: E402
|
||||
from src.db import init_db, get_db # noqa: E402
|
||||
from src import merge_gate # noqa: E402
|
||||
from src import stage_engine # noqa: E402
|
||||
from src.stage_engine import AdvanceResult # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_db(monkeypatch):
|
||||
monkeypatch.setattr(_db.settings, "db_path", _test_db)
|
||||
if os.path.exists(_test_db):
|
||||
os.unlink(_test_db)
|
||||
init_db()
|
||||
# Reset the in-process counters so each test starts from a known baseline.
|
||||
merge_gate._MERGE_GATE_COUNTERS.update(
|
||||
retest_timeout_total=0, retest_infra_retry_total=0,
|
||||
retest_infra_exhausted_total=0, retest_skipped_current_total=0,
|
||||
last_infra_timeout_wi=None,
|
||||
)
|
||||
yield
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# merge_gate_status() snapshot + counter increments.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc12_status_exposes_flags_and_counters():
|
||||
snap = merge_gate.merge_gate_status()
|
||||
for key in (
|
||||
"infra_tolerance_enabled", "infra_max_retries", "infra_retry_delay_s",
|
||||
"skip_when_current_enabled", "tree_kill_enabled", "retest_timeout_s",
|
||||
"retest_timeout_total", "retest_infra_retry_total",
|
||||
"retest_infra_exhausted_total", "retest_skipped_current_total",
|
||||
"last_infra_timeout_wi",
|
||||
):
|
||||
assert key in snap, f"missing /queue merge_gate key: {key}"
|
||||
|
||||
|
||||
def test_tc12_counters_track_infra_timeout_distinctly():
|
||||
merge_gate.note_retest_timeout("ORCH-110")
|
||||
merge_gate.note_retest_infra_retry()
|
||||
merge_gate.note_retest_infra_exhausted()
|
||||
merge_gate.note_retest_skipped_current()
|
||||
snap = merge_gate.merge_gate_status()
|
||||
assert snap["retest_timeout_total"] == 1
|
||||
assert snap["retest_infra_retry_total"] == 1
|
||||
assert snap["retest_infra_exhausted_total"] == 1
|
||||
assert snap["retest_skipped_current_total"] == 1
|
||||
# Distinguishable from a code-fault: the last infra-timeout WI is tracked here,
|
||||
# NOT in the merge-verify (code/merge) counters.
|
||||
assert snap["last_infra_timeout_wi"] == "ORCH-110"
|
||||
|
||||
|
||||
def test_tc12_status_never_raises(monkeypatch):
|
||||
# A broken settings attribute -> the snapshot degrades, never raises.
|
||||
monkeypatch.delattr(merge_gate.settings, "merge_retest_infra_tolerance_enabled", raising=False)
|
||||
snap = merge_gate.merge_gate_status()
|
||||
assert isinstance(snap, dict)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /queue carries the additive merge_gate block (and the legacy keys stay).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc12_queue_endpoint_includes_merge_gate_block():
|
||||
from src.main import queue
|
||||
out = asyncio.run(queue())
|
||||
assert "merge_gate" in out
|
||||
assert "infra_tolerance_enabled" in out["merge_gate"]
|
||||
# The pre-existing observability keys are untouched (additive only).
|
||||
assert "merge_verify" in out and "coverage" in out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The exhaustion alert is INFRA-specific + carries the clickable issue number,
|
||||
# distinct from the code-fault "Merge-gate still failing after N developer retries".
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc12_exhaustion_alert_is_infra_specific_and_clickable(monkeypatch):
|
||||
sent = {}
|
||||
|
||||
def _tg(msg):
|
||||
sent["msg"] = msg
|
||||
|
||||
monkeypatch.setattr(stage_engine, "send_telegram", _tg)
|
||||
monkeypatch.setattr(stage_engine, "set_issue_blocked", MagicMock())
|
||||
monkeypatch.setattr(stage_engine, "plane_add_comment", MagicMock())
|
||||
# link_for builds the clickable number; use a recognisable sentinel.
|
||||
monkeypatch.setattr(stage_engine, "link_for", lambda wi, **k: f"<a href='x'>{wi}</a>")
|
||||
monkeypatch.setattr(stage_engine.settings, "merge_retest_infra_max_retries", 2)
|
||||
|
||||
# Seed an exhausted budget for the task.
|
||||
conn = get_db()
|
||||
cur = conn.execute(
|
||||
"INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?,?,?,?,?)",
|
||||
("plane-ORCH-110", "ORCH-110", "orchestrator", "feature/ORCH-110-x", "deploy-staging"),
|
||||
)
|
||||
task_id = cur.lastrowid
|
||||
for _ in range(2):
|
||||
conn.execute(
|
||||
"INSERT INTO jobs (agent, repo, task_id, task_content) "
|
||||
"VALUES ('deployer','orchestrator',?, 'Note: merge-gate infra-timeout retry')",
|
||||
(task_id,),
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
res = AdvanceResult()
|
||||
stage_engine._handle_merge_gate_infra_retry(
|
||||
task_id, "deploy-staging", "orchestrator", "ORCH-110",
|
||||
"feature/ORCH-110-x", "re-test timeout after 900s", res,
|
||||
)
|
||||
assert res.note == "merge-gate-infra-retry-exhausted"
|
||||
assert "ORCH-110" in sent["msg"] # clickable issue number present
|
||||
assert "developer retries" not in sent["msg"] # NOT the code-fault wording
|
||||
assert "НЕ дефект кода" in sent["msg"] # explicitly infrastructural
|
||||
95
tests/test_orch110_retest_contract.py
Normal file
95
tests/test_orch110_retest_contract.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""ORCH-110 TC-11: the local re-test necessity contract (D4 / FR-4 / AC-6).
|
||||
|
||||
The chosen contract: run the local re-test IFF the pre-merge rebase actually moved
|
||||
HEAD (``main`` had moved -> a real semantic-conflict risk). When the rebase is a
|
||||
PROVEN no-op (HEAD unchanged -> branch already at origin/main, the CI/tester/staging-
|
||||
validated commit) the re-test is SKIPPED — it would be a redundant single point of
|
||||
false failure. On ANY uncertainty (an empty SHA) the re-test runs (fail-safe -> the
|
||||
red-rollback contract BR-6/AC-3 is never weakened).
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
os.environ.setdefault("ORCH_DB_PATH", os.path.join(tempfile.gettempdir(), "test_orch110_contract.db"))
|
||||
os.environ.setdefault("ORCH_REPOS_DIR", tempfile.gettempdir())
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from src import merge_gate # noqa: E402
|
||||
from src.qg import checks as qg # noqa: E402
|
||||
|
||||
_REPO = "orchestrator"
|
||||
_BRANCH = "feature/ORCH-110-x"
|
||||
_WI = "ORCH-110"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gate(monkeypatch):
|
||||
"""Real check_branch_mergeable, mocked primitives; record control flow."""
|
||||
state = {"retest": 0, "released": 0, "rebased": 0}
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_enabled", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_gate_repos", "", raising=False)
|
||||
monkeypatch.setattr(qg.settings, "premerge_rebase_always", True, raising=False)
|
||||
monkeypatch.setattr(qg.settings, "merge_retest_skip_when_current_enabled", True, raising=False)
|
||||
monkeypatch.setattr(merge_gate, "acquire_merge_lease", lambda *a, **k: (True, "lease acquired"))
|
||||
monkeypatch.setattr(merge_gate, "branch_is_behind_main", lambda r, b: True)
|
||||
|
||||
def _rebase(r, b):
|
||||
state["rebased"] += 1
|
||||
return True, "rebased onto origin/main"
|
||||
|
||||
def _release(r, b=None):
|
||||
state["released"] += 1
|
||||
|
||||
def _retest(r, b):
|
||||
state["retest"] += 1
|
||||
return True, "re-test green"
|
||||
|
||||
monkeypatch.setattr(merge_gate, "auto_rebase_onto_main", _rebase)
|
||||
monkeypatch.setattr(merge_gate, "release_merge_lease", _release)
|
||||
monkeypatch.setattr(merge_gate, "retest_branch", _retest)
|
||||
return state
|
||||
|
||||
|
||||
def _set_head_shas(monkeypatch, shas):
|
||||
seq = list(shas)
|
||||
monkeypatch.setattr(merge_gate, "head_sha", lambda r, b: seq.pop(0) if seq else "")
|
||||
|
||||
|
||||
def test_tc11_noop_rebase_skips_retest_lease_held(gate, monkeypatch):
|
||||
"""Proven no-op rebase (HEAD unchanged) -> skip re-test, PASS, lease HELD."""
|
||||
_set_head_shas(monkeypatch, ["sha_same", "sha_same"])
|
||||
ok, reason = qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert ok is True
|
||||
assert "re-test skipped" in reason
|
||||
assert gate["retest"] == 0 # re-test NOT run
|
||||
assert gate["released"] == 0 # lease HELD until the merge
|
||||
|
||||
|
||||
def test_tc11_head_moved_runs_retest(gate, monkeypatch):
|
||||
"""A real catch-up (HEAD moved) -> re-test RUNS (the ORCH-043 risk is real)."""
|
||||
_set_head_shas(monkeypatch, ["sha_old", "sha_new"])
|
||||
ok, reason = qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert ok is True
|
||||
assert reason == "rebased onto main, re-test green"
|
||||
assert gate["retest"] == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shas", [["", "sha_new"], ["sha_old", ""], ["", ""]])
|
||||
def test_tc11_uncertain_sha_runs_retest_failsafe(gate, monkeypatch, shas):
|
||||
"""Cannot prove a no-op (empty SHA) -> re-test RUNS (fail-safe, never skips on
|
||||
uncertainty)."""
|
||||
_set_head_shas(monkeypatch, shas)
|
||||
ok, reason = qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert ok is True
|
||||
assert gate["retest"] == 1 # re-test still runs on uncertainty
|
||||
|
||||
|
||||
def test_tc11_skip_bumps_observability_counter(gate, monkeypatch):
|
||||
"""The skip increments the read-only observability counter (D6)."""
|
||||
merge_gate._MERGE_GATE_COUNTERS["retest_skipped_current_total"] = 0
|
||||
_set_head_shas(monkeypatch, ["sha_same", "sha_same"])
|
||||
qg.check_branch_mergeable(_REPO, _WI, _BRANCH)
|
||||
assert merge_gate._MERGE_GATE_COUNTERS["retest_skipped_current_total"] == 1
|
||||
185
tests/test_orch110_retest_lifecycle.py
Normal file
185
tests/test_orch110_retest_lifecycle.py
Normal file
@@ -0,0 +1,185 @@
|
||||
"""ORCH-110 TC-01 / TC-02: process-group tree-kill of orchestrator-spawned pytest.
|
||||
|
||||
Covers D1 (FR-2 / BR-3 / AC-4): a timeout in ``merge_gate.retest_branch`` /
|
||||
``coverage_gate.measure_coverage`` must tree-kill the WHOLE subprocess subtree
|
||||
(children + grandchildren) — the orphan-leak root cause of the ORCH-109 incident —
|
||||
not just the direct child.
|
||||
|
||||
Fully deterministic and offline:
|
||||
* TC-01 runs a REAL throwaway process that spawns a long-sleeping grandchild via
|
||||
``sys.executable`` (always present), then asserts the grandchild is dead after a
|
||||
``run_in_process_group`` timeout — the concrete "no orphan survives" proof.
|
||||
* TC-01b / TC-02 assert the two call sites delegate to ``run_in_process_group``
|
||||
with the right tree-kill kwargs and map a timed-out ProcResult to their existing
|
||||
contract (``(False, "re-test timeout after <T>s")`` / ``None``).
|
||||
No network, no Plane/Gitea, no Claude CLI.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
os.environ.setdefault("ORCH_DB_PATH", os.path.join(tempfile.gettempdir(), "test_orch110_lifecycle.db"))
|
||||
os.environ.setdefault("ORCH_REPOS_DIR", tempfile.gettempdir())
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
import sys # noqa: E402
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from src import coverage_gate as cg # noqa: E402
|
||||
from src import merge_gate # noqa: E402
|
||||
from src import proc_group # noqa: E402
|
||||
from src.proc_group import ProcResult, run_in_process_group # noqa: E402
|
||||
|
||||
_POSIX = hasattr(os, "killpg") and hasattr(os, "setsid")
|
||||
|
||||
|
||||
def _alive(pid: int) -> bool:
|
||||
"""True iff ``pid`` is a live (non-reaped) process."""
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
return True
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-01: real grandchild is tree-killed on timeout (the FR-2 / AC-4 proof).
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.skipif(not _POSIX, reason="process-group tree-kill is POSIX-only")
|
||||
def test_tc01_grandchild_orphan_is_tree_killed_on_timeout(tmp_path):
|
||||
pidfile = tmp_path / "grandchild.pid"
|
||||
script = tmp_path / "spawner.py"
|
||||
# The leader spawns a long-sleeping grandchild, records its pid, then sleeps far
|
||||
# past the timeout. A naive subprocess.run(timeout=) would kill ONLY the leader
|
||||
# and leak the grandchild (the ORCH-109 bug). The process-group tree-kill must
|
||||
# reap the grandchild too.
|
||||
script.write_text(
|
||||
"import subprocess, sys, time\n"
|
||||
"gc = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(120)'])\n"
|
||||
"open(sys.argv[1], 'w').write(str(gc.pid))\n"
|
||||
"time.sleep(120)\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
res = run_in_process_group(
|
||||
[sys.executable, str(script), str(pidfile)],
|
||||
cwd=str(tmp_path), timeout=2, grace_s=1, tree_kill=True,
|
||||
)
|
||||
assert res.timed_out is True
|
||||
|
||||
# Wait for the leader to actually have written the grandchild pid.
|
||||
deadline = time.time() + 5
|
||||
while time.time() < deadline and not pidfile.exists():
|
||||
time.sleep(0.05)
|
||||
assert pidfile.exists(), "spawner never recorded the grandchild pid"
|
||||
gc_pid = int(pidfile.read_text().strip())
|
||||
|
||||
# The grandchild must die (be reaped) within a short window after the tree-kill.
|
||||
deadline = time.time() + 8
|
||||
while time.time() < deadline and _alive(gc_pid):
|
||||
time.sleep(0.1)
|
||||
assert not _alive(gc_pid), "grandchild orphan survived the timeout tree-kill (FR-2/AC-4)"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not _POSIX, reason="process-group runner is POSIX-only")
|
||||
def test_tc01_green_and_red_returncodes_passthrough(tmp_path):
|
||||
"""The runner returns the real exit code on a normal (non-timeout) completion."""
|
||||
ok = run_in_process_group(
|
||||
[sys.executable, "-c", "import sys; sys.exit(0)"],
|
||||
cwd=str(tmp_path), timeout=30, tree_kill=True,
|
||||
)
|
||||
assert ok.timed_out is False and ok.returncode == 0
|
||||
red = run_in_process_group(
|
||||
[sys.executable, "-c", "import sys; sys.exit(3)"],
|
||||
cwd=str(tmp_path), timeout=30, tree_kill=True,
|
||||
)
|
||||
assert red.timed_out is False and red.returncode == 3
|
||||
|
||||
|
||||
def test_tc01_fallback_when_tree_kill_disabled(tmp_path, monkeypatch):
|
||||
"""tree_kill=False -> degrades to subprocess.run (never-break); still times out."""
|
||||
res = run_in_process_group(
|
||||
[sys.executable, "-c", "import time; time.sleep(30)"],
|
||||
cwd=str(tmp_path), timeout=1, tree_kill=False,
|
||||
)
|
||||
assert res.timed_out is True
|
||||
|
||||
|
||||
def test_tc01_fallback_on_non_posix(tmp_path, monkeypatch):
|
||||
"""No os.killpg/setsid (non-POSIX) -> the plain subprocess.run fallback runs."""
|
||||
monkeypatch.setattr(proc_group, "_tree_kill_supported", lambda: False)
|
||||
res = run_in_process_group(
|
||||
[sys.executable, "-c", "import sys; sys.exit(0)"],
|
||||
cwd=str(tmp_path), timeout=30,
|
||||
)
|
||||
assert res.timed_out is False and res.returncode == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-01b: retest_branch delegates to the process-group runner with the right
|
||||
# kwargs and maps a timeout to its existing contract.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc01b_retest_branch_uses_process_group_and_maps_timeout(tmp_path, monkeypatch):
|
||||
wt = tmp_path / "wt"
|
||||
wt.mkdir()
|
||||
monkeypatch.setattr(merge_gate, "get_worktree_path", lambda repo, branch: str(wt))
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_retest_timeout_s", 900, raising=False)
|
||||
monkeypatch.setattr(merge_gate.settings, "subprocess_tree_kill_enabled", True, raising=False)
|
||||
monkeypatch.setattr(merge_gate.settings, "agent_kill_grace_seconds", 7, raising=False)
|
||||
|
||||
captured = {}
|
||||
|
||||
def _fake(cmd, *, cwd, timeout, env=None, grace_s=5.0, tree_kill=True):
|
||||
captured.update(cmd=cmd, cwd=cwd, timeout=timeout, grace_s=grace_s, tree_kill=tree_kill)
|
||||
return ProcResult(returncode=None, stdout="", stderr="", timed_out=True)
|
||||
|
||||
monkeypatch.setattr(merge_gate, "run_in_process_group", _fake)
|
||||
ok, reason = merge_gate.retest_branch("orchestrator", "feature/x")
|
||||
assert ok is False
|
||||
assert reason == "re-test timeout after 900s"
|
||||
# FR-2: spawned via the process-group runner with tree-kill + the agent grace.
|
||||
assert captured["tree_kill"] is True
|
||||
assert captured["grace_s"] == 7
|
||||
assert captured["cmd"][:3] == ["python", "-m", "pytest"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-02: coverage_gate.measure_coverage — sibling orphan source (BR-3). Timeout
|
||||
# tree-kills + returns None; uses the same runner with tree-kill from settings.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc02_measure_coverage_timeout_tree_kill_returns_none(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(cg, "ensure_worktree", lambda r, b: str(tmp_path))
|
||||
monkeypatch.setattr(cg.settings, "subprocess_tree_kill_enabled", True, raising=False)
|
||||
monkeypatch.setattr(cg.settings, "agent_kill_grace_seconds", 5, raising=False)
|
||||
monkeypatch.setattr(cg.settings, "coverage_run_timeout_s", 900, raising=False)
|
||||
|
||||
captured = {}
|
||||
|
||||
def _fake(cmd, *, cwd, timeout, env=None, grace_s=5.0, tree_kill=True):
|
||||
captured.update(tree_kill=tree_kill, grace_s=grace_s, timeout=timeout)
|
||||
return ProcResult(returncode=None, stdout="", stderr="", timed_out=True)
|
||||
|
||||
monkeypatch.setattr(cg, "run_in_process_group", _fake)
|
||||
assert cg.measure_coverage("orchestrator", "feature/x") is None
|
||||
assert captured["tree_kill"] is True
|
||||
assert captured["grace_s"] == 5
|
||||
assert captured["timeout"] == 900
|
||||
|
||||
|
||||
def test_tc02_measure_coverage_respects_tree_kill_killswitch(tmp_path, monkeypatch):
|
||||
"""With the tree-kill kill-switch off, the runner is asked for the fallback path."""
|
||||
monkeypatch.setattr(cg, "ensure_worktree", lambda r, b: str(tmp_path))
|
||||
monkeypatch.setattr(cg.settings, "subprocess_tree_kill_enabled", False, raising=False)
|
||||
captured = {}
|
||||
|
||||
def _fake(cmd, *, cwd, timeout, env=None, grace_s=5.0, tree_kill=True):
|
||||
captured.update(tree_kill=tree_kill)
|
||||
return ProcResult(returncode=None, stdout="", stderr="", timed_out=True)
|
||||
|
||||
monkeypatch.setattr(cg, "run_in_process_group", _fake)
|
||||
assert cg.measure_coverage("orchestrator", "feature/x") is None
|
||||
assert captured["tree_kill"] is False
|
||||
Reference in New Issue
Block a user