diff --git a/.openclaw/agents/tester.md b/.openclaw/agents/tester.md index c149c83..b0d9474 100644 --- a/.openclaw/agents/tester.md +++ b/.openclaw/agents/tester.md @@ -29,6 +29,17 @@ tools: ТОЛЬКО потом выноси вердикт. Любой FAIL/смок-сбой → `result: FAIL`; всё зелёное → `result: PASS`. +> **ORCH-116 — детерминированный раннер ведёт эту стадию для in-scope репо.** На `testing` для +> self-hosting `orchestrator` (репо с тест-контрактом) стадию теперь ведёт **детерминированный код** +> (`src/test_runner.py`, перехват в `launch_job` **до** `_spawn`, как `deploy-finalizer`/ +> `staging-runner`) — он исполняет тот же регресс `pytest tests/` в worktree ветки + read-only smoke +> (`/health`, `/status`, `/queue` + блок `serial_gate`), маппит exit-код в `result:` тем же +> контрактом `0 → PASS / иначе → FAIL`, пишет `13-test-report.md` и инициирует неизменный гейт +> `check_tests_passed`. LLM-шаги ниже остаются **fallback'ом** под выключенным kill-switch +> (`ORCH_TEST_RUNNER_ENABLED=false`) или для репо без тест-контракта. Контракт артефакта / гейт / +> machine-key `result:` — неизменны. Детали: +> `docs/work-items/ORCH-116/06-adr/ADR-001-deterministic-test-runner.md`. + **Алгоритм:** 1. **Окружение:** `curl -s http://localhost:8500/health`. 2. **Тесты — в worktree ветки задачи, НЕ в общем `/repos/orchestrator`.** Прогоняй `pytest` из diff --git a/.task-dev.md b/.task-dev.md index 75f0fd3..69f6747 100644 --- a/.task-dev.md +++ b/.task-dev.md @@ -1,4 +1,4 @@ -Work item: ORCH-115 +Work item: ORCH-116 Repo: orchestrator -Branch: feature/ORCH-115-orch-replace-llm-deployer-with +Branch: feature/ORCH-116-orch-replace-llm-tester-with-d Stage: development \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 62b1fde..da41afd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -493,6 +493,63 @@ Plane, **не** Quality Gate и **не** стадия). Детали — `docs/work-items/ORCH-123/06-adr/ADR-001-host-side-staging-execution-and-env-classification.md`, сквозной `docs/architecture/adr/adr-0049-host-side-docker-execution-boundary.md`. +## Детерминированный test-раннер вместо LLM-тестера (ORCH-116) +Второй реализованный срез determinization-roadmap (ORCH-118 A5, `needs-hybrid-fallback`): на стадии +`testing` для self-hosting `orchestrator` **LLM-агент `tester` заменён детерминированным кодом** +(`src/test_runner.py`). PASS/FAIL-ядро было деривируемо (регресс `pytest` + read-only smoke) — теперь +его делает leaf, перехватываемый в `launch_job` **до `_spawn`** (прецедент `deploy-finalizer`/ +`post-deploy-monitor`/`staging-runner`, `launcher.py:397/402/405`). **Инвариант (NFR-1):** замена +*продюсера* артефакта, **не** гейта — контракт `13-test-report.md`, гейт/`_parse_tests_verdict`/имя +`check_tests_passed`, `STAGE_TRANSITIONS`, machine-verdict `result:` (+ legacy `verdict:`/`status:`), +схема БД — **байт-в-байт не тронуты**. Аддитивно, под kill-switch, never-raise, fail-closed, гибрид +(LLM строго off-control-path). +- **Перехват (D1):** `launch_job` — `if job.agent=="tester" and test_runner.should_intercept(job)` + → `_run_test_runner_job` (зеркало `_run_staging_runner_job`): синхронно ведёт `jobs`-строку через + `mark_job`, возвращает `None` (нет `agent_runs`). Дискриминатор — роль `tester` **И** стадия задачи + `testing` (defense-in-depth: `tester` — единственный агент входа в `testing`, коллизии стадий нет, в + отличие от общей роли `deployer`/ORCH-115) **И** `applies(repo)`; `should_intercept` never-raise → + `False` → штатный `_spawn` (fail-safe к LLM-пути). +- **Раннер (D2-D7):** leaf по образцу `staging_runner`/`self_deploy`/`proc_group` (на импорте только + `config`/`proc_group`; `db`/`git_worktree`/`self_deploy`/`qg.checks`/`stage_engine`/`notifications` + — лениво). `applies` = kill-switch `test_runner_enabled` + скоуп `test_runner_repos` (пусто → + self-hosting only) **И** резолв тест-контракта `_has_test_contract` (BR-9: репо без контракта → + LLM-tester, enduro-trails 1:1 даже если руками в CSV). Исполняет `python -m pytest + ` **в worktree ветки** (анти checkout-гонка) через `proc_group` + (tree-kill, таймаут `test_runner_timeout_s=900`) + опц. read-only smoke (`/health`/`/status`/`/queue` + + блок `serial_gate`, stdlib `urllib`; транзиент → ограниченный ретрай, не-200/нет блока → немедленный + FAIL; `test_runner_smoke_enabled`); маппит exit-код **единым** `self_deploy.map_exit_code_to_status` + в токенах `result:` (`0→PASS`/иначе/None→`FAIL`, fail-closed; smoke-провал AND-ится в `FAIL`); пишет + `13-test-report.md` (`author_agent: test-runner`/`model_used: n/a`, 52c-схема) + best-effort push в + фичеветку; вызывает **существующий** `advance_stage(current_stage="testing", finished_agent="tester")` + — без новых рёбер/исходов (lease ORCH-114 берётся внутри `advance_stage`). +- **Анти-коллизия 52c-`status:` ↔ парсер (D6.1, специфично для tester):** `_parse_tests_verdict` читает + вердикт из **трёх** равноранговых полей (`verdict:`/`status:`/`result:`) с negative-token-priority → + 52c-обязательное `status:` раннера **ВСЕГДА выровнено** по вердикту (`PASS → status: success`, `FAIL + → status: failed`), иначе негативный токен в `status:` при `result: PASS` дал бы ложный FAIL. Прибито + unit-тестом через неизменённый парсер. +- **Двухуровневый исход (D5, анти-ORCH-110):** сюита **исполнилась** (реальный exit-код) → verdict→ + advance (FAIL → тот же откат `testing → development` + developer-retry, что у LLM); сюита **не + исполнилась** (tool-error: spawn-error/таймаут/`returncode None`) → инфра-сбой ≠ код-фейл → bounded + **DEFER** (re-queue **`tester`**-джоба + restart-safe маркер `test-runner infra-retry` в + `task_content`, без правки схемы БД), на исчерпании `test_runner_infra_max_retries` → fail-closed + `FAIL` + advance + alert. Никогда тихий advance/ложный green; не клинит очередь; не жжёт + developer-retry на транзиентной инфре. +- **Флаги** (`config.py`, дефолт = боевое): `test_runner_enabled` (kill-switch, env + `ORCH_TEST_RUNNER_ENABLED`), `test_runner_repos` (CSV; **пусто → self-hosting only**), + `test_runner_target=tests/`, `test_runner_timeout_s=900`, `test_runner_smoke_enabled`, + `test_runner_infra_max_retries=2`, `test_runner_infra_retry_delay_s=30`. Откат = + `ORCH_TEST_RUNNER_ENABLED=false` → на `testing` снова LLM-`tester` через `_spawn` **байт-в-байт**. + Наблюдаемость — in-process счётчики + read-only блок `test_runner` в `GET /queue` + один структурный + лог-вердикт на прогон (различает код-фейл и tool-error). **Гибрид (BR-8/NFR-7):** LLM строго + off-control-path — детерминированный раннер единственный продюсер `result:`; будущий триаж падений не + выносит/не переопределяет вердикт и не добавляет ребро в `STAGE_TRANSITIONS` (Phase 1 не реализован). + Норматив сопровождения ORCH-118: обновлены `llm-call-sites.md`/`llm-determinization-roadmap.md`/ + `llm-usage-policy.md` (A5/rank 2 — реализован, машинные блоки/инвариант «ровно один `first_slice`» + целы) + `tester.md` (LLM-ветвь — fallback). Покрытие — `tests/test_orch116_test_runner.py` + (TC-01…TC-14) + зелёные LLM-анти-дрейф тесты (TC-15). Детали — + `docs/work-items/ORCH-116/06-adr/ADR-001-deterministic-test-runner.md`, сквозной + `docs/architecture/adr/adr-0049-deterministic-test-runner.md`. + ## Машинный журнал уроков (ORCH-098) Шаг 1 («Фундамент», F2) эпика саморазвития: формализует свободнотекстовые «уроки» из `memory/` в **машинную структурированную таблицу отклонений конвейера** `lessons`, фундамент для будущих diff --git a/docs/architecture/README.md b/docs/architecture/README.md index d52f5b9..171e987 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -11,6 +11,7 @@ - **Quality Gates** (`src/qg/checks.py`) — проверки выхода со стадии, реестр `QG_CHECKS`. - **Agent Launcher** (`src/agents/launcher.py`) — запуск Claude CLI агентов в изолированном git worktree, мониторинг, auto-advance. Модель/эффорт каждого агента резолвятся из config (`resolve_agent_model`/`resolve_agent_effort`, ORCH-41), а не из frontmatter промпта. **ORCH-74:** имя модели валидируется форматом `^claude-…$` (`is_valid_model`) перед `--model`; невалидное → лог + откат на следующий уровень/CLI-дефолт (never-break, как `VALID_EFFORTS` для эффорта). Тот же предикат гардит inline-чтение `--fallback-model`. **ORCH-109 ([adr-0040](adr/adr-0040-agent-timeout-budgets-and-launch-model-stamp.md)):** (1) резолвенная **модель стампится в `agent_runs.model` в момент launch** (`_spawn`, объединённый `UPDATE … SET model=?, effort=?` рядом со стампом эффорта ORCH-087; пустой резолв → `NULL`; never-raise) → модель видна не-`null` при любом исходе прогона, включая timeout-kill (`exit_code=-9`), и in-flight в `GET /metrics`/`GET /queue` (`get_running_agents` уже отдаёт `model`); постфактум `record_usage` (`model=COALESCE(?, model)`) остаётся **обогащением**, не единственным источником истины. (2) **Per-role wall-clock бюджеты** через выделенные ключи `agent_timeout_developer_s=3600`/`agent_timeout_reviewer_s=3000` (лестница `_resolve_timeout`: `agent_timeout_overrides_json` → выделенный ключ роли → `agent_timeout_seconds=1800`; прочие роли — байт-в-байт; малформный/вне-диапазонный конфиг → дефолт + WARNING). Инвариант reaper ORCH-065 сохранён синхронным поднятием `reaper_max_running_s` 3600→**5400** (`5400 > max(timeout)3600 + grace20`). FR-5 анти-salvage — структурно: продвижение гейтится `if exit_code==0`, timeout-kill → `_finalize_job` (retry/fail), не advance. `STAGE_TRANSITIONS`/`QG_CHECKS`/схема БД не тронуты. - **Staging-runner** (`src/staging_runner.py`, ORCH-115 — [adr-0048](adr/adr-0048-deterministic-staging-runner.md)) — чистый **never-raise** leaf (паттерн `self_deploy`/`proc_group`/`staging_verdict`), заменяющий **LLM-агента `deployer` на стадии `deploy-staging`** детерминированным кодом (первый реализованный срез determinization-roadmap, A6/`replace-deterministic-now`). Перехват в `launch_job` **до `_spawn`** (рядом с D1/D2 `deploy-finalizer`/`post-deploy-monitor`): дискриминатор — **стадия задачи** `deploy-staging` **И** `applies(repo)` (kill-switch `staging_runner_enabled` + скоуп `staging_runner_repos`, пусто → self-hosting only), `should_intercept` never-raise → `False` → штатный `_spawn` (fail-safe к LLM). Раннер (зеркало `run_deploy_finalizer`) исполняет ту же staging-сюиту (`docker exec orchestrator-staging … staging_check.py`) через `proc_group` (tree-kill, таймаут `staging_runner_timeout_s=600`), маппит exit-код единым контрактом `self_deploy.map_exit_code_to_status` (`0→SUCCESS`/иначе→`FAILED`; ORCH-061 infra-tolerance остаётся внутри `staging_check.py`), пишет `15-staging-log.md` (тот же machine-key `staging_status:`, `author_agent: staging-runner`/`model_used: n/a`) + best-effort push в фичеветку, и вызывает **существующий** `advance_stage(current_stage="deploy-staging", finished_agent="deployer")` — без новых рёбер/исходов; transition-lease ORCH-114 берётся внутри `advance_stage` (раннер не трогает). **Двухуровневый исход (анти-ORCH-110):** сюита исполнилась → verdict→advance (FAILED → тот же откат `deploy-staging → development` + developer-retry, что у LLM); tool-error/таймаут → bounded defer (`staging_runner_infra_max_retries`) → fail-closed `FAILED` + alert на исчерпании (инфра ≠ код-фейл, никогда тихий advance/ложный green). `STAGE_TRANSITIONS`/`QG_CHECKS`/`check_staging_status`/`_parse_staging_status`/machine-verdict/схема БД — **байт-в-байт не тронуты** (замена *продюсера*, не гейта). Наблюдаемость — in-process счётчики + блок `staging_runner` в `GET /queue`. Откат — `ORCH_STAGING_RUNNER_ENABLED=false` (прежний LLM-deployer-путь байт-в-байт). Детали — `docs/work-items/ORCH-115/06-adr/ADR-001-deterministic-staging-runner.md`. **ORCH-123 (фикс стратегии исполнения — [adr-0049](adr/adr-0049-host-side-docker-execution-boundary.md)):** раннер ORCH-115 вызывал `docker exec` **изнутри** прод-контейнера, где **нет docker CLI** (образ несёт только `openssh-client git curl`, `Dockerfile:11`) → `FileNotFoundError` → постоянный environment-дефект ложно откатывался как код-фейл `deploy-staging → development` (инцидент ORCH-116). Фикс: `build_staging_command` теперь исполняет ту же `docker exec … staging_check.py … --mode stub` **host-side** через доверенный ssh-канал `ssh ''` (зеркало `self_deploy`/`image_freshness`, флаг `ORCH_STAGING_RUNNER_EXEC_HOST_SIDE=true`, дефолт; меняется **инициатор/канал**, не сама сюита — она по-прежнему бежит внутри `orchestrator-staging` 8501). Двухуровневый исход ORCH-115 заменён **трёхсторонней классификацией** (`classify_staging_outcome`): `suite-ran` (распознанный exit-код без env-маркера в stderr → доверяем коду: `0→SUCCESS`/`≠0→FAILED`+прежний откат, анти-over-tolerance), `permanent-env` (spawn-error/нет ssh-target/126-127/env-маркер `No such container`/`Cannot connect to the Docker daemon` → **немедленный infra-HOLD**: без отката, без developer-retry, DEFER пропускается), `transient-infra` (timeout/ssh 255 → bounded DEFER, на исчерпании → infra-HOLD, **не** прежний fail-closed FAILED+откат). Корневой инвариант: «сюита **не** исполнилась» (environment ИЛИ инфра) **никогда** не оканчивается код-фейл-откатом — закрывает RCA-класс ORCH-110 на staging-ребре. Plus prod-like preflight host-side канала в `main.lifespan` (best-effort, never-blocks). `STAGE_TRANSITIONS`/`QG_CHECKS`/`check_staging_status`/machine-verdict/схема БД — байт-в-байт не тронуты. Детали — `docs/work-items/ORCH-123/06-adr/ADR-001-host-side-staging-execution-and-env-classification.md`. +- **Test-runner** (`src/test_runner.py`, ORCH-116 — [adr-0049](adr/adr-0049-deterministic-test-runner.md)) — чистый **never-raise** leaf (зеркало `staging_runner`), заменяющий **LLM-агента `tester` на стадии `testing`** детерминированным кодом (**второй** реализованный срез determinization-roadmap, A5/`needs-hybrid-fallback`). Перехват в `launch_job` **до `_spawn`** (рядом с D1/D2/ORCH-115): дискриминатор — роль `tester` **И** стадия задачи `testing` (defense-in-depth: `tester` — единственный агент входа в `testing`) **И** `applies(repo)` (kill-switch `test_runner_enabled` + скоуп `test_runner_repos`, пусто → self-hosting only, **И** резолв тест-контракта `_has_test_contract` — репо без контракта → LLM-tester, BR-9). `should_intercept` never-raise → `False` → штатный `_spawn` (fail-safe к LLM). Раннер (зеркало `run_staging_gate`) исполняет регресс `python -m pytest ` **в worktree ветки** (анти checkout-гонка) через `proc_group` (tree-kill, таймаут `test_runner_timeout_s=900`) + опц. **read-only smoke** (`/health`/`/status`/`/queue` + блок `serial_gate`, `test_runner_smoke_enabled`; транзиентная недостижимость — ограниченный ретрай, не-200/нет блока — немедленный FAIL), маппит exit-код единым контрактом `self_deploy.map_exit_code_to_status` в токенах `result:` (`0→PASS`/иначе→`FAIL`; smoke-провал AND-ится в `FAIL`), пишет `13-test-report.md` (тот же machine-key `result:`, `author_agent: test-runner`/`model_used: n/a`; **52c-`status:` ВСЕГДА выровнен по вердикту** `success`/`failed` — иначе негативный токен в `status:` при `result: PASS` дал бы ложный FAIL через `_parse_tests_verdict`, D6.1) + best-effort push в фичеветку, и вызывает **существующий** `advance_stage(current_stage="testing", finished_agent="tester")` — без новых рёбер/исходов; transition-lease ORCH-114 берётся внутри `advance_stage` (раннер не трогает). **Двухуровневый исход (анти-ORCH-110):** сюита исполнилась → verdict→advance (FAIL → тот же откат `testing → development` + developer-retry, что у LLM); tool-error/таймаут (сюита НЕ исполнилась) → bounded defer (re-queue **`tester`**-джоба, `test_runner_infra_max_retries`) → fail-closed `FAIL` + INFRA-alert на исчерпании (инфра ≠ код-фейл, никогда тихий advance/ложный green, не жжёт developer-retry на транзиентной инфре). **Гибрид (BR-8/NFR-7):** LLM строго off-control-path — детерминированный раннер единственный продюсер `result:`; будущий off-control-path триаж падений не выносит/не переопределяет вердикт и не добавляет ребро в `STAGE_TRANSITIONS` (в Phase 1 не реализован). `STAGE_TRANSITIONS`/`QG_CHECKS`/`check_tests_passed`/`_parse_tests_verdict`/machine-verdict/схема БД — **байт-в-байт не тронуты** (замена *продюсера*, не гейта). Наблюдаемость — in-process счётчики + блок `test_runner` в `GET /queue`. Откат — `ORCH_TEST_RUNNER_ENABLED=false` (прежний LLM-tester-путь байт-в-байт). Детали — `docs/work-items/ORCH-116/06-adr/ADR-001-deterministic-test-runner.md`. - **Queue** (`src/queue_worker.py`, ORCH-1) — персистентная очередь задач (SQLite `jobs`), atomic claim, max_concurrency, ретраи, restart-safe. **ORCH-026:** `claim_next_job` гейтит задачи с незавершёнными зависимостями (`job_deps`, `NOT EXISTS`) без занятия слота; декларации/циклы — leaf `src/task_deps.py`. - **Job-reaper** (`src/job_reaper.py`, ORCH-065 — [adr-0011](adr/adr-0011-job-reaper-lease-reclaim.md)) — фоновый daemon-поток (каркас `reconciler`), стартует/останавливается в `main.lifespan` (после `reconciler.start()` / перед `worker.stop()`). Детектирует «мёртвый» `running`-job **без рестарта** процесса (Tier-1 мёртвый `jobs.pid` после `reaper_dead_ticks` тиков; Tier-2 `agent_runs.exit_code` записан, а job ещё `running`; Tier-3 backstop `reaper_max_running_s`) и приводит строку к корректному статусу через те же контракты (`_try_advance_stage`/`_finalize_job`, gate-driven; exit≠0/неизвестно → `attempts`. `STAGE_TRANSITIONS`/`QG_CHECKS`/`check_*`/machine-verdict/схемы существующих таблиц — байт-в-байт. Подробнее ниже (§ «Единое владение side-effectful переходами»). Детали — `docs/work-items/ORCH-114/06-adr/ADR-001-transition-ownership-lease-and-stage-cas.md`. @@ -447,7 +448,14 @@ ORCH-079 синхронизирует витрину с кодом и закры (перехват в `launch_job` до `_spawn`, как D1/D2) — A6 переходит из «план» в «реализовано». Карта `llm-call-sites.md` (A6), `llm-determinization-roadmap.md` (rank 1, инвариант «ровно один `first_slice`») и анти-дрейф-тесты обновляются **атомарно с кодом** в development (норматив - сопровождения NFR-6). Второй кандидат — `tester`-гибрид (rank 2) — остаётся планом. + сопровождения NFR-6). +- **Второй срез реализован (ORCH-116 — [adr-0049](adr/adr-0049-deterministic-test-runner.md)):** + rank-2 кандидат **tester (`result:`)** заменён детерминированным `src/test_runner.py` (тем же + перехватом до `_spawn`) — A5 переходит из «план» в «реализовано». `needs-hybrid-fallback`-природа + сохранена: детерминированное ядро (exit-код `pytest` + smoke) вынесено в раннер, LLM-ветвь жива как + fallback / будущий off-control-path триаж (не продюсер `result:`). Машинные блоки карты/roadmap + держат `first_slice=yes` у rank 1 (deployer) и `no`/`hybrid_needed=yes` у rank 2 (tester) — инвариант + не нарушен; правки атомарны с кодом (NFR-6). - ADR: [adr-0047](adr/adr-0047-llm-usage-policy-and-call-site-map.md); детально — `docs/work-items/ORCH-118/06-adr/ADR-001-llm-call-site-map-and-determinization-roadmap.md`. diff --git a/docs/architecture/internals.md b/docs/architecture/internals.md index 781743f..43f905b 100644 --- a/docs/architecture/internals.md +++ b/docs/architecture/internals.md @@ -109,6 +109,19 @@ LLM). Leaf `src/staging_runner.py` (зеркало `run_deploy_finalizer`) ис `self_deploy.map_exit_code_to_status`, пишет `15-staging-log.md` (тот же machine-key `staging_status:`) и вызывает существующий `advance_stage(finished_agent="deployer")` (см. §5). Так LLM-агент `deployer` на `deploy-staging` исчезает из happy-path; `STAGE_TRANSITIONS`/`QG_CHECKS`/схема БД не тронуты. +**ORCH-116 ([adr-0049](adr/adr-0049-deterministic-test-runner.md)):** тем же паттерном (рядом с +ORCH-115) перехватывается джоб `tester` на стадии `testing` для in-scope репо с тест-контрактом +(дискриминатор — роль `tester` **И** `tasks.stage == "testing"` **И** `test_runner.applies(repo)` под +kill-switch `test_runner_enabled`, скоуп `test_runner_repos`, резолв `_has_test_contract`; пусто → +self-hosting only; `should_intercept` never-raise → `False` → штатный `_spawn`, fail-safe к LLM). Leaf +`src/test_runner.py` (зеркало `run_staging_gate`) исполняет регресс `pytest ` **в worktree +ветки** через `proc_group` (tree-kill, таймаут `test_runner_timeout_s`) + read-only smoke, маппит +exit-код `self_deploy.map_exit_code_to_status` в токенах `result:` (`0→PASS`/иначе→`FAIL`), пишет +`13-test-report.md` (тот же machine-key `result:`; 52c-`status:` выровнен по вердикту — D6.1) и вызывает +существующий `advance_stage(finished_agent="tester")` (см. §5). Двухуровневый исход (анти-ORCH-110): +сюита НЕ исполнилась → bounded defer re-queue **`tester`**-джоба, не код-фейл-откат. Так LLM-агент +`tester` на `testing` исчезает из happy-path; `STAGE_TRANSITIONS`/`QG_CHECKS`/`check_tests_passed`/схема +БД не тронуты. ### 5. Auto-advance (`launcher._try_advance_stage`) diff --git a/docs/architecture/llm-call-sites.md b/docs/architecture/llm-call-sites.md index 4c361c2..b4f2a1f 100644 --- a/docs/architecture/llm-call-sites.md +++ b/docs/architecture/llm-call-sites.md @@ -57,7 +57,7 @@ | **A2** | `.openclaw/agents/architect.md` | стадия `architecture` | architect | `06-adr/`, `07` | — | `check_architecture_done:62` (наличие 06-adr/07) | ~80–200k / 5–20 мин | да (через S0) | **P** | `keep-LLM` | архитектурное решение / ADR — настоящее суждение | | **A3** | `.openclaw/agents/developer.md` | стадия `development` | developer | код + PR | — | `check_ci_green:82` (+ `check_branch_mergeable:657`) — CI/merge | ~150–400k / 10–40 мин | да (через S0) | **P** | `keep-LLM` | написание кода — настоящее суждение; гейт судит CI/merge, не самоотчёт | | **A4** | `.openclaw/agents/reviewer.md` | стадия `review` | reviewer | `12-review.md` | `verdict:` | `check_reviewer_verdict:336` (`verdict:`) | ~100–300k / 5–25 мин | да (через S0) | **C** | `keep-LLM` | control path, но вердикт «приемлемость кода/решения» **НЕ деривируем** из exit-кода — настоящее суждение | -| **A5** | `.openclaw/agents/tester.md` | стадия `testing` | tester | `13-test-report.md` | `result:` | `check_tests_passed:182` → `_parse_tests_verdict:226` (`result:`) | ~60–150k / 5–20 мин | да (через S0) | **C** | `needs-hybrid-fallback` | **avoidable**: PASS/FAIL = exit-code `pytest`+smoke (деривируем); LLM нужен лишь на триаж падений / маппинг TC↔критерии | +| **A5** | `.openclaw/agents/tester.md` | стадия `testing` | tester | `13-test-report.md` | `result:` | `check_tests_passed:182` → `_parse_tests_verdict:226` (`result:`) | ~60–150k / 5–20 мин | да (через S0; для in-scope репо на `testing` — **нет**, перехват до `_spawn`) | **C** | `needs-hybrid-fallback` | **avoidable, СРЕЗ РЕАЛИЗОВАН (ORCH-116):** на `testing` для self-hosting `orchestrator` (репо с тест-контрактом) вердикт `result:` производит детерминированный `src/test_runner.py` (перехват в `launch_job` до `_spawn`, как D1/D2) — exit-код `pytest` в worktree + read-only smoke. LLM-ветвь остаётся **fallback'ом** под выключенным флагом / для репо без контракта / как будущий **off-control-path** триаж падений (маппинг TC↔критерии), который НЕ выносит и НЕ переопределяет `result:` (гибрид-природа `needs-hybrid-fallback` сохранена) | | **A6** | `.openclaw/agents/deployer.md` | стадии `deploy-staging` / `deploy` | deployer | `15-staging-log.md` / `14-deploy-log.md` | `staging_status:` / `deploy_status:` | `check_staging_status:599` → `_parse_staging_status:538` (`staging_status:`); `check_deploy_status:473` → `_parse_deploy_status:413` (`deploy_status:`) | ~40–120k / 3–15 мин | да (через S0; для in-scope репо на `deploy-staging` — **нет**, перехват до `_spawn`) | **C** | `replace-deterministic-now` | **avoidable, СРЕЗ РЕАЛИЗОВАН (ORCH-115):** на `deploy-staging` для self-hosting `orchestrator` вердикт `staging_status:` производит детерминированный `src/staging_runner.py` (перехват в `launch_job` до `_spawn`, как D1/D2) — маппинг exit-кода `staging_check.py`; прод уже детерминирован Phase A/B/C (ORCH-036). LLM-ветвь остаётся fallback'ом под выключенным флагом / для не-self репо | | **D1** | `src/agents/launcher.py:389` (перехват в `launch_job` **до** `_spawn`; «Not an LLM spawn» `407`) | post-deploy edge | deploy-finalizer | jobs-row | — | — | — (детерминированный) | **нет** (слот, перехват до `_spawn`) | — | `already-deterministic` (эталон) | Занимает слот агента, но LLM не консультируется — рабочий прецедент замены | | **D2** | `src/agents/launcher.py:394` (перехват в `launch_job` **до** `_spawn`; «Not an LLM spawn» `428`) | post-deploy observation | post-deploy-monitor | jobs-row | — | — | — (детерминированный) | **нет** (слот, перехват до `_spawn`) | — | `already-deterministic` (эталон) | Тик наблюдения; LLM не консультируется | @@ -154,7 +154,13 @@ `replace-deterministic-now` без ввода второго транспорта (раннер LLM не зовёт) — карта/инвариант «единственный транспорт S0» соблюдены. Машинный `ORCH-118-INVENTORY-BLOCK` сохраняет deployer как `avoidable=yes`/`axis=C` (LLM-ветвь жива как fallback под выключенным флагом / для не-self репо). - Второй кандидат (`tester`) остаётся follow-up'ом по роли. + **Второй срез — tester (`result:`)** — реализован **ORCH-116** (`src/test_runner.py`, тем же + перехватом до `_spawn`): на `testing` для in-scope репо вердикт `result:` производит + детерминированный код (exit-код `pytest` в worktree + read-only smoke), не LLM. Это + `needs-hybrid-fallback`, тоже без второго транспорта (раннер LLM не зовёт). Машинный + `ORCH-118-INVENTORY-BLOCK` сохраняет tester как `avoidable=yes`/`axis=C`/ + `classification=needs-hybrid-fallback` — LLM-ветвь жива как fallback (выключенный флаг / репо без + контракта) и как будущий **off-control-path** триаж, который не выносит `result:`. - **Анти-дрейф тесты:** `tests/test_llm_call_site_inventory.py` (TC-01…TC-06, TC-09, TC-12, TC-13, TC-14) и `tests/test_llm_determinization_docs.py` (TC-07/08/11). Дискриминатор всех проверок — **«консультирует LLM», а не «спавнит subprocess»**. diff --git a/docs/architecture/llm-determinization-roadmap.md b/docs/architecture/llm-determinization-roadmap.md index 3f6579a..329f861 100644 --- a/docs/architecture/llm-determinization-roadmap.md +++ b/docs/architecture/llm-determinization-roadmap.md @@ -39,12 +39,24 @@ замены агента уже снят рабочим паттерном. 4. **`replace-deterministic-now`, без hybrid-fallback** (в отличие от tester). -## 2. Второй кандидат — **tester (гибрид)** +## 2. Второй кандидат — **tester (гибрид)** — ✅ РЕАЛИЗОВАН (ORCH-116) + +> **Статус: реализовано.** Срез выполнен в **ORCH-116** — `src/test_runner.py` (перехват в +> `launch_job` до `_spawn`, как `D1`/`D2`/ORCH-115): на стадии `testing` для self-hosting +> `orchestrator` вердикт `result:` производит детерминированный код (exit-код `pytest tests/` в +> worktree ветки + read-only smoke `/health`/`/status`/`/queue`+`serial_gate`, маппинг через +> `self_deploy.map_exit_code_to_status` в токенах `PASS`/`FAIL`), а не LLM. Под kill-switch +> `test_runner_enabled` + скоуп `test_runner_repos` (пусто → self-hosting only) + резолв тест-контракта +> (репо без контракта → LLM-tester, fail-safe). Контракт артефакта/гейта `check_tests_passed`/ +> `STAGE_TRANSITIONS`/схема БД — не тронуты. Запись `rank 2` в машинном блоке §4 сохраняется +> (`first_slice = no`, `hybrid_needed = yes` — инвариант карты) как фиксация второго среза. Детерминированное ядро (`pytest` + smoke даёт PASS/FAIL по exit-коду) покрывает основной вердикт; LLM-фолбэк сохраняется **только** на суждение, не сводимое к exit-коду: **триаж падений** и **маппинг TC ↔ критерии приёмки**. Поэтому `needs-hybrid-fallback`, а не `replace-deterministic-now`: поверхность -суждения шире и объём работы больше. +суждения шире и объём работы больше. В Phase 1 (ORCH-116) детерминированное ядро вынесено в раннер; +off-control-path LLM-триаж (он **не** выносит и **не** переопределяет `result:`, **не** добавляет ребро +в `STAGE_TRANSITIONS`) зафиксирован как Phase 2 follow-up по роли и в этом срезе не реализуется. ## 3. Общие требования к каждому follow-up'у diff --git a/docs/architecture/llm-usage-policy.md b/docs/architecture/llm-usage-policy.md index c4b40a0..f0bccd6 100644 --- a/docs/architecture/llm-usage-policy.md +++ b/docs/architecture/llm-usage-policy.md @@ -99,3 +99,10 @@ Call-site является **avoidable LLM control path** тогда и толь детерминированный `src/staging_runner.py` производит `staging_status:` без `_spawn` (перехват до него, как `D1`/`D2`) — раннер **LLM не зовёт** и **второй транспорт не вводит**, поэтому инвариант «единственный транспорт S0» соблюдён (TC-12 зелёный). Это образец для последующих срезов roadmap'а. +- **Реализованный срез (ORCH-116).** ORCH-116 снял A5/tester тем же паттерном: детерминированный + `src/test_runner.py` производит `result:` на `testing` для in-scope репо без `_spawn` (перехват до + него, как `D1`/`D2`) — exit-код `pytest` в worktree + read-only smoke. Это `needs-hybrid-fallback` + (детерминированное ядро вынесено; LLM-фолбэк на не-деривируемое суждение — триаж падений / маппинг + TC↔критерии — остаётся **off-control-path** и **не** производит `result:`). Раннер **LLM не зовёт** и + **второй транспорт не вводит** → инвариант «единственный транспорт S0» соблюдён (TC-12 зелёный). + Будущий off-control-path триаж — **не** новый транспорт control-path-консультации (он вне control-path). diff --git a/docs/overview/tech-agents.md b/docs/overview/tech-agents.md index 7d919ed..05f3f31 100644 --- a/docs/overview/tech-agents.md +++ b/docs/overview/tech-agents.md @@ -54,6 +54,14 @@ Machine-verdict ключи читаются гейтами **только из Y под выключенным флагом / для не-self репо и продолжает вести прод-стадию `deploy`. Подробнее — [конвейер](tech-pipeline.md) и [карта LLM-консультаций](../architecture/llm-call-sites.md). +Особенность (ORCH-116): на стадии `testing` для self-hosting `orchestrator` LLM-`tester` заменён +**детерминированным test-раннером** (`src/test_runner.py`) — его PASS/FAIL-ядро деривируемо (exit-код +`pytest` в worktree + read-only smoke), вердикт `result:` производит детерминированный код. Это +гибрид (`needs-hybrid-fallback`): LLM-промпт `tester` остаётся fallback'ом под выключенным флагом / для +репо без тест-контракта, а будущий off-control-path триаж падений не выносит и не переопределяет +`result:`. Подробнее — [конвейер](tech-pipeline.md) и +[карта LLM-консультаций](../architecture/llm-call-sites.md). + ## Человек как седьмая роль Человек не пишет артефакты конвейера, но принимает два решения, которые не делегированы diff --git a/docs/overview/tech-pipeline.md b/docs/overview/tech-pipeline.md index a4d35e5..b7c678c 100644 --- a/docs/overview/tech-pipeline.md +++ b/docs/overview/tech-pipeline.md @@ -44,6 +44,17 @@ created → analysis → architecture → development → review → testing → > на стадии снова работает LLM-`deployer` байт-в-байт. Это первый реализованный срез > determinization-roadmap (см. `docs/architecture/llm-determinization-roadmap.md`). +> **Детерминированный test-раннер (ORCH-116).** На стадии `testing` для self-hosting `orchestrator` +> работу ведёт **детерминированный код** (`src/test_runner.py`), а не LLM-агент `tester`: он +> перехватывается в `launch_job` до запуска агента (тем же паттерном, что staging-раннер), исполняет +> регресс `pytest` в worktree ветки + read-only smoke, маппит exit-код в `result:` и инициирует **тот +> же** гейт `check_tests_passed`. Это замена *продюсера* артефакта, а не гейта: контракт +> `13-test-report.md`, имя/семантика `check_tests_passed`/`_parse_tests_verdict`, `STAGE_TRANSITIONS` +> — не изменились. Под kill-switch `test_runner_enabled` (скоуп `test_runner_repos`, пусто → +> self-hosting only; репо без тест-контракта → LLM-tester); при выключении снова работает LLM-`tester` +> байт-в-байт. Это второй реализованный срез determinization-roadmap (гибрид: LLM-фолбэк остаётся на +> off-control-path триаж, не на вынесение `result:`). + ## Под-гейты деплойного ребра — врезки, не стадии На переходе `deploy-staging → deploy` исполняются четыре под-гейта в нормативном порядке diff --git a/docs/overview/tech-quality-security.md b/docs/overview/tech-quality-security.md index abcae9c..602e319 100644 --- a/docs/overview/tech-quality-security.md +++ b/docs/overview/tech-quality-security.md @@ -53,7 +53,10 @@ control-path и его вердикт деривируем из exit-кодов анти-дрейф тестами. **Первый срез реализован (ORCH-115):** на `deploy-staging` для self-hosting `orchestrator` LLM-`deployer` заменён детерминированным `src/staging_runner.py` (вердикт `staging_status:` = маппинг exit-кода staging-сюиты); LLM-ветвь остаётся fallback'ом, гейт -`check_staging_status` не тронут. Замена второго кандидата (`tester`) — follow-up по роли. +`check_staging_status` не тронут. **Второй срез реализован (ORCH-116):** на `testing` для self-hosting +`orchestrator` LLM-`tester` заменён детерминированным `src/test_runner.py` (вердикт `result:` = exit-код +`pytest` + read-only smoke); это гибрид (`needs-hybrid-fallback`) — LLM-ветвь остаётся fallback'ом / +будущим off-control-path триажем, гейт `check_tests_passed`/`_parse_tests_verdict` не тронут. - Карта вызовов LLM: [`../architecture/llm-call-sites.md`](../architecture/llm-call-sites.md) - Нормативная политика: [`../architecture/llm-usage-policy.md`](../architecture/llm-usage-policy.md) diff --git a/src/agents/launcher.py b/src/agents/launcher.py index 015eefd..643a4e5 100644 --- a/src/agents/launcher.py +++ b/src/agents/launcher.py @@ -406,6 +406,17 @@ class AgentLauncher: from .. import staging_runner if staging_runner.should_intercept(job): return self._run_staging_runner_job(job) + # ORCH-116: deterministic test-runner intercept (BEFORE _spawn). The LLM + # ``tester`` on the ``testing`` stage (self-hosting scope, repo with a + # test-contract) is replaced by a DETERMINISTIC test-runner (same precedent + # as deploy-finalizer / post-deploy-monitor / staging-runner). ``tester`` is + # the only agent entering ``testing``; the stage guard is defense-in-depth. + # Kill-switch off / out of scope / no contract -> should_intercept False -> + # the prior LLM tester runs via _spawn byte-for-byte. + if job.get("agent") == "tester": + from .. import test_runner + if test_runner.should_intercept(job): + return self._run_test_runner_job(job) return self._spawn( job["agent"], job["repo"], @@ -457,6 +468,28 @@ class AgentLauncher: pass return None + def _run_test_runner_job(self, job: dict): + """ORCH-116: run the deterministic test gate for a tester job on `testing`. + + Not an LLM spawn — there is no subprocess/monitor of an agent, so we mark the + jobs row done/failed here (mirror of _run_staging_runner_job). The runner + never-raises, but we guard anyway so a runner fault can't wedge the worker. + Returns None (no agent_run row, _spawn not called). + """ + from ..db import mark_job + from .. import test_runner + try: + test_runner.run_test_gate(job) + mark_job(job["id"], "done") + logger.info(f"test-runner job {job['id']} done") + except Exception as e: + logger.error(f"test-runner job {job['id']} failed: {e}") + try: + mark_job(job["id"], "failed", error=f"test-runner error: {e}") + except Exception: + pass + return None + def _run_post_deploy_monitor_job(self, job: dict): """ORCH-021: run one deterministic post-deploy monitor tick for a job. diff --git a/src/config.py b/src/config.py index 5c7d9f8..b754f5b 100644 --- a/src/config.py +++ b/src/config.py @@ -475,6 +475,57 @@ class Settings(BaseSettings): staging_runner_infra_retry_delay_s: int = 30 staging_runner_exec_host_side: bool = True + # ORCH-116: deterministic test-runner replacing the LLM `tester` agent on the + # `testing` stage for the self-hosting orchestrator (second realised slice of the + # determinization-roadmap, A5/tester needs-hybrid-fallback; first was ORCH-115/ + # deployer). A new leaf src/test_runner.py (never-raise) is intercepted in + # launch_job BEFORE _spawn (mirroring the deploy-finalizer / post-deploy-monitor / + # staging-runner precedent, launcher.py:397/402/405): it runs the SAME regression + # `python -m pytest ` in the task worktree via proc_group (tree-kill) + + # the read-only smoke (/health,/status,/queue + serial_gate block), maps the + # exit-code -> result: PASS|FAIL via the existing self_deploy.map_exit_code_to_status + # contract, writes 13-test-report.md, and initiates the EXISTING check_tests_passed + # gate exactly as a finished LLM-tester would. The artifact contract, the gate, + # STAGE_TRANSITIONS and the DB schema are byte-for-byte UNCHANGED — this only + # replaces the *producer* of the artifact. Pattern = staging_runner_* / coverage_gate_*. + # See docs/work-items/ORCH-116/06-adr/ADR-001-deterministic-test-runner.md and + # docs/architecture/adr/adr-0049-deterministic-test-runner.md. + # test_runner_enabled -> SINGLE kill-switch (env ORCH_TEST_RUNNER_ENABLED). + # False -> the intercept never fires -> the prior + # LLM tester runs on testing via _spawn byte-for-byte + # as before ORCH-116 (D8/AC-7). + # test_runner_repos -> CSV scope (env ORCH_TEST_RUNNER_REPOS). Empty -> + # self-hosting only (orchestrator) via + # is_self_hosting_repo; non-empty -> membership. + # A repo with no resolvable test-contract is never + # intercepted even if listed (BR-9, _has_test_contract). + # test_runner_target -> pytest target of the test-contract (env + # ORCH_TEST_RUNNER_TARGET; convention merge_retest_target). + # test_runner_timeout_s -> wall-clock budget for the pytest regression (env + # ORCH_TEST_RUNNER_TIMEOUT_S). Malformed/non-positive -> + # default + WARNING (never-break). Aligned with the + # cross-cutting budget invariant ORCH-065/109/110 WITHOUT + # touching reaper_max_running_s (D9): it replaces the + # <=1800s LLM testing window (agent_timeout_seconds) with a + # bounded <=900s deterministic one (Σ on the edge does not grow). + # test_runner_smoke_enabled -> optional read-only smoke (env ORCH_TEST_RUNNER_SMOKE_ENABLED). + # False -> pytest exit-code is the sole signal (smoke skipped). + # test_runner_infra_max_retries -> tool-error (suite did NOT execute: spawn-error / timeout / + # returncode None) bounded DEFER budget before a fail-closed + # FAIL (env ORCH_TEST_RUNNER_INFRA_MAX_RETRIES). Mirrors + # staging_runner_infra_max_retries — infra hiccup is NOT a + # code-fault, so it never burns a developer-retry until the + # budget is exhausted (D5, anti ORCH-110). + # test_runner_infra_retry_delay_s-> delay before the re-queued tester job (env + # ORCH_TEST_RUNNER_INFRA_RETRY_DELAY_S). + test_runner_enabled: bool = True + test_runner_repos: str = "" + test_runner_target: str = "tests/" + test_runner_timeout_s: int = 900 + test_runner_smoke_enabled: bool = True + test_runner_infra_max_retries: int = 2 + test_runner_infra_retry_delay_s: int = 30 + # ORCH-098 (FND/F2): machine lessons-journal — additive `lessons` table + leaf # src/lessons.py (never-raise observer, by образцу serial_gate/coverage_gate/ # metrics). The journal is an OBSERVER, never a Quality Gate: writing a lesson diff --git a/src/main.py b/src/main.py index 9655e91..874c29c 100644 --- a/src/main.py +++ b/src/main.py @@ -251,6 +251,7 @@ async def queue(): from . import checkout_hygiene from . import transition_lease from . import staging_runner + from . import test_runner from .disk_watchdog import disk_watchdog from .build_cache_pruner import build_cache_pruner return { @@ -304,6 +305,11 @@ async def queue(): # failed/tool_error/deferred counters, so a code-fail FAILED is distinguishable # from an infra tool-error. Additive block; never-raise. "staging_runner": staging_runner.snapshot(), + # ORCH-116 (FR-8 / AC-13): deterministic test-runner observability (read-only) + # — kill-switch, scope, target, timeout/smoke/infra budget + run/pass/fail/ + # tool_error/deferred counters, so a code-fail FAIL is distinguishable from an + # infra tool-error. Additive block; never-raise. + "test_runner": test_runner.snapshot(), # ORCH-098 (FR-4 / AC-4): lessons-journal observability (read-only) — # kill-switch + counts by type/status + last N lessons. Additive block; # never-raise (snapshot() returns {"enabled": ...} minimum on error). diff --git a/src/test_runner.py b/src/test_runner.py new file mode 100644 index 0000000..fd80970 --- /dev/null +++ b/src/test_runner.py @@ -0,0 +1,705 @@ +"""Deterministic test-runner (ORCH-116). + +The ``testing`` stage for the self-hosting ``orchestrator`` repo was driven by the +LLM ``tester`` agent, but its PASS/FAIL core is purely deterministic +(``.openclaw/agents/tester.md`` steps 1-3): run the regression ``pytest tests/`` in +the task worktree, do a read-only smoke (``/health`` / ``/status`` / ``/queue`` + +``serial_gate`` block), map the exit-code to a verdict (``0 -> PASS``, else ``FAIL``) +and write ``13-test-report.md`` with the machine frontmatter ``result: PASS|FAIL``. +This leaf replaces that LLM consultation with deterministic code, intercepted in +``launcher.launch_job`` BEFORE ``_spawn`` (the ``deploy-finalizer`` / +``post-deploy-monitor`` / ``staging-runner`` reserved-agent precedent, +``launcher.py:397/402/405``). It is the second realised slice of the +determinization-roadmap (A5/tester, ``needs-hybrid-fallback``; the first was +ORCH-115/deployer — ``src/staging_runner.py``). + +Hybrid nature (the key difference from ORCH-115): tester is +``needs-hybrid-fallback``, not ``replace-deterministic-now``. Its PASS/FAIL core is +fully derivable (pytest exit-code + smoke) and that is what this deterministic runner +owns; a future OPTIONAL **off-control-path** LLM triage/diagnosis after a +deterministic FAIL stays allowed (a separate role/job that never writes/overrides +``result:`` and never adds a STAGE_TRANSITIONS edge). Phase 1 does NOT implement the +triage but the architecture does not forbid it (D11/FR-9/AC-12). + +What is and is NOT changed (NFR-1, the critical invariant): + * UNCHANGED — the artifact contract (``13-test-report.md`` with ``result: + PASS|FAIL``), the gate ``check_tests_passed`` / ``_parse_tests_verdict``, + ``STAGE_TRANSITIONS``, the DB schema. This module replaces only the *producer* of + the artifact, never the gate that reads it. + * NEW — a deterministic producer + a launch_job intercept. Under a kill-switch + + repo-scope CSV + a test-contract resolve; fail-safe back to the LLM path when off + / out of scope / no contract. + +This module is a **leaf** (mirror of ``staging_runner`` / ``self_deploy`` / +``proc_group``): it imports only ``config`` / ``logging`` / ``proc_group`` at module +load; ``db`` / ``git_worktree`` / ``self_deploy.map_exit_code_to_status`` / +``qg.checks`` / ``stage_engine.advance_stage`` / ``notifications`` are imported LAZILY +inside functions so the heavy ``stage_engine`` is never pulled at import and no import +cycle forms. Every public function honours a **never-raise** contract (AC-9): a test +infra hiccup can never crash the worker / wedge the queue. + +Two-level outcome (D5 — the key safety decision, anti ORCH-110): + * the suite EXECUTED (a real exit-code, 0 or non-zero) -> trust the code: + ``0 -> PASS -> advance``; ``!=0 -> FAIL -> the existing rollback testing -> + development`` (same developer-retry path as a FAIL LLM verdict). A smoke failure + AND-s into ``FAIL`` (deterministic), it is NOT a tool-error. + * the suite did NOT execute (tool-error: spawn-error / timeout / ``returncode is + None``) -> an infra fault, NOT a code fault -> a bounded DEFER (re-queue a fresh + ``tester`` job with a delay + a restart-safe marker). On budget exhaustion -> + fail-closed ``FAIL`` + advance + alert. So the runner NEVER does a silent advance + / false green, and NEVER wedges the queue, but does NOT burn a developer-retry on + transient infra. +""" + +import logging +import time +import urllib.error +import urllib.request + +from .config import settings +from . import proc_group + +logger = logging.getLogger("orchestrator.test_runner") + +# Default wall-clock budget for the pytest regression (D9). Kept <= the LLM testing +# window it replaces (agent_timeout_seconds=1800) so Σ(work on the testing edge) does +# not grow and the cross-cutting reaper invariant (ORCH-065/109/110) holds WITHOUT +# touching reaper_max_running_s. 900s = ~74% headroom over the ~517s regression suite. +_DEFAULT_TIMEOUT_S = 900 + +_GIT_TIMEOUT = 60 + +# Read-only smoke knobs (D3). Transient unreachability (connection refused / timeout) +# is retried briefly before a FAIL so a single prod-8500 blip does not roll back a +# healthy branch; a reachable-but-wrong-form response (non-200 / no serial_gate block) +# is an immediate FAIL. +_SMOKE_TIMEOUT_S = 5 +_SMOKE_MAX_ATTEMPTS = 3 +_SMOKE_BACKOFF_S = 1.0 + +# Restart-safe DEFER marker (counted from the persisted jobs queue, mirror of +# staging_runner._INFRA_RETRY_MARKER / stage_engine._merge_infra_retry_count). Embedded +# verbatim in the re-queued job's task_content so a service restart never resets the +# infra-retry budget. +_INFRA_RETRY_MARKER = "test-runner infra-retry" + +# In-process observability counters (mirror staging_runner._STAGING_RUNNER_COUNTERS). +_TEST_RUNNER_COUNTERS: dict = { + "runs": 0, # run_test_gate entered + "pass": 0, # suite ran, exit 0 + smoke ok -> PASS + "fail": 0, # suite ran non-zero / smoke failed, OR infra budget exhausted -> FAIL + "tool_error": 0, # suite did NOT execute (spawn-error / timeout / None) + "deferred": 0, # bounded infra DEFER (re-queued) +} + + +def _bump(key: str) -> None: + """Increment an observability counter. Never raises.""" + try: + _TEST_RUNNER_COUNTERS[key] += 1 + except Exception: # noqa: BLE001 - observability must never break a decision + pass + + +# --------------------------------------------------------------------------- +# Conditionality (D8 / FR-7 / AC-7 / AC-8) +# --------------------------------------------------------------------------- +def _has_test_contract(repo: str) -> bool: + """Whether a test-contract (regression + smoke) is resolvable for ``repo`` (BR-9). + + In Phase 1 the contract is known by default ONLY for the self-hosting repo + (``orchestrator`` — ``pytest tests/`` + the read-only smoke); for every other repo + there is no contract yet (Phase 2 project test-contract) -> ``applies`` is False -> + the prior LLM-tester runs (fail-safe, AC-8). This makes AC-8 checkable: even if a + non-self repo (e.g. ``enduro-trails``) is hand-added to ``test_runner_repos``, + ``_has_test_contract`` returns False -> it is never intercepted. never-raise. + """ + try: + from .qg.checks import is_self_hosting_repo + return is_self_hosting_repo(repo) + except Exception as e: # noqa: BLE001 - never-raise contract + logger.warning("test_runner._has_test_contract error for %s: %s", repo, e) + return False + + +def applies(repo: str) -> bool: + """Whether the deterministic test-runner is REAL for ``repo``. + + Mirrors ``staging_runner.applies`` / ``coverage_gate``: + * ``test_runner_enabled=False`` -> always False (global kill-switch); the legacy + LLM-tester path runs on ``testing`` via ``_spawn``. + * ``test_runner_repos`` (CSV) non-empty -> only the listed repos. + * empty CSV -> only the self-hosting repo (``orchestrator``). + * AND a test-contract is resolvable for the repo (BR-9, ``_has_test_contract``). + Checked FIRST in ``should_intercept`` (local, no network, no DB) so a disabled flag + costs nothing. never-raise -> False (fail-safe to the prior LLM path). + """ + try: + if not settings.test_runner_enabled: + return False + raw = (settings.test_runner_repos or "").strip() + if raw: + allowed = {r.strip().lower() for r in raw.split(",") if r.strip()} + in_scope = (repo or "").strip().lower() in allowed + else: + # Lazy import keeps this module a leaf (no qg import at module load). + from .qg.checks import is_self_hosting_repo + in_scope = is_self_hosting_repo(repo) + if not in_scope: + return False + return _has_test_contract(repo) + except Exception as e: # noqa: BLE001 - never-raise contract + logger.warning("test_runner.applies error for %s: %s", repo, e) + return False + + +def should_intercept(job: dict) -> bool: + """True iff this ``tester`` job is the deterministic test-suite job (D1). + + ``tester`` is the ONLY agent that enters the ``testing`` stage + (``STAGE_TRANSITIONS["review"]["agent"]``), so there is no stage collision as + there was for the shared ``deployer`` role (ORCH-115); the + ``tasks.stage == "testing"`` guard is kept as defense-in-depth (R-1) — it stops a + stray future ``tester`` job outside ``testing`` from being intercepted. Intercept + iff ``agent == "tester"`` AND ``applies(repo)`` AND ``tasks.stage == "testing"``. + + never-raise -> False (a DB-lookup failure falls through to ``_spawn``, fail-safe to + the prior LLM path). + """ + try: + if (job.get("agent") or "") != "tester": + return False + # applies() FIRST (local, no DB): disabled flag -> zero DB overhead. + if not applies(job.get("repo")): + return False + task_id = job.get("task_id") + if task_id is None: + return False + from .db import get_db + conn = get_db() + row = conn.execute("SELECT stage FROM tasks WHERE id=?", (task_id,)).fetchone() + conn.close() + if not row: + return False + return (row[0] or "") == "testing" + except Exception as e: # noqa: BLE001 - never-raise contract + logger.warning("test_runner.should_intercept error: %s", e) + return False + + +# --------------------------------------------------------------------------- +# Suite execution (D3 / FR-2 / NFR-3 / AC-9 / AC-11) +# --------------------------------------------------------------------------- +def build_test_command() -> list[str]: + """Build the canonical regression argv (the same command the LLM-tester ran). + + ``python -m pytest -q`` (default ``tests/``, convention + ``merge_retest_target``). Self-hosting safety (BR-7 / AC-10 / TC-13): NO restart of + 8500, NO ``docker compose up orchestrator`` / ``--build``, NO force-push, NO + ``.env`` edit — the runner only executes pytest in the task worktree and does + read-only GETs. + """ + target = (settings.test_runner_target or "tests/").strip() or "tests/" + return ["python", "-m", "pytest", target, "-q"] + + +def _resolve_timeout() -> int: + """Resolve ``test_runner_timeout_s`` (malformed/non-positive -> default + WARNING, + never-break — mirror of ``staging_runner._resolve_timeout`` / + ``merge_gate._resolve_retest_timeout``).""" + raw = getattr(settings, "test_runner_timeout_s", _DEFAULT_TIMEOUT_S) + try: + t = int(raw) + if t > 0: + return t + logger.warning( + "test_runner_timeout_s non-positive (%r) -> default %ds", raw, _DEFAULT_TIMEOUT_S + ) + except (TypeError, ValueError): + logger.warning( + "test_runner_timeout_s malformed (%r) -> default %ds", raw, _DEFAULT_TIMEOUT_S + ) + return _DEFAULT_TIMEOUT_S + + +def run_test_suite(repo: str, branch: str) -> proc_group.ProcResult: + """Execute the pytest regression IN THE TASK WORKTREE, tree-killed on timeout. + + Runs in ``git_worktree.get_worktree_path(repo, branch)`` (NOT the shared + ``/repos/orchestrator`` — anti checkout-race, the same context coverage-gate / + merge-gate re-test use) through ``proc_group.run_in_process_group`` (ORCH-110) so a + hung pytest subtree is killed whole (no orphans, AC-11). Never raises (proc_group + degrades any OS error to a safe ``ProcResult``; a missing worktree -> a ProcResult + with ``returncode is None`` -> the tool-error DEFER path). + """ + cmd = build_test_command() + timeout = _resolve_timeout() + try: + grace = float(getattr(settings, "agent_kill_grace_seconds", 5) or 5) + except (TypeError, ValueError): + grace = 5.0 + try: + from .git_worktree import get_worktree_path + wt = get_worktree_path(repo, branch) + except Exception as e: # noqa: BLE001 - never-raise -> tool-error DEFER + logger.error("run_test_suite: worktree error for %s/%s: %s", repo, branch, e) + return proc_group.ProcResult(returncode=None, stdout="", stderr=str(e), timed_out=False) + return proc_group.run_in_process_group( + cmd, + cwd=wt, + timeout=timeout, + grace_s=grace, + tree_kill=bool(getattr(settings, "subprocess_tree_kill_enabled", True)), + ) + + +# --------------------------------------------------------------------------- +# Read-only smoke (D3 / FR-2 / AC-10) — stdlib urllib, never-raise +# --------------------------------------------------------------------------- +def _http_get(url: str) -> tuple[int, str]: + """GET ``url`` -> (http_code, body). Network/timeout -> (0, ""). Never raises. + + Mirror of ``post_deploy._http_status`` (stdlib only, no httpx import at module + load — keeps the leaf pure). ``urllib`` raises ``HTTPError`` for >=400 responses; + that is treated as a real status code (so a 5xx is observed, not swallowed).""" + try: + with urllib.request.urlopen(url, timeout=_SMOKE_TIMEOUT_S) as resp: # noqa: S310 + body = resp.read(8192).decode("utf-8", "replace") + return int(getattr(resp, "status", resp.getcode())), body + except urllib.error.HTTPError as e: + try: + body = e.read(8192).decode("utf-8", "replace") + except Exception: # noqa: BLE001 + body = "" + return int(e.code), body + except Exception as e: # noqa: BLE001 - URLError / socket timeout / anything + logger.warning("test_runner smoke GET error for %s: %s", url, e) + return 0, "" + + +def run_smoke() -> tuple[bool, str]: + """Read-only smoke against the running orchestrator (D3). Returns (ok, detail). + + Strictly read-only GETs (BR-7/AC-10): ``/health`` (HTTP 200), ``/status`` (HTTP + 200), ``/queue`` (HTTP 200 AND the ``serial_gate`` block present, ORCH-088). A + transient unreachability (code 0 = connection refused / timeout) is retried up to + ``_SMOKE_MAX_ATTEMPTS`` with a short backoff before FAIL (anti-flap, so a single + prod blip does not roll back a healthy branch); a reachable-but-wrong-form response + (non-200 / missing serial_gate) is an immediate FAIL. never-raise -> (False, detail) + on any unexpected error.""" + try: + base = (settings.post_deploy_base_url or "http://localhost:8500").rstrip("/") + except Exception as e: # noqa: BLE001 - never-raise + logger.warning("test_runner.run_smoke: base url error: %s", e) + return False, f"smoke base-url error: {e}" + + # (path, validator(code, body) -> (ok, immediate_fail, note)) + def _ok_200(code: int, body: str) -> tuple[bool, bool, str]: + if code == 200: + return True, False, "" + if code == 0: + return False, False, "unreachable" # transient -> retryable + return False, True, f"HTTP {code}" # reachable wrong form -> immediate FAIL + + def _queue_ok(code: int, body: str) -> tuple[bool, bool, str]: + if code == 0: + return False, False, "unreachable" + if code != 200: + return False, True, f"HTTP {code}" + if "serial_gate" not in (body or ""): + return False, True, "no serial_gate block" + return True, False, "" + + checks = (("/health", _ok_200), ("/status", _ok_200), ("/queue", _queue_ok)) + for path, validator in checks: + url = base + path + note = "unreachable" + for attempt in range(1, _SMOKE_MAX_ATTEMPTS + 1): + code, body = _http_get(url) + ok, immediate_fail, note = validator(code, body) + if ok: + break + if immediate_fail: + return False, f"smoke {path} failed: {note}" + # transient (unreachable) -> bounded retry before declaring FAIL. + if attempt < _SMOKE_MAX_ATTEMPTS: + time.sleep(_SMOKE_BACKOFF_S) + else: + return False, f"smoke {path} {note} after {_SMOKE_MAX_ATTEMPTS} attempts" + return True, "smoke ok (/health, /status, /queue + serial_gate)" + + +# --------------------------------------------------------------------------- +# exit-code -> result: (D4 / FR-3 / AC-3) — reuse the single contract, no 2nd mapping +# --------------------------------------------------------------------------- +def map_exit_code_to_result(exit_code) -> str: + """``0 -> PASS``; non-zero / None / non-int -> ``FAIL`` (fail-closed). + + A thin token-translator over ``self_deploy.map_exit_code_to_status`` — the SAME + pure, unit-tested ``0->SUCCESS`` contract the deploy-finalizer / staging-runner use + (BR-4: no second, drifting mapping). Only the tokens differ (``result:`` uses + ``PASS``/``FAIL``, the ``_TESTS_POSITIVE_TOKENS`` / ``_TESTS_NEGATIVE_TOKENS`` the + gate reads), not the logic.""" + from .self_deploy import map_exit_code_to_status as _map + return "PASS" if _map(exit_code) == "SUCCESS" else "FAIL" + + +# --------------------------------------------------------------------------- +# Artifact 13-test-report.md (D6 / FR-4 / AC-2) — mirror write_staging_log +# --------------------------------------------------------------------------- +def build_test_report( + work_item_id: str, exit_code, result: str, stdout: str = "", + smoke: str = "skipped", *, tool_error: bool = False, +) -> str: + """Render a ``13-test-report.md`` body whose ``result:`` frontmatter is the verdict + ``check_tests_passed`` / ``_parse_tests_verdict`` reads (contract UNCHANGED, AC-2). + Carries the mandatory 52c schema; ``author_agent: test-runner`` / ``model_used: + n/a`` honestly reflect the DETERMINISTIC producer. The machine key ``result:`` and + its UPPERCASE value are NOT changed. + + D6.1 (the tester-specific mine, absent in ORCH-115): ``_parse_tests_verdict`` reads + the verdict from THREE equal-rank fields — ``verdict:`` / ``status:`` / ``result:`` + — with negative-token priority. The 52c-mandatory ``status:`` is therefore read by + the SAME parser, so it MUST be aligned with the verdict and never contradict + ``result:``: ``PASS -> status: success`` (``SUCCESS`` is neither a positive nor a + negative token; the positive ``PASS`` comes from ``result:`` -> the parser gives + True); ``FAIL -> status: failed`` (``FAILED`` is a negative token, consistent with + ``FAIL`` -> False). A ``status:`` literal carrying a negative token at ``result: + PASS`` would be a false FAIL of a healthy run (reviewer ≥P1). + + Written as a literal block (mirror of ``staging_runner.build_staging_log``) so the + machine-read frontmatter is byte-exact; only the frontmatter is machine-read, the + body is informational.""" + import datetime + created = datetime.date.today().isoformat() + sub_status = "success" if result == "PASS" else "failed" + + tail = "" + if stdout: + tail_text = stdout.strip()[-1500:] + if tail_text: + tail = f"\npytest stdout (tail):\n```\n{tail_text}\n```\n" + + note = ( + "Regression suite did NOT execute (tool-error) and the infra-retry budget was " + "exhausted -> fail-closed FAIL." + if tool_error + else f"pytest exit-code `{exit_code}` -> `result: {result}` (smoke: {smoke})." + ) + + return ( + "---\n" + f"result: {result}\n" + f"work_item: {work_item_id}\n" + "stage: testing\n" + "author_agent: test-runner\n" + f"status: {sub_status}\n" + f"created_at: {created}\n" + "model_used: n/a\n" + f"exit_code: {exit_code}\n" + f"smoke: {smoke}\n" + "---\n\n" + "# Test Gate Log (deterministic runner, ORCH-116)\n\n" + f"{note}\n\n" + "Вердикт зафиксирован детерминированным test-раннером (ORCH-116), не LLM. " + "PASS/FAIL = exit-код `pytest` + read-only smoke (`/health`, `/status`, " + "`/queue` + блок `serial_gate`).\n" + f"{tail}" + ) + + +def write_test_report( + repo: str, work_item_id: str, branch: str, exit_code, result: str, + stdout: str = "", smoke: str = "skipped", *, tool_error: bool = False, +) -> bool: + """Write ``13-test-report.md`` into the task worktree (so ``check_tests_passed`` + reads it) and best-effort commit+push it to the FEATURE BRANCH. Returns True iff + the file was written. Never raises. + + Mirror of ``staging_runner.write_staging_log``: the actor name is ``test-runner`` + and the log is pushed only to the feature branch — there is NO separate PR-merge of + the log into ``main`` (the gate reads the worktree first via ``_repo_path``; + excluding any direct work on ``main`` strengthens AC-10 / BR-7). The feature branch + is merged into ``main`` later by the normal merge-gate path.""" + import os + import subprocess + from .git_worktree import get_worktree_path + + rel = f"docs/work-items/{work_item_id}/13-test-report.md" + try: + wt = get_worktree_path(repo, branch) + except Exception as e: # noqa: BLE001 - never-raise + logger.error("write_test_report: worktree error for %s/%s: %s", repo, branch, e) + return False + + path = os.path.join(wt, rel) + content = build_test_report( + work_item_id, exit_code, result, stdout, smoke, tool_error=tool_error + ) + try: + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(content) + except OSError as e: + logger.error("write_test_report: write error at %s: %s", path, e) + return False + + # Best-effort commit + push to the feature branch (the gate also falls back to + # origin/main). ORCH-101: HOME + email domain from Settings; the actor NAME stays + # the platform literal `test-runner` (deterministic system-actor commits). + _email = f"test-runner@{settings.git_email_domain}" + git_env = { + **os.environ, + "HOME": settings.agent_home_dir, + "GIT_AUTHOR_NAME": "test-runner", + "GIT_AUTHOR_EMAIL": _email, + "GIT_COMMITTER_NAME": "test-runner", + "GIT_COMMITTER_EMAIL": _email, + } + try: + subprocess.run(["git", "-C", wt, "add", rel], + capture_output=True, timeout=_GIT_TIMEOUT, env=git_env) + commit = subprocess.run( + ["git", "-C", wt, "commit", "-m", + f"test(ORCH-116): test gate {result} for {work_item_id}"], + capture_output=True, text=True, timeout=_GIT_TIMEOUT, env=git_env, + ) + if commit.returncode == 0: + subprocess.run(["git", "-C", wt, "push", "origin", branch], + capture_output=True, timeout=_GIT_TIMEOUT, env=git_env) + except (subprocess.SubprocessError, OSError) as e: + logger.warning("write_test_report: git commit/push best-effort failed: %s", e) + return True + + +# --------------------------------------------------------------------------- +# Existing-gate initiation (D7 / FR-5 / AC-4) — no new routing branch +# --------------------------------------------------------------------------- +def _advance(task_id, repo: str, work_item_id: str, branch: str) -> None: + """Initiate the SAME ``advance_stage`` evaluation a finished LLM-tester would + (``finished_agent="tester"`` on ``testing``): PASS -> ``testing -> deploy-staging``; + FAIL -> the existing rollback ``testing -> development`` + developer-retry + (``stage_engine.py:849``, matched by ``agent == "tester" and qg_name == + "check_tests_passed"`` — ``finished_agent="tester"`` is MANDATORY, R-2). No new + routing branch. The transition-lease (ORCH-114) is taken INSIDE advance_stage on + the side-effectful edge — the runner never touches it (task boundary O1). + never-raise.""" + try: + from . import stage_engine + stage_engine.advance_stage( + task_id=task_id, + current_stage="testing", + repo=repo, + work_item_id=work_item_id, + branch=branch, + finished_agent="tester", + ) + except Exception as e: # noqa: BLE001 - never-raise into the worker + logger.error( + "test_runner._advance: advance_stage failed for task %s (%s): %s", + task_id, work_item_id, e, + ) + + +# --------------------------------------------------------------------------- +# Two-level outcome (D5) — tool-error DEFER bookkeeping +# --------------------------------------------------------------------------- +def _infra_retry_count(task_id) -> int: + """How many times this task was re-queued by the tool-error DEFER path + (restart-safe; counted from the persisted jobs queue by the marker — mirror of + ``staging_runner._infra_retry_count``). Never raises -> 0 on error.""" + try: + from .db import get_db + conn = get_db() + n = conn.execute( + "SELECT COUNT(*) FROM jobs WHERE task_id=? AND task_content LIKE ?", + (task_id, f"%{_INFRA_RETRY_MARKER}%"), + ).fetchone()[0] + conn.close() + return int(n) + except Exception as e: # noqa: BLE001 - never-raise + logger.warning("test_runner._infra_retry_count error for %s: %s", task_id, e) + return 0 + + +def _handle_tool_error( + task_id, repo: str, work_item_id: str, branch: str, result: proc_group.ProcResult +) -> None: + """Suite did NOT execute (tool-error) -> bounded DEFER, then fail-closed (D5). + + Anti ORCH-110: an infra fault is NOT a code fault, so we re-queue a fresh + ``tester`` job (which re-enters this runner on the still-``testing`` task) with a + delay instead of an immediate FAIL-rollback that would burn a developer-retry. On + budget exhaustion -> write ``result: FAIL`` + advance (the existing rollback) + an + INFRA-specific alert (explicitly "not a code defect"). Never a silent advance / + false green; never wedges the queue. never-raise.""" + retries = _infra_retry_count(task_id) + try: + max_retries = int(settings.test_runner_infra_max_retries) + except (TypeError, ValueError): + max_retries = 2 + try: + delay = int(settings.test_runner_infra_retry_delay_s) + except (TypeError, ValueError): + delay = 30 + + if retries < max_retries: + _bump("deferred") + reason = "timeout" if result.timed_out else "suite did not execute (tool-error)" + task_desc = ( + f"Work item: {work_item_id}\nRepo: {repo}\nBranch: {branch}\n" + f"Stage: testing\nNote: {_INFRA_RETRY_MARKER} " + f"(attempt {retries + 1}/{max_retries}) — {reason}, retrying after {delay}s." + ) + try: + from .db import enqueue_job + new_job = enqueue_job( + "tester", repo, task_desc, task_id=task_id, available_at_delay_s=delay, + ) + logger.warning( + "Task %s (%s): regression suite did not execute (%s) -> infra-DEFER " + "(job_id=%s, attempt %d/%d)", + task_id, work_item_id, reason, new_job, retries + 1, max_retries, + ) + except Exception as e: # noqa: BLE001 - never-raise + logger.error("test_runner: infra-DEFER enqueue failed for %s: %s", task_id, e) + return + + # Budget exhausted -> fail-closed FAIL (terminal, never a false green). + _bump("fail") + logger.error( + "Task %s (%s): test tool-error DEFER budget exhausted (%d) -> fail-closed FAIL", + task_id, work_item_id, max_retries, + ) + write_test_report(repo, work_item_id, branch, result.returncode, "FAIL", + result.stdout, "skipped", tool_error=True) + _alert_infra_exhausted(work_item_id, max_retries) + _advance(task_id, repo, work_item_id, branch) + + +def _alert_infra_exhausted(work_item_id: str, max_retries: int) -> None: + """Best-effort Telegram alert that the regression suite never executed (infra, NOT + a code defect) after the retry budget. never-raise.""" + try: + from .notifications import send_telegram, link_for + send_telegram( + f"\U0001f6a8 {link_for(work_item_id)}: регресс-сюита не запустилась " + f"(инфра, НЕ дефект кода) после {max_retries} попыток — fail-closed FAIL, " + f"откат на development. Нужно проверить тест-окружение." + ) + except Exception as e: # noqa: BLE001 - never-raise + logger.warning("test_runner: infra-exhausted alert failed for %s: %s", work_item_id, e) + + +# --------------------------------------------------------------------------- +# Entry point (D2) — owns the full deterministic flow, mirror run_staging_gate +# --------------------------------------------------------------------------- +def run_test_gate(job: dict) -> None: + """Deterministic test gate for a ``tester`` job on ``testing``. + + Flow (mirror of ``staging_runner.run_staging_gate``): + 1. resolve ``work_item_id`` / ``branch`` by ``task_id``; + 2. execute the regression suite in the worktree (D3) -> ProcResult; + 3. suite EXECUTED -> map exit-code (+ smoke) -> ``result:``, write + ``13-test-report.md``, initiate the existing gate via ``advance_stage`` (D7); + 4. suite did NOT execute (tool-error) -> bounded DEFER / fail-closed (D5); + 5. observability counters + one structured verdict log (D10). + Never raises into the caller (the launcher marks the job done/failed).""" + started = time.time() + _bump("runs") + task_id = job.get("task_id") + repo = job.get("repo") + + # 1. resolve task fields. + work_item_id, branch = None, None + try: + from .db import get_db + conn = get_db() + row = conn.execute( + "SELECT work_item_id, branch FROM tasks WHERE id=?", (task_id,) + ).fetchone() + conn.close() + if row: + work_item_id, branch = row[0], row[1] + except Exception as e: # noqa: BLE001 - never-raise + logger.error("test_runner: task lookup failed for task_id=%s: %s", task_id, e) + if not work_item_id or not branch: + logger.error( + "test_runner: missing work_item_id/branch for task_id=%s — aborting", task_id + ) + return + + # 2-4. execute + classify + route — guarded so AC-9 (never-raise) holds even if an + # unexpected error escapes a sub-step (the worker must never crash on test infra; + # the task is left on testing for the reconciler/reaper to re-drive). + try: + result = run_test_suite(repo, branch) + duration_s = round(time.time() - started, 1) + suite_ran = (result.returncode is not None) and (not result.timed_out) + + if suite_ran: + # 3. trust the exit-code; AND-in the read-only smoke (D3). + exit_verdict = map_exit_code_to_result(result.returncode) + smoke_state = "skipped" + verdict = exit_verdict + if exit_verdict == "PASS" and bool(getattr(settings, "test_runner_smoke_enabled", True)): + smoke_ok, smoke_detail = run_smoke() + smoke_state = "ok" if smoke_ok else "failed" + if not smoke_ok: + verdict = "FAIL" + logger.warning( + "test_runner: pytest green but smoke FAILED for %s: %s", + work_item_id, smoke_detail, + ) + _bump("pass" if verdict == "PASS" else "fail") + logger.info( + "test_runner verdict: work_item=%s repo=%s exit_code=%s result=%s " + "smoke=%s duration_s=%s outcome=%s", + work_item_id, repo, result.returncode, verdict, smoke_state, duration_s, + "code-pass" if verdict == "PASS" else "code-fail", + ) + write_test_report(repo, work_item_id, branch, result.returncode, verdict, + result.stdout, smoke_state) + _advance(task_id, repo, work_item_id, branch) + return + + # 4. tool-error (suite did not execute) -> DEFER / fail-closed (D5). + _bump("tool_error") + logger.warning( + "test_runner verdict: work_item=%s repo=%s exit_code=%s result=%s " + "duration_s=%s outcome=tool-error (timed_out=%s)", + work_item_id, repo, result.returncode, "TOOL-ERROR", duration_s, result.timed_out, + ) + _handle_tool_error(task_id, repo, work_item_id, branch, result) + except Exception as e: # noqa: BLE001 - never-raise into the worker (AC-9) + logger.error( + "test_runner.run_test_gate: unexpected error for task %s (%s): %s", + task_id, work_item_id, e, + ) + + +# --------------------------------------------------------------------------- +# Observability (D10 / FR-8 / AC-13) +# --------------------------------------------------------------------------- +def snapshot() -> dict: + """Read-only test-runner summary for ``GET /queue`` (FR-8 / AC-13). + + Additive block; existing ``/queue`` keys are untouched. never-raise: any error -> + a minimal dict with the kill-switch state.""" + try: + return { + "enabled": bool(settings.test_runner_enabled), + "repos": getattr(settings, "test_runner_repos", "") or "", + "target": getattr(settings, "test_runner_target", "tests/") or "tests/", + "timeout_s": getattr(settings, "test_runner_timeout_s", _DEFAULT_TIMEOUT_S), + "smoke_enabled": bool(getattr(settings, "test_runner_smoke_enabled", True)), + "infra_max_retries": getattr(settings, "test_runner_infra_max_retries", 2), + "runs": _TEST_RUNNER_COUNTERS["runs"], + "pass": _TEST_RUNNER_COUNTERS["pass"], + "fail": _TEST_RUNNER_COUNTERS["fail"], + "tool_error": _TEST_RUNNER_COUNTERS["tool_error"], + "deferred": _TEST_RUNNER_COUNTERS["deferred"], + } + except Exception as e: # noqa: BLE001 - never-raise -> minimal dict + logger.warning("test_runner.snapshot error: %s", e) + return {"enabled": False} diff --git a/tests/test_orch116_test_runner.py b/tests/test_orch116_test_runner.py new file mode 100644 index 0000000..9a5b236 --- /dev/null +++ b/tests/test_orch116_test_runner.py @@ -0,0 +1,560 @@ +"""ORCH-116 — deterministic test-runner replacing the LLM tester on the `testing` +stage (self-hosting orchestrator). + +Covers Phase 1 (04-test-plan.yaml TC-01…TC-14): the launch_job intercept BEFORE +_spawn, the exit-code -> result: PASS|FAIL mapping, 13-test-report.md render + the +UNCHANGED gate contract (incl. the tester-specific D6.1 52c-`status:`↔parser +anti-collision), advance_stage initiation, kill-switch / scope / backward-compat for a +repo without a test-contract, never-raise / fail-safe, the bounded tool-error DEFER +(anti ORCH-110), process timeout (proc_group / worktree), read-only smoke, self-hosting +safety, the anti-drift invariant (STAGE_TRANSITIONS / QG_CHECKS / check_tests_passed / +_parse_tests_verdict / DB schema untouched), and the /queue observability block. + +No live Claude CLI, network or worktree git: the pytest subprocess, the smoke GETs and +advance_stage are mocked; the pure mapping/render is tested directly. (TC-15 — the LLM +call-site map anti-drift — lives in tests/test_llm_call_site_inventory.py / +test_llm_determinization_docs.py and is asserted green here too.) +""" +import os +import tempfile + +import pytest + +_test_db = os.path.join(tempfile.gettempdir(), "test_orch116_test_runner.db") +os.environ["ORCH_DB_PATH"] = _test_db +os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir() +os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token") +os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token") + +from unittest.mock import MagicMock # noqa: E402 + +import src.db as _db # noqa: E402 +from src.db import init_db, get_db, enqueue_job # noqa: E402 +from src import test_runner # noqa: E402 +from src import stage_engine # noqa: E402 +from src import config as cfg # noqa: E402 +from src.proc_group import ProcResult # noqa: E402 +from src.agents.launcher import AgentLauncher # noqa: E402 + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- +@pytest.fixture(autouse=True) +def fresh_db(monkeypatch, tmp_path): + monkeypatch.setattr(_db.settings, "db_path", _test_db) + if os.path.exists(_test_db): + os.unlink(_test_db) + init_db() + # Worktree artefacts land in tmp; git commit/push degrade gracefully (no repo). + monkeypatch.setattr("src.git_worktree.settings.worktrees_dir", str(tmp_path), raising=False) + # Runner ON, self-hosting scope (default). Smoke OFF by default so the PASS path + # does not hit the network; the dedicated smoke tests re-enable + mock it. + monkeypatch.setattr(cfg.settings, "test_runner_enabled", True, raising=False) + monkeypatch.setattr(cfg.settings, "test_runner_repos", "", raising=False) + monkeypatch.setattr(cfg.settings, "test_runner_smoke_enabled", False, raising=False) + monkeypatch.setattr(cfg.settings, "test_runner_infra_max_retries", 2, raising=False) + # Reset in-process counters between tests. + for k in test_runner._TEST_RUNNER_COUNTERS: + test_runner._TEST_RUNNER_COUNTERS[k] = 0 + yield + + +def _make_task(stage, repo="orchestrator", wi="ORCH-116", branch="feature/ORCH-116-x"): + conn = get_db() + cur = conn.execute( + "INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?, ?, ?, ?, ?)", + (f"plane-{wi}", wi, repo, branch, stage), + ) + tid = cur.lastrowid + conn.commit() + conn.close() + return tid + + +def _make_job(agent, repo, task_id, content="x"): + return enqueue_job(agent, repo, content, task_id=task_id) + + +def _job_dict(job_id, agent, repo, task_id): + return {"id": job_id, "agent": agent, "repo": repo, "task_id": task_id, "task_content": "x"} + + +def _read_report(repo, branch, wi): + from src.git_worktree import get_worktree_path + p = os.path.join(get_worktree_path(repo, branch), f"docs/work-items/{wi}/13-test-report.md") + with open(p, "r", encoding="utf-8") as f: + return f.read() + + +# --------------------------------------------------------------------------- +# TC-01 — applies(): kill-switch / scope / contract / fail-safe (FR-7 / AC-6/AC-7/AC-8) +# --------------------------------------------------------------------------- +def test_tc01_applies_killswitch_scope_and_contract(monkeypatch): + # enabled=False -> False (fall back to the LLM path) regardless of repo. + monkeypatch.setattr(cfg.settings, "test_runner_enabled", False) + assert test_runner.applies("orchestrator") is False + + # enabled, empty CSV -> self-hosting only (and a test-contract resolves for it). + monkeypatch.setattr(cfg.settings, "test_runner_enabled", True) + monkeypatch.setattr(cfg.settings, "test_runner_repos", "") + assert test_runner.applies("orchestrator") is True + assert test_runner.applies("enduro-trails") is False + assert test_runner.applies("") is False + + # BR-9 backward-compat: even hand-adding a non-self repo to the CSV -> False, + # because _has_test_contract is unresolved for it (Phase 1 = self only). + monkeypatch.setattr(cfg.settings, "test_runner_repos", "enduro-trails, orchestrator") + assert test_runner.applies("ENDURO-TRAILS") is False # no contract -> LLM-tester + assert test_runner.applies("orchestrator") is True # in scope + contract + assert test_runner.applies("other-repo") is False + + +def test_tc01_applies_never_raises(monkeypatch): + class Boom: + @property + def test_runner_enabled(self): + raise RuntimeError("boom") + monkeypatch.setattr(test_runner, "settings", Boom()) + assert test_runner.applies("orchestrator") is False + + +# --------------------------------------------------------------------------- +# TC-02 — exit-code -> verdict, single shared contract (FR-3 / AC-3) +# --------------------------------------------------------------------------- +def test_tc02_map_exit_code(): + from src import self_deploy + assert test_runner.map_exit_code_to_result(0) == "PASS" + for bad in (1, 2, 137, -9): + assert test_runner.map_exit_code_to_result(bad) == "FAIL" + for noncode in (None, "x", object()): + assert test_runner.map_exit_code_to_result(noncode) == "FAIL" + # Same underlying contract as the deploy-finalizer / staging-runner (no 2nd mapping): + # PASS iff SUCCESS, FAIL iff FAILED. + for v in (0, 1, None, "garbage"): + expect = "PASS" if self_deploy.map_exit_code_to_status(v) == "SUCCESS" else "FAIL" + assert test_runner.map_exit_code_to_result(v) == expect + + +# --------------------------------------------------------------------------- +# TC-03 — 13-test-report.md render: machine key + 52c schema + status alignment (FR-4 / AC-2) +# --------------------------------------------------------------------------- +def test_tc03_report_render_schema_and_status_alignment(): + body = test_runner.build_test_report("ORCH-116", 0, "PASS", "23 passed", smoke="ok") + assert "result: PASS" in body # UPPERCASE machine key + for field in ("work_item: ORCH-116", "stage: testing", + "author_agent: test-runner", "status: success", + "created_at:", "model_used: n/a"): + assert field in body, f"missing 52c field: {field}" + assert "smoke: ok" in body + assert "23 passed" in body # stdout tail copied into the body + + failed = test_runner.build_test_report("ORCH-116", 1, "FAIL", "boom", smoke="skipped") + assert "result: FAIL" in failed + # D6.1: status MUST be aligned with the verdict (failed, never `success`). + assert "status: failed" in failed + + +# --------------------------------------------------------------------------- +# TC-04 — generated report read by the UNCHANGED _parse_tests_verdict (AC-2 / D6.1) +# --------------------------------------------------------------------------- +def test_tc04_gate_parser_unchanged(): + from src.qg.checks import _parse_tests_verdict + ok, reason = _parse_tests_verdict( + test_runner.build_test_report("ORCH-116", 0, "PASS", "", smoke="ok")) + assert ok is True and "PASS" in reason + bad, reason2 = _parse_tests_verdict( + test_runner.build_test_report("ORCH-116", 2, "FAIL", "", smoke="skipped")) + assert bad is False and "FAIL" in reason2 + + +def test_tc04_status_field_never_false_negatives_a_pass(): + """The D6.1 mine: the 52c-mandatory `status:` is read by _parse_tests_verdict with + negative-token priority. A PASS report must NOT carry a status whose UPPERCASE is a + negative token (else a healthy run reads as FAIL).""" + from src.qg.checks import _parse_tests_verdict, _TESTS_NEGATIVE_TOKENS + body = test_runner.build_test_report("ORCH-116", 0, "PASS", "", smoke="ok") + # Extract the status: line from the frontmatter and assert it is token-safe. + status_line = next(ln for ln in body.splitlines() if ln.startswith("status:")) + status_val = status_line.split(":", 1)[1].strip().upper() + for neg in _TESTS_NEGATIVE_TOKENS: + assert neg not in status_val, f"PASS report status carries negative token {neg!r}" + assert _parse_tests_verdict(body)[0] is True + + +# --------------------------------------------------------------------------- +# TC-05 — launch_job intercepts tester on testing BEFORE _spawn (AC-1) +# --------------------------------------------------------------------------- +def test_tc05_launch_job_intercepts_before_spawn(monkeypatch): + tid = _make_task("testing") + jid = _make_job("tester", "orchestrator", tid) + lr = AgentLauncher() + spawn = MagicMock(return_value=999) + monkeypatch.setattr(lr, "_spawn", spawn) + run_gate = MagicMock() + monkeypatch.setattr(test_runner, "run_test_gate", run_gate) + + ret = lr.launch_job(_job_dict(jid, "tester", "orchestrator", tid)) + + assert ret is None # no agent_runs row + spawn.assert_not_called() # LLM path NOT taken + run_gate.assert_called_once() # deterministic runner ran + conn = get_db() + status = conn.execute("SELECT status FROM jobs WHERE id=?", (jid,)).fetchone()[0] + conn.close() + assert status == "done" + + +# --------------------------------------------------------------------------- +# TC-06 — discriminator: tester off-`testing` / non-tester / never-raise (AC-1 / R-1) +# --------------------------------------------------------------------------- +def test_tc06_stage_discriminator_off_testing_not_intercepted(monkeypatch): + tid = _make_task("development") # tester job but not the testing stage + jid = _make_job("tester", "orchestrator", tid) + assert test_runner.should_intercept(_job_dict(jid, "tester", "orchestrator", tid)) is False + # and launch_job falls through to _spawn for it. + lr = AgentLauncher() + spawn = MagicMock(return_value=42) + monkeypatch.setattr(lr, "_spawn", spawn) + ret = lr.launch_job(_job_dict(jid, "tester", "orchestrator", tid)) + assert ret == 42 + spawn.assert_called_once() + + +def test_tc06_non_tester_not_intercepted(): + tid = _make_task("testing") + jid = _make_job("developer", "orchestrator", tid) + assert test_runner.should_intercept(_job_dict(jid, "developer", "orchestrator", tid)) is False + + +def test_tc06_non_self_repo_not_intercepted(): + tid = _make_task("testing", repo="enduro-trails", wi="ET-9", branch="feature/ET-9-x") + jid = _make_job("tester", "enduro-trails", tid) + assert test_runner.should_intercept(_job_dict(jid, "tester", "enduro-trails", tid)) is False + + +def test_tc06_should_intercept_never_raises(monkeypatch): + # applies() True, but the DB lookup explodes -> should_intercept False (fall-through). + monkeypatch.setattr(test_runner, "applies", lambda repo: True) + monkeypatch.setattr(test_runner, "get_db", None, raising=False) + + def boom_get_db(): + raise RuntimeError("db down") + import src.db as dbmod + monkeypatch.setattr(dbmod, "get_db", boom_get_db) + assert test_runner.should_intercept( + {"agent": "tester", "repo": "orchestrator", "task_id": 1}) is False + + +# --------------------------------------------------------------------------- +# TC-07 — routing equivalence: PASS -> advance; FAIL -> same path (AC-4) +# --------------------------------------------------------------------------- +@pytest.mark.parametrize("rc,expected", [(0, "PASS"), (1, "FAIL")]) +def test_tc07_advance_initiated_like_llm(monkeypatch, rc, expected): + tid = _make_task("testing") + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=rc, stdout="out", stderr="", timed_out=False)) + advance = MagicMock() + monkeypatch.setattr(stage_engine, "advance_stage", advance) + + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + + advance.assert_called_once() + kwargs = advance.call_args.kwargs + assert kwargs["current_stage"] == "testing" + assert kwargs["finished_agent"] == "tester" + assert kwargs["work_item_id"] == "ORCH-116" + assert f"result: {expected}" in _read_report("orchestrator", "feature/ORCH-116-x", "ORCH-116") + + +# --------------------------------------------------------------------------- +# TC-08 — kill-switch: enabled=False -> LLM path via _spawn (AC-7) +# --------------------------------------------------------------------------- +def test_tc08_killswitch_falls_back_to_spawn(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_enabled", False) + tid = _make_task("testing") + jid = _make_job("tester", "orchestrator", tid) + lr = AgentLauncher() + spawn = MagicMock(return_value=7) + monkeypatch.setattr(lr, "_spawn", spawn) + run_gate = MagicMock() + monkeypatch.setattr(test_runner, "run_test_gate", run_gate) + + ret = lr.launch_job(_job_dict(jid, "tester", "orchestrator", tid)) + + assert ret == 7 + spawn.assert_called_once() # byte-for-byte the prior LLM-tester path + run_gate.assert_not_called() + + +# --------------------------------------------------------------------------- +# TC-09 — anti-drift NFR-1: STAGE_TRANSITIONS / QG_CHECKS / schema untouched (AC-6) +# --------------------------------------------------------------------------- +def test_tc09_pipeline_contract_unchanged(): + from src.stages import STAGE_TRANSITIONS + from src.qg.checks import QG_CHECKS + # The review->testing edge (tester) and testing->deploy-staging gate are byte-for-byte. + assert STAGE_TRANSITIONS["review"] == { + "next": "testing", "agent": "tester", "qg": "check_reviewer_verdict" + } + assert STAGE_TRANSITIONS["testing"] == { + "next": "deploy-staging", "agent": "deployer", "qg": "check_tests_passed" + } + assert "check_tests_passed" in QG_CHECKS + # No new ORCH-116 table: only the existing tables exist. + conn = get_db() + tables = {r[0] for r in conn.execute( + "SELECT name FROM sqlite_master WHERE type='table'").fetchall()} + conn.close() + assert not any("test_runner" in t for t in tables) + # The machine key is not renamed: the runner emits `result:`. + assert "result:" in test_runner.build_test_report("ORCH-116", 0, "PASS") + + +# --------------------------------------------------------------------------- +# TC-10 / TC-11 — two-level outcome + never-raise (anti ORCH-110, AC-5/AC-9) +# --------------------------------------------------------------------------- +def test_tc10_nonzero_exit_is_fail_and_advances(monkeypatch): + tid = _make_task("testing") + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=3, stdout="fail", stderr="", timed_out=False)) + advance = MagicMock() + monkeypatch.setattr(stage_engine, "advance_stage", advance) + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + advance.assert_called_once() + assert "result: FAIL" in _read_report("orchestrator", "feature/ORCH-116-x", "ORCH-116") + + +def test_tc10_timeout_defers_without_advance(monkeypatch): + tid = _make_task("testing") + monkeypatch.setattr(cfg.settings, "test_runner_infra_max_retries", 2) + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=None, stdout="", stderr="", timed_out=True)) + advance = MagicMock() + monkeypatch.setattr(stage_engine, "advance_stage", advance) + + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + + advance.assert_not_called() # NO silent advance / false green on infra hiccup + conn = get_db() + n = conn.execute( + "SELECT COUNT(*) FROM jobs WHERE task_id=? AND task_content LIKE ?", + (tid, f"%{test_runner._INFRA_RETRY_MARKER}%"), + ).fetchone()[0] + # The re-queued job must be a `tester` job (re-enters this runner), not a deployer. + agent = conn.execute( + "SELECT agent FROM jobs WHERE task_id=? AND task_content LIKE ?", + (tid, f"%{test_runner._INFRA_RETRY_MARKER}%"), + ).fetchone()[0] + conn.close() + assert n == 1 + assert agent == "tester" + assert test_runner._TEST_RUNNER_COUNTERS["deferred"] == 1 + assert test_runner._TEST_RUNNER_COUNTERS["tool_error"] == 1 + + +def test_tc10_tool_error_budget_exhausted_fails_closed(monkeypatch): + tid = _make_task("testing") + monkeypatch.setattr(cfg.settings, "test_runner_infra_max_retries", 0) # exhausted immediately + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=None, stdout="", stderr="", timed_out=True)) + advance = MagicMock() + monkeypatch.setattr(stage_engine, "advance_stage", advance) + alert = MagicMock() + monkeypatch.setattr(test_runner, "_alert_infra_exhausted", alert) + + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + + advance.assert_called_once() # fail-closed: write FAIL + advance (existing rollback) + alert.assert_called_once() + assert "result: FAIL" in _read_report("orchestrator", "feature/ORCH-116-x", "ORCH-116") + + +def test_tc11_run_gate_never_raises(monkeypatch): + tid = _make_task("testing") + + def boom(repo, branch): + raise RuntimeError("pytest exploded") + monkeypatch.setattr(test_runner, "run_test_suite", boom) + # Must NOT raise (AC-9): the worker is protected even on an unexpected error. + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + + +def test_tc11_launcher_contains_runner_fault(monkeypatch): + tid = _make_task("testing") + jid = _make_job("tester", "orchestrator", tid) + lr = AgentLauncher() + monkeypatch.setattr(lr, "_spawn", MagicMock()) + monkeypatch.setattr(test_runner, "run_test_gate", + MagicMock(side_effect=RuntimeError("kaboom"))) + ret = lr.launch_job(_job_dict(jid, "tester", "orchestrator", tid)) + assert ret is None + conn = get_db() + status = conn.execute("SELECT status FROM jobs WHERE id=?", (jid,)).fetchone()[0] + conn.close() + assert status == "failed" + + +# --------------------------------------------------------------------------- +# TC-12 — timeout resolution + worktree cwd + tree-kill (NFR-3/NFR-4 / AC-11) +# --------------------------------------------------------------------------- +def test_tc12_resolve_timeout_default_on_bad_value(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_timeout_s", 1234) + assert test_runner._resolve_timeout() == 1234 + for bad in (0, -5, "abc", None): + monkeypatch.setattr(cfg.settings, "test_runner_timeout_s", bad) + assert test_runner._resolve_timeout() == test_runner._DEFAULT_TIMEOUT_S + + +def test_tc12_pytest_runs_in_worktree_via_proc_group(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_timeout_s", 321) + monkeypatch.setattr(cfg.settings, "subprocess_tree_kill_enabled", True) + captured = {} + + def fake_run(cmd, *, cwd, timeout, grace_s, tree_kill): + captured.update(cmd=cmd, cwd=cwd, timeout=timeout, tree_kill=tree_kill) + return ProcResult(returncode=0, stdout="", stderr="", timed_out=False) + monkeypatch.setattr(test_runner.proc_group, "run_in_process_group", fake_run) + + test_runner.run_test_suite("orchestrator", "feature/ORCH-116-x") + from src.git_worktree import get_worktree_path + assert captured["cwd"] == get_worktree_path("orchestrator", "feature/ORCH-116-x") + assert captured["timeout"] == 321 + assert captured["tree_kill"] is True + assert captured["cmd"][:3] == ["python", "-m", "pytest"] + + +# --------------------------------------------------------------------------- +# TC-13 — self-hosting safety: no forbidden literals; smoke read-only (AC-10) +# --------------------------------------------------------------------------- +def test_tc13_command_has_no_dangerous_literals(): + cmd = " ".join(test_runner.build_test_command()) + forbidden = ("compose", "up -d", "--build", "8500", "force", "push", ".env", + "rm ", "restart", "docker") + for token in forbidden: + assert token not in cmd, f"forbidden literal {token!r} in runner command: {cmd}" + assert "pytest" in cmd + + +def test_tc13_smoke_is_read_only_and_checks_serial_gate(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_smoke_enabled", True) + monkeypatch.setattr(test_runner, "_SMOKE_BACKOFF_S", 0) + calls = [] + + def fake_get(url): + calls.append(url) + if url.endswith("/queue"): + return 200, '{"serial_gate": {"enabled": true}}' + return 200, "ok" + monkeypatch.setattr(test_runner, "_http_get", fake_get) + + ok, detail = test_runner.run_smoke() + assert ok is True + # All three read-only endpoints hit; serial_gate verified in /queue body. + assert any(u.endswith("/health") for u in calls) + assert any(u.endswith("/status") for u in calls) + assert any(u.endswith("/queue") for u in calls) + + +def test_tc13_smoke_missing_serial_gate_fails(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_smoke_enabled", True) + monkeypatch.setattr(test_runner, "_SMOKE_BACKOFF_S", 0) + monkeypatch.setattr(test_runner, "_http_get", + lambda url: (200, "{}")) # 200 but no serial_gate block + ok, detail = test_runner.run_smoke() + assert ok is False + assert "serial_gate" in detail + + +def test_tc13_smoke_unreachable_retries_then_fails(monkeypatch): + monkeypatch.setattr(cfg.settings, "test_runner_smoke_enabled", True) + monkeypatch.setattr(test_runner, "_SMOKE_BACKOFF_S", 0) + attempts = {"n": 0} + + def fake_get(url): + attempts["n"] += 1 + return 0, "" # always unreachable + monkeypatch.setattr(test_runner, "_http_get", fake_get) + ok, detail = test_runner.run_smoke() + assert ok is False + # bounded retry: more than one attempt was made before declaring FAIL. + assert attempts["n"] >= test_runner._SMOKE_MAX_ATTEMPTS + + +def test_tc13_pytest_green_but_smoke_fail_is_fail(monkeypatch): + tid = _make_task("testing") + monkeypatch.setattr(cfg.settings, "test_runner_smoke_enabled", True) + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=0, stdout="ok", stderr="", timed_out=False)) + monkeypatch.setattr(test_runner, "run_smoke", lambda: (False, "smoke /queue failed")) + advance = MagicMock() + monkeypatch.setattr(stage_engine, "advance_stage", advance) + + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + + advance.assert_called_once() # FAIL still advances (existing rollback) + body = _read_report("orchestrator", "feature/ORCH-116-x", "ORCH-116") + assert "result: FAIL" in body + assert "smoke: failed" in body + + +# --------------------------------------------------------------------------- +# TC-14 — observability + hybrid: /queue block + structured verdict log (AC-12/AC-13) +# --------------------------------------------------------------------------- +def test_tc14_snapshot_shape(): + snap = test_runner.snapshot() + for k in ("enabled", "repos", "target", "timeout_s", "smoke_enabled", + "infra_max_retries", "runs", "pass", "fail", "tool_error", "deferred"): + assert k in snap, f"snapshot missing key {k}" + + +def test_tc14_queue_endpoint_includes_block(): + import asyncio + from src import main + payload = asyncio.run(main.queue()) + assert "test_runner" in payload + assert "enabled" in payload["test_runner"] + + +def test_tc14_structured_verdict_log_distinguishes_outcomes(monkeypatch, caplog): + tid = _make_task("testing") + monkeypatch.setattr(stage_engine, "advance_stage", MagicMock()) + # code-pass + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=0, stdout="", stderr="", timed_out=False)) + with caplog.at_level("INFO", logger="orchestrator.test_runner"): + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + assert any("outcome=code-pass" in r.message for r in caplog.records) + + caplog.clear() + # tool-error + monkeypatch.setattr(cfg.settings, "test_runner_infra_max_retries", 2) + monkeypatch.setattr(test_runner, "run_test_suite", + lambda repo, branch: ProcResult(returncode=None, stdout="", stderr="", timed_out=True)) + with caplog.at_level("WARNING", logger="orchestrator.test_runner"): + test_runner.run_test_gate(_job_dict(1, "tester", "orchestrator", tid)) + assert any("outcome=tool-error" in r.message for r in caplog.records) + + +def test_tc14_snapshot_never_raises(monkeypatch): + class Boom: + def __getattr__(self, name): + raise RuntimeError("boom") + monkeypatch.setattr(test_runner, "settings", Boom()) + snap = test_runner.snapshot() + assert snap["enabled"] is False + + +# --------------------------------------------------------------------------- +# TC-15 — anti-drift of the LLM call-site map / roadmap stays green after our edits. +# --------------------------------------------------------------------------- +def test_tc15_llm_map_anti_drift_green(): + from tests import test_llm_call_site_inventory as inv + from tests import test_llm_determinization_docs as docs + # tester stays avoidable=yes / axis=C / needs-hybrid-fallback (LLM-branch = fallback). + inv.test_tc14_avoidable_set_fixed() + inv.test_tc13_control_path_axis_correct() + inv.test_tc04_classification_total_and_axis_consistent() + # roadmap: exactly one first_slice=yes (deployer); tester rank 2 hybrid. + docs.test_tc07_roadmap_completeness_and_first_slice() + docs.test_tc11_no_fabricated_followup_ids()