developer(ET): auto-commit from developer run_id=355
This commit is contained in:
125
docs/operations/PHANTOM_MERGE_RUNBOOK.md
Normal file
125
docs/operations/PHANTOM_MERGE_RUNBOOK.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Runbook — диагностика «фантомного merge» (ORCH-071)
|
||||
|
||||
> **Когда применять.** Задача дошла до `done` (или прод задеплоен «зелёным»), но есть
|
||||
> подозрение, что её ветка **не влита в `main`** — следующая задача срежет ветку от
|
||||
> устаревшего `main` и потеряет код предшественника (постмортем
|
||||
> `docs/history/LESSONS_2026-06-08_phantom-merge.md`). Этот runbook даёт 4 проверки
|
||||
> для **однозначной локализации** фантома.
|
||||
|
||||
С ORCH-071 такой исход блокируется автоматически: под-гейт `deploy → done`
|
||||
(`stage_engine._handle_merge_verify`) сначала **детерминированно вливает PR**
|
||||
(`merge_gate.merge_pr`, Gitea PR-merge API), затем **верифицирует merge**
|
||||
(`merge_gate.verify_merged_to_main`) и НЕ пускает задачу в `done`, пока merge не
|
||||
подтверждён (alert + HOLD). Этот runbook — для ручной перепроверки/инцидентов
|
||||
(в т.ч. при выключенном kill-switch `ORCH_MERGE_VERIFY_ENABLED=false`).
|
||||
|
||||
Подставьте значения:
|
||||
|
||||
```bash
|
||||
OWNER=admin # settings.gitea_owner
|
||||
REPO=orchestrator # репозиторий
|
||||
BRANCH=feature/ORCH-071-slug # ветка задачи
|
||||
GITEA=http://localhost:3000 # settings.gitea_url
|
||||
TOKEN=<gitea_token> # settings.gitea_token
|
||||
FILE=src/stage_engine.py # любой файл, гарантированно изменённый задачей
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Проверка 1 — Gitea API: список PR + флаги `merged`
|
||||
|
||||
Показывает, считает ли сам Gitea PR влитым.
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: token $TOKEN" \
|
||||
"$GITEA/api/v1/repos/$OWNER/$REPO/pulls?state=all" \
|
||||
| python3 -c 'import sys,json; \
|
||||
[print(p["number"], p["state"], "merged="+str(p.get("merged")), p["head"]["ref"]) \
|
||||
for p in json.load(sys.stdin)]'
|
||||
```
|
||||
|
||||
* **Фантом НЕ подтверждён (всё хорошо):** строка ветки `$BRANCH` имеет `merged=True`.
|
||||
* **Фантом подтверждён (по этому критерию):** PR ветки `state=open` / `merged=False`
|
||||
(или PR отсутствует), при том что задача в `done` / прод задеплоен.
|
||||
|
||||
---
|
||||
|
||||
## Проверка 2 — md5 прод-файлов vs `git show origin/main:<file>`
|
||||
|
||||
Сверяет содержимое файла на проде с тем, что лежит в `origin/main`.
|
||||
|
||||
```bash
|
||||
# в прод-контейнере (или через docker exec orchestrator):
|
||||
md5sum "/app/$FILE"
|
||||
|
||||
# содержимое того же файла из origin/main (на хосте, в клоне репо):
|
||||
git -C /home/slin/repos/$REPO fetch origin main -q
|
||||
git -C /home/slin/repos/$REPO show "origin/main:$FILE" | md5sum
|
||||
```
|
||||
|
||||
* **Совпало:** прод соответствует `main` (фантома нет ИЛИ задача не меняла этот файл —
|
||||
возьмите файл из проверки 3/diff'а ветки).
|
||||
* **Разошлось:** прод собран из ветки, а `main` его не получил → косвенный признак фантома.
|
||||
|
||||
---
|
||||
|
||||
## Проверка 3 — `git merge-base` ветки vs `main`
|
||||
|
||||
Главный детерминированный критерий: является ли HEAD ветки предком `origin/main`.
|
||||
|
||||
```bash
|
||||
git -C /home/slin/repos/$REPO fetch origin -q
|
||||
SHA=$(git -C /home/slin/repos/$REPO rev-parse "origin/$BRANCH")
|
||||
git -C /home/slin/repos/$REPO merge-base --is-ancestor "$SHA" origin/main \
|
||||
&& echo "MERGED: ветка влита в main" \
|
||||
|| echo "NOT MERGED: ветка НЕ предок origin/main (ФАНТОМ)"
|
||||
```
|
||||
|
||||
Это ровно та проверка, что выполняет `merge_gate.verify_merged_to_main` (rc=0 → влито).
|
||||
|
||||
* **`MERGED`:** фантома нет.
|
||||
* **`NOT MERGED`:** фантом подтверждён — `main` не содержит коммитов задачи.
|
||||
|
||||
---
|
||||
|
||||
## Проверка 4 — таймлайн деплой-логов
|
||||
|
||||
Восстанавливает порядок событий: был ли merge до/после деплоя, и был ли он вообще.
|
||||
|
||||
```bash
|
||||
# Вердикт деплоя + новое поле merge-верификации (ORCH-071):
|
||||
git -C /home/slin/repos/$REPO show "origin/$BRANCH:docs/work-items/<WI>/14-deploy-log.md" \
|
||||
| sed -n '1,12p' # frontmatter: deploy_status:, merged_to_main:
|
||||
|
||||
# Наблюдаемость под-гейта в живом сервисе:
|
||||
curl -s "$GITEA_HEALTH/queue" | python3 -c \
|
||||
'import sys,json; print(json.load(sys.stdin)["merge_verify"])'
|
||||
# -> {"enabled":..., "merge_verified_total":..., "not_merged_alerts_total":..., "last_alert_wi":...}
|
||||
|
||||
# Журнал хоста по деплою (sentinel-каталог задачи):
|
||||
ls -la /home/slin/repos/.deploy-state-$REPO/<WI>/
|
||||
cat /home/slin/repos/.deploy-state-$REPO/<WI>/hook.log
|
||||
```
|
||||
|
||||
* `deploy_status: SUCCESS` + `merged_to_main: false` → деплой прошёл, merge — нет
|
||||
(это и есть класс ORCH-071; задача должна быть удержана на `deploy`, не `done`).
|
||||
* `not_merged_alerts_total` растёт / `last_alert_wi == <WI>` → под-гейт уже поднял alert.
|
||||
|
||||
---
|
||||
|
||||
## Критерий «фантом подтверждён»
|
||||
|
||||
Фантомный merge считается **подтверждённым**, если выполняется ХОТЯ БЫ ОДНО из:
|
||||
|
||||
1. Проверка 1: PR ветки `state=open` / `merged=False` (или PR нет), а задача в `done`.
|
||||
2. Проверка 3: `merge-base --is-ancestor` вернул **NOT MERGED** (HEAD ветки не предок `origin/main`).
|
||||
3. Проверка 4: `14-deploy-log.md` имеет `deploy_status: SUCCESS` при `merged_to_main: false`.
|
||||
|
||||
Проверка 2 — вспомогательная (зависит от того, менял ли файл задачей), используется
|
||||
для подтверждения проверок 1/3.
|
||||
|
||||
### Что делать при подтверждённом фантоме
|
||||
|
||||
1. **Влить PR вручную** через Gitea (PR-merge API / UI) — НИКОГДА не `git push`/`--force` в `main` (INV-4).
|
||||
2. Повторить approve задачи (re-drive) — под-гейт переоценит: merge подтвердится → задача уйдёт в `done`.
|
||||
3. Если фантом случился при выключенном kill-switch — включить `ORCH_MERGE_VERIFY_ENABLED=true`.
|
||||
@@ -374,6 +374,27 @@ class Settings(BaseSettings):
|
||||
reaper_finalize_grace_s: int = 300
|
||||
lease_reclaim_enabled: bool = True
|
||||
|
||||
# ORCH-071: merge-verify under-gate on the `deploy -> done` edge. For the
|
||||
# self-hosting repo the `deploy` stage runs the DETERMINISTIC self-deploy path
|
||||
# (Phase A/B/C), where the LLM `deployer` agent — historically the ONLY actor
|
||||
# that merged the feature PR into `main` — never runs. Result: a "green" deploy
|
||||
# could reach `done` while the PR stayed `open` (phantom merge, postmortem
|
||||
# LESSONS_2026-06-08). This under-gate (врезка in advance_stage, NOT a new
|
||||
# STAGE_TRANSITIONS edge or registered QG) runs a deterministic merge-actor +
|
||||
# post-deploy verification before `done`: not-merged -> alert + HOLD (no done),
|
||||
# merged -> normal advance. Mirrors merge_gate_* / image_freshness_* rollout.
|
||||
# merge_verify_enabled -> global kill-switch; False -> strictly the prior
|
||||
# behaviour (no merge/verify), env ORCH_MERGE_VERIFY_ENABLED.
|
||||
# merge_verify_repos -> CSV of repos where the under-gate is REAL; empty ->
|
||||
# only the self-hosting repo (orchestrator). Mirrors
|
||||
# merge_gate_repos / self_deploy_repos.
|
||||
# merge_pr_timeout_s -> per Gitea merge/list HTTP call timeout.
|
||||
# merge_verify_timeout_s-> git fetch/merge-base timeout for the ancestor check.
|
||||
merge_verify_enabled: bool = True
|
||||
merge_verify_repos: str = ""
|
||||
merge_pr_timeout_s: int = 60
|
||||
merge_verify_timeout_s: int = 60
|
||||
|
||||
# Telegram notifications
|
||||
telegram_bot_token: str = ""
|
||||
telegram_chat_id: str = ""
|
||||
|
||||
@@ -147,6 +147,7 @@ async def queue():
|
||||
from .reconciler import reconciler
|
||||
from .job_reaper import reaper
|
||||
from . import post_deploy
|
||||
from . import merge_gate
|
||||
return {
|
||||
"counts": job_status_counts(),
|
||||
"max_concurrency": worker.max_concurrency,
|
||||
@@ -155,5 +156,6 @@ async def queue():
|
||||
"reconcile": reconciler.status(),
|
||||
"reaper": reaper.status(),
|
||||
"post_deploy": post_deploy.status(),
|
||||
"merge_verify": merge_gate.merge_verify_status(),
|
||||
"recent": recent_jobs(10),
|
||||
}
|
||||
|
||||
@@ -485,3 +485,193 @@ def pr_already_merged(repo: str, branch: str) -> bool:
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning("pr_already_merged check failed for %s/%s: %s", repo, branch, e)
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-071: deterministic merge-actor + post-deploy merge verification.
|
||||
#
|
||||
# For the self-hosting repo the `deploy` stage runs the deterministic self-deploy
|
||||
# path (Phase A/B/C) and the LLM `deployer` agent — historically the ONLY actor
|
||||
# that merged the feature PR into `main` — never runs. These two helpers close the
|
||||
# "phantom merge" gap (LESSONS_2026-06-08): a deterministic actor merges the PR via
|
||||
# the Gitea PR-merge API (NEVER a push/force-push to main, INV-4) and a verifier
|
||||
# confirms `main` actually received the commit before the pipeline reaches `done`.
|
||||
# Both wire into the `deploy -> done` under-gate (stage_engine._handle_merge_verify).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Lightweight in-process observability counters (D8). Reset only on process start;
|
||||
# surfaced read-only via `merge_verify_status()` in GET /queue. Never the source of
|
||||
# truth for any decision — purely informational.
|
||||
_MERGE_VERIFY_COUNTERS: dict = {
|
||||
"merge_verified_total": 0,
|
||||
"not_merged_alerts_total": 0,
|
||||
"last_alert_wi": None,
|
||||
}
|
||||
|
||||
|
||||
def note_merge_verified() -> None:
|
||||
"""Bump the 'merge verified -> done' counter (observability only). Never raises."""
|
||||
try:
|
||||
_MERGE_VERIFY_COUNTERS["merge_verified_total"] += 1
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def note_not_merged_alert(work_item_id: str | None) -> None:
|
||||
"""Bump the 'deploy succeeded but not merged' counter. Never raises."""
|
||||
try:
|
||||
_MERGE_VERIFY_COUNTERS["not_merged_alerts_total"] += 1
|
||||
_MERGE_VERIFY_COUNTERS["last_alert_wi"] = work_item_id
|
||||
except Exception: # noqa: BLE001 - observability must never break a decision
|
||||
pass
|
||||
|
||||
|
||||
def merge_verify_status() -> dict:
|
||||
"""Snapshot of the merge-verify under-gate for GET /queue. Never raises."""
|
||||
try:
|
||||
return {
|
||||
"enabled": bool(settings.merge_verify_enabled),
|
||||
"repos": settings.merge_verify_repos or "",
|
||||
"merge_verified_total": _MERGE_VERIFY_COUNTERS["merge_verified_total"],
|
||||
"not_merged_alerts_total": _MERGE_VERIFY_COUNTERS["not_merged_alerts_total"],
|
||||
"last_alert_wi": _MERGE_VERIFY_COUNTERS["last_alert_wi"],
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning("merge_verify_status error: %s", e)
|
||||
return {"enabled": False}
|
||||
|
||||
|
||||
def merge_verify_applies(repo: str) -> bool:
|
||||
"""Whether the ORCH-071 merge-verify under-gate is REAL for this repo.
|
||||
|
||||
Mirrors ``self_deploy_applies`` / ``image_freshness_applies`` (FR-5 / AC-10):
|
||||
* ``merge_verify_enabled=False`` -> always False (global kill-switch -> the
|
||||
pipeline behaves exactly as before ORCH-071 for everyone).
|
||||
* ``merge_verify_repos`` (CSV) non-empty -> real only for listed repos.
|
||||
* empty CSV -> real ONLY for the self-hosting repo (``orchestrator``); other
|
||||
repos keep the LLM-``deployer`` merge path unchanged (AC-4b).
|
||||
Never raises (any error -> False = no-op, the safe default).
|
||||
"""
|
||||
try:
|
||||
if not settings.merge_verify_enabled:
|
||||
return False
|
||||
raw = (settings.merge_verify_repos or "").strip()
|
||||
if raw:
|
||||
allowed = {r.strip().lower() for r in raw.split(",") if r.strip()}
|
||||
return (repo or "").strip().lower() in allowed
|
||||
# Lazy import keeps this a leaf-ish module (qg.checks imports merge_gate lazily).
|
||||
from .qg.checks import is_self_hosting_repo
|
||||
return is_self_hosting_repo(repo)
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning("merge_verify_applies error for %s: %s", repo, e)
|
||||
return False
|
||||
|
||||
|
||||
def merge_pr(repo: str, branch: str) -> tuple[bool, str]:
|
||||
"""Deterministically merge the open PR for ``branch`` via the Gitea PR-merge API.
|
||||
|
||||
The self-hosting deterministic merge-actor (FR-1 / D3). NEVER pushes or
|
||||
force-pushes ``main`` (INV-4/AC-8) — the ONLY mutation is the Gitea
|
||||
``POST /pulls/{index}/merge`` call, exactly what the LLM ``deployer`` used to do
|
||||
on non-self repos.
|
||||
|
||||
Algorithm:
|
||||
1. ``pr_already_merged`` -> True -> no-op ``(True, "already-merged")`` (INV-5/AC-9).
|
||||
2. ``GET /repos/{owner}/{repo}/pulls?state=open`` -> the open PR whose head ref
|
||||
== ``branch`` -> its index. No open PR -> ``(False, "no open PR")``.
|
||||
3. ``POST /repos/{owner}/{repo}/pulls/{index}/merge`` (Do: ``merge``) ->
|
||||
200/201 -> ``(True, "merged PR #<n>")``; otherwise ``(False, "<reason>")``.
|
||||
|
||||
Never-raise (INV-1/AC-9 / TC-09): any HTTP/parse error -> ``(False, reason)``.
|
||||
"""
|
||||
try:
|
||||
if pr_already_merged(repo, branch):
|
||||
logger.info("merge_pr: %s/%s already merged -> no-op", repo, branch)
|
||||
return True, "already-merged"
|
||||
|
||||
import httpx
|
||||
owner = settings.gitea_owner
|
||||
headers = {"Authorization": f"token {settings.gitea_token}"}
|
||||
base = f"{settings.gitea_url}/api/v1/repos/{owner}/{repo}"
|
||||
timeout = settings.merge_pr_timeout_s
|
||||
|
||||
resp = httpx.get(
|
||||
f"{base}/pulls", params={"state": "open"}, headers=headers, timeout=timeout
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
return False, f"list PRs failed: HTTP {resp.status_code}"
|
||||
index = None
|
||||
for pr in resp.json() or []:
|
||||
if pr.get("head", {}).get("ref") == branch:
|
||||
index = pr.get("number")
|
||||
break
|
||||
if index is None:
|
||||
return False, "no open PR"
|
||||
|
||||
m = httpx.post(
|
||||
f"{base}/pulls/{index}/merge",
|
||||
json={"Do": "merge"},
|
||||
headers=headers,
|
||||
timeout=timeout,
|
||||
)
|
||||
if m.status_code in (200, 201):
|
||||
logger.info("merge_pr: merged PR #%s for %s/%s", index, repo, branch)
|
||||
return True, f"merged PR #{index}"
|
||||
detail = (m.text or "").strip()[:200]
|
||||
logger.warning(
|
||||
"merge_pr: merge failed for %s/%s PR #%s: HTTP %s %s",
|
||||
repo, branch, index, m.status_code, detail,
|
||||
)
|
||||
return False, f"merge failed: HTTP {m.status_code}"
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning("merge_pr unexpected error for %s/%s: %s", repo, branch, e)
|
||||
return False, f"merge error: {e}"
|
||||
|
||||
|
||||
def verify_merged_to_main(repo: str, branch: str, sha: str) -> bool:
|
||||
"""Return True iff the deployed commit is confirmed merged into ``origin/main``.
|
||||
|
||||
Post-deploy verification (FR-2 / D4): the merge is confirmed when EITHER
|
||||
* ``pr_already_merged(repo, branch)`` is True (Gitea ``PR.merged == true``), OR
|
||||
* ``git merge-base --is-ancestor <sha> origin/main`` succeeds in the per-branch
|
||||
worktree (after ``git fetch origin main``), i.e. the validated SHA is an
|
||||
ancestor of the current ``origin/main``.
|
||||
|
||||
``sha`` is the validated commit (``image_freshness.validated_revision`` =
|
||||
worktree ``git rev-parse HEAD``). An empty ``sha`` makes the git branch
|
||||
inconclusive (only the PR-merged branch can then confirm).
|
||||
|
||||
Never-raise (INV-1/AC-7 / TC-04): any git/HTTP error -> ``False`` (= "not
|
||||
confirmed" -> fail-closed for ``done``: alert + HOLD). The exception is NEVER
|
||||
propagated into ``advance_stage``.
|
||||
"""
|
||||
try:
|
||||
if pr_already_merged(repo, branch):
|
||||
return True
|
||||
if not sha:
|
||||
logger.warning(
|
||||
"verify_merged_to_main: empty SHA for %s/%s and PR not known-merged",
|
||||
repo, branch,
|
||||
)
|
||||
return False
|
||||
try:
|
||||
wt = ensure_worktree(repo, branch)
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning(
|
||||
"verify_merged_to_main: worktree error for %s/%s: %s", repo, branch, e
|
||||
)
|
||||
return False
|
||||
subprocess.run(
|
||||
["git", "-C", wt, "fetch", "origin", "main"],
|
||||
capture_output=True, timeout=settings.merge_verify_timeout_s,
|
||||
)
|
||||
r = subprocess.run(
|
||||
["git", "-C", wt, "merge-base", "--is-ancestor", sha, "origin/main"],
|
||||
capture_output=True, timeout=settings.merge_verify_timeout_s,
|
||||
)
|
||||
return r.returncode == 0
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract
|
||||
logger.warning(
|
||||
"verify_merged_to_main unexpected error for %s/%s: %s", repo, branch, e
|
||||
)
|
||||
return False
|
||||
|
||||
@@ -349,3 +349,66 @@ def write_deploy_log(repo: str, work_item_id: str, branch: str, exit_code, statu
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("write_deploy_log: git commit/push best-effort failed: %s", e)
|
||||
return True
|
||||
|
||||
|
||||
def record_merged_to_main(repo: str, work_item_id: str, branch: str, merged: bool) -> bool:
|
||||
"""Stamp ``merged_to_main: true|false`` into 14-deploy-log.md frontmatter (ORCH-071).
|
||||
|
||||
Machine-readable observability for the merge-verify under-gate. ONLY the
|
||||
``merged_to_main:`` line is added/updated inside the YAML frontmatter block; the
|
||||
``deploy_status:`` field is left untouched, so the ``check_deploy_status`` /
|
||||
``_parse_deploy_status`` parsing contract is unchanged (TRZ §6 / AC §5).
|
||||
|
||||
Best-effort and idempotent: a missing log or any I/O error is logged and
|
||||
swallowed. Never raises.
|
||||
"""
|
||||
from .git_worktree import get_worktree_path
|
||||
|
||||
rel = f"docs/work-items/{work_item_id}/14-deploy-log.md"
|
||||
try:
|
||||
wt = get_worktree_path(repo, branch)
|
||||
except Exception as e: # noqa: BLE001 - never-raise
|
||||
logger.warning("record_merged_to_main: worktree error for %s/%s: %s", repo, branch, e)
|
||||
return False
|
||||
path = os.path.join(wt, rel)
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
except FileNotFoundError:
|
||||
logger.info("record_merged_to_main: no deploy log at %s (skip)", path)
|
||||
return False
|
||||
except OSError as e:
|
||||
logger.warning("record_merged_to_main: read error at %s: %s", path, e)
|
||||
return False
|
||||
|
||||
value = "true" if merged else "false"
|
||||
if not content.startswith("---"):
|
||||
# No frontmatter to amend — do not fabricate one (keep the contract minimal).
|
||||
logger.info("record_merged_to_main: no frontmatter in %s (skip)", path)
|
||||
return False
|
||||
parts = content.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return False
|
||||
fm_lines = parts[1].splitlines()
|
||||
new_lines = []
|
||||
replaced = False
|
||||
for ln in fm_lines:
|
||||
if ln.strip().lower().startswith("merged_to_main:"):
|
||||
new_lines.append(f"merged_to_main: {value}")
|
||||
replaced = True
|
||||
else:
|
||||
new_lines.append(ln)
|
||||
if not replaced:
|
||||
# Insert before the closing of the frontmatter block (append to the body).
|
||||
if new_lines and new_lines[0] == "":
|
||||
new_lines = new_lines[1:]
|
||||
new_lines.append(f"merged_to_main: {value}")
|
||||
new_fm = "\n".join(new_lines)
|
||||
new_content = "---\n" + new_fm.strip("\n") + "\n---" + parts[2]
|
||||
try:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write(new_content)
|
||||
except OSError as e:
|
||||
logger.warning("record_merged_to_main: write error at %s: %s", path, e)
|
||||
return False
|
||||
return True
|
||||
|
||||
@@ -346,6 +346,22 @@ def advance_stage(
|
||||
)
|
||||
return result
|
||||
|
||||
# --- ORCH-071 merge-verify under-gate (deploy -> done edge) ----------
|
||||
# The SINGLE choke-point that gates EVERY path into terminal `done`
|
||||
# (finalizer Phase C, reconciler F-1, job-reaper re-drive) on a CONFIRMED
|
||||
# merge of the feature PR into `main`. For the self-hosting repo the
|
||||
# deterministic self-deploy path never runs the LLM `deployer` that used to
|
||||
# merge the PR, so a green deploy could reach `done` while the PR stayed
|
||||
# `open` (phantom merge, ORCH-071). This врезка runs a deterministic
|
||||
# merge-actor + post-deploy verification BEFORE update_task_stage; if the
|
||||
# merge is not confirmed it HOLDs (alert, NO done, NO rollback) and returns
|
||||
# without advancing. Not a STAGE_TRANSITIONS edge / registered QG — it is an
|
||||
# edge sub-gate (mirrors the merge-gate врезка), so those contracts are
|
||||
# unchanged. No-op for non-self repos / kill-switch off (1:1 prior behaviour).
|
||||
if current_stage == "deploy" and next_stage == "done":
|
||||
if _handle_merge_verify(task_id, repo, work_item_id, branch, result):
|
||||
return result
|
||||
|
||||
# --- Advance ---------------------------------------------------------
|
||||
update_task_stage(task_id, next_stage)
|
||||
# Telegram live tracker: the analysis->architecture advance is the human
|
||||
@@ -1260,6 +1276,106 @@ def _deploy_finalize_defer_count(task_id: int) -> int:
|
||||
return n
|
||||
|
||||
|
||||
def _handle_merge_verify(task_id, repo, work_item_id, branch, result: AdvanceResult) -> bool:
|
||||
"""ORCH-071 merge-verify under-gate on the `deploy -> done` edge.
|
||||
|
||||
Returns:
|
||||
* ``True`` -> INTERVENED (HOLD): the merge is NOT confirmed -> alert +
|
||||
``set_issue_blocked`` (Plane non-terminal), task stays on `deploy`, NO
|
||||
``done``, NO rollback to development (not-merged is an INFRA defect, not a
|
||||
code fault -> ALERT-only, FR-3). The caller returns without advancing. A
|
||||
later re-drive (reaper / reconciler / re-approve) re-evaluates and, once the
|
||||
merge is fixed, lets the task advance to `done`.
|
||||
* ``False`` -> the merge is CONFIRMED (or the under-gate does not apply for
|
||||
this repo / kill-switch off) -> ``advance_stage`` proceeds to `done`
|
||||
unchanged (happy-path AC-4 / AC-4b).
|
||||
|
||||
Steps (D5):
|
||||
1. Conditionality (FR-5): not applicable -> return False (1:1 prior behaviour).
|
||||
2. Resolve the validated SHA; run the deterministic merge-actor
|
||||
``merge_gate.merge_pr`` (no-op if already merged, INV-5).
|
||||
3. ``merge_gate.verify_merged_to_main`` -> confirmed?
|
||||
* yes -> stamp ``merged_to_main: true``, return False (advance).
|
||||
* no -> alert + Blocked + stamp ``merged_to_main: false``, return True (HOLD).
|
||||
|
||||
Wrapped never-raise (INV-1/AC-7): any internal error is treated as "not
|
||||
confirmed" (HOLD + alert), never a propagated exception into ``advance_stage``.
|
||||
"""
|
||||
try:
|
||||
if not merge_gate.merge_verify_applies(repo):
|
||||
return False # non-self / kill-switch off -> behave exactly as before.
|
||||
|
||||
from . import image_freshness
|
||||
sha = image_freshness.validated_revision(repo, branch)
|
||||
|
||||
# Deterministic merge-actor (no-op if the PR is already merged, INV-5/AC-9).
|
||||
merged_ok, merge_msg = merge_gate.merge_pr(repo, branch)
|
||||
logger.info(
|
||||
f"Task {task_id}: merge-verify merge_pr -> ok={merged_ok} ({merge_msg})"
|
||||
)
|
||||
|
||||
confirmed = merge_gate.verify_merged_to_main(repo, branch, sha)
|
||||
if confirmed:
|
||||
merge_gate.note_merge_verified()
|
||||
try:
|
||||
self_deploy.record_merged_to_main(repo, work_item_id, branch, True)
|
||||
except Exception as e: # noqa: BLE001 - observability best-effort
|
||||
logger.warning(f"Task {task_id}: record merged_to_main(true) failed: {e}")
|
||||
logger.info(f"Task {task_id}: merge-verify CONFIRMED -> deploy->done allowed")
|
||||
return False
|
||||
|
||||
# Not confirmed -> alert + HOLD (no done, no rollback).
|
||||
merge_gate.note_not_merged_alert(work_item_id)
|
||||
try:
|
||||
self_deploy.record_merged_to_main(repo, work_item_id, branch, False)
|
||||
except Exception as e: # noqa: BLE001 - observability best-effort
|
||||
logger.warning(f"Task {task_id}: record merged_to_main(false) failed: {e}")
|
||||
msg = (
|
||||
f"deploy succeeded but not merged: {work_item_id} (repo={repo}, "
|
||||
f"branch={branch}). `main` НЕ получил commit задачи — задача удержана "
|
||||
f"на `deploy` (НЕ done). Нужно ручное вмешательство."
|
||||
)
|
||||
logger.warning(f"Task {task_id}: {msg}")
|
||||
if work_item_id:
|
||||
try:
|
||||
set_issue_blocked(work_item_id)
|
||||
except Exception as e: # noqa: BLE001 - never break the HOLD
|
||||
logger.warning(f"Task {task_id}: set_issue_blocked failed: {e}")
|
||||
try:
|
||||
plane_add_comment(
|
||||
work_item_id,
|
||||
"\U0001f6a8 Deploy прошёл, но PR НЕ влит в `main` "
|
||||
f"(merge: {merge_msg}). Задача удержана на `deploy` (НЕ done). "
|
||||
"Нужно влить PR вручную и повторить approve.",
|
||||
author="deployer",
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 - never break the HOLD
|
||||
logger.warning(f"Task {task_id}: plane not-merged comment failed: {e}")
|
||||
try:
|
||||
send_telegram(f"\U0001f6a8 {msg}")
|
||||
except Exception as e: # noqa: BLE001 - never break the HOLD
|
||||
logger.warning(f"Task {task_id}: not-merged telegram failed: {e}")
|
||||
result.alerted = True
|
||||
result.note = "merge-not-verified-hold"
|
||||
result.advanced = False
|
||||
return True
|
||||
except Exception as e: # noqa: BLE001 - never-raise contract (INV-1/AC-7)
|
||||
# Any internal error -> treat as "not confirmed" -> HOLD + alert, never crash.
|
||||
logger.error(f"Task {task_id}: _handle_merge_verify error: {e}")
|
||||
try:
|
||||
merge_gate.note_not_merged_alert(work_item_id)
|
||||
send_telegram(
|
||||
f"\U0001f6a8 {work_item_id}: ошибка merge-verify ({e}). "
|
||||
f"Задача удержана на `deploy` (НЕ done)."
|
||||
)
|
||||
except Exception: # noqa: BLE001 - best-effort alert
|
||||
pass
|
||||
result.alerted = True
|
||||
result.note = f"merge-verify-error: {e}"
|
||||
result.advanced = False
|
||||
return True
|
||||
|
||||
|
||||
def run_deploy_finalizer(job: dict):
|
||||
"""Phase C — deterministic finalizer (reserved-agent `deploy-finalizer`, no LLM).
|
||||
|
||||
|
||||
@@ -75,3 +75,23 @@ def _reset_webhook_secrets(monkeypatch):
|
||||
if db_path_env:
|
||||
monkeypatch.setattr(db_mod.settings, "db_path", db_path_env, raising=False)
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _disable_merge_verify(monkeypatch):
|
||||
"""ORCH-071: disable the merge-verify under-gate by default in ALL tests.
|
||||
|
||||
The under-gate (deploy -> done) runs a deterministic merge-actor + a
|
||||
post-deploy merge verification that make REAL Gitea/git calls. Leaving it ON
|
||||
by default would (a) reach the network from unrelated deploy->done tests and
|
||||
(b) make them pass/fail by ACCIDENT depending on whether the live Gitea still
|
||||
has the historical PR merged (a hidden CI flake). We therefore default it to
|
||||
its documented kill-switch OFF state (``merge_verify_enabled=False`` == 1:1
|
||||
pre-ORCH-071 behaviour). Tests that specifically target the under-gate
|
||||
(test_merge_verify / test_deploy_finalizer_merge_gate / test_merge_actor /
|
||||
test_deploy_restart_merge_recovery) re-enable it via their own monkeypatch
|
||||
AFTER this autouse fixture, scoping the feature ON to just those tests.
|
||||
"""
|
||||
from src import config as _cfg
|
||||
monkeypatch.setattr(_cfg.settings, "merge_verify_enabled", False, raising=False)
|
||||
yield
|
||||
|
||||
188
tests/test_deploy_finalizer_merge_gate.py
Normal file
188
tests/test_deploy_finalizer_merge_gate.py
Normal file
@@ -0,0 +1,188 @@
|
||||
"""ORCH-071 — Phase C finalizer x merge-verify under-gate (integration).
|
||||
|
||||
Covers TC-05 (FR-3/G2/AC-1: deploy SUCCESS but PR open -> NOT done + alert),
|
||||
TC-06 (AC-4: deploy SUCCESS + merge confirmed -> done) and TC-14 (AC-11: Phase B
|
||||
runs only on confirm_deploy; merge/verify never introduce an auto-deploy).
|
||||
|
||||
Mirrors tests/test_deploy_terminal_sync.py: the finalizer drives advance_stage,
|
||||
the deploy gate is forced green, and the merge-actor/verifier are mocked so the
|
||||
test stays deterministic (no real Gitea/git).
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
|
||||
_test_db = os.path.join(tempfile.gettempdir(), "test_orch_merge_verify.db")
|
||||
os.environ["ORCH_DB_PATH"] = _test_db
|
||||
os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir()
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
from unittest.mock import MagicMock # noqa: E402
|
||||
|
||||
import src.db as _db # noqa: E402
|
||||
from src.db import init_db, get_db # noqa: E402
|
||||
from src import stage_engine # noqa: E402
|
||||
from src import self_deploy # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_db(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(_db.settings, "db_path", _test_db)
|
||||
if os.path.exists(_test_db):
|
||||
os.unlink(_test_db)
|
||||
init_db()
|
||||
monkeypatch.setattr(self_deploy.settings, "repos_dir", str(tmp_path))
|
||||
monkeypatch.setattr(self_deploy.settings, "host_repos_dir", str(tmp_path))
|
||||
monkeypatch.setattr(stage_engine.self_deploy, "write_deploy_log", MagicMock(return_value=True))
|
||||
# The under-gate is disabled by conftest default; these tests target it.
|
||||
monkeypatch.setattr(stage_engine.merge_gate.settings, "merge_verify_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.merge_gate.settings, "merge_verify_repos", "")
|
||||
# The merged_to_main stamp is an observability side effect (no log file here).
|
||||
monkeypatch.setattr(
|
||||
stage_engine.self_deploy, "record_merged_to_main", MagicMock(return_value=True)
|
||||
)
|
||||
# ORCH-021 post-deploy monitor is orthogonal; keep it off for these tests.
|
||||
monkeypatch.setattr(stage_engine.post_deploy.settings, "post_deploy_monitor_enabled", False)
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def silence_side_effects(monkeypatch):
|
||||
for name in (
|
||||
"notify_stage_change", "notify_qg_failure", "notify_approve_requested",
|
||||
"send_telegram", "plane_notify_stage", "plane_notify_qg", "plane_add_comment",
|
||||
"set_issue_in_review", "set_issue_needs_input", "set_issue_in_progress",
|
||||
"set_issue_blocked", "set_issue_done", "set_issue_analysis",
|
||||
"set_issue_awaiting_deploy", "set_issue_deploying", "set_issue_monitoring",
|
||||
):
|
||||
monkeypatch.setattr(stage_engine, name, MagicMock())
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "release_merge_lease", MagicMock())
|
||||
|
||||
|
||||
def _pass(*a, **k):
|
||||
return (True, "ok")
|
||||
|
||||
|
||||
def _make_task(stage, repo="orchestrator", branch="feature/ORCH-071-x", wi="ORCH-071"):
|
||||
conn = get_db()
|
||||
cur = conn.execute(
|
||||
"INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?, ?, ?, ?, ?)",
|
||||
(f"plane-{wi}", wi, repo, branch, stage),
|
||||
)
|
||||
task_id = cur.lastrowid
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return task_id
|
||||
|
||||
|
||||
def _stage(task_id):
|
||||
conn = get_db()
|
||||
row = conn.execute("SELECT stage FROM tasks WHERE id=?", (task_id,)).fetchone()
|
||||
conn.close()
|
||||
return row[0]
|
||||
|
||||
|
||||
def _force_deploy_gate_green(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS, "check_deploy_status": _pass},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-05 (AC-1): deploy_status=SUCCESS but PR open -> task is HELD (not done) + alert.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc05_success_but_not_merged_holds_and_alerts(monkeypatch):
|
||||
self_deploy.write_marker("orchestrator", "ORCH-071", self_deploy.RESULT, "0")
|
||||
_force_deploy_gate_green(monkeypatch)
|
||||
# The merge-actor finds no merge and the verifier confirms NOT merged.
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "merge_pr", MagicMock(return_value=(False, "no open PR")))
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "verify_merged_to_main", MagicMock(return_value=False))
|
||||
|
||||
task_id = _make_task("deploy")
|
||||
stage_engine.run_deploy_finalizer(
|
||||
{"task_id": task_id, "repo": "orchestrator", "id": 1, "agent": "deploy-finalizer"}
|
||||
)
|
||||
|
||||
# AC-1 PASS: the task did NOT reach done and was Blocked for manual handling.
|
||||
assert _stage(task_id) == "deploy"
|
||||
assert stage_engine.set_issue_blocked.called
|
||||
assert not stage_engine.set_issue_done.called
|
||||
assert not stage_engine.set_issue_monitoring.called
|
||||
# An alert was sent ("deploy succeeded but not merged").
|
||||
assert stage_engine.send_telegram.called
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-06 (AC-4): deploy_status=SUCCESS + merge confirmed -> done (happy-path).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc06_success_and_merged_reaches_done(monkeypatch):
|
||||
self_deploy.write_marker("orchestrator", "ORCH-071", self_deploy.RESULT, "0")
|
||||
_force_deploy_gate_green(monkeypatch)
|
||||
merge_pr = MagicMock(return_value=(True, "merged PR #1"))
|
||||
verify = MagicMock(return_value=True)
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "merge_pr", merge_pr)
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "verify_merged_to_main", verify)
|
||||
|
||||
task_id = _make_task("deploy")
|
||||
stage_engine.run_deploy_finalizer(
|
||||
{"task_id": task_id, "repo": "orchestrator", "id": 1, "agent": "deploy-finalizer"}
|
||||
)
|
||||
|
||||
assert _stage(task_id) == "done"
|
||||
# The deterministic merge-actor + verifier both ran on the deploy->done edge.
|
||||
assert merge_pr.called
|
||||
assert verify.called
|
||||
# Self-hosting: terminal status -> Monitoring (post_deploy off here -> Done set).
|
||||
assert not stage_engine.set_issue_blocked.called
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-14 (AC-11): a plain Approved on `deploy` (confirm_deploy=False) is a no-op —
|
||||
# Phase B (prod deploy) requires "Confirm Deploy", and merge/verify do NOT run
|
||||
# (the under-gate never introduces an auto-deploy).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc14_plain_approved_on_deploy_is_noop_no_merge(monkeypatch):
|
||||
monkeypatch.setattr(stage_engine.self_deploy.settings, "self_deploy_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.self_deploy.settings, "self_deploy_repos", "")
|
||||
monkeypatch.setattr(stage_engine.settings, "deploy_require_manual_approve", True)
|
||||
|
||||
merge_pr = MagicMock()
|
||||
verify = MagicMock()
|
||||
initiate = MagicMock(return_value=(True, "ok"))
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "merge_pr", merge_pr)
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "verify_merged_to_main", verify)
|
||||
monkeypatch.setattr(stage_engine.self_deploy, "initiate_deploy", initiate)
|
||||
|
||||
task_id = _make_task("deploy")
|
||||
# finished_agent=None + confirm_deploy=False == a plain Approved on `deploy`.
|
||||
result = stage_engine.advance_stage(
|
||||
task_id, "deploy", "orchestrator", "ORCH-071", "feature/ORCH-071-x",
|
||||
finished_agent=None, confirm_deploy=False,
|
||||
)
|
||||
|
||||
assert result.note == "approved-on-deploy-noop"
|
||||
assert _stage(task_id) == "deploy"
|
||||
# No prod deploy initiated and the merge-verify under-gate never fired.
|
||||
assert not initiate.called
|
||||
assert not merge_pr.called
|
||||
assert not verify.called
|
||||
|
||||
|
||||
def test_tc14_confirm_deploy_initiates_phase_b(monkeypatch):
|
||||
monkeypatch.setattr(stage_engine.self_deploy.settings, "self_deploy_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.self_deploy.settings, "self_deploy_repos", "")
|
||||
monkeypatch.setattr(stage_engine.settings, "deploy_require_manual_approve", True)
|
||||
initiate = MagicMock(return_value=(True, "ok"))
|
||||
monkeypatch.setattr(stage_engine.self_deploy, "initiate_deploy", initiate)
|
||||
|
||||
task_id = _make_task("deploy")
|
||||
stage_engine.advance_stage(
|
||||
task_id, "deploy", "orchestrator", "ORCH-071", "feature/ORCH-071-x",
|
||||
finished_agent=None, confirm_deploy=True,
|
||||
)
|
||||
# Only the dedicated "Confirm Deploy" signal initiates the prod deploy.
|
||||
assert initiate.called
|
||||
116
tests/test_deploy_restart_merge_recovery.py
Normal file
116
tests/test_deploy_restart_merge_recovery.py
Normal file
@@ -0,0 +1,116 @@
|
||||
"""ORCH-071 TC-10 (AC-3/G3) — merge survives a restart during Phase B (smoke).
|
||||
|
||||
Scenario: the prod container "dies" during Phase B BEFORE the feature PR is merged
|
||||
(the holder of the merge step is gone). Because the merge runs in the
|
||||
restart-surviving Phase C finalizer (deploy->done under-gate), a re-drive of the
|
||||
finalizer in the NEW container catches the merge up: it merges the PR, the verifier
|
||||
turns green and the task finally reaches ``done`` — never stuck without an alert and
|
||||
never ``done`` without a confirmed merge.
|
||||
|
||||
The first finalizer pass models "died before merge": the merge-actor cannot complete
|
||||
and the verifier is red -> HOLD + alert (task stays on ``deploy``). The second pass
|
||||
models the re-drive after the restart: the merge lands, verify is green -> ``done``.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
|
||||
_test_db = os.path.join(tempfile.gettempdir(), "test_orch_merge_recovery.db")
|
||||
os.environ["ORCH_DB_PATH"] = _test_db
|
||||
os.environ["ORCH_REPOS_DIR"] = tempfile.gettempdir()
|
||||
os.environ.setdefault("ORCH_GITEA_TOKEN", "test-token")
|
||||
os.environ.setdefault("ORCH_PLANE_API_TOKEN", "test-token")
|
||||
|
||||
from unittest.mock import MagicMock # noqa: E402
|
||||
|
||||
import src.db as _db # noqa: E402
|
||||
from src.db import init_db, get_db # noqa: E402
|
||||
from src import stage_engine # noqa: E402
|
||||
from src import self_deploy # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_db(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(_db.settings, "db_path", _test_db)
|
||||
if os.path.exists(_test_db):
|
||||
os.unlink(_test_db)
|
||||
init_db()
|
||||
monkeypatch.setattr(self_deploy.settings, "repos_dir", str(tmp_path))
|
||||
monkeypatch.setattr(self_deploy.settings, "host_repos_dir", str(tmp_path))
|
||||
monkeypatch.setattr(stage_engine.self_deploy, "write_deploy_log", MagicMock(return_value=True))
|
||||
monkeypatch.setattr(stage_engine.merge_gate.settings, "merge_verify_enabled", True)
|
||||
monkeypatch.setattr(stage_engine.merge_gate.settings, "merge_verify_repos", "")
|
||||
monkeypatch.setattr(
|
||||
stage_engine.self_deploy, "record_merged_to_main", MagicMock(return_value=True)
|
||||
)
|
||||
monkeypatch.setattr(stage_engine.post_deploy.settings, "post_deploy_monitor_enabled", False)
|
||||
yield
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def silence_side_effects(monkeypatch):
|
||||
for name in (
|
||||
"notify_stage_change", "notify_qg_failure", "notify_approve_requested",
|
||||
"send_telegram", "plane_notify_stage", "plane_notify_qg", "plane_add_comment",
|
||||
"set_issue_in_review", "set_issue_needs_input", "set_issue_in_progress",
|
||||
"set_issue_blocked", "set_issue_done", "set_issue_analysis",
|
||||
"set_issue_awaiting_deploy", "set_issue_deploying", "set_issue_monitoring",
|
||||
):
|
||||
monkeypatch.setattr(stage_engine, name, MagicMock())
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "release_merge_lease", MagicMock())
|
||||
|
||||
|
||||
def _stage(task_id):
|
||||
conn = get_db()
|
||||
row = conn.execute("SELECT stage FROM tasks WHERE id=?", (task_id,)).fetchone()
|
||||
conn.close()
|
||||
return row[0]
|
||||
|
||||
|
||||
def test_tc10_merge_recovers_after_restart(monkeypatch):
|
||||
self_deploy.write_marker("orchestrator", "ORCH-071", self_deploy.RESULT, "0")
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS, "check_deploy_status": lambda *a, **k: (True, "ok")},
|
||||
)
|
||||
|
||||
# Stateful merge: the FIRST attempt (pre-restart) cannot complete; the SECOND
|
||||
# (the re-driven finalizer after the restart) merges and the verifier goes green.
|
||||
state = {"attempts": 0, "merged": False}
|
||||
|
||||
def fake_merge_pr(repo, branch):
|
||||
state["attempts"] += 1
|
||||
if state["attempts"] == 1:
|
||||
return (False, "interrupted by restart")
|
||||
state["merged"] = True
|
||||
return (True, "merged PR #1")
|
||||
|
||||
def fake_verify(repo, branch, sha):
|
||||
return state["merged"]
|
||||
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "merge_pr", fake_merge_pr)
|
||||
monkeypatch.setattr(stage_engine.merge_gate, "verify_merged_to_main", fake_verify)
|
||||
|
||||
conn = get_db()
|
||||
cur = conn.execute(
|
||||
"INSERT INTO tasks (plane_id, work_item_id, repo, branch, stage) VALUES (?, ?, ?, ?, ?)",
|
||||
("plane-ORCH-071", "ORCH-071", "orchestrator", "feature/ORCH-071-x", "deploy"),
|
||||
)
|
||||
task_id = cur.lastrowid
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
job = {"task_id": task_id, "repo": "orchestrator", "id": 1, "agent": "deploy-finalizer"}
|
||||
|
||||
# Pass 1 (process died before merge): HOLD — not done, alerted, Blocked.
|
||||
stage_engine.run_deploy_finalizer(job)
|
||||
assert _stage(task_id) == "deploy"
|
||||
assert stage_engine.set_issue_blocked.called
|
||||
assert not stage_engine.set_issue_done.called
|
||||
|
||||
# Pass 2 (finalizer re-driven after restart): merge lands, verify green -> done.
|
||||
stage_engine.run_deploy_finalizer(job)
|
||||
assert _stage(task_id) == "done"
|
||||
assert state["merged"] is True
|
||||
135
tests/test_merge_actor.py
Normal file
135
tests/test_merge_actor.py
Normal file
@@ -0,0 +1,135 @@
|
||||
"""ORCH-071 — deterministic merge-actor (merge_gate.merge_pr).
|
||||
|
||||
Covers TC-07 (FR-1: merge via Gitea PR-merge API, no push/force-push), TC-08
|
||||
(AC-9: idempotency — already-merged -> no-op), TC-09 (AC-7: never-raise) and TC-13
|
||||
(AC-8/INV-2: self-hosting safety — no prod restart, no direct/force push to main).
|
||||
Gitea HTTP is mocked; the actor must NEVER shell out to git/docker/ssh.
|
||||
"""
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from src import merge_gate
|
||||
|
||||
|
||||
class _Resp:
|
||||
def __init__(self, status_code, payload=None, text=""):
|
||||
self.status_code = status_code
|
||||
self._payload = payload if payload is not None else []
|
||||
self.text = text
|
||||
|
||||
def json(self):
|
||||
return self._payload
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _settings(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_enabled", True)
|
||||
monkeypatch.setattr(merge_gate.settings, "gitea_url", "http://gitea.test")
|
||||
monkeypatch.setattr(merge_gate.settings, "gitea_owner", "admin")
|
||||
monkeypatch.setattr(merge_gate.settings, "gitea_token", "tok")
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_pr_timeout_s", 5)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-07: an OPEN PR -> the actor calls Gitea POST /pulls/{index}/merge (Do: merge).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc07_merge_actor_calls_gitea_merge(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
|
||||
branch = "feature/ORCH-071-x"
|
||||
get_calls, post_calls = [], []
|
||||
|
||||
def fake_get(url, params=None, headers=None, timeout=None):
|
||||
get_calls.append((url, params))
|
||||
return _Resp(200, [{"head": {"ref": branch}, "number": 7}])
|
||||
|
||||
def fake_post(url, json=None, headers=None, timeout=None):
|
||||
post_calls.append((url, json))
|
||||
return _Resp(200)
|
||||
|
||||
monkeypatch.setattr(httpx, "get", fake_get)
|
||||
monkeypatch.setattr(httpx, "post", fake_post)
|
||||
|
||||
ok, msg = merge_gate.merge_pr("orchestrator", branch)
|
||||
assert ok is True
|
||||
assert "PR #7" in msg
|
||||
# POST hit the PR-merge API endpoint with Do=merge.
|
||||
assert len(post_calls) == 1
|
||||
url, body = post_calls[0]
|
||||
assert url.endswith("/repos/admin/orchestrator/pulls/7/merge")
|
||||
assert body == {"Do": "merge"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-08 (AC-9): already-merged PR -> no-op (no second merge, no Gitea error).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc08_idempotent_already_merged(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: True)
|
||||
|
||||
def must_not_call(*a, **k):
|
||||
raise AssertionError("no Gitea call must be made when already merged")
|
||||
|
||||
monkeypatch.setattr(httpx, "get", must_not_call)
|
||||
monkeypatch.setattr(httpx, "post", must_not_call)
|
||||
|
||||
ok, msg = merge_gate.merge_pr("orchestrator", "feature/ORCH-071-x")
|
||||
assert ok is True
|
||||
assert msg == "already-merged"
|
||||
|
||||
|
||||
def test_tc08_no_open_pr_is_not_an_error(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(httpx, "get", lambda *a, **k: _Resp(200, []))
|
||||
ok, msg = merge_gate.merge_pr("orchestrator", "feature/ORCH-071-x")
|
||||
assert ok is False
|
||||
assert msg == "no open PR"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-09 (AC-7): a Gitea HTTP error -> (False, reason), exception not propagated.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc09_never_raise_on_http_error(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
|
||||
def boom(*a, **k):
|
||||
raise httpx.ConnectError("gitea unreachable")
|
||||
|
||||
monkeypatch.setattr(httpx, "get", boom)
|
||||
ok, msg = merge_gate.merge_pr("orchestrator", "feature/ORCH-071-x")
|
||||
assert ok is False
|
||||
assert "merge error" in msg
|
||||
|
||||
|
||||
def test_tc09_merge_endpoint_non_2xx_is_false(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(
|
||||
httpx, "get", lambda *a, **k: _Resp(200, [{"head": {"ref": "feature/ORCH-071-x"}, "number": 3}])
|
||||
)
|
||||
monkeypatch.setattr(httpx, "post", lambda *a, **k: _Resp(409, text="conflict"))
|
||||
ok, msg = merge_gate.merge_pr("orchestrator", "feature/ORCH-071-x")
|
||||
assert ok is False
|
||||
assert "HTTP 409" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-13 (AC-8/INV-2): the merge-actor NEVER shells out (no git push/force-push,
|
||||
# no docker/ssh prod restart) — the only side effect is the Gitea PR-merge API.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc13_no_shell_out_no_force_push(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(
|
||||
httpx, "get", lambda *a, **k: _Resp(200, [{"head": {"ref": "feature/ORCH-071-x"}, "number": 9}])
|
||||
)
|
||||
monkeypatch.setattr(httpx, "post", lambda *a, **k: _Resp(200))
|
||||
|
||||
subprocess_calls = []
|
||||
monkeypatch.setattr(
|
||||
merge_gate.subprocess, "run",
|
||||
lambda cmd, *a, **k: subprocess_calls.append(cmd),
|
||||
)
|
||||
|
||||
ok, _ = merge_gate.merge_pr("orchestrator", "feature/ORCH-071-x")
|
||||
assert ok is True
|
||||
# No subprocess (git/docker/ssh) was invoked by the merge-actor at all.
|
||||
assert subprocess_calls == []
|
||||
126
tests/test_merge_verify.py
Normal file
126
tests/test_merge_verify.py
Normal file
@@ -0,0 +1,126 @@
|
||||
"""ORCH-071 — post-deploy merge verification + rollout conditionality.
|
||||
|
||||
Covers TC-01..04 (FR-2/G1/AC-1/AC-7: verify_merged_to_main), TC-11 (AC-4b: non-self
|
||||
repo no-op) and TC-12 (AC-10: kill-switch). All deterministic: git/HTTP are mocked,
|
||||
the verifier honours the never-raise contract.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from src import merge_gate
|
||||
|
||||
|
||||
class _R:
|
||||
"""Minimal stand-in for a completed subprocess result (returncode only)."""
|
||||
|
||||
def __init__(self, rc):
|
||||
self.returncode = rc
|
||||
self.stdout = ""
|
||||
self.stderr = ""
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _enable(monkeypatch):
|
||||
# The conftest disables the under-gate by default; these tests target it, so
|
||||
# re-enable the feature and pin the scope to self-hosting only (empty CSV).
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_enabled", True)
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_repos", "")
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_timeout_s", 5)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-01: validated SHA is an ancestor of origin/main (merge-base rc=0) -> True.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc01_verify_true_when_sha_is_ancestor(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(merge_gate, "ensure_worktree", lambda r, b: "/wt")
|
||||
|
||||
calls = []
|
||||
|
||||
def fake_run(cmd, *a, **k):
|
||||
calls.append(cmd)
|
||||
# fetch -> rc 0; merge-base --is-ancestor -> rc 0 (is ancestor).
|
||||
return _R(0)
|
||||
|
||||
monkeypatch.setattr(merge_gate.subprocess, "run", fake_run)
|
||||
assert merge_gate.verify_merged_to_main("orchestrator", "feature/ORCH-071-x", "abc123") is True
|
||||
# The verifier consulted git merge-base --is-ancestor on origin/main.
|
||||
assert any("merge-base" in c and "--is-ancestor" in c and "origin/main" in c for c in calls)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-02: PR.merged==true short-circuits to True even if git is unavailable.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc02_verify_true_when_pr_merged_even_without_git(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: True)
|
||||
|
||||
def boom(*a, **k):
|
||||
raise RuntimeError("git must NOT be consulted when PR is already merged")
|
||||
|
||||
monkeypatch.setattr(merge_gate, "ensure_worktree", boom)
|
||||
monkeypatch.setattr(merge_gate.subprocess, "run", boom)
|
||||
assert merge_gate.verify_merged_to_main("orchestrator", "feature/ORCH-071-x", "") is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-03: not an ancestor (rc=1) AND PR not merged -> False (phantom merge).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc03_verify_false_when_phantom(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(merge_gate, "ensure_worktree", lambda r, b: "/wt")
|
||||
|
||||
def fake_run(cmd, *a, **k):
|
||||
if "merge-base" in cmd:
|
||||
return _R(1) # NOT an ancestor.
|
||||
return _R(0) # fetch ok.
|
||||
|
||||
monkeypatch.setattr(merge_gate.subprocess, "run", fake_run)
|
||||
assert merge_gate.verify_merged_to_main("orchestrator", "feature/ORCH-071-x", "abc123") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-04 (AC-7): never-raise — a git/OS error -> False, exception not propagated.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc04_verify_never_raises_on_git_error(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", lambda r, b: False)
|
||||
monkeypatch.setattr(merge_gate, "ensure_worktree", lambda r, b: "/wt")
|
||||
|
||||
def boom(*a, **k):
|
||||
raise OSError("git exploded")
|
||||
|
||||
monkeypatch.setattr(merge_gate.subprocess, "run", boom)
|
||||
# No exception escapes; the conservative verdict is "not confirmed".
|
||||
assert merge_gate.verify_merged_to_main("orchestrator", "feature/ORCH-071-x", "abc123") is False
|
||||
|
||||
|
||||
def test_tc04_verify_never_raises_on_http_error(monkeypatch):
|
||||
def boom(r, b):
|
||||
raise RuntimeError("gitea down")
|
||||
|
||||
monkeypatch.setattr(merge_gate, "pr_already_merged", boom)
|
||||
assert merge_gate.verify_merged_to_main("orchestrator", "feature/ORCH-071-x", "abc123") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-11 (AC-4b): non-self repo -> under-gate is a no-op (merge stays with deployer).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc11_non_self_repo_does_not_apply(monkeypatch):
|
||||
# Empty CSV -> only the self-hosting repo is in scope.
|
||||
assert merge_gate.merge_verify_applies("orchestrator") is True
|
||||
assert merge_gate.merge_verify_applies("enduro-trails") is False
|
||||
|
||||
|
||||
def test_tc11_csv_scopes_to_listed_repos(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_repos", "enduro-trails")
|
||||
assert merge_gate.merge_verify_applies("enduro-trails") is True
|
||||
# When the CSV is set, the self repo is NOT auto-included.
|
||||
assert merge_gate.merge_verify_applies("orchestrator") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TC-12 (AC-10): kill-switch off -> applies False for everyone (1:1 prior behaviour).
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc12_kill_switch_disables_under_gate(monkeypatch):
|
||||
monkeypatch.setattr(merge_gate.settings, "merge_verify_enabled", False)
|
||||
assert merge_gate.merge_verify_applies("orchestrator") is False
|
||||
assert merge_gate.merge_verify_applies("enduro-trails") is False
|
||||
@@ -53,6 +53,29 @@ def test_tc15_finalizer_log_roundtrips_through_parser():
|
||||
assert ok_f is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-071 TC-15: the deploy-status parsing contract is UNCHANGED by the new
|
||||
# merge-verify under-gate. The ``merged_to_main:`` observability field the
|
||||
# under-gate stamps into 14-deploy-log.md must NOT influence ``deploy_status:``
|
||||
# parsing — the gate keeps reading ONLY the ``deploy_status:`` frontmatter.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc15_merged_to_main_field_does_not_affect_deploy_status():
|
||||
ok_s, _ = _parse_deploy_status(
|
||||
"---\ndeploy_status: SUCCESS\nmerged_to_main: false\n---\n\nbody"
|
||||
)
|
||||
# deploy_status is the ONLY field read: SUCCESS stays SUCCESS regardless of
|
||||
# the merged_to_main observability stamp (which the under-gate enforces
|
||||
# separately, outside this parser).
|
||||
assert ok_s is True
|
||||
ok_f, _ = _parse_deploy_status(
|
||||
"---\ndeploy_status: FAILED\nmerged_to_main: true\n---\n\nbody"
|
||||
)
|
||||
assert ok_f is False
|
||||
# merged_to_main alone (no deploy_status) is NOT a verdict.
|
||||
ok_n, _ = _parse_deploy_status("---\nmerged_to_main: true\n---\n")
|
||||
assert ok_n is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-061 / TC-04 + TC-05: infra-tolerant staging verdict (pure logic, AC-2/AC-3).
|
||||
#
|
||||
|
||||
@@ -39,3 +39,21 @@ def test_tc16_deploy_staging_transition_unchanged():
|
||||
|
||||
def test_tc16_done_is_terminal():
|
||||
assert get_next_stage("done") is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-071 TC-16: the merge-verify under-gate is an EDGE sub-gate врезанный in
|
||||
# advance_stage (like the merge-gate), NOT a new STAGE_TRANSITIONS edge and NOT a
|
||||
# new registered QG. The state machine + QG registry must stay untouched.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_tc16_merge_verify_adds_no_stage_or_qg():
|
||||
# The deploy->done edge keeps its single gate (no second registered QG).
|
||||
assert STAGE_TRANSITIONS["deploy"]["qg"] == "check_deploy_status"
|
||||
# No new stage was introduced for merge verification.
|
||||
assert "merge-verify" not in STAGE_TRANSITIONS
|
||||
assert "merge_verify" not in STAGE_TRANSITIONS
|
||||
|
||||
from src.qg.checks import QG_CHECKS
|
||||
# The under-gate is NOT a registered quality-gate check.
|
||||
assert "check_merged_to_main" not in QG_CHECKS
|
||||
assert "check_merge_verify" not in QG_CHECKS
|
||||
|
||||
Reference in New Issue
Block a user