fix(merge-gate): tolerate re-test infra-timeout + tree-kill spawned pytest

Eliminate the false `deploy-staging -> development` rollback that fired when the
merge-gate local re-test timed out (infra/resource) on a green CI + tester +
staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget
under CPU starvation from orphaned pytest processes -> timeout misrouted as a
code fault -> developer-retry loop -> manual gate).

Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched
byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable
name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push
main) and the no-prod-restart rule are preserved.

- D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage
  pytest in its own process group (start_new_session) and tree-kills the WHOLE
  group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by
  merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak.
  Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the
  prior subprocess.run.
- D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/
  other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded
  re-queue, task stays on deploy-staging, no rollback / no developer-retry); a
  red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert.
- D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD
  already CI/tester/staging-validated); fail-safe runs the re-test on any
  uncertainty. Flag merge_retest_skip_when_current_enabled.
- D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation;
  reaper_max_running_s invariant preserved without change.
- D6: in-process counters + read-only merge_gate block in GET /queue; appended
  ("ORCH-110","classify_retest_failure","src/merge_gate.py") to
  MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/
  .env.example) updated in the same PR.

Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after
incident regression). Full suite green (1988 passed).

Refs: ORCH-110

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 10:14:55 +03:00
committed by deployer
parent cf602b4810
commit 651b9af7c3
24 changed files with 1816 additions and 50 deletions

View File

@@ -44,7 +44,8 @@ Invariants (ADR-001 §7, never broken):
artefact / decides. It never calls the deploy hook, never restarts the prod
container, never pushes / force-pushes ``main``.
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and lazily
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` /
``proc_group`` (the ORCH-110 stdlib-only process-group runner) and lazily
``qg.checks.is_self_hosting_repo`` / ``db`` / ``notifications``; it never imports
``stage_engine``.
"""
@@ -52,11 +53,11 @@ This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and laz
import json
import logging
import os
import subprocess
import sys
from .config import settings
from .git_worktree import ensure_worktree, get_worktree_path
from .proc_group import run_in_process_group # ORCH-110 D1: tree-kill on timeout
logger = logging.getLogger("orchestrator.coverage_gate")
@@ -152,22 +153,25 @@ def measure_coverage(repo: str, branch: str) -> float | None:
"-q",
]
timeout = settings.coverage_run_timeout_s
try:
subprocess.run(cmd, cwd=wt, capture_output=True, text=True, timeout=timeout)
except subprocess.TimeoutExpired:
# ORCH-110 (D1 / FR-2 / BR-3): run the coverage suite in its OWN process group so
# a timeout tree-kills the WHOLE subtree (the sibling orphan-leak source of the
# ORCH-109 incident), not just the direct child. The metric is read from the JSON
# file regardless of the exit code, so a non-timeout spawn/OS error (returncode
# None) just falls through to "no coverage json produced" -> None, byte-for-byte
# the prior fail-open contract.
res = run_in_process_group(
cmd,
cwd=wt,
timeout=timeout,
tree_kill=bool(getattr(settings, "subprocess_tree_kill_enabled", True)),
grace_s=settings.agent_kill_grace_seconds,
)
if res.timed_out:
logger.warning(
"measure_coverage: pytest --cov timed out after %ss for %s/%s",
timeout, repo, branch,
)
return None
except FileNotFoundError:
logger.warning(
"measure_coverage: pytest / pytest-cov not available for %s/%s", repo, branch
)
return None
except (subprocess.SubprocessError, OSError) as e:
logger.warning("measure_coverage: pytest --cov error for %s/%s: %s", repo, branch, e)
return None
data = None
try: