fix(merge-gate): tolerate re-test infra-timeout + tree-kill spawned pytest
Eliminate the false `deploy-staging -> development` rollback that fired when the merge-gate local re-test timed out (infra/resource) on a green CI + tester + staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget under CPU starvation from orphaned pytest processes -> timeout misrouted as a code fault -> developer-retry loop -> manual gate). Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push main) and the no-prod-restart rule are preserved. - D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage pytest in its own process group (start_new_session) and tree-kills the WHOLE group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak. Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the prior subprocess.run. - D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/ other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded re-queue, task stays on deploy-staging, no rollback / no developer-retry); a red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert. - D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated); fail-safe runs the re-test on any uncertainty. Flag merge_retest_skip_when_current_enabled. - D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation; reaper_max_running_s invariant preserved without change. - D6: in-process counters + read-only merge_gate block in GET /queue; appended ("ORCH-110","classify_retest_failure","src/merge_gate.py") to MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/ .env.example) updated in the same PR. Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after incident regression). Full suite green (1988 passed). Refs: ORCH-110 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -44,7 +44,8 @@ Invariants (ADR-001 §7, never broken):
|
||||
artefact / decides. It never calls the deploy hook, never restarts the prod
|
||||
container, never pushes / force-pushes ``main``.
|
||||
|
||||
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and lazily
|
||||
This module is a **leaf**: it imports only ``config`` / ``git_worktree`` /
|
||||
``proc_group`` (the ORCH-110 stdlib-only process-group runner) and lazily
|
||||
``qg.checks.is_self_hosting_repo`` / ``db`` / ``notifications``; it never imports
|
||||
``stage_engine``.
|
||||
"""
|
||||
@@ -52,11 +53,11 @@ This module is a **leaf**: it imports only ``config`` / ``git_worktree`` and laz
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from .config import settings
|
||||
from .git_worktree import ensure_worktree, get_worktree_path
|
||||
from .proc_group import run_in_process_group # ORCH-110 D1: tree-kill on timeout
|
||||
|
||||
logger = logging.getLogger("orchestrator.coverage_gate")
|
||||
|
||||
@@ -152,22 +153,25 @@ def measure_coverage(repo: str, branch: str) -> float | None:
|
||||
"-q",
|
||||
]
|
||||
timeout = settings.coverage_run_timeout_s
|
||||
try:
|
||||
subprocess.run(cmd, cwd=wt, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
# ORCH-110 (D1 / FR-2 / BR-3): run the coverage suite in its OWN process group so
|
||||
# a timeout tree-kills the WHOLE subtree (the sibling orphan-leak source of the
|
||||
# ORCH-109 incident), not just the direct child. The metric is read from the JSON
|
||||
# file regardless of the exit code, so a non-timeout spawn/OS error (returncode
|
||||
# None) just falls through to "no coverage json produced" -> None, byte-for-byte
|
||||
# the prior fail-open contract.
|
||||
res = run_in_process_group(
|
||||
cmd,
|
||||
cwd=wt,
|
||||
timeout=timeout,
|
||||
tree_kill=bool(getattr(settings, "subprocess_tree_kill_enabled", True)),
|
||||
grace_s=settings.agent_kill_grace_seconds,
|
||||
)
|
||||
if res.timed_out:
|
||||
logger.warning(
|
||||
"measure_coverage: pytest --cov timed out after %ss for %s/%s",
|
||||
timeout, repo, branch,
|
||||
)
|
||||
return None
|
||||
except FileNotFoundError:
|
||||
logger.warning(
|
||||
"measure_coverage: pytest / pytest-cov not available for %s/%s", repo, branch
|
||||
)
|
||||
return None
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("measure_coverage: pytest --cov error for %s/%s: %s", repo, branch, e)
|
||||
return None
|
||||
|
||||
data = None
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user