fix(merge-gate): tolerate re-test infra-timeout + tree-kill spawned pytest

Eliminate the false `deploy-staging -> development` rollback that fired when the
merge-gate local re-test timed out (infra/resource) on a green CI + tester +
staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget
under CPU starvation from orphaned pytest processes -> timeout misrouted as a
code fault -> developer-retry loop -> manual gate).

Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched
byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable
name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push
main) and the no-prod-restart rule are preserved.

- D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage
  pytest in its own process group (start_new_session) and tree-kills the WHOLE
  group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by
  merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak.
  Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the
  prior subprocess.run.
- D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/
  other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded
  re-queue, task stays on deploy-staging, no rollback / no developer-retry); a
  red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert.
- D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD
  already CI/tester/staging-validated); fail-safe runs the re-test on any
  uncertainty. Flag merge_retest_skip_when_current_enabled.
- D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation;
  reaper_max_running_s invariant preserved without change.
- D6: in-process counters + read-only merge_gate block in GET /queue; appended
  ("ORCH-110","classify_retest_failure","src/merge_gate.py") to
  MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/
  .env.example) updated in the same PR.

Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after
incident regression). Full suite green (1988 passed).

Refs: ORCH-110

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 10:14:55 +03:00
committed by deployer
parent cf602b4810
commit 651b9af7c3
24 changed files with 1816 additions and 50 deletions

View File

@@ -164,11 +164,32 @@ ORCH_TRACKER_LIVE_STATUS_TIMEOUT_S=3
# DEFER_MAX_ATTEMPTS -> defer retries before escalation (avoids livelock).
ORCH_MERGE_GATE_ENABLED=true
ORCH_MERGE_GATE_REPOS=
ORCH_MERGE_RETEST_TIMEOUT_S=600
# ORCH-110 (D5): re-test budget raised 600 -> 900 (74% headroom over the observed
# 516.7s suite). Cross-invariant (ORCH-065/109): keep ORCH_REAPER_MAX_RUNNING_S
# (5400) > Σ(deploy-staging gate-work) + grace if you raise this — see
# docs/work-items/ORCH-110/07-infra-requirements.md.
ORCH_MERGE_RETEST_TIMEOUT_S=900
ORCH_MERGE_RETEST_TARGET=tests/
ORCH_MERGE_LOCK_TIMEOUT_S=300
ORCH_MERGE_DEFER_DELAY_S=60
ORCH_MERGE_DEFER_MAX_ATTEMPTS=5
# ORCH-110: merge-gate re-test infra-timeout tolerance + tree-kill of the
# orchestrator-spawned pytest subprocess (re-test + coverage). Each default = the
# desired prod behaviour; each flag is an independent kill-switch (off ->
# byte-for-byte pre-ORCH-110). The tree-kill grace reuses ORCH_AGENT_KILL_GRACE_SECONDS.
# SUBPROCESS_TREE_KILL_ENABLED -> D1: spawn re-test/coverage pytest in its
# own process group; tree-kill the WHOLE group on timeout (no orphan grandchildren).
# MERGE_RETEST_INFRA_TOLERANCE_ENABLED -> D3: a re-test TIMEOUT is a transient
# (bounded infra-retry, NOT a code-fault rollback to development).
# MERGE_RETEST_INFRA_MAX_RETRIES -> D3: infra-retry budget before an infra-alert.
# MERGE_RETEST_INFRA_RETRY_DELAY_S -> D3: delay before the staging-deployer re-run.
# MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED-> D4: skip the local re-test when the
# pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated).
ORCH_SUBPROCESS_TREE_KILL_ENABLED=true
ORCH_MERGE_RETEST_INFRA_TOLERANCE_ENABLED=true
ORCH_MERGE_RETEST_INFRA_MAX_RETRIES=2
ORCH_MERGE_RETEST_INFRA_RETRY_DELAY_S=120
ORCH_MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED=true
# ORCH-026 Level A: unconditional pre-merge rebase. With the flag ON (default),
# check_branch_mergeable ALWAYS rebases the branch onto origin/main under the held
# merge-lease (not only when behind) — a deterministic structural anti-phantom on