fix(merge-gate): tolerate re-test infra-timeout + tree-kill spawned pytest
Eliminate the false `deploy-staging -> development` rollback that fired when the merge-gate local re-test timed out (infra/resource) on a green CI + tester + staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget under CPU starvation from orphaned pytest processes -> timeout misrouted as a code fault -> developer-retry loop -> manual gate). Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push main) and the no-prod-restart rule are preserved. - D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage pytest in its own process group (start_new_session) and tree-kills the WHOLE group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak. Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the prior subprocess.run. - D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/ other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded re-queue, task stays on deploy-staging, no rollback / no developer-retry); a red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert. - D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated); fail-safe runs the re-test on any uncertainty. Flag merge_retest_skip_when_current_enabled. - D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation; reaper_max_running_s invariant preserved without change. - D6: in-process counters + read-only merge_gate block in GET /queue; appended ("ORCH-110","classify_retest_failure","src/merge_gate.py") to MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/ .env.example) updated in the same PR. Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after incident regression). Full suite green (1988 passed). Refs: ORCH-110 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
23
.env.example
23
.env.example
@@ -164,11 +164,32 @@ ORCH_TRACKER_LIVE_STATUS_TIMEOUT_S=3
|
||||
# DEFER_MAX_ATTEMPTS -> defer retries before escalation (avoids livelock).
|
||||
ORCH_MERGE_GATE_ENABLED=true
|
||||
ORCH_MERGE_GATE_REPOS=
|
||||
ORCH_MERGE_RETEST_TIMEOUT_S=600
|
||||
# ORCH-110 (D5): re-test budget raised 600 -> 900 (74% headroom over the observed
|
||||
# 516.7s suite). Cross-invariant (ORCH-065/109): keep ORCH_REAPER_MAX_RUNNING_S
|
||||
# (5400) > Σ(deploy-staging gate-work) + grace if you raise this — see
|
||||
# docs/work-items/ORCH-110/07-infra-requirements.md.
|
||||
ORCH_MERGE_RETEST_TIMEOUT_S=900
|
||||
ORCH_MERGE_RETEST_TARGET=tests/
|
||||
ORCH_MERGE_LOCK_TIMEOUT_S=300
|
||||
ORCH_MERGE_DEFER_DELAY_S=60
|
||||
ORCH_MERGE_DEFER_MAX_ATTEMPTS=5
|
||||
# ORCH-110: merge-gate re-test infra-timeout tolerance + tree-kill of the
|
||||
# orchestrator-spawned pytest subprocess (re-test + coverage). Each default = the
|
||||
# desired prod behaviour; each flag is an independent kill-switch (off ->
|
||||
# byte-for-byte pre-ORCH-110). The tree-kill grace reuses ORCH_AGENT_KILL_GRACE_SECONDS.
|
||||
# SUBPROCESS_TREE_KILL_ENABLED -> D1: spawn re-test/coverage pytest in its
|
||||
# own process group; tree-kill the WHOLE group on timeout (no orphan grandchildren).
|
||||
# MERGE_RETEST_INFRA_TOLERANCE_ENABLED -> D3: a re-test TIMEOUT is a transient
|
||||
# (bounded infra-retry, NOT a code-fault rollback to development).
|
||||
# MERGE_RETEST_INFRA_MAX_RETRIES -> D3: infra-retry budget before an infra-alert.
|
||||
# MERGE_RETEST_INFRA_RETRY_DELAY_S -> D3: delay before the staging-deployer re-run.
|
||||
# MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED-> D4: skip the local re-test when the
|
||||
# pre-merge rebase was a proven no-op (HEAD already CI/tester/staging-validated).
|
||||
ORCH_SUBPROCESS_TREE_KILL_ENABLED=true
|
||||
ORCH_MERGE_RETEST_INFRA_TOLERANCE_ENABLED=true
|
||||
ORCH_MERGE_RETEST_INFRA_MAX_RETRIES=2
|
||||
ORCH_MERGE_RETEST_INFRA_RETRY_DELAY_S=120
|
||||
ORCH_MERGE_RETEST_SKIP_WHEN_CURRENT_ENABLED=true
|
||||
# ORCH-026 Level A: unconditional pre-merge rebase. With the flag ON (default),
|
||||
# check_branch_mergeable ALWAYS rebases the branch onto origin/main under the held
|
||||
# merge-lease (not only when behind) — a deterministic structural anti-phantom on
|
||||
|
||||
Reference in New Issue
Block a user