Eliminate the false `deploy-staging -> development` rollback that fired when the
merge-gate local re-test timed out (infra/resource) on a green CI + tester +
staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget
under CPU starvation from orphaned pytest processes -> timeout misrouted as a
code fault -> developer-retry loop -> manual gate).
Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched
byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable
name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push
main) and the no-prod-restart rule are preserved.
- D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage
pytest in its own process group (start_new_session) and tree-kills the WHOLE
group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by
merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak.
Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the
prior subprocess.run.
- D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/
other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded
re-queue, task stays on deploy-staging, no rollback / no developer-retry); a
red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert.
- D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD
already CI/tester/staging-validated); fail-safe runs the re-test on any
uncertainty. Flag merge_retest_skip_when_current_enabled.
- D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation;
reaper_max_running_s invariant preserved without change.
- D6: in-process counters + read-only merge_gate block in GET /queue; appended
("ORCH-110","classify_retest_failure","src/merge_gate.py") to
MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/
.env.example) updated in the same PR.
Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after
incident regression). Full suite green (1988 passed).
Refs: ORCH-110
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The deploy-edge merge-gate re-test bounced ORCH-111 back to development again
with `3 failed, 1916 passed, 14 errors in 444.79s` — a resource-exhaustion
signature, NOT a code defect. This is the SECOND occurrence of the identical
flake on this branch (cf. 4311720).
Evidence the branch is sound:
- Watchdog-only change (watchdog/** + docker-compose.yml + docs). It touches no
src/, no STAGE_TRANSITIONS/QG_CHECKS/check_*, and none of the failing test
files (tests/test_stage_engine.py, tests/test_orch109_timeout_model.py).
- The failures/errors are OUTSIDE this branch's scope:
test_stage_engine.py::TestStagingInfraTolerance tc02/tc13/tc14 and
test_orch109_timeout_model.py::TestContractsUnchanged::test_tc12. They pass in
isolation (4 passed/5.9s) and were ERRORS (subprocess timeouts), not assertion
failures — a systemic host failure, not logic.
- No pytest-randomly/-xdist installed -> deterministic order; merge-gate re-test
and a local run execute the same order on the same code.
- The failed run took 444.79s vs a clean local full run of 204.72s (2x slower):
the orphaned-pytest CPU-starvation incident ORCH-111 itself alerts on. By
design ORCH-111 only observes; it does not reap (ADR BR-3).
Full `pytest tests/` is green locally: 1933 passed, 0 failed, 0 errors in
204.72s (well under the 600s merge_retest budget), and the local run was FASTER
than the prior retrigger's (267s) -> host load is currently low. Empty commit to
re-run CI + the pipeline now.
NOTE (operator): until the orphaned host pytest processes are cleaned up, the
merge-gate re-test can keep flaking. ORCH-111 detects them (proc_blocking,
default-off) but does not reap them (BR-3) -> manual host cleanup is the durable
fix; a follow-up work item for reap/remediation is recommended.
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merge-gate re-test bounced ORCH-111 to development with 1 failed + 40
errors in 488s — a resource-exhaustion signature, NOT a code defect:
- This branch is watchdog-only (watchdog/** + compose); it touches no src/,
no STAGE_TRANSITIONS/QG_CHECKS/check_*, and no tests/test_stage_engine.py.
- The failing tests (test_stage_engine.py::TestStagingInfraTolerance
tc02/tc12/tc13/tc14) are outside this branch's scope, pass in isolation
(5 passed/19s), and pass right after the new watchdog tests (105 passed).
tc14 takes NO fixtures yet "errored" — a systemic/host failure, not logic.
- Host load was ~10-12 on a 4-core box at re-test time (the exact orphaned-
pytest CPU-starvation incident ORCH-111 alerts on; ORCH-111 by design only
observes, it does not reap — BR-3).
Evidence the branch is sound: full `pytest tests/` is green locally
(1933 passed, 0 failed, 0 errors in 267s, well under the 600s budget) and
Gitea CI on the branch HEAD is green (push + pull_request). Empty commit to
re-run the pipeline now that host load has dropped (10.5 -> 6).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the observability gap between agent_hung (only tracked jobs by jobs.pid)
and orphaned pytest subprocesses the orchestrator launches itself
(merge_gate.retest_branch / coverage_gate.measure_coverage). On a timeout-kill of
the agent (-9, ORCH-109) the grand-child pytest reparents onto tini and keeps
running for days, starving CPU and failing merge-gate re-test — with no alert.
Strictly inside the observer (watchdog/** + the watchdog compose service):
- watchdog/collectors/proc.py: stdlib-only /proc scan (under pid: host),
read-only, never-raise -> []; pure parsers split from I/O (tested on a fake
/proc tree). Never reads /proc/<pid>/environ.
- watchdog/signals.py: pure proc_signals builder, per-entity
("proc_blocking", pid), active iff age_s > proc_age_s; actionable RU detail.
- watchdog/core.py: opt-in tick block (gated on proc_enabled -> zero overhead /
byte-for-byte when off) + RECOVERY synthesis for a vanished process through the
existing decide()/AlertState (no new anti-spam logic).
- watchdog/config.py: WATCHDOG_PROC_{ENABLED(false),AGE_MIN(60),PATTERNS(pytest),
COOLDOWN_S(1800)}; default threshold > max(merge_retest_timeout_s=600,
coverage_run_timeout_s=900) so a legit in-flight run never crosses it.
- docker-compose.yml: pid: host on orchestrator-watchdog ONLY (read-only privilege).
Anti-false-positive and no overlap with agent_hung are by construction (cmdline
scope + age threshold), not fragile cross-namespace PID matching.
Canon synced: WATCHDOG_PROC_* in .env.watchdog.example <-> .env.example block;
documented in LITE_SETUP.md and docs/architecture/README.md (architect). src/**,
/metrics, schema_version, STAGE_TRANSITIONS, QG_CHECKS, check_*, machine-verdict
and the DB schema are untouched; deploy rebuilds only the sidecar, prod
orchestrator is not restarted (NFR-3).
Tests: tests/watchdog/test_proc_blocking_signal.py (TC-01..TC-06),
test_proc_collector.py (/proc parsing), test_tick_proc_blocking_integration.py
(TC-07), plus pid: host and proc-config assertions. Full pytest tests/ green (1930).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the ORCH-109 entry was inserted above the ORCH-105 entry, the
ORCH-105 bullet had its body accidentally duplicated (the same
"слайдо-источник …" paragraph appeared twice in one bullet). Restore
the ORCH-105 entry to its canonical single-bodied form (byte-for-byte
identical to origin/main); the legitimate ORCH-109 additions are
untouched.
Refs: ORCH-109
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two additive, isolated launch-subsystem fixes from incident ORCH-104, without
touching STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict / DB schema.
D1 — launch-time model stamp: write the resolved model into agent_runs.model in
the SAME UPDATE as the effort stamp (ORCH-087), so the model is present from
launch, survives a timeout-kill (exit_code=-9), and is visible in-flight in
/metrics & /queue. record_usage stays an enrichment (model=COALESCE preserves the
launch stamp when the usage JSON model is None). never-raise (isolated try/except).
D3/D4 — dedicated per-role budgets: agent_timeout_developer_s=3600 /
agent_timeout_reviewer_s=3000 with a deterministic _resolve_timeout ladder
(overrides_json[agent] > dedicated role key > agent_timeout_seconds=1800; other
roles byte-for-byte). Malformed/non-positive config falls back to the global
default + WARNING (never-break). reaper_max_running_s raised 3600 -> 5400 in
lockstep to keep the ORCH-065 invariant (5400 > 3600 + 20 = 3620).
FR-4 (kill / in-flight visibility) and FR-5 (anti-salvage) are structural in the
existing code; pinned here by regression tests (tests/test_orch109_timeout_model.py,
TC-01..TC-12). Docs: .env.example, config passport, CHANGELOG, CLAUDE.md
(README/internals authored by architect in this branch).
Refs: ORCH-109
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Canonical staging_check.py (stub) exit 0; all REAL checks green,
C9a/C9b waived sandbox-infra (ORCH-061).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
