The deploy-edge merge-gate re-test bounced ORCH-111 back to development again
with `3 failed, 1916 passed, 14 errors in 444.79s` — a resource-exhaustion
signature, NOT a code defect. This is the SECOND occurrence of the identical
flake on this branch (cf. 4311720).
Evidence the branch is sound:
- Watchdog-only change (watchdog/** + docker-compose.yml + docs). It touches no
src/, no STAGE_TRANSITIONS/QG_CHECKS/check_*, and none of the failing test
files (tests/test_stage_engine.py, tests/test_orch109_timeout_model.py).
- The failures/errors are OUTSIDE this branch's scope:
test_stage_engine.py::TestStagingInfraTolerance tc02/tc13/tc14 and
test_orch109_timeout_model.py::TestContractsUnchanged::test_tc12. They pass in
isolation (4 passed/5.9s) and were ERRORS (subprocess timeouts), not assertion
failures — a systemic host failure, not logic.
- No pytest-randomly/-xdist installed -> deterministic order; merge-gate re-test
and a local run execute the same order on the same code.
- The failed run took 444.79s vs a clean local full run of 204.72s (2x slower):
the orphaned-pytest CPU-starvation incident ORCH-111 itself alerts on. By
design ORCH-111 only observes; it does not reap (ADR BR-3).
Full `pytest tests/` is green locally: 1933 passed, 0 failed, 0 errors in
204.72s (well under the 600s merge_retest budget), and the local run was FASTER
than the prior retrigger's (267s) -> host load is currently low. Empty commit to
re-run CI + the pipeline now.
NOTE (operator): until the orphaned host pytest processes are cleaned up, the
merge-gate re-test can keep flaking. ORCH-111 detects them (proc_blocking,
default-off) but does not reap them (BR-3) -> manual host cleanup is the durable
fix; a follow-up work item for reap/remediation is recommended.
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merge-gate re-test bounced ORCH-111 to development with 1 failed + 40
errors in 488s — a resource-exhaustion signature, NOT a code defect:
- This branch is watchdog-only (watchdog/** + compose); it touches no src/,
no STAGE_TRANSITIONS/QG_CHECKS/check_*, and no tests/test_stage_engine.py.
- The failing tests (test_stage_engine.py::TestStagingInfraTolerance
tc02/tc12/tc13/tc14) are outside this branch's scope, pass in isolation
(5 passed/19s), and pass right after the new watchdog tests (105 passed).
tc14 takes NO fixtures yet "errored" — a systemic/host failure, not logic.
- Host load was ~10-12 on a 4-core box at re-test time (the exact orphaned-
pytest CPU-starvation incident ORCH-111 alerts on; ORCH-111 by design only
observes, it does not reap — BR-3).
Evidence the branch is sound: full `pytest tests/` is green locally
(1933 passed, 0 failed, 0 errors in 267s, well under the 600s budget) and
Gitea CI on the branch HEAD is green (push + pull_request). Empty commit to
re-run the pipeline now that host load has dropped (10.5 -> 6).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the observability gap between agent_hung (only tracked jobs by jobs.pid)
and orphaned pytest subprocesses the orchestrator launches itself
(merge_gate.retest_branch / coverage_gate.measure_coverage). On a timeout-kill of
the agent (-9, ORCH-109) the grand-child pytest reparents onto tini and keeps
running for days, starving CPU and failing merge-gate re-test — with no alert.
Strictly inside the observer (watchdog/** + the watchdog compose service):
- watchdog/collectors/proc.py: stdlib-only /proc scan (under pid: host),
read-only, never-raise -> []; pure parsers split from I/O (tested on a fake
/proc tree). Never reads /proc/<pid>/environ.
- watchdog/signals.py: pure proc_signals builder, per-entity
("proc_blocking", pid), active iff age_s > proc_age_s; actionable RU detail.
- watchdog/core.py: opt-in tick block (gated on proc_enabled -> zero overhead /
byte-for-byte when off) + RECOVERY synthesis for a vanished process through the
existing decide()/AlertState (no new anti-spam logic).
- watchdog/config.py: WATCHDOG_PROC_{ENABLED(false),AGE_MIN(60),PATTERNS(pytest),
COOLDOWN_S(1800)}; default threshold > max(merge_retest_timeout_s=600,
coverage_run_timeout_s=900) so a legit in-flight run never crosses it.
- docker-compose.yml: pid: host on orchestrator-watchdog ONLY (read-only privilege).
Anti-false-positive and no overlap with agent_hung are by construction (cmdline
scope + age threshold), not fragile cross-namespace PID matching.
Canon synced: WATCHDOG_PROC_* in .env.watchdog.example <-> .env.example block;
documented in LITE_SETUP.md and docs/architecture/README.md (architect). src/**,
/metrics, schema_version, STAGE_TRANSITIONS, QG_CHECKS, check_*, machine-verdict
and the DB schema are untouched; deploy rebuilds only the sidecar, prod
orchestrator is not restarted (NFR-3).
Tests: tests/watchdog/test_proc_blocking_signal.py (TC-01..TC-06),
test_proc_collector.py (/proc parsing), test_tick_proc_blocking_integration.py
(TC-07), plus pid: host and proc-config assertions. Full pytest tests/ green (1930).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
