feat(watchdog): sidecar-watchdog F1b — monitoring brain in a separate container (ORCH-100)

Add the `watchdog/` package (thin Python-3.12 stdlib-only daemon) and the
`orchestrator-watchdog` compose service — the brain half of the domain-0
observability pair. F1a (ORCH-099) exposes GET /metrics raw signal; F1b reads it,
augments with host / container / dependency probes, runs each signal through a
generalised pure decision function (decide(signal_active, prev, now, cooldown),
a strict superset of disk_watchdog.decide_action) with per-signal in-memory
dedup/throttle/recovery, and alerts over its OWN independent Telegram channel.

Key properties (ADR-001):
- Observer separated from observed: separate container; /metrics not answering is
  itself the master `orch_down` alarm (debounced K ticks — no flap on a hiccup).
- Strictly read-only: docker.sock GET-only + mounted :ro (double guard), host
  paths :ro, no DB/disk writes, no process control — self-hosting-safe.
- never-raise on three levels (per-source/per-tick/per-send) + WATCHDOG_ENABLED
  kill-switch (disabled -> inert idle-loop, not exit).
- Disk anti-duplicate (D6): disk_watchdog (ORCH-063) stays sole owner of the 85%
  alert; sidecar carries orch_down + an opt-in 97% ceiling (default off).
- NO import from src/** (C-1); src/**, STAGE_TRANSITIONS, QG_CHECKS, check_*, DB
  schema — untouched. env_file optional so a missing .env.watchdog never breaks
  `docker compose up` for the prod orchestrator.

Tests: tests/watchdog/ (TC-01…TC-13) + full tests/ regression green (TC-14).
Docs: CHANGELOG, .env.example canon (WATCHDOG_*); architecture README + adr-0033
authored at the architecture stage.

Refs: ORCH-100

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 09:05:01 +03:00
committed by orchestrator-deployer
parent 1c08b3f62a
commit 259b507906
31 changed files with 2261 additions and 2 deletions

View File

@@ -465,3 +465,48 @@ ORCH_POST_DEPLOY_BASE_URL=http://localhost:8500
# DB title TEXT is unbounded). Default 200. An invalid/empty value gracefully
# degrades to 200 (the process never crashes on startup).
ORCH_QG0_TITLE_MAX=200
# ── ORCH-100 (FND/F1b): sidecar-watchdog (orchestrator-watchdog container) ─────
# The monitoring brain runs in a SEPARATE container with its OWN config. These
# keys are read by the watchdog package (watchdog/config.py), NOT by the
# orchestrator. At runtime they live in `.env.watchdog` (env_file of the
# orchestrator-watchdog service); this block is the canon. NO real secrets here.
# ENABLED -> kill-switch; false (or not starting the service) -> inert.
# INTERVAL_S -> seconds between ticks.
# HTTP_TIMEOUT_S -> per-request timeout (metrics / pings / docker / telegram).
# COOLDOWN_S -> re-alert throttle for a sustained signal (anti-spam).
# METRICS_URL -> orchestrator /metrics (host-network -> 127.0.0.1:8500).
# ORCH_DOWN_TICKS-> K consecutive /metrics failures before "орк не отвечает".
# MEM_PCT -> host memory used-% threshold.
# DISK_CRIT_* -> OPT-IN independent disk CEILING (disk_watchdog/ORCH-063 owns
# the 85% alert; this is a higher ceiling on the sidecar's own
# channel, OFF by default -> no double disk-alert, AC-5/D6).
# DISK_PATHS -> host paths measured for the opt-in ceiling.
# AGENT_HUNG_MIN -> runtime minutes before an agent with ~0 CPU is "hung".
# AGENT_CPU_FLOOR-> CPU fraction below which a long-running agent counts as hung.
# STAGE_STUCK_MIN-> minutes a task may sit in one stage before alerting.
# QUEUE_DEPTH -> queued-job depth threshold.
# CONTAINERS -> CSV of container names to watch (status != running/healthy).
# DOCKER_SOCK -> path to the read-only docker.sock inside the container.
# DEPS -> CSV of name=url dependency pings (empty -> no pings).
# TG_BOT_TOKEN / TG_CHAT_ID -> the sidecar's OWN Telegram bot/chat (independent
# of the orchestrator's; absent -> logs, does not send).
WATCHDOG_ENABLED=true
WATCHDOG_INTERVAL_S=30
WATCHDOG_HTTP_TIMEOUT_S=5
WATCHDOG_COOLDOWN_S=1800
WATCHDOG_METRICS_URL=http://127.0.0.1:8500/metrics
WATCHDOG_ORCH_DOWN_TICKS=3
WATCHDOG_MEM_PCT=90
WATCHDOG_DISK_CRIT_ENABLED=false
WATCHDOG_DISK_CRIT_PCT=97
WATCHDOG_DISK_PATHS=/repos,/app/data
WATCHDOG_AGENT_HUNG_MIN=20
WATCHDOG_AGENT_CPU_FLOOR=0.01
WATCHDOG_STAGE_STUCK_MIN=120
WATCHDOG_QUEUE_DEPTH=20
WATCHDOG_CONTAINERS=orchestrator
WATCHDOG_DOCKER_SOCK=/var/run/docker.sock
WATCHDOG_DEPS=
WATCHDOG_TG_BOT_TOKEN=
WATCHDOG_TG_CHAT_ID=