feat(disk-watchdog): host-FS fill heartbeat + Telegram alert at >=85% (ORCH-063)

Adds src/disk_watchdog.py — a background daemon thread modelled on
reconciler/job_reaper that measures host-FS fill via the mounted bind-paths
(/repos, /app/data) with shutil.disk_usage and Telegram-alerts the operator at
>= threshold (default 85%). The missing proactive signal: on 07.06.2026 the
mva154 host disk silently hit 100% and stalled the whole self-hosting pipeline.

- Pure decide_action(used_pct, threshold, prev, now, realert_s): alert on
  crossing up, cooldown re-alert, single recovery below threshold (unit-tested
  without a thread/timer; clock injected).
- measure_paths: shutil.disk_usage per path, dedup by st_dev, per-path
  never-raise (a broken path never fails the tick).
- Config flags ORCH_DISK_MONITOR_* with defensive validation (threshold 1..100,
  positive intervals -> default + warning). Kill-switch -> daemon does not start.
- Additive disk_monitor block in GET /queue; start/stop in main.lifespan.
- never-raise (per-path/per-tick/per-send); STAGE_TRANSITIONS/QG_CHECKS/check_*/
  DB schema untouched, no migration (anti-spam state in-memory).

Tests: tests/test_disk_watchdog.py (TC-01..TC-12, 18 cases); full suite green
(1296). Docs: INFRA.md, .env.example, CHANGELOG.md (architecture/README.md +
ADRs authored at architecture stage).

Refs: ORCH-063
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-09 18:51:37 +03:00
committed by orchestrator-deployer
parent 4d9251c698
commit 8759cb7df8
7 changed files with 817 additions and 0 deletions

View File

@@ -267,6 +267,25 @@ ORCH_REAPER_MAX_RUNNING_S=3600
ORCH_REAPER_FINALIZE_GRACE_S=300
ORCH_LEASE_RECLAIM_ENABLED=true
# ORCH-063: disk-watchdog — background heartbeat that measures HOST-FS fill via the
# mounted bind-paths (/repos, /app/data) with shutil.disk_usage (NOT the container
# overlay /) and Telegram-alerts the operator at >= threshold. On 07.06.2026 the
# mva154 host disk silently hit 100% and stalled the WHOLE self-hosting pipeline;
# this is the missing proactive signal. Daemon thread modelled on reconciler/reaper
# (start/stop in main.lifespan, /queue snapshot, never-raise). Anti-spam state is
# in-memory (no DB migration); the watchdog only READS fill and SENDS Telegram — it
# never touches the disk/container or restarts prod (self-hosting safety).
# DISK_MONITOR_ENABLED -> kill-switch; false -> the daemon does not start (1:1 as before).
# DISK_MONITOR_INTERVAL_S -> heartbeat measurement period, seconds (order of minutes).
# DISK_MONITOR_THRESHOLD_PCT -> fill % that triggers the alert (Owner-fixed 85; valid 1..100).
# DISK_MONITOR_REALERT_S -> cooldown between repeat alerts while above threshold (~6h).
# DISK_MONITOR_PATHS -> CSV of monitored HOST bind-paths; empty -> /repos,/app/data.
ORCH_DISK_MONITOR_ENABLED=true
ORCH_DISK_MONITOR_INTERVAL_S=300
ORCH_DISK_MONITOR_THRESHOLD_PCT=85
ORCH_DISK_MONITOR_REALERT_S=21600
ORCH_DISK_MONITOR_PATHS=/repos,/app/data
# ORCH-022: security-gate (secret-scanning + dependency audit) on the
# deploy-staging -> deploy edge, run FIRST among the edge sub-gates. Deterministic
# (no LLM): gitleaks (offline secret-scan, pinned Go binary in the image) + pip-audit