feat(post-deploy): post-deploy prod monitoring + degradation reaction (ORCH-021)

Extend pipeline responsibility past deploy->done: after the terminal
transition for an applicable repo, arm a ~15min observation window that
probes prod and reacts to a degradation the restart-time health-check
missed ("green deploy, red prod").

- src/post_deploy.py: new leaf module (config + lazy qg/db only).
  Sentinel-file restart-safe state (.post-deploy-state-<repo>/<wi>/),
  no DB migration. probe_signals/classify/decide_action/run_rollback,
  all never-raise.
- Reserved-agent job `post-deploy-monitor` (no-LLM, Variant B, calque of
  deploy-finalizer): self-requeues each tick via enqueue_job.
- Deterministic classify: DEGRADED iff >= fail_threshold consecutive
  health failures OR window 5xx ratio > 5xx_threshold; fail-safe HEALTHY.
- Self-hosting invariant (BR-5/AC-8): a tick NEVER restarts the prod
  orchestrator container -> orchestrator is ALWAYS ALERT_ONLY.
- Conditionality (ORCH-35/36/43/58): kill-switch + CSV repos, empty ->
  self-hosting only.
- QG_CHECKS / STAGE_TRANSITIONS / schema unchanged (AC-12).
- Docs: CHANGELOG, CLAUDE artefact list (16-post-deploy-log.md),
  architecture README, .env.example (ORCH_POST_DEPLOY_*).

Refs: ORCH-021

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-07 14:16:12 +00:00
committed by Dev Agent
parent 2bdba532d5
commit 2f4c553fd8
12 changed files with 1322 additions and 3 deletions

View File

@@ -265,6 +265,37 @@ class Settings(BaseSettings):
reconcile_notify_unblock: bool = True
reconcile_skip_blocked_enabled: bool = True
# ORCH-021: post-deploy production monitoring + degradation reaction. After
# the terminal deploy->done transition for an applicable repo, a reserved-agent
# `post-deploy-monitor` job (no LLM, modelled on deploy-finalizer) probes prod
# over a window and reacts to a degradation the restart-time health-check
# missed (class "green deploy, red prod", precedent ET-8). State is in sentinel
# files (.post-deploy-state-<repo>/<wi>/), no DB migration. See
# docs/architecture/adr/adr-0010-post-deploy-monitor.md.
# post_deploy_monitor_enabled -> global kill-switch (BR-8); False -> the
# pipeline is 1:1 as before ORCH-021 (no arm).
# post_deploy_repos -> CSV of repos where monitoring is REAL; empty
# -> only the self-hosting repo (orchestrator).
# Mirrors self_deploy_repos / merge_gate_repos.
# post_deploy_window_s -> observation window length (~15 min, BR-1).
# post_deploy_interval_s -> seconds between probe ticks.
# post_deploy_fail_threshold -> N CONSECUTIVE health failures -> DEGRADED.
# post_deploy_5xx_threshold -> window 5xx ratio above this -> DEGRADED.
# post_deploy_auto_rollback -> globally allow auto-rollback; True acts ONLY
# for non-self repos. For self-hosting the
# reaction is ALWAYS ALERT_ONLY (BR-5) — a tick
# NEVER restarts the prod orchestrator container.
# post_deploy_base_url -> base URL of the observed prod instance.
# Rollback target params reuse the existing deploy_prod_* settings (no dupes).
post_deploy_monitor_enabled: bool = True
post_deploy_repos: str = ""
post_deploy_window_s: int = 900
post_deploy_interval_s: int = 30
post_deploy_fail_threshold: int = 3
post_deploy_5xx_threshold: float = 0.5
post_deploy_auto_rollback: bool = False
post_deploy_base_url: str = "http://localhost:8500"
# Telegram notifications
telegram_bot_token: str = ""
telegram_chat_id: str = ""