fix(reconciler): stop F-2 livelock spam on synced terminal tasks + cache TTL

Reconciler F-2 spammed Telegram "<wi> разблокирована" every ~120s for a
fully-synchronized Done task (incident ET-002, 191+ msgs/night) after the
ORCH-066 Plane status model merge. Two stacked defects (defense in depth):

- D1 (selection): actionable states were told apart by bare UUID, so a Done
  issue aliased onto the approved UUID entered the approved branch. Now
  terminal states are excluded by Plane state GROUP (completed/cancelled),
  a project-independent discriminator robust to UUID aliasing; per-issue
  check with a logical-key fallback when the group is unavailable.
  get_project_states caches {uuid -> group} from the same /states/ fetch;
  new sibling accessor get_project_state_groups.
- D2 (notification): _note_unblock fired unconditionally after _dispatch.
  Now it only fires on a confirmed state change (stage before/after _dispatch;
  task-appears for the start case) — handlers' contracts untouched.
- TR-3: in-memory dedup guard {issue_id -> last unblocked state} as a backstop.
- TR-4: _STATES_CACHE lived for the whole process lifetime, so a new Plane
  status was invisible without a restart. Added TTL ORCH_PLANE_STATES_TTL_S
  (default 300s; 0 = previous lifetime cache) reusing reload_project_states();
  a failed refresh serves the stale-but-correct set, not enduro defaults.

STAGE_TRANSITIONS / QG_CHECKS / DB schema / handle_* contracts / F-1 / F-3
unchanged; never-raise preserved; self-hosting tick never restarts prod.
Observability: skipped_terminal_total / deduped_total in /queue reconcile block.

Tests: tests/test_reconciler_plane.py (TC-01..TC-10),
tests/test_plane_states_cache.py (TC-11/TC-12).

Refs: ORCH-068

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-08 05:06:29 +00:00
committed by deployer
parent 0605309602
commit 1d72c44587
8 changed files with 709 additions and 16 deletions

View File

@@ -265,6 +265,18 @@ class Settings(BaseSettings):
reconcile_notify_unblock: bool = True
reconcile_skip_blocked_enabled: bool = True
# ORCH-068: TTL for the per-project Plane states cache (_STATES_CACHE in
# plane_sync). Historically the cache lived for the whole process lifetime,
# so a status added to Plane after start was never seen without a restart
# ("stale set -> no pipeline action"). With a TTL the entry self-heals by
# re-fetching /states/ after it expires (invalidation reuses the existing
# reload_project_states() primitive — no duplicated reset logic).
# plane_states_ttl_s (env ORCH_PLANE_STATES_TTL_S):
# >0 -> seconds before a cache entry is re-fetched (default 300 = 5 min);
# 0 -> disable TTL -> strictly the previous lifetime cache (back-compat
# escape hatch). get_project_states return shape is unchanged.
plane_states_ttl_s: int = 300
# ORCH-021: post-deploy production monitoring + degradation reaction. After
# the terminal deploy->done transition for an applicable repo, a reserved-agent
# `post-deploy-monitor` job (no LLM, modelled on deploy-finalizer) probes prod