fix(reconciler): stop F-2 livelock spam on synced terminal tasks + cache TTL

Reconciler F-2 spammed Telegram "<wi> разблокирована" every ~120s for a
fully-synchronized Done task (incident ET-002, 191+ msgs/night) after the
ORCH-066 Plane status model merge. Two stacked defects (defense in depth):

- D1 (selection): actionable states were told apart by bare UUID, so a Done
  issue aliased onto the approved UUID entered the approved branch. Now
  terminal states are excluded by Plane state GROUP (completed/cancelled),
  a project-independent discriminator robust to UUID aliasing; per-issue
  check with a logical-key fallback when the group is unavailable.
  get_project_states caches {uuid -> group} from the same /states/ fetch;
  new sibling accessor get_project_state_groups.
- D2 (notification): _note_unblock fired unconditionally after _dispatch.
  Now it only fires on a confirmed state change (stage before/after _dispatch;
  task-appears for the start case) — handlers' contracts untouched.
- TR-3: in-memory dedup guard {issue_id -> last unblocked state} as a backstop.
- TR-4: _STATES_CACHE lived for the whole process lifetime, so a new Plane
  status was invisible without a restart. Added TTL ORCH_PLANE_STATES_TTL_S
  (default 300s; 0 = previous lifetime cache) reusing reload_project_states();
  a failed refresh serves the stale-but-correct set, not enduro defaults.

STAGE_TRANSITIONS / QG_CHECKS / DB schema / handle_* contracts / F-1 / F-3
unchanged; never-raise preserved; self-hosting tick never restarts prod.
Observability: skipped_terminal_total / deduped_total in /queue reconcile block.

Tests: tests/test_reconciler_plane.py (TC-01..TC-10),
tests/test_plane_states_cache.py (TC-11/TC-12).

Refs: ORCH-068

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-08 05:06:29 +00:00
committed by deployer
parent 0605309602
commit 1d72c44587
8 changed files with 709 additions and 16 deletions

View File

@@ -117,6 +117,15 @@ ORCH_RECONCILE_GRACE_OVERRIDES_JSON=
ORCH_RECONCILE_NOTIFY_UNBLOCK=true
ORCH_RECONCILE_SKIP_BLOCKED_ENABLED=true
# ORCH-068: TTL (seconds) for the per-project Plane states cache (plane_sync
# _STATES_CACHE). Historically the cache lived for the whole process lifetime,
# so a status added to Plane after start was invisible until a restart
# ("stale set -> no pipeline action"). With a TTL the entry self-heals by
# re-fetching /states/ once it expires (reuses reload_project_states()).
# >0 -> re-fetch after this many seconds (default 300 = 5 min);
# 0 -> disable TTL -> strictly the previous lifetime cache (back-compat).
ORCH_PLANE_STATES_TTL_S=300
# ORCH-065: job-reaper + proactive merge-lease reclaim. A background daemon thread
# (src/job_reaper.py, started LAST in main.lifespan after requeue_running_jobs) reaps
# zombie 'running' jobs whose monitor/process died before writing the terminal status