fix(reconciler): stop F-2 livelock spam on synced terminal tasks + cache TTL
Reconciler F-2 spammed Telegram "<wi> разблокирована" every ~120s for a
fully-synchronized Done task (incident ET-002, 191+ msgs/night) after the
ORCH-066 Plane status model merge. Two stacked defects (defense in depth):
- D1 (selection): actionable states were told apart by bare UUID, so a Done
issue aliased onto the approved UUID entered the approved branch. Now
terminal states are excluded by Plane state GROUP (completed/cancelled),
a project-independent discriminator robust to UUID aliasing; per-issue
check with a logical-key fallback when the group is unavailable.
get_project_states caches {uuid -> group} from the same /states/ fetch;
new sibling accessor get_project_state_groups.
- D2 (notification): _note_unblock fired unconditionally after _dispatch.
Now it only fires on a confirmed state change (stage before/after _dispatch;
task-appears for the start case) — handlers' contracts untouched.
- TR-3: in-memory dedup guard {issue_id -> last unblocked state} as a backstop.
- TR-4: _STATES_CACHE lived for the whole process lifetime, so a new Plane
status was invisible without a restart. Added TTL ORCH_PLANE_STATES_TTL_S
(default 300s; 0 = previous lifetime cache) reusing reload_project_states();
a failed refresh serves the stale-but-correct set, not enduro defaults.
STAGE_TRANSITIONS / QG_CHECKS / DB schema / handle_* contracts / F-1 / F-3
unchanged; never-raise preserved; self-hosting tick never restarts prod.
Observability: skipped_terminal_total / deduped_total in /queue reconcile block.
Tests: tests/test_reconciler_plane.py (TC-01..TC-10),
tests/test_plane_states_cache.py (TC-11/TC-12).
Refs: ORCH-068
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -117,6 +117,15 @@ ORCH_RECONCILE_GRACE_OVERRIDES_JSON=
|
||||
ORCH_RECONCILE_NOTIFY_UNBLOCK=true
|
||||
ORCH_RECONCILE_SKIP_BLOCKED_ENABLED=true
|
||||
|
||||
# ORCH-068: TTL (seconds) for the per-project Plane states cache (plane_sync
|
||||
# _STATES_CACHE). Historically the cache lived for the whole process lifetime,
|
||||
# so a status added to Plane after start was invisible until a restart
|
||||
# ("stale set -> no pipeline action"). With a TTL the entry self-heals by
|
||||
# re-fetching /states/ once it expires (reuses reload_project_states()).
|
||||
# >0 -> re-fetch after this many seconds (default 300 = 5 min);
|
||||
# 0 -> disable TTL -> strictly the previous lifetime cache (back-compat).
|
||||
ORCH_PLANE_STATES_TTL_S=300
|
||||
|
||||
# ORCH-065: job-reaper + proactive merge-lease reclaim. A background daemon thread
|
||||
# (src/job_reaper.py, started LAST in main.lifespan after requeue_running_jobs) reaps
|
||||
# zombie 'running' jobs whose monitor/process died before writing the terminal status
|
||||
|
||||
Reference in New Issue
Block a user