Reconciler F-1 could not tell "stuck by a lost webhook" from "escalated:
max developer retries reached, waiting for a human". With CI green and a
reviewer that kept sending REQUEST_CHANGES up to the cap, every tick
re-unblocked development -> review -> rollback -> re-unblock (incident
ET-013, infinite bounce: wasted agent runs, Telegram spam, parasitic load
on the shared self-hosting instance).
Add two pre-gate guards in Reconciler._reconcile_gate_task (after the
existing analysis/no-gate/active-job/grace guards, before the gate
pre-evaluation), each an early silent return (no advance, no unblocked_total
increment, no notifications):
- Guard 1 (escalated, deterministic, no network, checked first):
developer_retry_count(task_id) >= MAX_DEVELOPER_RETRIES. Promote
stage_engine._developer_retry_count to public developer_retry_count
(single source of truth; private alias kept). Limit from the constant,
not a literal 3.
- Guard 2 (explicit human Plane gate, Variant A, no DB migration): new
never-raise plane_sync.fetch_issue_state + Reconciler._is_blocked_or_needs_input;
any error/None/unresolved project -> conservative skip. New sub-flag
ORCH_RECONCILE_SKIP_BLOCKED_ENABLED mutes only the networked Guard 2.
F-2 unchanged: Blocked/Needs Input are outside {in_progress, approved,
rejected} so they are never replayed (regression test added). DB schema,
STAGE_TRANSITIONS, QG_CHECKS, never-raise, analysis carve-out and
kill-switches untouched.
Refs: ORCH-060
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging check exit code 1 (C9a/C9b). Live inspection inside orchestrator-staging
proves the production webhook handler is correct: get_project_states(SANDBOX).in_progress
= 84a76f65..., but scripts/staging_check.py hardcodes the enduro fallback b873d9eb...
=> handler correctly classifies the webhook as "no pipeline action". Fix belongs in
scripts/staging_check.py (resolve SANDBOX in_progress dynamically), NOT in handle_status_start
or any ORCH-058 image-freshness code. Image under test = ORCH-058 merge commit 094b5e2f.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
E2E pipeline not triggered on staging webhook ("no pipeline action" on
state b873d9eb...); reproduces prior FAILED. Rolls task back to development.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Strategy-A freshness re-validation rebuilt 8501 from merged commit 094b5e2 and
re-ran staging_check; E2E C9a/C9b fail (Plane "In Progress"/started webhook ->
"no pipeline action", no task/branch/analyst-job). Machine verdict FAILED ->
rollback to development. Prod (8500) untouched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Close AC-11 documentation gap left by the prior developer run: the
ORCH-058 feature (staging-image provenance before BUILD-ONCE retag) was
implemented and green but never recorded in the golden-source docs.
- CHANGELOG.md: add the ORCH-058 [Unreleased]/Added entry (layers A+B,
validated_revision anchor, check_staging_image_fresh, EXPECTED_REVISION
hook guard, new ORCH_IMAGE_FRESHNESS_* flags, ADR/test refs).
- .env.example (canon): document ORCH_IMAGE_FRESHNESS_ENABLED /
ORCH_IMAGE_FRESHNESS_REPOS, mirroring the ORCH-036/043/053 precedent.
- docs/architecture/README.md: footer note design -> реализовано, aligning
it with the already-updated section.
Refs: ORCH-058
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-run of deploy-staging gate (merge-gate defer cycle). Canonical
staging_check.py (mode=stub) ran inside orchestrator-staging (8501);
all 10 checks passed (exit 0). No prod (8500) container touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Конвейер продвигается только входящими webhook; потерянное событие (502 на
ребилде, отсутствие ретраев у Plane/Gitea, неразрезолвленный sha→branch)
оставляет задачу молча застрявшей (класс инцидента ORCH-044). Новый фоновый
daemon-поток src/reconciler.py (паттерн queue_worker) доигрывает пропущенный
переход через те же штатные гейты/обработчики, что и webhook:
- F-1 gate-side: для задач stage≠done, без активного job и age(updated_at) ≥
grace_for_stage(stage) — read-only пред-оценка канонического QG; зелёный →
stage_engine.advance_stage(..., finished_agent=None); красный → тишина (спам
нотификаций структурно невозможен). analysis F-1 не трогает (человеческий гейт).
- F-2 plane-side: опрос Plane API per-project (plane_sync.list_issues_by_state,
курсорная пагинация, never-raise) → реплей In Progress/Approved/Rejected через
существующие handle_status_start/handle_verdict (async из sync-потока, asyncio.run).
- F-3: усиление sha→branch в handle_ci_status — БД-fallback по единственной
development-задаче repo (неоднозначность → не резолвим), debug→info.
- Анти-дубль на создании (db.create_task_atomic под process-wide Lock): гонка
reconcile↔webhook не плодит второй task/branch/worktree/analyst-job (AC-4).
- F-4 observability: лог-строка разблокировки + Telegram + блок reconcile в /queue.
Старт/стоп в main.lifespan (после worker.start() / перед worker.stop()),
restart-safe, never-raise на единицу работы. Kill-switches ORCH_RECONCILE_ENABLED
/ ORCH_RECONCILE_PLANE_ENABLED + grace-настройки. Схема БД и реестры
STAGE_TRANSITIONS/QG_CHECKS не менялись.
Тесты: test_reconciler.py, test_reconciler_plane.py, test_gitea_sha_resolve.py,
test_config.py (33 новых, 563 всего зелёные). Документация обновлена (golden source):
architecture/README.md, INFRA.md, README.md, CHANGELOG.md, adr-0007 → accepted.
Refs: ORCH-053
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live staging-stand suite (scripts/staging_check.py, stub mode) ran inside
orchestrator-staging: 10/10 checks PASS, exit code 0. Merge-gate edge
(deploy-staging → deploy) cleared for ORCH-043.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
