Three verified live-card defects in src/notifications.py (ORCH-067/087),
all additive and indication-only (STAGE_TRANSITIONS / QG_CHECKS / check_* /
transport / DB schema untouched; never-raise; revert = git revert):
- Деф.1 (D1): _STAGE_STATUS_LABEL covered 8 of 10 STAGE_TRANSITIONS keys —
deploy-staging and cancelled (ORCH-090) fell back to the misleading
"To Analyse". Added deploy-staging→"Deploying (staging)",
cancelled→"Cancelled"; replaced the runtime fallback for an UNMAPPED stage
with a neutral capitalized label (_neutral_stage_label). created stays an
explicit "To Analyse"; broken/None input degrades safely. Map completeness
is asserted programmatically from STAGE_TRANSITIONS.keys() (single source of
truth), not a static list.
- Деф.2 (D2): the stage-row loop drew ✅ for any stage with a finished agent
run regardless of position — after a rollback the card showed the absurd
"✅ Внедрение + 🔄 Разработка". Added read-only _pipeline_pos from the
STAGE_TRANSITIONS order and a suppression gate (✅ only when
current_pos >= _pipeline_pos(stage_key)); deploy-staging→deploy normalization
applied ONLY to the current position; is_active_stage untouched.
- Деф.3 (D3): _stage_line took only the LAST run (ORCH-069: developer 3 runs
Σ $3.98 rendered ~$0.00). It now aggregates ALL of the agent's runs with the
same per-run formulas as the task totals → strict convergence with
SUM(agent_runs) by task_id; model/effort/attempt come from the last run.
Tests: test_tracker_status_line.py (ORCH-091 TC-01..TC-03 + updated tc06);
new test_tracker_rollback_metrics.py (TC-05..TC-08). Full suite green (1370).
Docs: CHANGELOG + internals.md (architecture README already updated by architect).
Refs: ORCH-091
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review P1: a STOP while a self-hosting task is PARKED on `deploy` awaiting the
manual `Confirm Deploy` was classified as a critical merge/deploy window solely
because the task still held the per-repo merge-lease (held from merge-gate through
deploy->done). That window is fully reversible — nothing is merged or deployed yet
(the irreversible merge_pr runs later in _handle_merge_verify, always under an
INITIATED marker). So the cancel was DEFERRED to run_deploy_finalizer, which only
runs after Phase B (Confirm Deploy) — the very step the operator pressed STOP to
avoid. Result: the deferred cancel was never applied, the task wedged non-terminal
holding the lease, blocking the repo's serial-gate (ORCH-088) and merges.
Fix: gate the merge-lease branch of cancel.in_critical_window on an actively
RUNNING actor (_task_has_running_actor). Lease held + running deploy/merge job ->
still deferred (genuine in-flight step). Lease held + no running actor (idle
deploy parking) -> NOT critical -> immediate full reset, which itself releases the
lease (step 3c) and drives the task terminal. INITIATED-marker deferral unchanged.
Also fixes review P2 (AC-6): set_task_cancel_requested now returns the first-stamp
fact (rowcount), and the deferred branch only notifies on the first transition —
a repeated STOP while still deferred no longer spams duplicate notifications.
Tests: test_d7_lease_held_idle_parking_is_not_critical,
test_d7_lease_held_with_running_actor_still_critical,
test_d7_stop_on_deploy_awaiting_confirm_full_resets,
test_d7_repeated_stop_in_critical_window_no_duplicate_notify. Full suite green (1349).
Refs: ORCH-090
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce the dedicated Plane STOP status as a single declarative task-cancel
mechanism: stop the active agent (graceful SIGTERM cascade), cancel all jobs
(terminal `cancelled`, never requeued), remove the worktree + delete the remote
feature branch (never main, never force-push), drive the task to the new
system-terminal state `cancelled` and tombstone the natural keys so a later
"To Analyse" re-creates it from scratch (docs artefacts preserved). STOP during a
critical merge/deploy window is deferred until the irreversible step finishes
honestly. Also closes the relaunch hole: handle_status_start relaunch is gated to
the `analysis` stage; the only pipeline-start entry point remains "To Analyse".
Cross-cutting (adr-0026): the "task terminal" predicate is widened {done} ->
{done, cancelled} in serial_gate / task_deps / stages sink + reaper/worker
requeue guards. STAGE_TRANSITIONS exit-gates / QG_CHECKS / check_* are unchanged
(`cancelled` is a sink, not a new edge). Additive, never-raise, restart-safe,
under kill-switch ORCH_STOP_STATUS_ENABLED (off -> zero regression).
New: src/cancel.py (leaf), src/gitea.py (delete_remote_branch), tasks columns
cancelled_at/cancel_requested_at, jobs status `cancelled`, GET /queue `stop` block.
Tests: tests/test_stop_status.py (TC-01..TC-14 + D7); full suite green (1345).
Docs updated in-PR (architecture README, CLAUDE.md, README.md, .env.example,
CHANGELOG). ADR-001 D4 refinement: plane_issue_id is tombstoned too (the lookup
ORs on it) — original UUID recoverable from the parseable suffix.
Refs: ORCH-090
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add src/build_cache_pruner.py — a background daemon thread modelled 1:1 on
src/disk_watchdog.py that periodically runs STRICTLY `docker builder prune -f
--filter until=<until>` (BuildKit GC) on the HOST over ssh. It is the "second
half" of the disk-watchdog (ORCH-063): the watchdog signals, the pruner cleans.
Removes the root cause of the 07.06.2026 incident (build cache ~11GB -> disk
100% -> whole self-hosting pipeline down) automatically, без оператора.
ADR-001 (Variant A): host-over-ssh, same channel as image_freshness/self_deploy
(no docker CLI in the image). Touches ONLY the build cache — no image/system
prune, no image/container removal, never restarts the docker daemon or the prod
container (self-hosting safety). No ssh target -> tick is a no-op.
- src/config.py: ORCH_BUILD_CACHE_PRUNE_* flags + defensive validators
(interval/timeout >0, until ~ ^\d+[smhdw]?$, notify_min_gb >=0 -> safe default).
- src/main.py: start last (after disk_watchdog) / stop first in lifespan;
additive read-only build_cache_prune block in GET /queue.
- never-raise on two levels (per-command + per-tick); kill-switch
ORCH_BUILD_CACHE_PRUNE_ENABLED (false -> daemon does not start, 1:1 as before).
- STAGE_TRANSITIONS / QG_CHECKS / check_* / _parse_* / DB schema UNCHANGED;
last-run/last-result is in-memory (no migration).
- tests/test_build_cache_pruner.py: TC-01..TC-12 (23 cases, docker fully mocked).
- .env.example + CHANGELOG.md updated; INFRA.md / architecture docs already
carry the component (architecture stage).
Refs: ORCH-062
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds src/disk_watchdog.py — a background daemon thread modelled on
reconciler/job_reaper that measures host-FS fill via the mounted bind-paths
(/repos, /app/data) with shutil.disk_usage and Telegram-alerts the operator at
>= threshold (default 85%). The missing proactive signal: on 07.06.2026 the
mva154 host disk silently hit 100% and stalled the whole self-hosting pipeline.
- Pure decide_action(used_pct, threshold, prev, now, realert_s): alert on
crossing up, cooldown re-alert, single recovery below threshold (unit-tested
without a thread/timer; clock injected).
- measure_paths: shutil.disk_usage per path, dedup by st_dev, per-path
never-raise (a broken path never fails the tick).
- Config flags ORCH_DISK_MONITOR_* with defensive validation (threshold 1..100,
positive intervals -> default + warning). Kill-switch -> daemon does not start.
- Additive disk_monitor block in GET /queue; start/stop in main.lifespan.
- never-raise (per-path/per-tick/per-send); STAGE_TRANSITIONS/QG_CHECKS/check_*/
DB schema untouched, no migration (anti-spam state in-memory).
Tests: tests/test_disk_watchdog.py (TC-01..TC-12, 18 cases); full suite green
(1296). Docs: INFRA.md, .env.example, CHANGELOG.md (architecture/README.md +
ADRs authored at architecture stage).
Refs: ORCH-063
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
