Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes
agent_runs.exit_code FIRST, then does git push / PR / Plane comments before
_finalize_job, and the agent pid is already dead in that window — so the old
"exit_code recorded -> reap now" had no grace and could race a healthy job.
Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job)
BEFORE the atomic claim, so a reaper that lost the race had already enqueued
the next stage (dup advance / dup enqueue), violating ADR-001 Р-1.
Fix:
- Tier-2 grace: reap only once agent_runs.exit_code has been recorded for
>= reaper_finalize_grace_s (new setting, default 300s; > max finalization
window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New
finished_age_s column computed in get_running_jobs.
- claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the
reconciler pattern) to choose the terminal status, then atomically claim
'done' FIRST; only the claim winner runs the advance. A loser performs no
side effects -> no dup advance / dup enqueue.
Docs (golden source) updated in the same change: ADR-001, global adr-0011,
README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011
link). New tests cover the grace window, lost-claim no-side-effects, and the
already-advanced idempotent path.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The pr_already_merged guard was defined + unit-tested but consulted by zero
production code, while ADR-001 Р-3 / README / CHANGELOG claimed the merge path
consults it before a repeat merge (reviewer P1, ORCH-065 attempt 2/3). The
actual merge actor is the LLM deployer agent (it merges the feature PR at the
start of the `deploy` stage), so on a reaper re-drive of an already-merged PR
the deployer would blindly re-merge → Gitea error → false БАГ-8 rollback; AC-11
("no second merge") was not met deterministically.
Wire the guard at the real consultation point — the deployer prompt — so it
runs merge_gate.pr_already_merged before any (re-)merge and no-ops when the PR
is already merged. check_branch_mergeable is left untouched (AC-13: check_*
behaviour unchanged; it runs on the first deploy-staging→deploy edge, not on a
deploy-stage re-drive where the second-merge risk lives).
- .openclaw/agents/deployer.md: idempotent pre-merge guard step + general rule.
- src/merge_gate.py: docstring names the deployer-prompt consultation point.
- docs/architecture/README.md, CHANGELOG.md: state the consultation point so
golden-source matches implementation.
- tests/test_merge_gate.py: regression test asserting the deployer prompt wires
the guard (so it can't silently become dead code again).
pytest tests/ -q: 743 passed.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the "zombie jobs" incident class: job status was set only inside
the live launcher process, so a process death left jobs.status='running'
forever; at max_concurrency=1 one zombie blocked ALL projects' queue
(self-hosting risk). Adds a background daemon (src/job_reaper.py) with
three-tier liveness (dead-pid streak / known exit_code / max-running
backstop) whose only mutating write is an atomic terminal flip guarded by
WHERE status='running' (no double-process). For exit0 the canonical QG is
the source of truth via gate-driven advance, not "exit0".
Also proactively reclaims stale merge-lease (dead pid OR TTL) via file
delete only (no git ops), and makes merge finalization idempotent
(pr_already_merged guard + up-to-date short-circuit on re-drive).
New jobs.pid column via idempotent _ensure_column (no migration); pid
stamped in launcher._spawn after Popen. Reaper start/stop in lifespan;
"reaper" snapshot in GET /queue. Kill-switches: ORCH_REAPER_ENABLED,
ORCH_REAPER_INTERVAL_S, ORCH_REAPER_DEAD_TICKS, ORCH_REAPER_MAX_RUNNING_S,
ORCH_LEASE_RECLAIM_ENABLED.
Invariants unchanged (AC-13): STAGE_TRANSITIONS, QG_CHECKS registry,
check_branch_mergeable signature/behaviour, BUG-8 rollback, hook exit
codes. restart-safe, never-raise per unit of background work.
Docs: docs/architecture/README.md, CHANGELOG.md, .env.example.
Tests: tests/test_job_reaper.py, tests/test_merge_lease_reclaim.py,
tests/test_merge_gate.py (TC-16), tests/test_merge_gate_race.py (TC-17),
tests/test_queue.py, tests/test_config.py (TC-19/TC-20). 742 passed.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ORCH-058 staging rebuild (check_staging_image_fresh) builds the image with
the task git-worktree as the docker build context. A fresh worktree holds only
tracked files, but the Dockerfile did `COPY data/ ./data/` — and `data/` (the
SQLite dir) is gitignored, so it is absent from that context: `docker build`
failed with exit 1 ("BUILD-STAGING: docker build failed - aborting"), bouncing
the task off deploy-staging back to development in a loop.
The COPY was dead weight regardless: `data/` is always supplied at runtime as a
bind-mount volume (./data:/app/data, see docker-compose.yml) which shadows
anything baked into the image. Replace it with `RUN mkdir -p /app/data` so the
mountpoint exists without depending on the build context.
Regression guard: test_tc08b_dockerfile_does_not_copy_gitignored_data_dir
forbids COPY of any gitignored path (the worktree-context invariant).
Refs: ORCH-021
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extend pipeline responsibility past deploy->done: after the terminal
transition for an applicable repo, arm a ~15min observation window that
probes prod and reacts to a degradation the restart-time health-check
missed ("green deploy, red prod").
- src/post_deploy.py: new leaf module (config + lazy qg/db only).
Sentinel-file restart-safe state (.post-deploy-state-<repo>/<wi>/),
no DB migration. probe_signals/classify/decide_action/run_rollback,
all never-raise.
- Reserved-agent job `post-deploy-monitor` (no-LLM, Variant B, calque of
deploy-finalizer): self-requeues each tick via enqueue_job.
- Deterministic classify: DEGRADED iff >= fail_threshold consecutive
health failures OR window 5xx ratio > 5xx_threshold; fail-safe HEALTHY.
- Self-hosting invariant (BR-5/AC-8): a tick NEVER restarts the prod
orchestrator container -> orchestrator is ALWAYS ALERT_ONLY.
- Conditionality (ORCH-35/36/43/58): kill-switch + CSV repos, empty ->
self-hosting only.
- QG_CHECKS / STAGE_TRANSITIONS / schema unchanged (AC-12).
- Docs: CHANGELOG, CLAUDE artefact list (16-post-deploy-log.md),
architecture README, .env.example (ORCH_POST_DEPLOY_*).
Refs: ORCH-021
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The self-hosting orchestrator looped on deploy-staging -> development because
scripts/staging_check.py exited 1 on ANY failed check, so two infra-only checks
(C9a sandbox branch / C9b analyst-job — caused by SANDBOX bot accounts not being
members of the sandbox Plane project, NOT a pipeline regress) forced
staging_status: FAILED -> rollback -> loop, burning developer retries and tokens.
Direction (б) per ADR-001: classify staging checks as REAL (all pipeline checks,
fail-closed) vs SANDBOX_INFRA (narrow allowlist {C9a, C9b}, waivable). New leaf
module src/staging_verdict.py (stdlib-only, never-raise): classify_check +
compute_staging_verdict fold per-check results into a tolerant-but-fail-closed
verdict — any REAL failure -> FAILED/exit1 (safety net holds under any flag);
only C9a/C9b failed & tolerant -> SUCCESS/exit0 with waived list; only infra &
strict -> FAILED/exit1; any internal error -> FAILED/exit1 (never a false green).
staging_check.py now auto-classifies each check (public 3-tuple _items shape kept
as an ORCH-048 b6 regression guard), exposes categorized_items(), prints
INFRA-WAIVED/VERDICT lines, and exits via the verdict; new --strict flag forces
legacy strictness per-run. Kill-switch ORCH_STAGING_INFRA_TOLERANCE_ENABLED
(default true) restores legacy strict mode globally. launcher gains
action_stage_no_changes_note so "no changes to commit" on action stages is logged
as expected, not treated as under-delivery.
Contracts unchanged: STAGE_TRANSITIONS, QG_CHECKS registry, staging_status:/
deploy_status: frontmatter, hook exit-code (0/1/2), check_staging_status; no DB
migration. Docs: README, STAGING_CHECK.md, deployer.md, .env.example, CHANGELOG.
Refs: ORCH-061
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reconciler F-1 could not tell "stuck by a lost webhook" from "escalated:
max developer retries reached, waiting for a human". With CI green and a
reviewer that kept sending REQUEST_CHANGES up to the cap, every tick
re-unblocked development -> review -> rollback -> re-unblock (incident
ET-013, infinite bounce: wasted agent runs, Telegram spam, parasitic load
on the shared self-hosting instance).
Add two pre-gate guards in Reconciler._reconcile_gate_task (after the
existing analysis/no-gate/active-job/grace guards, before the gate
pre-evaluation), each an early silent return (no advance, no unblocked_total
increment, no notifications):
- Guard 1 (escalated, deterministic, no network, checked first):
developer_retry_count(task_id) >= MAX_DEVELOPER_RETRIES. Promote
stage_engine._developer_retry_count to public developer_retry_count
(single source of truth; private alias kept). Limit from the constant,
not a literal 3.
- Guard 2 (explicit human Plane gate, Variant A, no DB migration): new
never-raise plane_sync.fetch_issue_state + Reconciler._is_blocked_or_needs_input;
any error/None/unresolved project -> conservative skip. New sub-flag
ORCH_RECONCILE_SKIP_BLOCKED_ENABLED mutes only the networked Guard 2.
F-2 unchanged: Blocked/Needs Input are outside {in_progress, approved,
rejected} so they are never replayed (regression test added). DB schema,
STAGE_TRANSITIONS, QG_CHECKS, never-raise, analysis carve-out and
kill-switches untouched.
Refs: ORCH-060
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Close AC-11 documentation gap left by the prior developer run: the
ORCH-058 feature (staging-image provenance before BUILD-ONCE retag) was
implemented and green but never recorded in the golden-source docs.
- CHANGELOG.md: add the ORCH-058 [Unreleased]/Added entry (layers A+B,
validated_revision anchor, check_staging_image_fresh, EXPECTED_REVISION
hook guard, new ORCH_IMAGE_FRESHNESS_* flags, ADR/test refs).
- .env.example (canon): document ORCH_IMAGE_FRESHNESS_ENABLED /
ORCH_IMAGE_FRESHNESS_REPOS, mirroring the ORCH-036/043/053 precedent.
- docs/architecture/README.md: footer note design -> реализовано, aligning
it with the already-updated section.
Refs: ORCH-058
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-deploy after a FAILED prod deploy wedged the task on `deploy`: the
sentinel markers (approve-requested/initiated/result) are keyed by the
stable work_item_id, so after the БАГ-8 rollback (deploy -> development)
and a developer fix, Phase B's idempotency-guard saw a STALE `initiated`
and became a no-op — the detached hook never re-launched and the
finalizer was never enqueued. Add self_deploy.clear_state (never-raise,
idempotent) and call it on the check_deploy_status FAILED rollback and at
the start of Phase A, so every fresh prod-deploy pass starts clean.
Also document the new ORCH_SELF_DEPLOY_* / ORCH_DEPLOY_* descriptors in
the canonical .env.example (CLAUDE.md rule #8, ТЗ §2.6), modelled on the
ORCH-043 merge-gate block (placeholders only, secrets not committed).
Contracts untouched: STAGE_TRANSITIONS, QG_CHECKS, _parse_deploy_status,
БАГ-8, merge-gate.
Refs: ORCH-036
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add the optional, backward-compatible SOURCE_IMAGE branch to
orchestrator-deploy-hook.sh: when set, retag the staging-validated image
onto TARGET_IMAGE (docker tag) before `up -d --no-build` instead of
rebuilding — guarantees prod runs the exact artefact that passed staging
(AC-7 / TC-14). Unset -> prior behaviour; exit-code contract (0/1/2) and
health-loop untouched.
Update golden-source docs (AC-13): rewrite deployer.md `deploy` stage from
"paper SUCCESS" to the executable self-deploy (Phase A/B/C, no self-restart
from inside the container) and add the ORCH-036 CHANGELOG entry.
Refs: ORCH-036
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Конвейер продвигается только входящими webhook; потерянное событие (502 на
ребилде, отсутствие ретраев у Plane/Gitea, неразрезолвленный sha→branch)
оставляет задачу молча застрявшей (класс инцидента ORCH-044). Новый фоновый
daemon-поток src/reconciler.py (паттерн queue_worker) доигрывает пропущенный
переход через те же штатные гейты/обработчики, что и webhook:
- F-1 gate-side: для задач stage≠done, без активного job и age(updated_at) ≥
grace_for_stage(stage) — read-only пред-оценка канонического QG; зелёный →
stage_engine.advance_stage(..., finished_agent=None); красный → тишина (спам
нотификаций структурно невозможен). analysis F-1 не трогает (человеческий гейт).
- F-2 plane-side: опрос Plane API per-project (plane_sync.list_issues_by_state,
курсорная пагинация, never-raise) → реплей In Progress/Approved/Rejected через
существующие handle_status_start/handle_verdict (async из sync-потока, asyncio.run).
- F-3: усиление sha→branch в handle_ci_status — БД-fallback по единственной
development-задаче repo (неоднозначность → не резолвим), debug→info.
- Анти-дубль на создании (db.create_task_atomic под process-wide Lock): гонка
reconcile↔webhook не плодит второй task/branch/worktree/analyst-job (AC-4).
- F-4 observability: лог-строка разблокировки + Telegram + блок reconcile в /queue.
Старт/стоп в main.lifespan (после worker.start() / перед worker.stop()),
restart-safe, never-raise на единицу работы. Kill-switches ORCH_RECONCILE_ENABLED
/ ORCH_RECONCILE_PLANE_ENABLED + grace-настройки. Схема БД и реестры
STAGE_TRANSITIONS/QG_CHECKS не менялись.
Тесты: test_reconciler.py, test_reconciler_plane.py, test_gitea_sha_resolve.py,
test_config.py (33 новых, 563 всего зелёные). Документация обновлена (golden source):
architecture/README.md, INFRA.md, README.md, CHANGELOG.md, adr-0007 → accepted.
Refs: ORCH-053
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deterministic (no-LLM) sub-gate on the deploy-staging -> deploy edge that
catches a feature branch up to the CURRENT origin/main, re-tests the combined
tree, and serialises merges with a per-repo file lease — so two green parallel
branches can no longer break main (self-hosting safety for the orchestrator repo).
- src/merge_gate.py: branch_is_behind_main, auto_rebase_onto_main (push
--force-with-lease ONLY the task branch, NEVER main), retest_branch, and a
file merge-lease (atomic O_CREAT|O_EXCL, holder-aware release, stale reclaim).
Strict never-raise contract; all git ops in the per-branch worktree.
- src/qg/checks.py: check_branch_mergeable composes the primitives under the
lease; registered in QG_CHECKS. Conditional rollout (merge_gate_enabled /
merge_gate_repos, default self-hosting only).
- src/stage_engine.py: sub-gate hook on deploy-staging (not a new stage). PASS ->
advance; "merge-lock busy" -> DEFER (re-queue with available_at, anti-deadlock
at max_concurrency=1, capped); conflict/red re-test -> rollback to development
+ developer retry (capped by MAX_DEVELOPER_RETRIES). Lease released on
deploy->done / rollback / PR-merged webhook.
- src/db.py: enqueue_job(available_at_delay_s=...) for the defer (no schema change).
- src/webhooks/gitea.py: holder-aware lease release on PR-merged.
- src/config.py + .env.example: ORCH_MERGE_* settings.
Docs: README + adr-0006 (architect) already cover the design; CHANGELOG updated.
Tests: test_merge_gate.py, test_qg_merge_gate.py, test_merge_gate_race.py,
test_stage_engine.py::TestMergeGate, test_config.py, QG-registry snapshot.
Full suite: 535 passed.
Refs: ORCH-043
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both compose services (orchestrator, orchestrator-staging) now declare
user: "1000:1000" so pipeline artifacts (git worktree, docs/work-items
commits) are created as slin:slin on the host — git pull/reset under slin
no longer fail with permission errors. docker.sock access preserved via
group_add: ["999"]. SSH mount target aligned with the launcher-forced
HOME=/home/slin (/root/.ssh -> /home/slin/.ssh). launcher.py and Dockerfile
unchanged. INFRA.md and CHANGELOG.md updated; host-prerequisites (P-1..P-4)
documented.
Refs: ORCH-040
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ORCH-042: new ORCH_TRACKER_MODE (Settings.tracker_mode, default edit) selects
the live-tracker card behaviour. bump mode re-creates the card at the bottom of
the chat on every update (delete_telegram + send silently + repoint message_id),
keeping the "one card per task" invariant: <=1 new message per call, repoint
only on successful send, delete result never gates the send. New never-raising
delete_telegram helper. Anything != "bump" resolves to edit (zero regression).
Also russify/cosmetic-fix the card text (both modes): "Подтверждение BRD" label,
✅ after approve-gate, Russian stage labels, "📦 Внедрено". Docs updated in the
same PR (CHANGELOG, internals.md, .env.example).
Refs: ORCH-042
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
B6 false-FAILed because it built the project registry from the
launcher process-env via a host-path hack (sys.path.insert +
importlib.reload), not from the running staging instance. Run from the
host, ORCH_PROJECTS_JSON is unset -> default ET+ORCH registry -> false
FAIL -> spurious deploy-staging -> development rollback.
Variant (v) per ADR-001: remove the host-path hack; canonically run the
suite INSIDE orchestrator-staging via docker exec so src.projects
resolves from /app (PYTHONPATH) with .env.staging. Verdict logic
extracted into pure _evaluate_b6(known) -> (passed, detail) +
_known_project_ids_from_registry() / _run_b6() with deterministic FAIL on
source unavailability. deployer.md and STAGING_CHECK.md updated to the
docker exec command. src/projects.py, .env* and checks A/B4/B5/C
untouched.
Refs: ORCH-048
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
При заворотах на development task_desc теперь несёт дословный must-fix текст
(P0/P1 ревьюера, причина FAIL тестера) вместо одной ссылки на файл — developer-
агент видит суть претензий сразу и не повторяет ту же ошибку, экономя retry-
бюджет и токены общего инстанса.
- Новый defensive-модуль src/review_parse.py (never-raise): extract_review_findings
(P0/P1 из 12-review.md ## Findings), extract_test_failures (фрагмент тела
13-test-report.md: pytest output / FAIL-строки / Итог), усечение по лимиту.
- Две rollback-ветки stage_engine: встраивают текст + сохраняют ссылку на полный
файл; graceful-фоллбэк на ссылку-строку при битом/пустом артефакте.
- Последовательность отката, retry-счётчик, поля AdvanceResult, реестр QG_CHECKS
не менялись.
- Доки: README (Stage Engine / Откаты), CHANGELOG.
- Тесты: tests/test_review_parse.py, test_stage_engine.py::TestRollbackTaskDescEmbedding.
Refs: ORCH-046
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_parse_tests_verdict now accepts three equal-rank machine-readable
frontmatter fields in 13-test-report.md — result: (canonical tester
output), verdict: and status: (legacy/enduro-trails). Any one non-empty
field suffices; a negative token in any field stays authoritative.
Fixes the producer/consumer contract mismatch where the tester emits
`result: PASS` (per .openclaw/agents/tester.md) but the gate only read
verdict:/status:, causing a testing->development rollback loop until
MAX_DEVELOPER_RETRIES (observed on ORCH-17). Token sets frozen and gate
signature/QG_CHECKS unchanged for full backward compatibility.
Refs: ORCH-047
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
check_tests_passed/_parse_tests_verdict gated the testing -> deploy-staging
transition on `verdict:`/`status:` in 13-test-report.md, but the tester agent
prompt (.openclaw/agents/tester*) documents `result: PASS | FAIL` as THE
machine-readable field. A report that followed the contract literally
(ORCH-017: only `result: PASS`, no verdict:/status:) was bounced back to
development with a misleading "Tests FAILED". ORCH-016 only passed because its
report redundantly carried both `verdict:` and `result:`.
Treat `result:` as a first-class machine field alongside verdict/status; a
negative token in any field stays authoritative (ET-013 contract preserved).
Self-hosting QG fix: unblocks every project whose tester emits only `result:`.
Docs updated in-PR: CHANGELOG, architecture README machine-keys note.
Tests: test_qg.py::TestCheckTestsPassed::test_result_pass_only_passes / _fail_only_fails.
Refs: ORCH-017
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
notify_approve_requested now embeds two HTML <a> links into the single
notifying approve-gate message: a Gitea branch-view link to 01-brd.md and a
Plane issue browser link. Adds ORCH_PLANE_WEB_URL (external Plane web URL,
fallback to plane_api_url) with a loopback-guard that omits the Plane link
when the resolved base is localhost/empty (no broken localhost URLs in prod).
Each link is built independently and omitted on missing data; the message and
the "flip to Approved" call to action are always sent as exactly one ping. The
shared send_telegram helper is left untouched (min blast radius for the
self-hosting prod container). Dynamic labels are html.escaped; parse_mode=HTML
preserved. QG registry / stages / approve handler unchanged.
Docs updated in-PR: CHANGELOG, .env.example, INFRA env map.
Tests: test_notify_approve_links.py, test_analysis_approve_flow_links.py.
Refs: ORCH-017
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
