Закрывает P1 из 12-review.md (attempt 2/3): бизнес-витрина docs/overview/
не отражала новый операторский Plane-статус «Оценка» (ORCH-020).
- business.md: бизнес-способность «Оценка задачи до запуска» + Сценарий 7.
- presentation.md: статус-жест «Оценка» на слайдах 8/9/15.
- tech-pipeline.md: новая секция «Оценка задачи: статус «Оценка»» +
переформулировано устаревшее «управляющих статусов ровно три» в семейство
операторских статусов-действий (запуск / гейты / STOP / Оценка).
- tech-integrations.md: то же переформулирование + запись прогноза/факта в Plane.
- tech-data-model.md: таблица task_estimates в списке вспомогательных.
- tech-observability.md: блок estimator в GET /queue + пункт «Оценка» карточки.
Также (P3): ужесточён тавтологичный assert TC-20 (был `... or True`) —
теперь проверяет, что ни один таргет ребра STAGE_TRANSITIONS не равен estimate.
Машинные факты витрины (tests/test_system_docs.py) и control-path не тронуты;
ruff на изменённом файле чист; полный pytest зелёный (2248 passed).
Refs: ORCH-020
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a deterministic task-estimation side-mechanism: a new operator Plane
status «Оценка» (third action-status, family STOP/Confirm Deploy) triggers a
new never-raise leaf src/estimator.py that forecasts cost/time/tokens/story
points {1,2,3,5,8} from the history of completed tasks (no LLM — ADR-001 D1),
writes the forecast to Plane (estimate_point + comment), the Telegram card and
the additive task_estimates ledger (UPSERT by work_item_id), then returns the
issue to Backlog. On task completion the fact (from usage.py) is written to
Plane `point`. Massivity is free (Plane multi-select -> N webhooks); re-estimate
is idempotent; anti-disruption skips in-flight tasks; anti-loop (Backlog matches
no trigger branch).
INVARIANT (NFR-1/NFR-3): the estimator is an observer/producer, never a Quality
Gate or stage — STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys /
existing table schemas and the hot launch path (resolve_agent_model/_spawn) are
byte-for-byte untouched. fail-closed: `estimate` absent from _DEFAULT_STATES ->
a board without the status is inert (zero regression, enduro untouched).
- config: ORCH_ESTIMATOR_* flags (kill-switch + CSV scope empty->self-hosting
only, bootstrap defaults, story-point cost thresholds, wall cap).
- db: additive task_estimates table + record_estimate/set_actual/get_estimate/
estimates_snapshot + read-only completed_task_stats history aggregate.
- plane_sync: «Оценка»->estimate name map (NOT in _DEFAULT_STATES); set_issue_
backlog/set_issue_point/set_issue_estimate_point + get_project_estimate_points
(all via ORCH-117 write-guard, best-effort/fail-safe).
- webhooks/plane: fail-closed estimate branch + handle_estimate (anti-disruption,
auto-return to Backlog, off-loop).
- stage_engine: best-effort fact-on-done врезка (after terminal decision).
- notifications: «Оценка» card line (read from ledger, omitted when empty).
- main: read-only `estimator` GET /queue block + optional POST/GET /estimate.
- onboarding canon + Lite/Bundled/ONBOARDING docs: 23rd status «Оценка» (group
unstarted, never terminal); anti-drift count pins bumped 22->23.
- tests: tests/test_orch020_estimator.py (TC-01..TC-20), full suite green.
Refs: ORCH-020
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Create docs/operations/FAQ_STOP.md — a user-facing "вопрос → ответ" FAQ for
Plane board users explaining the STOP status: what it does, how to cancel a
task, step-by-step consequences (agent stops → jobs cancelled → working branch
and worktree removed → task → cancelled → Telegram+Plane; docs artifacts
preserved, main/prod untouched), deferred cancellation in the critical
merge/deploy window, explicit "STOP does NOT revert merged/deployed code"
(revert is a separate task), restart only via "To Analyse" from scratch, no-op
causes, where to observe the result, and STOP/Approved/Confirm Deploy disambig.
docs-only: src/**, STAGE_TRANSITIONS, QG_CHECKS, check_*, machine-verdict keys
and the DB schema are byte-for-byte untouched. STOP behaviour source of truth
remains ORCH-090 (adr-0026); the FAQ documents and links to it (link-first:
machine details markers/lease/tombstone given by reference, not duplicated).
Add two-way cross-links: docs/overview/business.md (Сценарий 6) and
docs/overview/tech-pipeline.md (Отмена: STOP → cancelled) → FAQ; FAQ → overview
+ ADR ORCH-090. Structure guarded by deterministic anti-drift test
tests/test_faq_stop_doc.py (offline, no network/LLM/subprocess; mirrors
tests/test_lite_setup_doc.py): existence + 8 section anchors + fact bricks +
cross-links + claim-level negative scan (sentence-level, not bare substrings,
so it never false-fails on correctly negating phrases — TR-3, with a
non-evergreen self-check). Full pytest tests/ green (2227 passed).
Refs: ORCH-108
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merge-gate's auto_rebase_onto_main silently dropped the ORCH-119
CHANGELOG bullet during a same-anchor 3-way merge: origin/main's
ORCH-120/126 entries were kept while the ORCH-119 insertion was lost.
Re-spliced the entry verbatim under ## [Unreleased] alongside 120/126.
Refs: ORCH-119
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Description section of 00-business-request.md always read the literal `TBD`,
losing the source-backed Plane-issue request context. Render the ACTUAL issue
`description` on both creation paths:
- Direct path A (serial_gate N/A): start_pipeline passes `description` to
_create_initial_docs.
- Deferred path B (ORCH-088, dominates on self-hosting): persist `description`
durable in the additive `tasks.description` column inside the same atomic INSERT
in create_task_atomic (race-safe vs ORCH-053 anti-dup claim), read it in
launcher._spawn -> _materialize_deferred_branch at claim (no network in the hot
claim path, NFR-4).
Pure render helper _render_business_request with a fail-safe fallback marker for
empty/None/unreadable descriptions (never breaks task creation); Gitea 422 stays a
no-op (idempotent). STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys
and the base CREATE TABLE tasks are byte-for-byte unchanged; the ORCH-088
anti-stale-base invariant is preserved (only the data source is enriched).
Tests: tests/test_orch119_business_request.py (TC-01 mandatory red->green
regression; TC-02..TC-07). Updated the ORCH-088 serial-gate spy for the additive
_create_initial_docs arg.
Refs: ORCH-119
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address reviewer P1 (ось ORCH-011/ORCH-079, правило агентов №6): витрина
описывала паузу serial-gate как исключительно операторскую, но ORCH-120
добавил движковый авто-park/unpark на analyst Needs Input.
- tech-pipeline.md: абзац пауз теперь называет два источника (оператор +
авто-park движком на Needs Input, флаг analyst_needs_input_autopause_enabled,
скоуп self-hosting, симметричный unpark на resume).
- tech-observability.md: пункт пауз в GET /queue — оба источника.
- tech-agents.md: when-applicable сигнальный канал 01-questions.md у analyst
(строка таблицы + поясняющая врезка; не machine-verdict, не deliverable).
- CHANGELOG: запись ORCH-120 дополнена строкой про обновление витрины.
tests/test_system_docs.py зелёный (29 passed). src/STAGE_TRANSITIONS/QG_CHECKS
не тронуты — docs-only.
Refs: ORCH-120
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Activates and completes the previously dead "analyst asks BLOCKING questions ->
01-questions.md -> Needs Input" path. Four coordinated changes, additive, under
kill-switch, self-hosting scope, never-raise; STAGE_TRANSITIONS / QG_CHECKS /
check_* / machine-verdict keys / DB schema are byte-for-byte UNCHANGED (the flow
is a pre-gate engine branch, NOT a Quality Gate; 01-questions.md is a SIGNAL
artifact, NOT a machine-verdict).
- D1 contract + canon: analyst.md documents the 01-questions.md channel (blocking
questions -> Needs Input, do NOT fabricate deliverables) + resume behaviour; new
skeleton docs/_templates/01-questions.md; PIPELINE_DOCS.md manifest row + 01-
prefix note.
- D2 freshness-supersede (DQ-2): pure offline mtime predicate questions_active in
the new leaf src/analyst_questions.py (a full FRESH package supersedes a stale
untouched 01-questions.md -> no Needs-Input loop, AC-6).
- D3 priority: questions take priority over "files ready" in
_handle_analysis_approved_flow (_decide_analysis_outcome + _emit_analysis_*);
off/out-of-scope runs the ORIGINAL byte-for-byte order (AC-9).
- D4 auto-park: set_task_paused on Needs Input via the ORCH-124 pause axis so the
repo serial-gate FIFO is not wedged while waiting for a human (AC-4); D5 resume +
unpark (clear_task_paused) in handle_status_start (analysis branch).
Flags (config.py, safe defaults): analyst_questions_gate_enabled /
analyst_questions_gate_repos (empty -> self-hosting only) /
analyst_needs_input_autopause_enabled.
Tests: test_orch120_analyst_needs_input.py (TC-01 regress + TC-02/03/06/09/10),
test_orch120_serial_gate_needs_input.py (TC-04), test_orch120_resume_unpark.py
(TC-05), test_orch120_questions_artifact_canon.py (TC-08), assert in
test_agent_prompts_canon.py (TC-07). Full suite green (2205 passed).
Refs: ORCH-120
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Queued analyst-jobs hung forever even with ORCH_SERIAL_GATE_ENABLED=false
(incident ORCH-124/125, job 2286: queued + run_id=759/760 + pid=35/42 +
started_at=NULL — physically impossible). No path returning a job to
'queued' reset its run-ownership (run_id/pid); after a container restart a
reused pid made pid_alive(stale)=True, so the job-reaper Tier-1 saw a phantom
'running' and at max_concurrency=1 wedged the claim of the whole shared queue.
Enforce the invariant `status='queued' ⇒ run_id IS NULL AND pid IS NULL AND
started_at IS NULL` on existing columns (no schema change):
- D1 forward-cleanup: requeue_running_jobs / mark_job('queued') /
mark_job_transient / reap_running_job('queued') reset run_id=NULL, pid=NULL
in the same UPDATE that clears started_at; atomic status-guards preserved.
- D2 clean claim: claim_next_job resets pid/run_id on the queued->running flip
(defense-in-depth) so the row carries pid IS NULL until _spawn stamps it.
- D4 self-heal + observability: db.find_impossible_queued_jobs /
sanitize_impossible_queued run at startup (main.lifespan) and on each reaper
tick (JobReaper.sanitize_impossible_queued_once, never-raise); counter
impossible_queued_total in the GET /queue reaper block. Kill-switch
ORCH_IMPOSSIBLE_QUEUED_SANITIZE_ENABLED (default on; gates only the D4 sweep).
- D5: reaper Tier-1 unchanged — the fix restores its precondition (pid reflects
THIS run). Marked invariants ORCH-065/113/114/099 preserved.
Tests: tests/test_orch126_queued_stale_run.py (TC-01 mandatory regression
red->green; TC-02..TC-10). Full pytest tests/ -q green (2189 passed).
Docs: internals.md (run-ownership invariant section), .env.example, CHANGELOG;
cross-cutting adr-0052.
Refs: ORCH-126
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses reviewer REQUEST_CHANGES (run 768) on ORCH-124 — docs-only,
no src/tests touched, fix scope unchanged.
P1: update docs/overview/ showcase for the new serial-gate "pause without
blocking" axis (changed task-routing functionality, ORCH-011/ORCH-079):
- tech-pipeline.md: FIFO exception "pause without blocking" next to freeze
- tech-data-model.md: durable signal tasks.paused_at on the Task row
- tech-observability.md: paused/reason in serial_gate GET /queue block +
operator endpoints POST /serial-gate/pause|resume
P2: strip leaked tool-call trailing tags (</content>/</invoke>) from 4
golden-source docs of this PR (06-adr/ADR-001, adr-0051,
08-data-requirements.md, 10-tech-risks.md).
CHANGELOG "Доки" bullet extended accordingly. Full suite green (2178 passed);
test_system_docs.py green (machine-checked showcase facts intact).
Refs: ORCH-124
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The deterministic test-runner gate (full `pytest tests/`) failed on
test_orch123_staging_runner_exec.py::test_r2_held_deploy_staging_not_rolled_back
once ORCH-124 reached the testing stage.
Root cause (pre-existing latent regress, surfaced — not introduced — by
ORCH-124): the fixture isolated `worktrees_dir` but not `repos_dir`.
`check_staging_status` falls back to `<repos_dir>/<repo>` (and its
origin/main) when the feature worktree is absent. After ORCH-123 merged,
the real `/repos/orchestrator/docs/work-items/ORCH-123/15-staging-log.md`
(verdict SUCCESS) exists on disk, so the intended-RED staging gate read it
and went green -> advance_stage was called -> the R-2 assertion failed.
Order-dependent: the test passed alone, failed in the full suite.
Fix: isolate `settings.repos_dir` to an empty tmp subdir in the fixture
(mirroring the existing worktrees_dir isolation) so the staging gate is
deterministically "not found" -> red, regardless of suite ordering. The
ORCH-123 R-2 invariant (a held deploy-staging task is never rolled back to
development, adr-0049/ADR-001 D4) is preserved and strengthened — the fix
only restores the test's stated premise. src/** / STAGE_TRANSITIONS /
QG_CHECKS / check_* untouched (test-only change).
Refs: ORCH-124
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes incident ORCH-116/ORCH-123: serial_gate defined a repo's "active task"
purely by machine stage (tasks.stage NOT IN ('done','cancelled')). Plane statuses
Backlog/Blocked/Needs-Input (layer-B indication, ORCH-066) do NOT change
tasks.stage (layer A), so a paused predecessor was indistinguishable from an active
one and held the FIFO gate closed against an urgent successor — the urgent fix
could not start until the paused task was formally done.
Introduces an explicit, durable, DB-resolvable per-task "park" signal — additive
nullable column tasks.paused_at (pattern of cancelled_at/track) — and a new
ORTHOGONAL scheduler "pause" axis. The serial-gate "active task" predicate becomes
`stage NOT IN ('done','cancelled') AND paused_at IS NULL` across all three points
(build_claim_clause / repo_has_active_task / _per_repo_snapshot). The terminal set
{done,cancelled} in serial_gate/task_deps/stages.py is byte-for-byte unchanged
(adr-0026 not regressed): task_deps/stages.py do NOT read paused_at, so a paused
declared dependency and an active repo_freeze STILL block (pause never bypasses
them — different axes). Anti-stale-base on resume relies on the existing deferred
branch cut (ORCH-088) + pre-merge auto_rebase_onto_main + merge-gate re-test
(ORCH-026/093/110) — no new rebase machinery.
Additive, under an independent sub-flag, never-raise, restart-safe; hot-claim
fail-OPEN and freeze fail-CLOSED preserved. STAGE_TRANSITIONS / QG_CHECKS / check_*
/ machine-verdict keys / existing table schemas are byte-for-byte untouched (this is
a queue-scheduler + observability change, not a Quality Gate).
- src/db.py: additive tasks.paused_at column (_ensure_column) + set/clear/is helpers
- src/serial_gate.py: _pause_layer_enabled() + pause-term in the 3 points; `paused`
list + per-job `reason` (freeze>dependency>active-task>null) in the /queue snapshot
- src/config.py + .env.example: serial_gate_pause_enabled (default True = true no-op)
- src/main.py: POST /serial-gate/pause|resume?work_item=<id> (by образцу unfreeze)
- tests/test_orch124_serial_gate_pause.py: TC-01 mandatory incident regress + TC-02..15
- CHANGELOG.md: [Unreleased] entry
ADR: docs/work-items/ORCH-124/06-adr/ADR-001-serial-gate-pause-without-blocking.md
Cross-cutting: docs/architecture/adr/adr-0051-serial-gate-pause-without-blocking.md
Refs: ORCH-124
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Recovery from the merge-gate rebase-conflict bounce. The feature branch was
rebased onto origin/main (which had merged ORCH-123). The single conflicting
hunk — docs/architecture/README.md — was resolved during the rebase: kept
ORCH-123's host-side staging-runner line AND the ORCH-116 test-runner bullet.
This follow-up commit reconciles the remainder:
- Renumber the global sweeping ADR adr-0049 -> adr-0050. ORCH-123 took adr-0049
(adr-0049-host-side-docker-execution-boundary.md) on main while ORCH-116 was
in flight, so ORCH-116 yields to the merged task and moves to the next free
number. Mechanical cross-reference reconciliation only (git mv + title + every
test-runner reference across README/internals/CLAUDE/CHANGELOG/config.py +
06-adr/ADR-001 + 12-review). Main's adr-0049 host-side references are left
byte-for-byte untouched. No design/verdict content was altered.
- Restore the ORCH-116 CHANGELOG entry that the CHANGELOG auto-merge silently
dropped (both ORCH-123 and ORCH-116 inserted at the same [Unreleased] anchor;
git kept only ORCH-123).
- Add the missing ORCH_TEST_RUNNER_* keys to .env.example (parity with the
ORCH_STAGING_RUNNER_* block; ORCH-101 canon of start keys).
Refs: ORCH-116
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second realised slice of the determinization-roadmap (ORCH-118 A5,
needs-hybrid-fallback): on the `testing` stage for the self-hosting
`orchestrator` repo the LLM `tester` agent is replaced by a deterministic
test-runner (src/test_runner.py), intercepted in launch_job BEFORE _spawn
(deploy-finalizer / post-deploy-monitor / staging-runner precedent).
It runs the regression `python -m pytest <target>` in the task worktree via
proc_group (tree-kill) + an optional read-only smoke (/health, /status, /queue
+ serial_gate), maps the exit-code -> result: PASS|FAIL via the existing
self_deploy.map_exit_code_to_status contract, writes 13-test-report.md and
initiates the EXISTING check_tests_passed gate exactly as a finished LLM-tester.
Invariant (NFR-1): only the *producer* changes — the artifact contract
(13-test-report.md / result:), the gate check_tests_passed / _parse_tests_verdict,
STAGE_TRANSITIONS and the DB schema are byte-for-byte UNCHANGED. Additive, under
a kill-switch (test_runner_enabled), never-raise, fail-closed, self-hosting scope,
two-level outcome (tool-error DEFER, anti ORCH-110), hybrid (LLM strictly
off-control-path). 52c-`status:` is aligned with the verdict (D6.1) so the
three-field _parse_tests_verdict never false-negatives a PASS.
Docs (ORCH-118 NFR-6, atomic with code): llm-call-sites.md (A5 implemented),
llm-determinization-roadmap.md (rank 2 implemented), llm-usage-policy.md,
README/internals/overview, tester.md, CLAUDE.md, CHANGELOG.md. Coverage:
tests/test_orch116_test_runner.py (TC-01..TC-14); LLM anti-drift tests green.
Full suite: 2137 passed.
Refs: ORCH-116
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ORCH-115 deterministic staging-runner ran `docker exec` FROM INSIDE the prod
`orchestrator` container, which ships only `openssh-client git curl` — no `docker`
CLI (Dockerfile:11). `Popen(["docker", ...])` hit FileNotFoundError -> a PERMANENT
environment defect that was mis-routed as a code-fail rollback
`deploy-staging -> development` (burning developer-retries). Incident ORCH-116:
every self-hosting task reaching deploy-staging was doomed to a false rollback.
Fix (adr-0049, additive, flag-gated, never-raise, self-hosting scope; the gate /
artifact contract / STAGE_TRANSITIONS / DB schema are byte-for-byte unchanged):
- D1: build_staging_command() wraps the SAME `docker exec ... staging_check.py
... --mode stub` in `ssh <user@host> '<...>'` so it runs HOST-SIDE over the
existing trusted ssh channel (mirror self_deploy / image_freshness). New flag
staging_runner_exec_host_side (default True). No docker CLI/SDK added to the
image, docker.sock not used in-container (D2 security).
- D3: three-way classify_staging_outcome (suite-ran / permanent-env /
transient-infra), disambiguating the exit=1 collision by scanning stderr.
- D4: invariant "infra != code-fail" — permanent-env / exhausted transient-infra
end in an infra-HOLD (no rollback, no developer-retry), NOT a false FAILED
rollback (supersedes ORCH-115 D5). A really-executed failing suite still rolls
back (anti-over-tolerance). R-2 verified: a held deploy-staging task is not
rolled back by the reconciler.
- D5: prod-like preflight() of the host-side channel at startup (main.lifespan,
best-effort, never blocks).
- D8: snapshot adds permanent_env / exec_host_side / preflight.
Docs (golden source, same PR): INFRA.md execution-boundary section,
architecture/README.md, CLAUDE.md, CHANGELOG.md, .env.example.
Tests: tests/test_orch123_staging_runner_exec.py (TC-01 mandatory regression
red->green; TC-02..TC-14 + R-2). ORCH-115 anti-drift green (3 tests updated for
the D1/D4/D8 supersession). Full suite: 2131 passed.
Refs: ORCH-123
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the LLM `deployer` agent on the `deploy-staging` stage (self-hosting
orchestrator) with a deterministic staging-runner intercepted in launch_job
BEFORE _spawn (the deploy-finalizer / post-deploy-monitor reserved-agent
precedent). The runner executes the SAME staging suite, maps the exit-code to
`staging_status:` via the existing self_deploy.map_exit_code_to_status contract,
writes 15-staging-log.md, and initiates the UNCHANGED check_staging_status gate
exactly as a finished LLM-deployer would.
Invariant (NFR-1): this replaces only the *producer* of the artifact — the
artifact contract, the gate / _parse_staging_status / check_staging_status name,
STAGE_TRANSITIONS, the machine-verdict key `staging_status:` and the DB schema are
byte-for-byte unchanged. Additive, under a kill-switch + repo-scope CSV,
never-raise, fail-safe back to the LLM path.
Two-level outcome (D5, anti ORCH-110): suite executed -> verdict -> advance
(FAILED -> the existing deploy-staging -> development rollback + developer-retry,
same as a FAILED LLM verdict); tool-error (suite did not execute) -> bounded DEFER
-> fail-closed FAILED + alert on exhaustion (infra != code fault; never a silent
advance / false green).
First implemented slice of the LLM determinization roadmap (ORCH-118 A6,
replace-deterministic-now).
- New leaf src/staging_runner.py (never-raise; proc_group tree-kill + timeout)
- launch_job intercept + _run_staging_runner_job (mirror _run_deploy_finalizer_job)
- config: ORCH_STAGING_RUNNER_* keys (enabled/repos/timeout/infra-retry budget)
- GET /queue staging_runner observability block
- docs: llm-call-sites/roadmap/usage-policy (A6 implemented; machine blocks +
single-transport invariant intact), deployer.md (LLM branch -> fallback),
CLAUDE.md, CHANGELOG.md, overview (tech-pipeline/tech-agents/tech-quality-security),
.env.example
- tests/test_orch115_staging_runner.py (TC-01..TC-13); LLM anti-drift green (TC-14)
Refs: ORCH-115
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ORCH-118 (inventory-first, docs+tests only): publish an evidence-based map of
every place the orchestrator's control flow consumes (or can consume) an LLM
judgment, mark the control-path axis (C control-path vs P artifact-producer),
define "avoidable LLM control path" as a checkable two-bit predicate, classify
each call-site, and order the deterministic-replacement roadmap. Pin the map to
code with offline structural anti-drift tests.
- docs/architecture/llm-call-sites.md — map + machine-readable inventory block
+ control-path axis + classification + keep-LLM justifications + deterministic
non-agent paths (FR-1/FR-2/FR-3/FR-8).
- docs/architecture/llm-determinization-roadmap.md — ordered candidates BY ROLE,
savings sourced from agent_runs, recommended first slice = deployer staging
(FR-4). No fabricated follow-up Plane-IDs (R3/NFR-6).
- docs/architecture/llm-usage-policy.md — normative principle, keep/replace
criteria via the axis, definition of "avoidable LLM control path" (FR-5/FR-8).
- tests/test_llm_call_site_inventory.py — TC-01/02/03/04/05/06/09/12/13/14.
- tests/test_llm_determinization_docs.py — TC-07/08/11.
- CHANGELOG.md + docs/overview/tech-quality-security.md — golden-source sync (AC-8).
Avoidable LLM control paths = {tester, deployer}; control-path-keep = {reviewer};
not-control-path (P) = {analyst, architect, developer}. Single LLM transport =
launcher._spawn (S0); no alternative transport (TC-12). Runtime untouched:
STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys / DB schema are
byte-for-byte; no replacement runners implemented (FR-7). Full suite: 2081 passed.
Refs: ORCH-118
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the root class of incident ORCH-114: a pytest/worktree process performed a
REAL write (PATCH issues state=<Done> + comment) against the PRODUCTION Plane
project, because test/staging processes inherit the live Plane token
(PLANE_HEADERS/PROJECT_ID are captured at import — a post-hoc env/token swap is a
no-op) and nothing forced them to write only to the sandbox. Symmetric to the
existing _no_telegram autouse floor.
- New pure never-raise leaf src/plane_write_guard.py (decide/audit_block/
audit_allow), wired into the 3 plane_sync write primitives (update_issue_state /
add_comment / _set_issue_state_direct) via _guard_allows_write, AT CALL TIME,
before any network step. Active ONLY in a test process (pytest in sys.modules /
PYTEST_CURRENT_TEST); live + staging runtimes (uvicorn) are a strict no-op.
- In a test process: default-deny. A write is allowed iff opt-in
(plane_test_write_enabled) AND target project in the sandbox allowlist
(plane_test_sandbox_projects, default = the one SANDBOX id). Prod is blocked even
with opt-in (allowlist sandbox-only); unresolved project -> block (fail-closed).
- Independent second layer: tests/conftest.py::_plane_sandbox_only autouse floor.
Intentionally NO prod-block kill-switch (anti back-door, NFR-6).
- Audit: block -> loud ERROR; sandbox-allow -> INFO.
- Bypass fixtures for the 3 (+1) pre-existing tests that assert on the mocked
write primitive's httpx call (header/URL/state logic), the guard is no Quality
Gate: STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict / DB schema
untouched.
- Tests: tests/test_orch117_plane_write_isolation.py (TC-01 mandatory ORCH-114
regression + TC-02..TC-14). Docs: CLAUDE.md, architecture/README.md,
operations/INFRA.md, .env.example, CHANGELOG.md.
Refs: ORCH-117
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the REQUEST_CHANGES findings on ORCH-114 (durable transition-ownership
lease + expected-stage CAS):
P1 — documentation = golden source:
- .env.example: add ORCH_TRANSITION_LEASE_ENABLED / ORCH_TRANSITION_LEASE_REPOS
(canon of 100% start keys, ORCH-101), next to the other gate kill-switches.
- CLAUDE.md: add the ORCH-114 passport section (mechanism, invariant, flags,
ADR links) so a future agent editing advance_stage/reaper/webhooks finds the
ownership invariant in the first mandatory-read doc (ORCH-078 traceability index).
P2 — should-fix:
- docs/overview/ (system showcase, ORCH-011): add transition_lease to
tech-data-model.md (helper tables), tech-observability.md (/queue blocks) and
tech-architecture.md (components).
- ADR-001 D4 alignment: the four side-effectful-edge rollback handlers
(_handle_merge_gate_rollback / _handle_security_gate / _handle_coverage_gate /
_handle_image_freshness) now write `development` through the expected-stage CAS
via a shared _rollback_stage_cas helper (defence against the rollback↔done
contradiction, BR-6) instead of a bare unconditional update_task_stage. Under the
held lease the sole owner always wins; a lost race aborts WITHOUT side effects.
Kill-switch off / out-of-scope repo -> degenerates to the prior write -> 1:1.
- Test isolation: make tests/test_webhooks.py order-independent by pinning the
proj-1 registry per-test (mirrors test_webhook_dedup.proj_registry); it had only
passed by relying on import order. Drop the needless module-level ORCH_DB_PATH
setdefault in test_orch114 (fresh_db already isolates db_path).
New regression tests (TC-11): in-region rollback writes route through CAS;
rollback CAS wins when at expected stage; rollback CAS-lost does NOT clobber `done`;
kill-switch-off rollback degenerates to the unconditional write.
ruff clean (src/stage_engine.py, src/transition_lease.py); full suite 2052 passed.
Refs: ORCH-114
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the root class of the ORCH-110/111/112/113 incident chain: side-effectful
stage transitions had no single ownership. `advance_stage` is re-enterable and wrote
the stage with a bare `UPDATE ... WHERE id=?` (no compare-and-swap), while >=5 actors
(monitor / Plane-webhook / reconciler F-1 / job-reaper / deploy-finalizer) enter the
same transition independently. A concurrent or post-restart re-entry therefore
re-applied irreversible effects (merge_pr / coverage-ratchet / image-rebuild /
prod-deploy initiation) and produced a contradictory rollback<->done (incident
ORCH-111, job 1914 / PR #130).
Two complementary layers, both additive, under one kill-switch, never-raise:
1. Durable transition-lease (new table `transition_lease`) — owner-exclusion on
ENTRY to the side-effectful region: a second actor that sees a LIVE owner does
not start the heavy sub-gates at all (prevention, not post-hoc repair).
2. Expected-stage CAS (`db.update_task_stage_cas`) — atomicity on the stage WRITE:
a lost race aborts with NO side effect. Also closes the 6 paths that write the
stage in bypass of advance_stage (gitea x5 + plane rollback).
Owner liveness = owner_pid + owner_boot_id (NOT a heartbeat — a blocking 900s merge
re-test cannot beat one; ADR-001 D3), making restart recovery free (a fresh boot_id
renders every prior lease stale -> reclaimed by recover_on_startup). The lease has no
own TTL: its hard age ceiling is the reaper Tier-3 backstop reaper_max_running_s, so
the cross-cutting budget invariant ORCH-065/109/110/113 is untouched.
Generalises ORCH-113 finalizer-liveness (process-local, Tier-2, deploy-staging) to a
durable cross-path lease: the reaper consults it on all relevant paths (defer live,
reclaim dead; Tier-3 ignores the marker -> bounded; a reap force-releases the lease);
reconciler F-1 and the Plane webhook defer on an active lease; main.lifespan calls
recover_on_startup() after requeue_running_jobs. finalizer_liveness.py is unchanged
(it remains the kill-switch-off fallback).
Scope self-hosting (transition_lease_repos="" -> orchestrator only; enduro untouched).
Kill-switch ORCH_TRANSITION_LEASE_ENABLED=false -> CAS degenerates to the prior
unconditional update_task_stage, lease inert, reaper -> ORCH-113 fallback (byte-for-
byte pre-ORCH-114). STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys /
existing table schemas — byte-for-byte (one additive table, no epoch column on tasks).
Observability: read-only `transition_lease` block in GET /queue + a Telegram alert on
forced/stale reclaim + optional POST /transition-lease/release?work_item=<id>.
Coverage: tests/test_orch114_transition_ownership.py (TC-01 mandatory regression of
the ORCH-111 class — red before fix, green after; TC-02..TC-14). Full suite green
(2048 passed); the 4 webhook tests that spied on the removed gitea.update_task_stage
were updated to spy on the new commit_stage_cas write path.
ADR: docs/work-items/ORCH-114/06-adr/ADR-001-transition-ownership-lease-and-stage-cas.md
Cross-cutting: docs/architecture/adr/adr-0045-transition-ownership-lease-and-stage-cas.md
Refs: ORCH-114
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-deploy git pull blocked on a dirty shared main checkout (manual/abandoned
WIP from a failed/cancelled task) — incident ORCH-111: "Your local changes to
src/config.py would be overwritten by merge" wedged the prod deploy and required
manual intervention (a group risk on self-hosting).
The deploy hook (--deploy) now converges the deploy-base to a clean, current
origin/main BEFORE the pull (git fetch + reset --hard origin/main + a SCOPED
`git clean -fd`, NEVER -x), strictly preserving the rollback/log artefacts
(.deploy-prev-image-* / deploy-hook.log via -e), gitignored .env/data/*.db/build
(no -x), and sibling/.git state (out of clean scope). Gated by CHECKOUT_HYGIENE
env injected by self_deploy.build_deploy_command only when the new pure never-raise
leaf src/checkout_hygiene.py says applies(repo) (kill-switch + self-hosting scope).
Convergence after failed/cancelled is this same deploy-time self-heal — cancel_task
is NOT extended and no background janitor is introduced. Observability: the hook
writes a `hygiene` sentinel, the Phase-C finalizer reads it and sends a best-effort
Telegram alert.
Additive, under kill-switch (ORCH_CHECKOUT_HYGIENE_ENABLED, default true; off ->
bare `git pull origin main` 1:1 before ORCH-112), never-raise, self-hosting scope.
STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict keys / DB schema / the
hook exit-code contract (0/1/2, ORCH-036) are byte-for-byte untouched.
Coverage: tests/test_deploy_checkout_hygiene.py (TC-01..TC-10; real-hook shell
simulation in a temp git repo, no network/prod/ssh, + unit). TC-01 is the
mandatory ORCH-111 regression (RED before the fix, GREEN after). Docs golden
source updated in the same PR (CLAUDE.md, CHANGELOG.md, .env.example; INFRA.md /
architecture/README.md / adr-0044 written at the architecture stage).
Refs: ORCH-112
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On the deploy-staging -> deploy edge the live monitor stamps
agent_runs.finished_at FIRST, then runs the heavy edge sub-gates
(security/merge-gate re-test/coverage/image-freshness) in-thread for MINUTES
and only THEN _finalize_job. Reaper Tier-2 measures finished_age_s from
finished_at, so past reaper_finalize_grace_s it treated the live, long
finalizer as dead and independently re-ran the advance -> a second re-test
went red -> false rollback deploy-staging -> development while the original
finalizer concurrently merged the PR (incident ORCH-111, job 1914).
Add a process-local finalizer-ownership registry (src/finalizer_liveness.py,
never-raise): the monitor mark()s ownership right after the exit_code stamp and
clear()s it in a try/finally around the (verbatim-extracted) finalization tail,
so an exception in the monitor thread still releases ownership and a genuinely
dead finalizer is reaped. The reaper Tier-2 consults the marker only when the
kill-switch is on AND the task stage == deploy-staging AND ownership is active
-> DEFER (no second advance) and fall through to the Tier-3 backstop, which
ignores the marker (a stuck/dead finalizer is still reaped in bounded time).
In-memory is authoritative (monitor + reaper are daemon threads of one uvicorn
process); restart is covered by the startup requeue_running_jobs.
Additive, global kill-switch reaper_finalizer_liveness_enabled (default True;
false -> reaper byte-for-byte prior). STAGE_TRANSITIONS / QG_CHECKS / every
check_* / machine-verdict keys / DB schema unchanged; grace/ceiling and the
ORCH-065/109/110 budget invariant untouched; never restarts prod, never pushes
main. Observability: finalizer_defers_total + finalizer_owned in GET /queue.
Tests: tests/test_orch113_reaper_finalizer_liveness.py (TC-01..TC-08, incl. the
mandatory ORCH-111 regression: red before the fix, green after).
Refs: ORCH-113
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging suite 8/10 PASS, REAL failed: none; C9a/C9b infra-waived (ORCH-061).
staging_status: SUCCESS
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eliminate the false `deploy-staging -> development` rollback that fired when the
merge-gate local re-test timed out (infra/resource) on a green CI + tester +
staging branch (incident ORCH-109/PR #129: a 516.7s suite blew its 600s budget
under CPU starvation from orphaned pytest processes -> timeout misrouted as a
code fault -> developer-retry loop -> manual gate).
Additive, 5 independent kill-switches, never-raise, self-hosting scope. Untouched
byte-for-byte: STAGE_TRANSITIONS, the QG_CHECKS registry, check_branch_mergeable
name/semantics, machine-verdict keys, the DB schema. INV-4 (never push/force-push
main) and the no-prod-restart rule are preserved.
- D1: new stdlib-only leaf src/proc_group.py runs the spawned re-test/coverage
pytest in its own process group (start_new_session) and tree-kills the WHOLE
group on timeout (os.killpg SIGTERM->grace->SIGKILL); used by
merge_gate.retest_branch and coverage_gate.measure_coverage. No orphan leak.
Fallback never-break: subprocess_tree_kill_enabled=False / non-POSIX -> the
prior subprocess.run.
- D2/D3: merge_gate.classify_retest_failure distinguishes timeout/red/lock-busy/
other; an infra timeout routes to _handle_merge_gate_infra_retry (bounded
re-queue, task stays on deploy-staging, no rollback / no developer-retry); a
red re-test / conflict still rolls back (BR-6). Exhaustion -> one infra alert.
- D4: skip the local re-test when the pre-merge rebase was a proven no-op (HEAD
already CI/tester/staging-validated); fail-safe runs the re-test on any
uncertainty. Flag merge_retest_skip_when_current_enabled.
- D5: merge_retest_timeout_s 600 -> 900 + _resolve_retest_timeout validation;
reaper_max_running_s invariant preserved without change.
- D6: in-process counters + read-only merge_gate block in GET /queue; appended
("ORCH-110","classify_retest_failure","src/merge_gate.py") to
MAIN_REGRESSION_MARKERS. Docs (README/internals overview/CLAUDE/CHANGELOG/
.env.example) updated in the same PR.
Tests: tests/test_orch110_*.py (TC-01..TC-12, incl. the red-before/green-after
incident regression). Full suite green (1988 passed).
Refs: ORCH-110
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The deploy-edge merge-gate re-test bounced ORCH-111 back to development again
with `3 failed, 1916 passed, 14 errors in 444.79s` — a resource-exhaustion
signature, NOT a code defect. This is the SECOND occurrence of the identical
flake on this branch (cf. 4311720).
Evidence the branch is sound:
- Watchdog-only change (watchdog/** + docker-compose.yml + docs). It touches no
src/, no STAGE_TRANSITIONS/QG_CHECKS/check_*, and none of the failing test
files (tests/test_stage_engine.py, tests/test_orch109_timeout_model.py).
- The failures/errors are OUTSIDE this branch's scope:
test_stage_engine.py::TestStagingInfraTolerance tc02/tc13/tc14 and
test_orch109_timeout_model.py::TestContractsUnchanged::test_tc12. They pass in
isolation (4 passed/5.9s) and were ERRORS (subprocess timeouts), not assertion
failures — a systemic host failure, not logic.
- No pytest-randomly/-xdist installed -> deterministic order; merge-gate re-test
and a local run execute the same order on the same code.
- The failed run took 444.79s vs a clean local full run of 204.72s (2x slower):
the orphaned-pytest CPU-starvation incident ORCH-111 itself alerts on. By
design ORCH-111 only observes; it does not reap (ADR BR-3).
Full `pytest tests/` is green locally: 1933 passed, 0 failed, 0 errors in
204.72s (well under the 600s merge_retest budget), and the local run was FASTER
than the prior retrigger's (267s) -> host load is currently low. Empty commit to
re-run CI + the pipeline now.
NOTE (operator): until the orphaned host pytest processes are cleaned up, the
merge-gate re-test can keep flaking. ORCH-111 detects them (proc_blocking,
default-off) but does not reap them (BR-3) -> manual host cleanup is the durable
fix; a follow-up work item for reap/remediation is recommended.
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merge-gate re-test bounced ORCH-111 to development with 1 failed + 40
errors in 488s — a resource-exhaustion signature, NOT a code defect:
- This branch is watchdog-only (watchdog/** + compose); it touches no src/,
no STAGE_TRANSITIONS/QG_CHECKS/check_*, and no tests/test_stage_engine.py.
- The failing tests (test_stage_engine.py::TestStagingInfraTolerance
tc02/tc12/tc13/tc14) are outside this branch's scope, pass in isolation
(5 passed/19s), and pass right after the new watchdog tests (105 passed).
tc14 takes NO fixtures yet "errored" — a systemic/host failure, not logic.
- Host load was ~10-12 on a 4-core box at re-test time (the exact orphaned-
pytest CPU-starvation incident ORCH-111 alerts on; ORCH-111 by design only
observes, it does not reap — BR-3).
Evidence the branch is sound: full `pytest tests/` is green locally
(1933 passed, 0 failed, 0 errors in 267s, well under the 600s budget) and
Gitea CI on the branch HEAD is green (push + pull_request). Empty commit to
re-run the pipeline now that host load has dropped (10.5 -> 6).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the observability gap between agent_hung (only tracked jobs by jobs.pid)
and orphaned pytest subprocesses the orchestrator launches itself
(merge_gate.retest_branch / coverage_gate.measure_coverage). On a timeout-kill of
the agent (-9, ORCH-109) the grand-child pytest reparents onto tini and keeps
running for days, starving CPU and failing merge-gate re-test — with no alert.
Strictly inside the observer (watchdog/** + the watchdog compose service):
- watchdog/collectors/proc.py: stdlib-only /proc scan (under pid: host),
read-only, never-raise -> []; pure parsers split from I/O (tested on a fake
/proc tree). Never reads /proc/<pid>/environ.
- watchdog/signals.py: pure proc_signals builder, per-entity
("proc_blocking", pid), active iff age_s > proc_age_s; actionable RU detail.
- watchdog/core.py: opt-in tick block (gated on proc_enabled -> zero overhead /
byte-for-byte when off) + RECOVERY synthesis for a vanished process through the
existing decide()/AlertState (no new anti-spam logic).
- watchdog/config.py: WATCHDOG_PROC_{ENABLED(false),AGE_MIN(60),PATTERNS(pytest),
COOLDOWN_S(1800)}; default threshold > max(merge_retest_timeout_s=600,
coverage_run_timeout_s=900) so a legit in-flight run never crosses it.
- docker-compose.yml: pid: host on orchestrator-watchdog ONLY (read-only privilege).
Anti-false-positive and no overlap with agent_hung are by construction (cmdline
scope + age threshold), not fragile cross-namespace PID matching.
Canon synced: WATCHDOG_PROC_* in .env.watchdog.example <-> .env.example block;
documented in LITE_SETUP.md and docs/architecture/README.md (architect). src/**,
/metrics, schema_version, STAGE_TRANSITIONS, QG_CHECKS, check_*, machine-verdict
and the DB schema are untouched; deploy rebuilds only the sidecar, prod
orchestrator is not restarted (NFR-3).
Tests: tests/watchdog/test_proc_blocking_signal.py (TC-01..TC-06),
test_proc_collector.py (/proc parsing), test_tick_proc_blocking_integration.py
(TC-07), plus pid: host and proc-config assertions. Full pytest tests/ green (1930).
Refs: ORCH-111
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the ORCH-109 entry was inserted above the ORCH-105 entry, the
ORCH-105 bullet had its body accidentally duplicated (the same
"слайдо-источник …" paragraph appeared twice in one bullet). Restore
the ORCH-105 entry to its canonical single-bodied form (byte-for-byte
identical to origin/main); the legitimate ORCH-109 additions are
untouched.
Refs: ORCH-109
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two additive, isolated launch-subsystem fixes from incident ORCH-104, without
touching STAGE_TRANSITIONS / QG_CHECKS / check_* / machine-verdict / DB schema.
D1 — launch-time model stamp: write the resolved model into agent_runs.model in
the SAME UPDATE as the effort stamp (ORCH-087), so the model is present from
launch, survives a timeout-kill (exit_code=-9), and is visible in-flight in
/metrics & /queue. record_usage stays an enrichment (model=COALESCE preserves the
launch stamp when the usage JSON model is None). never-raise (isolated try/except).
D3/D4 — dedicated per-role budgets: agent_timeout_developer_s=3600 /
agent_timeout_reviewer_s=3000 with a deterministic _resolve_timeout ladder
(overrides_json[agent] > dedicated role key > agent_timeout_seconds=1800; other
roles byte-for-byte). Malformed/non-positive config falls back to the global
default + WARNING (never-break). reaper_max_running_s raised 3600 -> 5400 in
lockstep to keep the ORCH-065 invariant (5400 > 3600 + 20 = 3620).
FR-4 (kill / in-flight visibility) and FR-5 (anti-salvage) are structural in the
existing code; pinned here by regression tests (tests/test_orch109_timeout_model.py,
TC-01..TC-12). Docs: .env.example, config passport, CHANGELOG, CLAUDE.md
(README/internals authored by architect in this branch).
Refs: ORCH-109
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Canonical staging_check.py (stub) exit 0; all REAL checks green,
C9a/C9b waived sandbox-infra (ORCH-061).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merge-gate re-test runs under the orchestrator's prod env, where the
operator legitimately set ORCH_AGENT_FALLBACK_MODEL and changed
ORCH_AGENT_MODEL_DEFAULT / ORCH_AGENT_EFFORT_*. Two ORCH-41-era tests
asserted SHIPPED defaults through the env-backed settings singleton and
failed 3/3 there, while Gitea CI (clean env) stayed green. Branch
ORCH-009 touches neither src/ nor these tests - latent non-hermetic
landmine on main, detonated by the prod env change.
- test_resolve_agent_effort.py: autouse fixture now mirrors the sibling
model-file baseline (pins shipped model/fallback fields) so the
flag-assembly tests are env-independent.
- test_resolve_agent_model.py: fixture also resets agent_fallback_model;
test_fallback_model_disabled_by_default now asserts the CLASS field
default (the actual ORCH-074 ADR-001 G4 invariant: shipped default
is ""), never-break is_valid_model asserts unchanged byte-for-byte.
Clean-env behaviour is byte-equivalent (fixtures pin exactly what an
empty env yields). Full suite: 1713 passed (was 2 failed / 1711).
Refs: ORCH-009
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator capability to bring a NEW project online in one pass, fully
outside the runtime and the pipeline (src/** byte-exact, no kill-switch
needed — activation is an explicit human CLI run). Reference = the
orchestrator repo itself (ORCH-52b/c/d/e canons).
* onboarding/repo-skeleton/ — parametrized kit of a new repo: 6 agent
prompt templates per canon 52d/92 (5 ru + deployer en with the
shared-host guardrail frame), reviewer doc-gate (REQUEST_CHANGES),
CLAUDE.md passport, AGENTS.md, CONTRIBUTING.md, docs/ skeleton with
mandatory operations/INFRA.md, .env.example; {{NAME}} placeholders +
stdlib render, dictionary onboarding/placeholders.json (bijection
held by tests). Canon is NOT forked: docs/_templates + docs/_standards
are live-copied from the checkout at materialization time (BR-2/D3).
* scripts/onboard_project.py — plan (default, GET-only, zero mutations)
/ apply (idempotent ensure, no delete ops at all) / verify (registry
round-trip via the actual projects._parse_projects_json, all 22 state
names incl. fail-closed Confirm Deploy/STOP, labels, webhook, kit
completeness, unresolved-placeholder scan). Closed read-only src
import list (ADR D4); state groups fixed per ADR D5 (STOP→cancelled,
terminal groups only Done/Cancelled/STOP); Gitea webhook reuses the
single global ORCH_GITEA_WEBHOOK_SECRET (TR-6); initial push ONLY
into a freshly created empty repo (INV-4 untouched); never restarts
prod / never edits .env / deletes nothing (NFR-2); secrets masked
(NFR-3); Plane CE API gaps degrade to manual-step (fail-safe).
* docs/operations/ONBOARDING.md runbook + SETUP_WEBHOOKS.md generalized
per-repo; CLAUDE.md / docs/architecture/README.md / CHANGELOG.md
updated in the same PR (golden source).
* Anti-drift tests: test_onboarding_kit.py / test_onboarding_script.py
(mocked, no network) / test_onboarding_invariants.py (snapshots of
STAGE_TRANSITIONS/QG_CHECKS, closed CLI import list, reference
.openclaw/agents/ prompts untouched). Full regression: 1713 passed.
Refs: ORCH-009
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging suite (docker exec orchestrator-staging, port 8501) exit 0.
All REAL checks green; C9a/C9b INFRA-WAIVED (ORCH-061).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Merge-gate auto_rebase_onto_main bounced this branch back: ORCH-100 landed
in main first and claimed global ADR number adr-0033 (adr-0033-sidecar-watchdog),
while this branch had created adr-0033-lessons-journal. Resolved the genuine
land race:
- rebased feature/ORCH-098-fnd onto current origin/main (linear history)
- resolved docs/architecture/README.md component-list conflict — both the
Lessons-journal and Sidecar-watchdog bullets now coexist
- renamed docs/architecture/adr/adr-0033-lessons-journal.md →
adr-0034-lessons-journal.md (next free global ADR number) + fixed the
in-file header
- updated all cross-references (CLAUDE.md, README.md, work-item ADR-001,
12-review.md) 0033→0034 for the lessons journal; ORCH-100's adr-0033
(sidecar) left intact
- recovered the ORCH-098 CHANGELOG entry silently dropped by the rebase
auto-merge (now above ORCH-100, ADR ref corrected to 0034)
No code semantics changed; src/** auto-merged cleanly (ORCH-100 did not
touch src/**). ruff: n/a locally (CI). pytest tests/ -q: 1630 passed.
Refs: ORCH-098
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 1 ("Foundation", F2) of the self-improvement epic: formalise free-text
"lessons" from memory/ into a machine-readable `lessons` table — the foundation
for the future retrospective agent (E2), the RICE prioritiser (E3) and Стрим.
- src/lessons.py: pure never-raise observer leaf (record/get/update/snapshot),
kill-switch only, NO repo scope (observer-only; records about any repo incl.
enduro; repo cut on the read side). Slug-convention constants.
- src/db.py: additive idempotent `lessons` table in init_db() (+3 indexes);
nullable attribution columns from the start (NFR-6, _ensure_column forward-safe);
helpers record_lesson/get_lessons/update_lesson/lessons_snapshot/
lessons_recent_dup_exists (auto-dedup window).
- 4 auto-detectors (best-effort, source="auto", deduped): gate_failure
(_handle_qg_failure_rollbacks), merge_hold (_handle_merge_verify HOLD),
transient_retry (launcher._finalize_transient budget-exhaustion), deploy_degraded
(post-deploy DEGRADED -> set_repo_freeze).
- src/main.py: GET /lessons, POST /lessons, POST /lessons/{id} + read-only
`lessons` block in GET /queue; off-switch -> {"enabled": false}.
- src/config.py: lessons_enabled / lessons_query_limit_default / lessons_dedup_window_s.
- tests/test_lessons.py: TC-01..TC-12 (unit + integration), all green.
- Docs: CLAUDE.md, docs/architecture/README.md (component + schema + API), CHANGELOG.
Invariant: the journal is an OBSERVER, not a Quality Gate — STAGE_TRANSITIONS /
QG_CHECKS / check_* / machine-verdict / existing table schemas are byte-for-byte
untouched; enduro not affected. never-raise on every public fn + injection.
Refs: ORCH-098
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_queue.py::TestRetry::test_finalize_job_requeue_then_fail failed in the
self-hosting environment because launcher._finalize_job classifies a non-zero
exit by reading the tail of <settings.runs_dir>/<run_id>.log. settings.runs_dir
defaults to the live prod dir /app/data/runs, which on the host holds REAL
accumulated agent logs; a real 2.log containing "429" flips the expected
'permanent' classification to 'transient', requeueing the job instead of
marking it 'failed'. This is ambient prod pollution, not a code fault.
Add an autouse _isolate_runs_dir fixture (mirroring _no_telegram /
_disable_merge_verify) that redirects settings.runs_dir to a per-test tmp dir
so _run_log_path() resolves to a non-existent file and classify_log_file()
returns the documented 'permanent' default. Full suite: 1617 passed. src/**
untouched.
Refs: ORCH-100
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the `watchdog/` package (thin Python-3.12 stdlib-only daemon) and the
`orchestrator-watchdog` compose service — the brain half of the domain-0
observability pair. F1a (ORCH-099) exposes GET /metrics raw signal; F1b reads it,
augments with host / container / dependency probes, runs each signal through a
generalised pure decision function (decide(signal_active, prev, now, cooldown),
a strict superset of disk_watchdog.decide_action) with per-signal in-memory
dedup/throttle/recovery, and alerts over its OWN independent Telegram channel.
Key properties (ADR-001):
- Observer separated from observed: separate container; /metrics not answering is
itself the master `orch_down` alarm (debounced K ticks — no flap on a hiccup).
- Strictly read-only: docker.sock GET-only + mounted :ro (double guard), host
paths :ro, no DB/disk writes, no process control — self-hosting-safe.
- never-raise on three levels (per-source/per-tick/per-send) + WATCHDOG_ENABLED
kill-switch (disabled -> inert idle-loop, not exit).
- Disk anti-duplicate (D6): disk_watchdog (ORCH-063) stays sole owner of the 85%
alert; sidecar carries orch_down + an opt-in 97% ceiling (default off).
- NO import from src/** (C-1); src/**, STAGE_TRANSITIONS, QG_CHECKS, check_*, DB
schema — untouched. env_file optional so a missing .env.watchdog never breaks
`docker compose up` for the prod orchestrator.
Tests: tests/watchdog/ (TC-01…TC-13) + full tests/ regression green (TC-14).
Docs: CHANGELOG, .env.example canon (WATCHDOG_*); architecture README + adr-0033
authored at the architecture stage.
Refs: ORCH-100
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up ORCH-040: legacy root:root files in /repos broke worktree creation
under uid 1000 with a raw "Permission denied" (agent never started, no diagnosis).
Three additive, kill-switch-reversible layers; STAGE_TRANSITIONS / QG_CHECKS /
check_* / machine-verdict keys / DB schema are byte-for-byte unchanged.
- D1: ensure_worktree classifies the permission class and raises an actionable
RuntimeError (cause + chown command + INFRA.md ref); non-permission errors keep
the prior raw-stderr contract; kill-switch off -> contract 1:1 as before ORCH-057.
- D2: new never-raise leaf src/fs_normalize.py — scan_ownership (TTL-cached,
early-exit per root), applies()-first scope (empty CSV -> self-hosting only),
opt-in normalize() that chowns ONLY when privileged (no-op under uid 1000).
- D3: best-effort startup detect in main.lifespan (WARNING + Telegram on mismatch,
never-fatal); read-only fs_ownership block in GET /queue; POST /fs-normalize/check.
Claim is NOT blocked — the clear early outcome is delivered by D1 at launch.
- Docs/config: .env.example flags + CHANGELOG (architecture README / adr-0031 /
INFRA.md procedure already landed on the branch).
- Tests: test_fs_normalize.py, test_git_worktree_perm.py,
test_fs_normalize_startup.py, test_api_queue.py (TC-01..TC-12). Full suite green.
Refs: ORCH-057
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FND/F1a: add a versioned read-only JSON endpoint GET /metrics that exposes the
orchestrator's own raw state for the future observability sidecar F1b — active
task stages, job queue, agent-liveness (pid/runtime/cpu_ticks), and cost/tokens.
The orchestrator emits ONLY raw signal it alone knows; thresholds/alerts/history
live in the separate sidecar (observer separated from observed, BRD §1).
- src/metrics.py: new leaf collector build_metrics() (never-raise per section,
serial_gate.snapshot() pattern); envelope schema_version/generated_at/clk_tck +
stages/queue/agents/cost. _read_cpu_ticks(pid) reads utime+stime from
/proc/<pid>/stat (null on None/dead/non-Linux pid — never raises).
- src/main.py: thin @app.get("/metrics") wrapper (style of GET /queue).
- src/db.py: read-only helpers get_running_agents() (dedicated SELECT, not an
extension of the hot-path get_running_jobs()), agent_cost_totals(),
queue_retry_stats(); job_status_counts() default dict gains the cancelled key.
- src/config.py: metrics_endpoint_enabled kill-switch (default True), env
ORCH_METRICS_ENABLED via explicit validation_alias so the documented switch
actually controls the flag.
- docs: README API table row + CHANGELOG entry (contract section already added
by architect); .env.example ORCH_METRICS_ENABLED.
Strictly read-only / never-raise: STAGE_TRANSITIONS / QG_CHECKS / check_* /
machine-verdict keys / DB schema untouched; /health//status//queue byte-for-byte.
Tests: tests/test_metrics.py (TC-01..TC-11) + env-alias tests in test_config.py.
Full suite green (1482).
Refs: ORCH-099
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer P1 (ORCH-027 attempt 2): inserting the ORCH-027 changelog
block duplicated the adjacent ORCH-095 entry — its paragraph body was
repeated verbatim, corrupting a golden-source doc and another work
item's artifact (CLAUDE.md §3). Remove the duplicate half, leaving a
single ORCH-095 body. ORCH-027 entry untouched (already correct).
Refs: ORCH-027
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
measure_coverage hardcoded "python" for the coverage subprocess; the prod container
and the CI runner expose "python3" (a bare "python" may be absent), and pytest-cov
lives in exactly the running interpreter's environment. Use sys.executable so the
measurement always runs under the same interpreter as the orchestrator.
Refs: ORCH-027
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a deterministic (no-LLM) coverage sub-gate that blocks coverage
degradation before a task branch merges into `main`. Existing gates judge only by
the FACT of passing (check_ci_green / check_tests_passed / merge-gate re-test), not
by completeness — so a batch autonomous run (ORCH-088) silently erodes coverage.
Pattern mirrors the security-gate (ORCH-022): leaf src/coverage_gate.py (never-raise)
+ thin check_coverage_gate in QG_CHECKS + _handle_coverage_gate splice in advance_stage,
run AFTER merge-gate (measured on the caught-up HEAD that lands in main) and BEFORE
image-freshness (fail before the expensive docker rebuild).
- measure_coverage: pytest --cov=src --cov-report=json in the per-branch worktree ->
line coverage %; None on tool error -> fail-open + WARNING by default (FR-6).
- compute_coverage_verdict (pure): absolute | baseline | both + epsilon (NFR-4 anti-flap);
baseline None -> bootstrap (absolute-only).
- coverage_baseline DB table (additive, CREATE TABLE IF NOT EXISTS) + ratchet-up in
_handle_merge_verify (deploy->done): atomic compare-and-set under merge-lease, never
decreases; bootstrap on first merge.
- Artefact 18-coverage-report.md (coverage_status: frontmatter, single source of truth);
GET /queue `coverage` block; FAIL -> Telegram; optional POST /coverage/baseline override.
- Flags ORCH_COVERAGE_* (kill-switch + self-hosting-only scope) -> enduro untouched;
STAGE_TRANSITIONS / existing check_* / verdict keys byte-for-byte unchanged (NFR-5/AC-8).
- pytest-cov==5.0.0 added to requirements.txt.
Tests: tests/test_coverage_gate.py (TC-01..TC-15). Frozen QG-registry anti-regress
tests + deploy-staging edge tests updated for the new sub-gate. Full suite green.
Docs: README / adr-0029 / PIPELINE_DOCS / 18-coverage-report.md template (architecture
stage) + CHANGELOG / CLAUDE.md / .env.example (this PR).
Refs: ORCH-027
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
render_task_tracker sends/edits the live card with parse_mode=HTML. _fmt_minutes
returns the literal "<1м" for a sub-minute stage; interpolated raw into HTML text
Telegram parsed "<1м" as an opening tag -> editMessageText 400 can't parse
entities -> edit_telegram EDIT_FAILED -> update_task_tracker early return
(anti-duplicate ORCH-087) -> the card froze (incident ORCH-093, message_id 18854).
Close the whole "unescaped data in HTML text" class per ADR-001: a module-local
_esc(x)=html.escape(str(x)) (never-raise) wraps every DATA slot (durations, status
label, model, effort, token/cost metrics) exactly once at the render boundary in
render_task_tracker/_stage_line. Source functions stay HTML-agnostic (_fmt_minutes
still returns "<1м"; escape on the boundary renders it visually identical as
<1м, so the visible format is unchanged). Intentional MARKUP slots (num_html /
link_for / _done_link / already-escaped esc_title) are NOT escaped, so the issue
number stays a clickable <a> tag and nothing is double-escaped.
A previously-frozen card auto-recovers on the next stage transition (a new safe
render edits in place, 200) — no new code, no touch to edit_telegram /
update_task_tracker / the orphan ledger, so the ORCH-087 anti-duplicate invariant
is preserved (a transient edit failure still does not spawn a new card).
STAGE_TRANSITIONS / QG_CHECKS / check_* / notification transport / DB schema are
untouched. New tests/test_tracker_html_escape.py (TC-01..TC-11); full suite green.
Refs: ORCH-095
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A DB stage=done task with 0 active jobs flapped in Plane between `Awaiting
Deploy` and `Monitoring after Deploy` instead of holding `Done` (verified live
on ORCH-061, task 47): the three deploy-phase setters were terminal-blind, so
any stale/duplicate/unknown caller under the bot token re-stamped an
intermediate status over the terminal Done, forever.
- New leaf src/deploy_status_guard.py (pure, never-raise, config-gated): decide()
-> ALLOW | CONVERGE_DONE | SUPPRESS on the entry of set_issue_awaiting_deploy /
set_issue_deploying / set_issue_monitoring. A deploy-phase status is legitimate
iff the task is non-terminal OR (done AND post-deploy window active); otherwise
done converges to Done idempotently, cancelled is suppressed (FR-2, D1/D2).
- D3: move post_deploy.arm_monitor ABOVE the terminal-sync block in advance_stage
so window_active is True when the legitimate first Monitoring is set (the task
is already DB-done by then); a re-drive after the window closes converges to Done.
- D4: run_post_deploy_monitor no-ops without a status PATCH / re-queue when the
task became cancelled mid-window (zombie-tick guard, FR-3).
- D5: additive `reason` kwarg on the three setters + one structured log line per
verdict (work_item/caller/target/db_stage/window_active/verdict); new read-only
db.get_task_by_work_item_id; post_deploy.window_active helper.
- Flags deploy_status_guard_enabled (kill-switch -> 1:1) / deploy_status_guard_repos
(CSV; empty = self-hosting only). STAGE_TRANSITIONS / QG_CHECKS / check_* /
machine-verdict keys / DB schema untouched (reads existing tasks.stage).
Tests: TC-01..TC-12 across 5 new test modules + config flags; updated the
reason-kwarg assertions in test_deploy_terminal_sync / test_deploy_approve.
Full regress green (1413). Docs: CHANGELOG, CLAUDE.md, docs/architecture/README.md
(status -> реализовано), .env.example.
Refs: ORCH-094
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
merge_pr now wraps ONLY the mutating POST /pulls/{n}/merge in a bounded
exponential-backoff retry-loop on TRANSIENT outcomes (405 "try again later",
408, any 5xx, network/timeout, and 409|422 while the PR is still mergeable);
TERMINAL outcomes (403/404/real conflict via mergeable==False) -> fast honest
False, so the ORCH-071/081 not-merged HOLD backstop is unchanged. Fixes the
ORCH-063 false HOLD + manual re-merge on Gitea's post-push mergeability hiccup.
ensure_open_pr gains an "already fully in main" guard (_branch_fully_in_main,
git merge-base --is-ancestor HEAD origin/main) BEFORE creating a PR -> new
"already-in-main" outcome avoids the garbage empty PR on a re-driven finalizer;
_handle_merge_verify skips merge_pr on that outcome and lets the authoritative
SHA-in-main check confirm -> done (not a HOLD). git error of the guard fails
OPEN to the create path.
New ORCH_MERGE_RETRY_* settings (kill-switch merge_retry_enabled -> one-shot,
max_attempts=3, backoff base=2/max=5). INV-4 (merge only via Gitea PR-merge API,
never push/force-push main), never-raise, STAGE_TRANSITIONS/QG_CHECKS/DB schema
unchanged. Docs (README merge-verify section, CLAUDE.md, CHANGELOG, .env.example)
updated in the same PR. Tests: test_merge_gate.py TC-01..12, test_config.py
TC-13, test_merge_verify.py TC-14..16; full suite green (1389).
Refs: ORCH-093
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three verified live-card defects in src/notifications.py (ORCH-067/087),
all additive and indication-only (STAGE_TRANSITIONS / QG_CHECKS / check_* /
transport / DB schema untouched; never-raise; revert = git revert):
- Деф.1 (D1): _STAGE_STATUS_LABEL covered 8 of 10 STAGE_TRANSITIONS keys —
deploy-staging and cancelled (ORCH-090) fell back to the misleading
"To Analyse". Added deploy-staging→"Deploying (staging)",
cancelled→"Cancelled"; replaced the runtime fallback for an UNMAPPED stage
with a neutral capitalized label (_neutral_stage_label). created stays an
explicit "To Analyse"; broken/None input degrades safely. Map completeness
is asserted programmatically from STAGE_TRANSITIONS.keys() (single source of
truth), not a static list.
- Деф.2 (D2): the stage-row loop drew ✅ for any stage with a finished agent
run regardless of position — after a rollback the card showed the absurd
"✅ Внедрение + 🔄 Разработка". Added read-only _pipeline_pos from the
STAGE_TRANSITIONS order and a suppression gate (✅ only when
current_pos >= _pipeline_pos(stage_key)); deploy-staging→deploy normalization
applied ONLY to the current position; is_active_stage untouched.
- Деф.3 (D3): _stage_line took only the LAST run (ORCH-069: developer 3 runs
Σ $3.98 rendered ~$0.00). It now aggregates ALL of the agent's runs with the
same per-run formulas as the task totals → strict convergence with
SUM(agent_runs) by task_id; model/effort/attempt come from the last run.
Tests: test_tracker_status_line.py (ORCH-091 TC-01..TC-03 + updated tc06);
new test_tracker_rollback_metrics.py (TC-05..TC-08). Full suite green (1370).
Docs: CHANGELOG + internals.md (architecture README already updated by architect).
Refs: ORCH-091
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review P1: a STOP while a self-hosting task is PARKED on `deploy` awaiting the
manual `Confirm Deploy` was classified as a critical merge/deploy window solely
because the task still held the per-repo merge-lease (held from merge-gate through
deploy->done). That window is fully reversible — nothing is merged or deployed yet
(the irreversible merge_pr runs later in _handle_merge_verify, always under an
INITIATED marker). So the cancel was DEFERRED to run_deploy_finalizer, which only
runs after Phase B (Confirm Deploy) — the very step the operator pressed STOP to
avoid. Result: the deferred cancel was never applied, the task wedged non-terminal
holding the lease, blocking the repo's serial-gate (ORCH-088) and merges.
Fix: gate the merge-lease branch of cancel.in_critical_window on an actively
RUNNING actor (_task_has_running_actor). Lease held + running deploy/merge job ->
still deferred (genuine in-flight step). Lease held + no running actor (idle
deploy parking) -> NOT critical -> immediate full reset, which itself releases the
lease (step 3c) and drives the task terminal. INITIATED-marker deferral unchanged.
Also fixes review P2 (AC-6): set_task_cancel_requested now returns the first-stamp
fact (rowcount), and the deferred branch only notifies on the first transition —
a repeated STOP while still deferred no longer spams duplicate notifications.
Tests: test_d7_lease_held_idle_parking_is_not_critical,
test_d7_lease_held_with_running_actor_still_critical,
test_d7_stop_on_deploy_awaiting_confirm_full_resets,
test_d7_repeated_stop_in_critical_window_no_duplicate_notify. Full suite green (1349).
Refs: ORCH-090
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce the dedicated Plane STOP status as a single declarative task-cancel
mechanism: stop the active agent (graceful SIGTERM cascade), cancel all jobs
(terminal `cancelled`, never requeued), remove the worktree + delete the remote
feature branch (never main, never force-push), drive the task to the new
system-terminal state `cancelled` and tombstone the natural keys so a later
"To Analyse" re-creates it from scratch (docs artefacts preserved). STOP during a
critical merge/deploy window is deferred until the irreversible step finishes
honestly. Also closes the relaunch hole: handle_status_start relaunch is gated to
the `analysis` stage; the only pipeline-start entry point remains "To Analyse".
Cross-cutting (adr-0026): the "task terminal" predicate is widened {done} ->
{done, cancelled} in serial_gate / task_deps / stages sink + reaper/worker
requeue guards. STAGE_TRANSITIONS exit-gates / QG_CHECKS / check_* are unchanged
(`cancelled` is a sink, not a new edge). Additive, never-raise, restart-safe,
under kill-switch ORCH_STOP_STATUS_ENABLED (off -> zero regression).
New: src/cancel.py (leaf), src/gitea.py (delete_remote_branch), tasks columns
cancelled_at/cancel_requested_at, jobs status `cancelled`, GET /queue `stop` block.
Tests: tests/test_stop_status.py (TC-01..TC-14 + D7); full suite green (1345).
Docs updated in-PR (architecture README, CLAUDE.md, README.md, .env.example,
CHANGELOG). ADR-001 D4 refinement: plane_issue_id is tombstoned too (the lookup
ORs on it) — original UUID recoverable from the parseable suffix.
Refs: ORCH-090
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add src/build_cache_pruner.py — a background daemon thread modelled 1:1 on
src/disk_watchdog.py that periodically runs STRICTLY `docker builder prune -f
--filter until=<until>` (BuildKit GC) on the HOST over ssh. It is the "second
half" of the disk-watchdog (ORCH-063): the watchdog signals, the pruner cleans.
Removes the root cause of the 07.06.2026 incident (build cache ~11GB -> disk
100% -> whole self-hosting pipeline down) automatically, без оператора.
ADR-001 (Variant A): host-over-ssh, same channel as image_freshness/self_deploy
(no docker CLI in the image). Touches ONLY the build cache — no image/system
prune, no image/container removal, never restarts the docker daemon or the prod
container (self-hosting safety). No ssh target -> tick is a no-op.
- src/config.py: ORCH_BUILD_CACHE_PRUNE_* flags + defensive validators
(interval/timeout >0, until ~ ^\d+[smhdw]?$, notify_min_gb >=0 -> safe default).
- src/main.py: start last (after disk_watchdog) / stop first in lifespan;
additive read-only build_cache_prune block in GET /queue.
- never-raise on two levels (per-command + per-tick); kill-switch
ORCH_BUILD_CACHE_PRUNE_ENABLED (false -> daemon does not start, 1:1 as before).
- STAGE_TRANSITIONS / QG_CHECKS / check_* / _parse_* / DB schema UNCHANGED;
last-run/last-result is in-memory (no migration).
- tests/test_build_cache_pruner.py: TC-01..TC-12 (23 cases, docker fully mocked).
- .env.example + CHANGELOG.md updated; INFRA.md / architecture docs already
carry the component (architecture stage).
Refs: ORCH-062
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds src/disk_watchdog.py — a background daemon thread modelled on
reconciler/job_reaper that measures host-FS fill via the mounted bind-paths
(/repos, /app/data) with shutil.disk_usage and Telegram-alerts the operator at
>= threshold (default 85%). The missing proactive signal: on 07.06.2026 the
mva154 host disk silently hit 100% and stalled the whole self-hosting pipeline.
- Pure decide_action(used_pct, threshold, prev, now, realert_s): alert on
crossing up, cooldown re-alert, single recovery below threshold (unit-tested
without a thread/timer; clock injected).
- measure_paths: shutil.disk_usage per path, dedup by st_dev, per-path
never-raise (a broken path never fails the tick).
- Config flags ORCH_DISK_MONITOR_* with defensive validation (threshold 1..100,
positive intervals -> default + warning). Kill-switch -> daemon does not start.
- Additive disk_monitor block in GET /queue; start/stop in main.lifespan.
- never-raise (per-path/per-tick/per-send); STAGE_TRANSITIONS/QG_CHECKS/check_*/
DB schema untouched, no migration (anti-spam state in-memory).
Tests: tests/test_disk_watchdog.py (TC-01..TC-12, 18 cases); full suite green
(1296). Docs: INFRA.md, .env.example, CHANGELOG.md (architecture/README.md +
ADRs authored at architecture stage).
Refs: ORCH-063
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
src/frontmatter.py grows from a single-key reader into the full machine
contract: reader (read_frontmatter_value, unchanged), one parse primitive
(parse_frontmatter), writer (render/write_frontmatter), schema validator
(validate_schema/REQUIRED_FIELDS, warning-only by default) and a shared
strip_frontmatter helper. The five verdict gates (check_reviewer_verdict,
_parse_tests_verdict, _parse_deploy_status, _parse_staging_status,
parse_security_status) now read through the single parse_frontmatter point
instead of duplicated ad-hoc YAML logic; review_parse._strip_frontmatter and
security_gate.extract_security_findings reuse the shared helper.
Strictly backward compatible + never-raise: STAGE_TRANSITIONS, the QG_CHECKS
composition, verdict semantics (incl. ORCH-047 three-field tester + negative
token priority), reason-strings and worktree->origin/main fallback are 1:1.
The schema validator never influences a gate verdict by default; hard-fail is
reserved behind the frontmatter_validation_strict kill-switch (default False).
New formal handoff spec docs/_standards/HANDOFF_PROTOCOL.md ("stage -> required
output" + required frontmatter schema), aligned 1:1 with PIPELINE_DOCS.md.
Tests: test_frontmatter.py (TC-01..07), test_qg_verdicts.py (TC-08..15),
test_security_gate.py (TC-12), test_stages_invariants.py (TC-16). Full
tests/ green (1212).
Refs: ORCH-076
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging suite 8/10 PASS, exit 0. All REAL checks green; C9a/C9b
sandbox-infra waived (ORCH-061). staging_status: SUCCESS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lift the two HUMAN gates that block an autonomous batch run (epic ORCH-088):
the BRD gate (analysis: manual Approved) and the prod-deploy gate (deploy
Phase A: manual Confirm Deploy, ORCH-059). Selective (a Plane label on the
issue), declarative, reversible, and WITHOUT touching a single technical check.
Additive, mirroring the conditional sub-gates (ORCH-035/043/058/088): leaf
src/labels.py (never-raise) + two point insertions + config flags.
STAGE_TRANSITIONS / QG_CHECKS / check_* / DB schema are NOT touched.
- autoApprove: врезка in _handle_analysis_approved_flow (files_ok branch) ->
set_issue_approved + log/Telegram/Plane-comment + advance_stage(
finished_agent=None) — the SAME path a human Approved takes (approved-via-
status -> analysis->architecture + mark_brd_review_ended). No duplicated
transition logic; re-entrancy safe.
- autoDeploy: врезка in _handle_self_deploy_phase_a after advance to deploy +
clear_state -> log/Telegram/Plane-comment + _handle_self_deploy_phase_b
(INITIATED marker, Deploying, finalizer). Only the indicative human steps are
skipped. BR-5 holds structurally: Phase A is reached only after the green edge
sub-gates, so autoDeploy can never deploy a broken build.
- plane_sync: fetch_issue_labels (None on error != []), get_project_labels
({normalized_name->uuid}, TTL cache, ambiguity sentinel), set_issue_approved.
- config flags: auto_label_enabled (kill-switch), auto_approve_label/
auto_deploy_label, auto_label_repos (empty -> self-hosting only),
auto_label_states_ttl_s. applies() (local) checked FIRST; has_label (network)
only when applies==True -> zero network / zero regression when disabled (AC-8).
- Fail-safe (never auto on doubt), transparency via log+Telegram+Plane+card,
read-only auto_labels block in GET /queue.
- Tests TC-01..TC-26 across 7 modules; docs (CLAUDE.md, architecture README,
CHANGELOG) updated in the same PR.
Refs: ORCH-089
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging check 8/10 PASS, exit 0. All REAL checks green; C9a/C9b
sandbox-infra waived (ORCH-061).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Этап 1 (serial e2e) пакетного автономного режима. Новая задача репо не входит
в analysis (analyst-job не выбирается, ветка не режется), пока в репо есть более
ранняя незавершённая задача (FIFO, t2.id < jobs.task_id) ИЛИ репо заморожен.
- src/serial_gate.py — новый leaf (never-raise): build_claim_clause (fail-OPEN),
is_repo_frozen (fail-CLOSED), set/clear_repo_freeze, serial_gate_applies, snapshot.
- src/db.py — идемпотентная миграция repo_freeze + serial_gate-фрагмент в claim_next_job.
- src/webhooks/plane.py + src/agents/launcher.py — отложенный срез ветки: start_pipeline
не создаёт Gitea-ветку/docs для применимого репо; релокация в _materialize_deferred_branch
на момент claim analyst-job (база = свежий origin/main с кодом предшественника, AC-6).
- src/stage_engine.py — post-deploy DEGRADED → durable per-repo freeze + Telegram-алерт.
- src/main.py — блок serial_gate в GET /queue + POST /serial-gate/unfreeze.
- src/config.py — serial_gate_enabled / serial_gate_repos / serial_gate_freeze_enabled.
FIFO-уточнение реализации (FR-2): ADR-001 D1 фиксировал t2.id != jobs.task_id; при !=
пакет одновременно созданных свежих задач взаимно блокировался бы (дедлок). t2.id <
jobs.task_id допускает самую раннюю задачу и сериализует остальные, сохраняя AC-1/R-7.
STAGE_TRANSITIONS / QG_CHECKS / check_* — без изменений. Аддитивно, под kill-switch,
never-raise, restart-safe; при выключенном флаге — нулевая регрессия (enduro не затронут).
Тесты: TC-01..TC-22 (test_serial_gate*.py + test_queue_endpoint.py); полный прогон 1114 зелёных.
Docs: README (serial gate / /queue / API / БД), CLAUDE.md, CHANGELOG.md, .env.example.
Refs: ORCH-088
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_spawn_stamps_resolved_effort упал в CI с PermissionError на '/app':
launcher._spawn хардкодил output_path='/app/data/runs/{run_id}.log' и
os.makedirs('/app/data/runs'). В контейнере /app есть, на CI-хосте
(act_runner hostexecutor) — нет, makedirs бросает -> красный CI.
Фикс корня (не только теста): базовый каталог per-run логов вынесен в
Settings.runs_dir (env ORCH_RUNS_DIR, дефолт '/app/data/runs' = прод 1:1).
Новый хелпер _run_log_path(run_id) — единый источник пути, использован в
_spawn + три прежних inline-строки логов/алертов. Тест monkeypatch-ит
settings.runs_dir на tmp_path -> окружение-независим (проверено прогоном
с принудительно недоступным /app). pytest tests/ -q: 1090 passed.
STAGE_TRANSITIONS/QG_CHECKS/схема БД не тронуты. Docs: README env-таблица,
CHANGELOG.
Refs: ORCH-087
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Staging check suite passed (8/10 PASS, exit 0). C9a/C9b waived as
sandbox-infra (ORCH-061); all REAL checks green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Закрывает F-1-пробел ORCH-068: терминал-исключение и in-memory dedup
(изначально только F-2) распространены на gate-side путь реконсилятора,
устраняя ложное «🔧 reconciler: ET-002 done разблокирована (потерян
webhook)» (особенно после рестарта).
- D1: новый _resolve_issue_status — один сетевой резолв Plane-статуса
задачи за тик (states, groups, state_uuid) после дешёвых локальных
гардов; never-raise -> ({}, {}, None) при сбое.
- D2: безусловный терминал-скип ДО Guard 2 (группа Plane completed/
cancelled, fallback на логические ключи done/cancelled, либо стадия в
БД орка ∈ {done, cancelled}); skipped_terminal_total++, не подчинён
reconcile_skip_blocked_enabled.
- D3: _is_blocked_or_needs_input переиспользует резолв D1 (опц. аргументы,
_UNSET -> самостоятельный резолв для прямых/легаси-вызовов; 1:1).
- D4: вызов _note_unblock на F-1 теперь передаёт state_uuid -> dedup
работает на обоих путях (deduped_total++ на повторе).
Анти-регресс: легитимный unblock не-терминальной застрявшей задачи
по-прежнему advance + один Telegram. STAGE_TRANSITIONS / QG_CHECKS /
схема БД / сигнатуры advance_*/_note_unblock / форма status() / новые
флаги — без изменений; never-raise сохранён.
Тесты: tests/test_reconciler.py TC-86-01..09/11,
tests/test_reconciler_plane.py TC-86-10. Полный прогон зелёный (1069).
Refs: ORCH-086
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add "disable_web_page_preview": True to the JSON payload of both
low-level Telegram primitives — send_telegram (POST /sendMessage) and
edit_telegram (POST /editMessageText). Telegram no longer expands the
Plane "Modern project management" link-preview banner under every
tracker card (bump/edit) and notify/alert message, which the default
bump mode (ORCH-067) was duplicating on each transition.
Single-point fix at the primitive level — all consumers
(update_task_tracker, notify_approve_requested, notify_error, stage
alerts from launcher/stage_engine) inherit it without code changes.
parse_mode: HTML is preserved so the ORCH-NNN issue link stays
clickable; disable_notification, bump/edit logic, the one-card-per-task
invariant, return contracts and never-raise are untouched. Unconditional,
no kill-switch (ADR-001).
Tests: tests/test_link_preview_disabled.py (TC-01..06). Docs: CHANGELOG,
CLAUDE.md, docs/architecture/README.md (Notifications component).
Refs: ORCH-080
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the missing invariant "by merge-verify time the branch has an open
code-PR". The pipeline created a PR only on the developer path with a fresh
worktree commit (launcher._ensure_pr), so a branch (e.g. after a manual main
restore) could reach the deploy->done merge-verify under-gate PR-less ->
merge_pr returned "no open PR" -> a FALSE HOLD (ORCH-074 incident).
- merge_gate.ensure_open_pr(repo, branch) -> (status, detail): idempotent
leaf-actor (never-raise). GET open PRs filtered head==branch AND base==main
(identical to merge_pr/ORCH-073 FR-3 — auto docs-PR is not a code-PR) ->
existed; else POST -> created; 409/422 race -> re-GET -> existed (no dup);
any other error -> failed.
- stage_engine._handle_merge_verify: врезка after validated_revision and
BEFORE merge_pr. created|existed -> proceed; failed -> honest HOLD via new
_hold_pr_create_failed (note "pr-create-failed-hold", text distinguishable
from the not-merged HOLD; task stays on deploy, NO rollback).
- launcher._ensure_pr delegated to ensure_open_pr (single PR-creation path,
shared head==branch & base==main filter); the developer-only trigger is
unchanged.
- ORCH-073 protection untouched & authoritative: merge is confirmed ONLY by
verify_merged_to_main (SHA-in-main) + check_main_regression. Real un-merged
code still HOLDs.
- Kill-switch ORCH_MERGE_VERIFY_AUTOCREATE_PR_ENABLED (default true); scope =
merge_verify_applies (self-hosting / merge_verify_repos); non-self -> no-op;
false -> ORCH-074 behaviour 1:1. No DB migration; main never push/force-push.
- Append ORCH-082 marker to MAIN_REGRESSION_MARKERS (append-only convention).
- conftest defaults the autocreate flag OFF (mirrors merge_verify_enabled) so
unrelated deploy->done tests stay 1:1 (no network).
Tests: tests/test_orch082_ensure_pr.py (TC-01..05),
tests/test_orch082_merge_verify_autocreate.py (TC-06..12). Docs: README
merge-verify block (ORCH-082), CHANGELOG, .env.example.
Refs: ORCH-082
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_agent_effort returned '' for all agents in prod because empty
ORCH_AGENT_EFFORT_*= env vars clobber pydantic class-defaults, leaving no
non-empty floor to fall back to -> --effort never reached the Claude CLI.
Add a level-4 per-role floor in resolve_agent_effort (src/agents/launcher.py):
_agent_effort_floor reads the declared class-default of agent_effort_<agent>
(model_fields[...].default), which a present-but-empty env cannot override.
Floor applies only when levels 1-3 are empty and BEFORE validation, so a typo
(non-empty) still drops to '' (never-break ORCH-41) and explicit env/override
still wins (priority preserved). config.py: agent_effort_developer high->xhigh
(single source of truth; floor follows automatically).
Refs: ORCH-081
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
G1: remove the dead `model:` line from all 6 .openclaw/agents/*.md prompts —
launcher never read it; config (agent_model_*) is the single source of truth.
G2: add is_valid_model helper (format check ^claude-…$) applied inside
resolve_agent_model's resolution cascade and at the inline --fallback-model
read in _spawn. An invalid name is logged and skipped to the next valid level
(in the limit: no --model flag), never passed to the CLI, never raises. Format
check chosen over an allowlist for forward-compatibility (ADR-001).
G3 (routing) and G4 (fallback) intentionally NOT enabled — all agents stay on
claude-opus-4-8; agent_fallback_model stays "".
Docs (golden source) updated in the same change: README model/effort table +
validation, CLAUDE.md, .env.example (ORCH_AGENT_MODEL_*/EFFORT_*/FALLBACK_MODEL),
CHANGELOG. Tests: test_agent_frontmatter_no_model.py (G1), extended
test_resolve_agent_model.py (G2 never-break).
Refs: ORCH-074
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Level A — merge/deploy serialization within one repo: reuse the existing
ORCH-043/065 merge-lease (no new mechanism); the only new logic is an
unconditional pre-merge rebase in check_branch_mergeable — under the held
lease, auto_rebase_onto_main is ALWAYS called when premerge_rebase_always
(default True), not just when the branch is behind. No-op on an up-to-date
branch (rebase keeps HEAD, force-with-lease -> "Everything up-to-date", CI
not triggered). Kill-switch off -> ORCH-043 behaviour 1:1.
Level B — declarative task dependencies: additive job_deps table
(CREATE ... IF NOT EXISTS, no live-DB migration); claim_next_job gate
(NOT EXISTS) defers a job whose depends-on tasks are not yet 'done' without
occupying a max_concurrency slot; inert on empty job_deps -> zero regression.
New leaf src/task_deps.py (never-raise): is_task_ready (fail-open), DFS cycle
detection + Blocked/alert, declare/ingest_plane_relations (db source never
hits the network on the hot path), snapshot. Telegram waiting-line, /queue
observability, reconciler skip + cycle backstop, reaper untouched.
Invariants unchanged: STAGE_TRANSITIONS, QG_CHECKS registry (dep gate is a
claim_next_job врезка, not a registered QG), DB schema of existing tables,
HTTP endpoints; non-self repos remain a no-op on empty deps/scope.
Flags: ORCH_PREMERGE_REBASE_ALWAYS, ORCH_TASK_DEPS_ENABLED, ORCH_TASK_DEPS_SOURCE.
Docs: docs/architecture/README.md, CLAUDE.md, .env.example, CHANGELOG.md,
adr-0015. Tests: tests/test_orch026_*.py (64 tests); full suite 991 green.
Refs: ORCH-026
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root-cause fix for main erosion (phantom merge): code of ORCH-067/069 reached
`done` while absent from origin/main (only their auto docs-PRs landed).
- FR-1: verify_merged_to_main confirms merge ONLY by `git merge-base
--is-ancestor <validated_sha> origin/main`; the OR-branch pr_already_merged is
removed (a merged PR no longer confirms). Empty SHA / git error -> False.
- FR-2: pr_already_merged demoted to merge_pr idempotency-guard; counts a PR only
when merged & head.ref==<branch> & base.ref=="main" (explicit in-loop filter).
- FR-3: merge_pr selects the open code-PR by head==<branch> AND base==main.
- FR-5: new deterministic check_main_regression in _handle_merge_verify (after
confirmed SHA-in-main, before done) verifies MAIN_REGRESSION_MARKERS still in
origin/main; deterministic count==0 -> alert "main regressed" + HOLD (NOT done,
no rollback); git error of the grep -> fail-open. Kill-switch
ORCH_REGRESSION_GUARD_ENABLED; non-self -> no-op.
- FR-4: root .gitattributes `CHANGELOG.md merge=union` so Unreleased edits
auto-merge on rebase without conflict (branch not rolled back).
Invariants unchanged (STAGE_TRANSITIONS, QG_CHECKS, deploy-status, merge-gate,
image-freshness, DB schema, external HTTP API); non-self repos no-op (INV-5);
never-raise (INV-1); merge only via Gitea PR-API (INV-2).
Docs: CHANGELOG, .env.example (README/ADR updated by architect). Tests:
tests/test_orch073_*.py (TC-01..18); existing merge-gate tests updated for the
new code-PR filter.
Refs: ORCH-073
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Canonical run inside orchestrator-staging (ORCH-048): exit 0, all REAL
checks green; C9a/C9b waived as known sandbox-infra (ORCH-061).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the hardcoded `len(name) > 80` cap in the QG-0 entry validation
(_qg0_errors) with a configurable Settings.qg0_title_max (env
ORCH_QG0_TITLE_MAX, default 200). The 80-char cap was a hygiene limit, not
structural, so valid 81-200 char titles were rejected without a business
reason. The limit is read dynamically per call and the error text interpolates
the active value.
Graceful degradation (AC-3, self-hosting safety): an empty/non-numeric env
value no longer crashes the process on startup. A field_validator(mode="before")
intercepts the raw env before int-parsing and falls back to 200 (never raises),
suppressing pydantic ValidationError.
Additive and backward-compatible (default 200 > old 80). Invariants unchanged:
STAGE_TRANSITIONS, QG_CHECKS registry, DB schema, slug [:30], lower limits,
soft-QG-0 warning path, API.
Refs: ORCH-069
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Закрывает P0/P1 ревью (attempt 2/3): документация = golden source.
- CHANGELOG.md: запись ORCH-067 в [Unreleased] (bump-дефолт, статус-строка
карточки по модели ORCH-066, кликабельный номер задачи, новые флаги).
- CLAUDE.md: раздел «Нотификации / Telegram live-tracker» (ТЗ §5).
- .env.example: ORCH_TRACKER_MODE=bump (синхрон с новым дефолтом) +
ORCH_TRACKER_LIVE_STATUS / _TTL_S / _TIMEOUT_S.
Refs: ORCH-067
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add CHANGELOG entry for the phantom-merge fix (merge-verify sub-gate,
deterministic merge actor, post-deploy verification, kill-switch).
Addresses P0 blocker from reviewer (attempt 2/3): docs = golden source
per CLAUDE.md §2/§6 and AC-5.
Refs: ORCH-071
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging check suite passed against orchestrator-staging (8501), exit 0.
All REAL pipeline checks green; sandbox-infra C9a/C9b waived per ORCH-061.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reconciler F-2 spammed Telegram "<wi> разблокирована" every ~120s for a
fully-synchronized Done task (incident ET-002, 191+ msgs/night) after the
ORCH-066 Plane status model merge. Two stacked defects (defense in depth):
- D1 (selection): actionable states were told apart by bare UUID, so a Done
issue aliased onto the approved UUID entered the approved branch. Now
terminal states are excluded by Plane state GROUP (completed/cancelled),
a project-independent discriminator robust to UUID aliasing; per-issue
check with a logical-key fallback when the group is unavailable.
get_project_states caches {uuid -> group} from the same /states/ fetch;
new sibling accessor get_project_state_groups.
- D2 (notification): _note_unblock fired unconditionally after _dispatch.
Now it only fires on a confirmed state change (stage before/after _dispatch;
task-appears for the start case) — handlers' contracts untouched.
- TR-3: in-memory dedup guard {issue_id -> last unblocked state} as a backstop.
- TR-4: _STATES_CACHE lived for the whole process lifetime, so a new Plane
status was invisible without a restart. Added TTL ORCH_PLANE_STATES_TTL_S
(default 300s; 0 = previous lifetime cache) reusing reload_project_states();
a failed refresh serves the stale-but-correct set, not enduro defaults.
STAGE_TRANSITIONS / QG_CHECKS / DB schema / handle_* contracts / F-1 / F-3
unchanged; never-raise preserved; self-hosting tick never restarts prod.
Observability: skipped_terminal_total / deduped_total in /queue reconcile block.
Tests: tests/test_reconciler_plane.py (TC-01..TC-10),
tests/test_plane_states_cache.py (TC-11/TC-12).
Refs: ORCH-068
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Приводит статусы доски Plane к смыслу стадий конвейера, сохраняя
инвариант «статус — индикация, а не управление». Меняется только слой B
(отображение: src/plane_sync.py + точки выставления статуса в
stage_engine.py/webhooks/plane.py/reconciler.py); слой A — машина стадий
src/stages.py::STAGE_TRANSITIONS — остаётся байт-в-байт неизменным (AC-21).
- 6 новых логических ключей статуса (to_analyse, analysis, code_review,
awaiting_deploy, deploying, monitoring) + сеттеры и диспетчер
set_issue_stage_state.
- Project-relative alias-fallback (BR-12): новый ключ деградирует на
базовый UUID того же проекта → нулевая регрессия для enduro-trails.
- Самодеплой (ORCH-036) индицирует фазы: Awaiting Deploy / Deploying;
terminal-sync для self-hosting → Monitoring after Deploy, для прочих →
терминальный Done.
- Post-deploy монитор (ORCH-021): HEALTHY → Done, DEGRADED → Blocked
(только индикация; self-hosting ALERT_ONLY, прод не трогается, BR-5).
- Reconciler: триггер старта/резюма на To Analyse; Guard 2 учитывает
новые активные ожидания без расширения skip-set на алиасах.
- never-raise контракт сеттеров и резолвера состояний сохранён.
- Раскатка — созданием статусов в Plane оператором, без kill-switch.
Инварианты не менялись: STAGE_TRANSITIONS, QG_CHECKS (12 чеков),
check_deploy_status, exit-код-контракт хука, merge-gate, схема БД.
ADR: docs/work-items/ORCH-066/06-adr/ADR-001-plane-status-model.md
Тесты: test_plane_status_model, test_plane_to_analyse_resume,
test_plane_status_failclosed + TC в существующих наборах. 774 passed.
Refs: ORCH-066
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging check suite passed (8/10, exit 0): all REAL checks green;
C9a/C9b waived as known sandbox-infra (ORCH-061).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split the overloaded `Approved` Plane status: it served BOTH as the human BRD
gate on `analysis` AND as the silent Phase B prod-deploy trigger on `deploy`
(ORCH-036), so a routine approve could launch a self-hosting prod restart.
ORCH-059 introduces a dedicated logical status `confirm_deploy` ("Confirm
Deploy") that triggers ONLY Phase B on `deploy`; `Approved` stays purely a
pipeline gate.
- plane_sync: map "Confirm Deploy" -> "confirm_deploy" in _PLANE_NAME_TO_KEY;
intentionally absent from _DEFAULT_STATES => fail-closed (no UUID -> .get
yields None, no KeyError, no blind deploy).
- webhooks/plane: handle_issue_updated routes "Confirm Deploy" (fail-closed
.get) to new handle_confirm_deploy (guarded to stage=="deploy") ->
_try_advance_stage(confirm_deploy=True).
- stage_engine: advance_stage gains keyword-only confirm_deploy=False; Phase B
block returns early for deploy+finished_agent is None but only initiates the
deploy when confirm_deploy=True; a plain Approved is a deterministic no-op
(returns before check_deploy_status -> no false БАГ-8 rollback).
- Phase A CTA now asks the operator for "Confirm Deploy", not "Approved".
Contracts unchanged: STAGE_TRANSITIONS, QG_CHECKS, check_deploy_status, hook
exit codes, Phases A/C, merge-gate, DB schema. Conditional like ORCH-35/36
(self-hosting only). Docs updated (CLAUDE.md, architecture/README.md, CHANGELOG).
Refs: ORCH-059
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
TC-17 seeded 17-security-report.md via get_worktree_path() which resolves to
settings.worktrees_dir (default /repos/_wt) -> the test wrote into the real shared
host worktree path. In CI that dir is owned by another user -> PermissionError.
Monkeypatch git_worktree.settings.worktrees_dir to tmp_path/_wt (same pattern as
test_git_worktree.py / test_merge_gate.py). Prod logic untouched.
Add a deterministic (no-LLM) security sub-gate on the deploy-staging -> deploy
edge, run FIRST (before merge-gate ORCH-043 and image-freshness ORCH-058) so it
fails cheaply before any expensive rebase/rebuild, and scans origin/main..HEAD
before rebase so a task is never blamed for a CVE introduced by an updated main.
Why: the autonomous pipeline merged branches into main with no check for a leaked
secret or a vulnerable dependency. For the self-hosting orchestrator (one shared
prod instance serving every project from a shared DB) a single leak/CVE landed in
the prod of all projects (CLAUDE.md self-hosting, section 8).
- New leaf src/security_gate.py (never-raise): gitleaks (offline, fail-closed on
tool error => secrets guarantee is unconditional) + pip-audit (best-effort;
unreachable CVE feed degrades fail-open + loud warning by default, strict via
security_dep_audit_fail_closed). Verdict lives ONLY in 17-security-report.md
YAML frontmatter (write -> read-back single source of truth); FAIL is
authoritative; missing/broken frontmatter => fail-closed.
- check_security_gate thin wrapper registered in QG_CHECKS (lazy import, no cycle).
- _handle_security_gate wired FIRST in advance_stage deploy-staging block: FAIL ->
rollback to development + developer-retry (cap MAX_DEVELOPER_RETRIES); task_desc
carries verbatim findings (ORCH-046 pattern). No merge-lease release (runs before
lease acquire). Self-hosting safe: only reads/scans/writes, never deploys.
- Conditional rollout (security_gate_enabled + security_gate_repos; empty scope ->
self-hosting only). 6 new ORCH_SECURITY_* settings.
- Infra: pinned gitleaks Go binary in Dockerfile (+curl/ca-certificates), pip-audit
in requirements.txt, versioned .gitleaks.toml at repo root.
- STAGE_TRANSITIONS and DB schema unchanged.
Docs: docs/architecture/README.md (marked realized), CLAUDE.md (artifact 17),
CHANGELOG.md. Tests: test_security_gate.py, test_qg_security.py,
test_stage_engine_security_gate.py + updated registry/edge snapshots.
Refs: ORCH-022
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Canonical staging_check.py run inside orchestrator-staging:
8/10 PASS, all REAL checks green, C9a/C9b infra-waived (ORCH-061),
exit 0 → advance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tier-2 reaped a LIVE, still-finalizing monitor: _monitor_agent writes
agent_runs.exit_code FIRST, then does git push / PR / Plane comments before
_finalize_job, and the agent pid is already dead in that window — so the old
"exit_code recorded -> reap now" had no grace and could race a healthy job.
Worse, _reap_known_outcome ran the advance (advance_stage -> enqueue_job)
BEFORE the atomic claim, so a reaper that lost the race had already enqueued
the next stage (dup advance / dup enqueue), violating ADR-001 Р-1.
Fix:
- Tier-2 grace: reap only once agent_runs.exit_code has been recorded for
>= reaper_finalize_grace_s (new setting, default 300s; > max finalization
window). A live finalizing monitor is never reaped (FR-1.3/AC-3). New
finished_age_s column computed in get_running_jobs.
- claim-before-act for exit0: evaluate the canonical QG READ-ONLY (the
reconciler pattern) to choose the terminal status, then atomically claim
'done' FIRST; only the claim winner runs the advance. A loser performs no
side effects -> no dup advance / dup enqueue.
Docs (golden source) updated in the same change: ADR-001, global adr-0011,
README, internals, .env.example, CHANGELOG (also fixes the P3 broken adr-0011
link). New tests cover the grace window, lost-claim no-side-effects, and the
already-advanced idempotent path.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The pr_already_merged guard was defined + unit-tested but consulted by zero
production code, while ADR-001 Р-3 / README / CHANGELOG claimed the merge path
consults it before a repeat merge (reviewer P1, ORCH-065 attempt 2/3). The
actual merge actor is the LLM deployer agent (it merges the feature PR at the
start of the `deploy` stage), so on a reaper re-drive of an already-merged PR
the deployer would blindly re-merge → Gitea error → false БАГ-8 rollback; AC-11
("no second merge") was not met deterministically.
Wire the guard at the real consultation point — the deployer prompt — so it
runs merge_gate.pr_already_merged before any (re-)merge and no-ops when the PR
is already merged. check_branch_mergeable is left untouched (AC-13: check_*
behaviour unchanged; it runs on the first deploy-staging→deploy edge, not on a
deploy-stage re-drive where the second-merge risk lives).
- .openclaw/agents/deployer.md: idempotent pre-merge guard step + general rule.
- src/merge_gate.py: docstring names the deployer-prompt consultation point.
- docs/architecture/README.md, CHANGELOG.md: state the consultation point so
golden-source matches implementation.
- tests/test_merge_gate.py: regression test asserting the deployer prompt wires
the guard (so it can't silently become dead code again).
pytest tests/ -q: 743 passed.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the "zombie jobs" incident class: job status was set only inside
the live launcher process, so a process death left jobs.status='running'
forever; at max_concurrency=1 one zombie blocked ALL projects' queue
(self-hosting risk). Adds a background daemon (src/job_reaper.py) with
three-tier liveness (dead-pid streak / known exit_code / max-running
backstop) whose only mutating write is an atomic terminal flip guarded by
WHERE status='running' (no double-process). For exit0 the canonical QG is
the source of truth via gate-driven advance, not "exit0".
Also proactively reclaims stale merge-lease (dead pid OR TTL) via file
delete only (no git ops), and makes merge finalization idempotent
(pr_already_merged guard + up-to-date short-circuit on re-drive).
New jobs.pid column via idempotent _ensure_column (no migration); pid
stamped in launcher._spawn after Popen. Reaper start/stop in lifespan;
"reaper" snapshot in GET /queue. Kill-switches: ORCH_REAPER_ENABLED,
ORCH_REAPER_INTERVAL_S, ORCH_REAPER_DEAD_TICKS, ORCH_REAPER_MAX_RUNNING_S,
ORCH_LEASE_RECLAIM_ENABLED.
Invariants unchanged (AC-13): STAGE_TRANSITIONS, QG_CHECKS registry,
check_branch_mergeable signature/behaviour, BUG-8 rollback, hook exit
codes. restart-safe, never-raise per unit of background work.
Docs: docs/architecture/README.md, CHANGELOG.md, .env.example.
Tests: tests/test_job_reaper.py, tests/test_merge_lease_reclaim.py,
tests/test_merge_gate.py (TC-16), tests/test_merge_gate_race.py (TC-17),
tests/test_queue.py, tests/test_config.py (TC-19/TC-20). 742 passed.
Refs: ORCH-065
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The ORCH-058 staging rebuild (check_staging_image_fresh) builds the image with
the task git-worktree as the docker build context. A fresh worktree holds only
tracked files, but the Dockerfile did `COPY data/ ./data/` — and `data/` (the
SQLite dir) is gitignored, so it is absent from that context: `docker build`
failed with exit 1 ("BUILD-STAGING: docker build failed - aborting"), bouncing
the task off deploy-staging back to development in a loop.
The COPY was dead weight regardless: `data/` is always supplied at runtime as a
bind-mount volume (./data:/app/data, see docker-compose.yml) which shadows
anything baked into the image. Replace it with `RUN mkdir -p /app/data` so the
mountpoint exists without depending on the build context.
Regression guard: test_tc08b_dockerfile_does_not_copy_gitignored_data_dir
forbids COPY of any gitignored path (the worktree-context invariant).
Refs: ORCH-021
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extend pipeline responsibility past deploy->done: after the terminal
transition for an applicable repo, arm a ~15min observation window that
probes prod and reacts to a degradation the restart-time health-check
missed ("green deploy, red prod").
- src/post_deploy.py: new leaf module (config + lazy qg/db only).
Sentinel-file restart-safe state (.post-deploy-state-<repo>/<wi>/),
no DB migration. probe_signals/classify/decide_action/run_rollback,
all never-raise.
- Reserved-agent job `post-deploy-monitor` (no-LLM, Variant B, calque of
deploy-finalizer): self-requeues each tick via enqueue_job.
- Deterministic classify: DEGRADED iff >= fail_threshold consecutive
health failures OR window 5xx ratio > 5xx_threshold; fail-safe HEALTHY.
- Self-hosting invariant (BR-5/AC-8): a tick NEVER restarts the prod
orchestrator container -> orchestrator is ALWAYS ALERT_ONLY.
- Conditionality (ORCH-35/36/43/58): kill-switch + CSV repos, empty ->
self-hosting only.
- QG_CHECKS / STAGE_TRANSITIONS / schema unchanged (AC-12).
- Docs: CHANGELOG, CLAUDE artefact list (16-post-deploy-log.md),
architecture README, .env.example (ORCH_POST_DEPLOY_*).
Refs: ORCH-021
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-ran staging_check inside orchestrator-staging (exit 0); all REAL checks
green, C9a/C9b waived per ORCH-061.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Validated ORCH-061 infra-tolerance against live staging (8501): all REAL
checks pass, only sandbox-infra C9a/C9b fail and are waived → exit 0.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The self-hosting orchestrator looped on deploy-staging -> development because
scripts/staging_check.py exited 1 on ANY failed check, so two infra-only checks
(C9a sandbox branch / C9b analyst-job — caused by SANDBOX bot accounts not being
members of the sandbox Plane project, NOT a pipeline regress) forced
staging_status: FAILED -> rollback -> loop, burning developer retries and tokens.
Direction (б) per ADR-001: classify staging checks as REAL (all pipeline checks,
fail-closed) vs SANDBOX_INFRA (narrow allowlist {C9a, C9b}, waivable). New leaf
module src/staging_verdict.py (stdlib-only, never-raise): classify_check +
compute_staging_verdict fold per-check results into a tolerant-but-fail-closed
verdict — any REAL failure -> FAILED/exit1 (safety net holds under any flag);
only C9a/C9b failed & tolerant -> SUCCESS/exit0 with waived list; only infra &
strict -> FAILED/exit1; any internal error -> FAILED/exit1 (never a false green).
staging_check.py now auto-classifies each check (public 3-tuple _items shape kept
as an ORCH-048 b6 regression guard), exposes categorized_items(), prints
INFRA-WAIVED/VERDICT lines, and exits via the verdict; new --strict flag forces
legacy strictness per-run. Kill-switch ORCH_STAGING_INFRA_TOLERANCE_ENABLED
(default true) restores legacy strict mode globally. launcher gains
action_stage_no_changes_note so "no changes to commit" on action stages is logged
as expected, not treated as under-delivery.
Contracts unchanged: STAGE_TRANSITIONS, QG_CHECKS registry, staging_status:/
deploy_status: frontmatter, hook exit-code (0/1/2), check_staging_status; no DB
migration. Docs: README, STAGING_CHECK.md, deployer.md, .env.example, CHANGELOG.
Refs: ORCH-061
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Canonical staging_check run inside orchestrator-staging container
(ORCH-048). Exit code 1: branch never appeared in sandbox (C9a) and
analyst job never enqueued (C9b). staging_status: FAILED → rollback
to development per ORCH-35.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reconciler F-1 could not tell "stuck by a lost webhook" from "escalated:
max developer retries reached, waiting for a human". With CI green and a
reviewer that kept sending REQUEST_CHANGES up to the cap, every tick
re-unblocked development -> review -> rollback -> re-unblock (incident
ET-013, infinite bounce: wasted agent runs, Telegram spam, parasitic load
on the shared self-hosting instance).
Add two pre-gate guards in Reconciler._reconcile_gate_task (after the
existing analysis/no-gate/active-job/grace guards, before the gate
pre-evaluation), each an early silent return (no advance, no unblocked_total
increment, no notifications):
- Guard 1 (escalated, deterministic, no network, checked first):
developer_retry_count(task_id) >= MAX_DEVELOPER_RETRIES. Promote
stage_engine._developer_retry_count to public developer_retry_count
(single source of truth; private alias kept). Limit from the constant,
not a literal 3.
- Guard 2 (explicit human Plane gate, Variant A, no DB migration): new
never-raise plane_sync.fetch_issue_state + Reconciler._is_blocked_or_needs_input;
any error/None/unresolved project -> conservative skip. New sub-flag
ORCH_RECONCILE_SKIP_BLOCKED_ENABLED mutes only the networked Guard 2.
F-2 unchanged: Blocked/Needs Input are outside {in_progress, approved,
rejected} so they are never replayed (regression test added). DB schema,
STAGE_TRANSITIONS, QG_CHECKS, never-raise, analysis carve-out and
kill-switches untouched.
Refs: ORCH-060
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging check exit code 1 (C9a/C9b). Live inspection inside orchestrator-staging
proves the production webhook handler is correct: get_project_states(SANDBOX).in_progress
= 84a76f65..., but scripts/staging_check.py hardcodes the enduro fallback b873d9eb...
=> handler correctly classifies the webhook as "no pipeline action". Fix belongs in
scripts/staging_check.py (resolve SANDBOX in_progress dynamically), NOT in handle_status_start
or any ORCH-058 image-freshness code. Image under test = ORCH-058 merge commit 094b5e2f.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
E2E pipeline not triggered on staging webhook ("no pipeline action" on
state b873d9eb...); reproduces prior FAILED. Rolls task back to development.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Strategy-A freshness re-validation rebuilt 8501 from merged commit 094b5e2 and
re-ran staging_check; E2E C9a/C9b fail (Plane "In Progress"/started webhook ->
"no pipeline action", no task/branch/analyst-job). Machine verdict FAILED ->
rollback to development. Prod (8500) untouched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per ADR-001 step 3 / AC-4: after the freshly rebuilt staging container is
healthy, run staging_check.py --mode stub against the fresh 8501 stand BEFORE
reporting success, so the EXACT artefact BUILD-ONCE retagged to prod is the one
validated on staging. Fail-closed: staging_check rc!=0 -> exit 1 (not promoted).
- Invoked inside the container (docker exec $TARGET_SERVICE) per the canonical
signature in scripts/staging_check.py header, --base-url http://localhost:$TARGET_PORT.
- Targets ONLY 8501 (staging), never 8500 (prod) - AC-9.
- --mode stub: fast, deterministic, no LLM spend (ADR).
- Static regression test test_tc07_build_staging_runs_staging_check_stub_after_health:
asserts staging_check.py + --mode stub present, runs after health, before exit 0,
fail-closed, and never hard-codes prod 8500.
Closes reviewer P0/P1 (ORCH-058 attempt 3): the committed --build-staging hook
recomputed GIT_SHA=$(git rev-parse HEAD) in $REPO (prod clone on `main`) and built
`docker build ... "$REPO"`, ignoring the caller-supplied BUILD_CONTEXT/GIT_SHA. On
the deploy-staging -> deploy edge the PR is not yet merged, so `main` HEAD != the
validated SHA -> the staging image got the wrong revision label and Strategy-B's
guard fail-closed on EVERY valid self-deploy (AC-6 deadlock). It also only did
`docker build` + exit 0 — never recreating 8501 nor health-checking — so
rebuild_staging_image's rc=0 ("rebuilt and healthy") was a lie (AC-4 unmet).
- Hook --build-staging now honours caller BUILD_CONTEXT (validated worktree) and
GIT_SHA, recreates orchestrator-staging on the fresh image and runs the 10x6s
health-check; build/health failure -> exit 1 (FAILED contract preserved).
- image_freshness.rebuild_staging_image: document why COMPOSE_PROFILE/TARGET_SERVICE/
TARGET_PORT are intentionally omitted (hook STAGING defaults -> 8501 only, P2).
- tests: assert the caller<->hook contract (builds from $BUILD_CONTEXT, no
`git rev-parse HEAD` recompute, recreates + health-checks 8501) so the P0
regression can't pass green again (P1).
Refs: ORCH-058
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Close AC-11 documentation gap left by the prior developer run: the
ORCH-058 feature (staging-image provenance before BUILD-ONCE retag) was
implemented and green but never recorded in the golden-source docs.
- CHANGELOG.md: add the ORCH-058 [Unreleased]/Added entry (layers A+B,
validated_revision anchor, check_staging_image_fresh, EXPECTED_REVISION
hook guard, new ORCH_IMAGE_FRESHNESS_* flags, ADR/test refs).
- .env.example (canon): document ORCH_IMAGE_FRESHNESS_ENABLED /
ORCH_IMAGE_FRESHNESS_REPOS, mirroring the ORCH-036/043/053 precedent.
- docs/architecture/README.md: footer note design -> реализовано, aligning
it with the already-updated section.
Refs: ORCH-058
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-deploy after a FAILED prod deploy wedged the task on `deploy`: the
sentinel markers (approve-requested/initiated/result) are keyed by the
stable work_item_id, so after the БАГ-8 rollback (deploy -> development)
and a developer fix, Phase B's idempotency-guard saw a STALE `initiated`
and became a no-op — the detached hook never re-launched and the
finalizer was never enqueued. Add self_deploy.clear_state (never-raise,
idempotent) and call it on the check_deploy_status FAILED rollback and at
the start of Phase A, so every fresh prod-deploy pass starts clean.
Also document the new ORCH_SELF_DEPLOY_* / ORCH_DEPLOY_* descriptors in
the canonical .env.example (CLAUDE.md rule #8, ТЗ §2.6), modelled on the
ORCH-043 merge-gate block (placeholders only, secrets not committed).
Contracts untouched: STAGE_TRANSITIONS, QG_CHECKS, _parse_deploy_status,
БАГ-8, merge-gate.
Refs: ORCH-036
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add the optional, backward-compatible SOURCE_IMAGE branch to
orchestrator-deploy-hook.sh: when set, retag the staging-validated image
onto TARGET_IMAGE (docker tag) before `up -d --no-build` instead of
rebuilding — guarantees prod runs the exact artefact that passed staging
(AC-7 / TC-14). Unset -> prior behaviour; exit-code contract (0/1/2) and
health-loop untouched.
Update golden-source docs (AC-13): rewrite deployer.md `deploy` stage from
"paper SUCCESS" to the executable self-deploy (Phase A/B/C, no self-restart
from inside the container) and add the ORCH-036 CHANGELOG entry.
Refs: ORCH-036
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-run of deploy-staging gate (merge-gate defer cycle). Canonical
staging_check.py (mode=stub) ran inside orchestrator-staging (8501);
all 10 checks passed (exit 0). No prod (8500) container touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Конвейер продвигается только входящими webhook; потерянное событие (502 на
ребилде, отсутствие ретраев у Plane/Gitea, неразрезолвленный sha→branch)
оставляет задачу молча застрявшей (класс инцидента ORCH-044). Новый фоновый
daemon-поток src/reconciler.py (паттерн queue_worker) доигрывает пропущенный
переход через те же штатные гейты/обработчики, что и webhook:
- F-1 gate-side: для задач stage≠done, без активного job и age(updated_at) ≥
grace_for_stage(stage) — read-only пред-оценка канонического QG; зелёный →
stage_engine.advance_stage(..., finished_agent=None); красный → тишина (спам
нотификаций структурно невозможен). analysis F-1 не трогает (человеческий гейт).
- F-2 plane-side: опрос Plane API per-project (plane_sync.list_issues_by_state,
курсорная пагинация, never-raise) → реплей In Progress/Approved/Rejected через
существующие handle_status_start/handle_verdict (async из sync-потока, asyncio.run).
- F-3: усиление sha→branch в handle_ci_status — БД-fallback по единственной
development-задаче repo (неоднозначность → не резолвим), debug→info.
- Анти-дубль на создании (db.create_task_atomic под process-wide Lock): гонка
reconcile↔webhook не плодит второй task/branch/worktree/analyst-job (AC-4).
- F-4 observability: лог-строка разблокировки + Telegram + блок reconcile в /queue.
Старт/стоп в main.lifespan (после worker.start() / перед worker.stop()),
restart-safe, never-raise на единицу работы. Kill-switches ORCH_RECONCILE_ENABLED
/ ORCH_RECONCILE_PLANE_ENABLED + grace-настройки. Схема БД и реестры
STAGE_TRANSITIONS/QG_CHECKS не менялись.
Тесты: test_reconciler.py, test_reconciler_plane.py, test_gitea_sha_resolve.py,
test_config.py (33 новых, 563 всего зелёные). Документация обновлена (golden source):
architecture/README.md, INFRA.md, README.md, CHANGELOG.md, adr-0007 → accepted.
Refs: ORCH-053
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Live staging-stand suite (scripts/staging_check.py, stub mode) ran inside
orchestrator-staging: 10/10 checks PASS, exit code 0. Merge-gate edge
(deploy-staging → deploy) cleared for ORCH-043.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Deterministic (no-LLM) sub-gate on the deploy-staging -> deploy edge that
catches a feature branch up to the CURRENT origin/main, re-tests the combined
tree, and serialises merges with a per-repo file lease — so two green parallel
branches can no longer break main (self-hosting safety for the orchestrator repo).
- src/merge_gate.py: branch_is_behind_main, auto_rebase_onto_main (push
--force-with-lease ONLY the task branch, NEVER main), retest_branch, and a
file merge-lease (atomic O_CREAT|O_EXCL, holder-aware release, stale reclaim).
Strict never-raise contract; all git ops in the per-branch worktree.
- src/qg/checks.py: check_branch_mergeable composes the primitives under the
lease; registered in QG_CHECKS. Conditional rollout (merge_gate_enabled /
merge_gate_repos, default self-hosting only).
- src/stage_engine.py: sub-gate hook on deploy-staging (not a new stage). PASS ->
advance; "merge-lock busy" -> DEFER (re-queue with available_at, anti-deadlock
at max_concurrency=1, capped); conflict/red re-test -> rollback to development
+ developer retry (capped by MAX_DEVELOPER_RETRIES). Lease released on
deploy->done / rollback / PR-merged webhook.
- src/db.py: enqueue_job(available_at_delay_s=...) for the defer (no schema change).
- src/webhooks/gitea.py: holder-aware lease release on PR-merged.
- src/config.py + .env.example: ORCH_MERGE_* settings.
Docs: README + adr-0006 (architect) already cover the design; CHANGELOG updated.
Tests: test_merge_gate.py, test_qg_merge_gate.py, test_merge_gate_race.py,
test_stage_engine.py::TestMergeGate, test_config.py, QG-registry snapshot.
Full suite: 535 passed.
Refs: ORCH-043
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging check suite passed 10/10 (exit 0), run canonically inside
orchestrator-staging via the Docker Engine API (docker exec equivalent).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both compose services (orchestrator, orchestrator-staging) now declare
user: "1000:1000" so pipeline artifacts (git worktree, docs/work-items
commits) are created as slin:slin on the host — git pull/reset under slin
no longer fail with permission errors. docker.sock access preserved via
group_add: ["999"]. SSH mount target aligned with the launcher-forced
HOME=/home/slin (/root/.ssh -> /home/slin/.ssh). launcher.py and Dockerfile
unchanged. INFRA.md and CHANGELOG.md updated; host-prerequisites (P-1..P-4)
documented.
Refs: ORCH-040
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ORCH-042: new ORCH_TRACKER_MODE (Settings.tracker_mode, default edit) selects
the live-tracker card behaviour. bump mode re-creates the card at the bottom of
the chat on every update (delete_telegram + send silently + repoint message_id),
keeping the "one card per task" invariant: <=1 new message per call, repoint
only on successful send, delete result never gates the send. New never-raising
delete_telegram helper. Anything != "bump" resolves to edit (zero regression).
Also russify/cosmetic-fix the card text (both modes): "Подтверждение BRD" label,
✅ after approve-gate, Russian stage labels, "📦 Внедрено". Docs updated in the
same PR (CHANGELOG, internals.md, .env.example).
Refs: ORCH-042
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Artifact-only production deploy verdict for ORCH-044. All gates green
(review APPROVED, tests PASS, staging SUCCESS 10/10). src/ runtime
changed → real rebuild+restart of prod orchestrator (8500) delegated to
Owner-run deploy hook (ORCH-36); prod container not touched by agent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging suite run inside orchestrator-staging via docker exec (canonical,
ADR-001). All 10/10 checks pass, exit 0. B6 now reads registry from the
running staging instance's own process-env -> sandbox present, prod ET/ORCH
absent, no false FAIL / spurious rollback.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
B6 false-FAILed because it built the project registry from the
launcher process-env via a host-path hack (sys.path.insert +
importlib.reload), not from the running staging instance. Run from the
host, ORCH_PROJECTS_JSON is unset -> default ET+ORCH registry -> false
FAIL -> spurious deploy-staging -> development rollback.
Variant (v) per ADR-001: remove the host-path hack; canonically run the
suite INSIDE orchestrator-staging via docker exec so src.projects
resolves from /app (PYTHONPATH) with .env.staging. Verdict logic
extracted into pure _evaluate_b6(known) -> (passed, detail) +
_known_project_ids_from_registry() / _run_b6() with deterministic FAIL on
source unavailability. deployer.md and STAGING_CHECK.md updated to the
docker exec command. src/projects.py, .env* and checks A/B4/B5/C
untouched.
Refs: ORCH-048
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging instance (8501) still runs a pre-ORCH-048 image without GET /projects,
so B6 deterministically FAILs (endpoint unavailable → no false PASS). Branch
code is correct; remediation is a host-side `--profile staging up -d --build`
of orchestrator-staging before re-running the gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Staging suite ran end-to-end against staging (8501, stub mode): 9/10 PASS,
exit 1. Failure is B6 — staging project registry not isolated (sees prod
ET/ORCH, sandbox absent), violating the INFRA isolation invariant. Gate is
authoritative and red → staging_status: FAILED (rollback to development).
Note: this is a staging .env/ORCH_PROJECTS_JSON misconfig, not an ORCH-046
code regression (same B6 as ORCH-047).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
При заворотах на development task_desc теперь несёт дословный must-fix текст
(P0/P1 ревьюера, причина FAIL тестера) вместо одной ссылки на файл — developer-
агент видит суть претензий сразу и не повторяет ту же ошибку, экономя retry-
бюджет и токены общего инстанса.
- Новый defensive-модуль src/review_parse.py (never-raise): extract_review_findings
(P0/P1 из 12-review.md ## Findings), extract_test_failures (фрагмент тела
13-test-report.md: pytest output / FAIL-строки / Итог), усечение по лимиту.
- Две rollback-ветки stage_engine: встраивают текст + сохраняют ссылку на полный
файл; graceful-фоллбэк на ссылку-строку при битом/пустом артефакте.
- Последовательность отката, retry-счётчик, поля AdvanceResult, реестр QG_CHECKS
не менялись.
- Доки: README (Stage Engine / Откаты), CHANGELOG.
- Тесты: tests/test_review_parse.py, test_stage_engine.py::TestRollbackTaskDescEmbedding.
Refs: ORCH-046
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_parse_tests_verdict now accepts three equal-rank machine-readable
frontmatter fields in 13-test-report.md — result: (canonical tester
output), verdict: and status: (legacy/enduro-trails). Any one non-empty
field suffices; a negative token in any field stays authoritative.
Fixes the producer/consumer contract mismatch where the tester emits
`result: PASS` (per .openclaw/agents/tester.md) but the gate only read
verdict:/status:, causing a testing->development rollback loop until
MAX_DEVELOPER_RETRIES (observed on ORCH-17). Token sets frozen and gate
signature/QG_CHECKS unchanged for full backward compatibility.
Refs: ORCH-047
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
check_tests_passed/_parse_tests_verdict gated the testing -> deploy-staging
transition on `verdict:`/`status:` in 13-test-report.md, but the tester agent
prompt (.openclaw/agents/tester*) documents `result: PASS | FAIL` as THE
machine-readable field. A report that followed the contract literally
(ORCH-017: only `result: PASS`, no verdict:/status:) was bounced back to
development with a misleading "Tests FAILED". ORCH-016 only passed because its
report redundantly carried both `verdict:` and `result:`.
Treat `result:` as a first-class machine field alongside verdict/status; a
negative token in any field stays authoritative (ET-013 contract preserved).
Self-hosting QG fix: unblocks every project whose tester emits only `result:`.
Docs updated in-PR: CHANGELOG, architecture README machine-keys note.
Tests: test_qg.py::TestCheckTestsPassed::test_result_pass_only_passes / _fail_only_fails.
Refs: ORCH-017
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
notify_approve_requested now embeds two HTML <a> links into the single
notifying approve-gate message: a Gitea branch-view link to 01-brd.md and a
Plane issue browser link. Adds ORCH_PLANE_WEB_URL (external Plane web URL,
fallback to plane_api_url) with a loopback-guard that omits the Plane link
when the resolved base is localhost/empty (no broken localhost URLs in prod).
Each link is built independently and omitted on missing data; the message and
the "flip to Approved" call to action are always sent as exactly one ping. The
shared send_telegram helper is left untouched (min blast radius for the
self-hosting prod container). Dynamic labels are html.escaped; parse_mode=HTML
preserved. QG registry / stages / approve handler unchanged.
Docs updated in-PR: CHANGELOG, .env.example, INFRA env map.
Tests: test_notify_approve_links.py, test_analysis_approve_flow_links.py.
Refs: ORCH-017
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the deploy-log pattern: deployer writes 15-staging-log.md on the
feature branch, then merges the artifact into origin/main so the
check_staging_status quality gate can read it via _staging_log_from_main()
(see src/qg/checks.py:489).
Verdict from the staging run on http://localhost:8501 (mode=stub):
staging_status: SUCCESS (10/10 checks PASS)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ORCH-10 root cause: PLANE_STATES was a global dict hardcoding enduro-trails
UUIDs. The webhook comparison only
matched ET UUID (b873d9eb) and silently ignored the ORCH in_progress UUID
(e331bfb3), blocking pipeline start for all orchestrator-project tasks.
Changes:
- src/plane_sync.py:
* Rename PLANE_STATES -> _DEFAULT_STATES (enduro UUIDs kept as safe fallback).
* PLANE_STATES preserved as alias to _DEFAULT_STATES (backward compat).
* Add get_project_states(project_id) -> {logical_key: state_uuid}:
fetches Plane API GET /projects/<id>/states/, maps by state name,
caches per project_id, falls back to _DEFAULT_STATES on API failure.
* Add _STATES_CACHE: dict, reload_project_states(project_id=None).
* Add _PLANE_NAME_TO_KEY mapping and _STAGE_TO_STATE_KEY for clean lookup.
* Add stage_to_state(stage, project_id) using get_project_states().
* update_issue_state() uses stage_to_state() instead of STAGE_TO_STATE dict.
* set_issue_{needs_input,in_review,blocked,done,in_progress,stage_state}()
all resolve state UUID via get_project_states(project_id) instead of
the global PLANE_STATES dict.
- src/webhooks/plane.py:
* handle_issue_updated: import get_project_states, resolve proj_states per
incoming project_id, compare new_state against proj_states["in_progress"],
proj_states["approved"], proj_states["rejected"].
* start_pipeline QG-0 blocked path: use get_project_states(plane_project_id)
instead of PLANE_STATES["blocked"].
- tests/test_orch10_states.py: 23 new tests covering:
* get_project_states returns correct UUIDs for both ET and ORCH projects.
* API failure / empty response / None project_id -> _DEFAULT_STATES fallback.
* Caching and reload_project_states (per-project and full flush).
* stage_to_state() per-project resolution.
* Webhook in_progress triggers pipeline for BOTH b873d9eb (ET) and e331bfb3 (ORCH).
* Webhook approved/rejected routes correctly per project.
* PLANE_STATES alias and _DEFAULT_STATES backward compat.
Stage 1/5 of staging environment for self-hosting (ORCH-7).
Adds orchestrator-staging compose service under staging profile, isolated DB, .env.staging.example, docs. Prod untouched; service inert until explicitly started.
- Add orchestrator-staging compose service under profile 'staging'
so normal 'docker compose up -d' does NOT start it.
- Port 8501 via command override; network_mode: host (no ports mapping needed).
- DB isolation via separate volume ./data/staging:/app/data — physically
separate from prod ./data/orchestrator.db on the host.
- ORCH_DB_PATH=/app/data/orchestrator.db explicit in env (same container
path, isolated by volume mount).
- Add .env.staging.example with all required keys and placeholders.
- Update .gitignore: add .env.staging and data/staging/ exclusions.
- Add docs/STAGING.md: how to start staging, architecture table, roadmap.
Refs: ORCH-31 (Stage 1 of 5)
Adds autouse fixture _reset_webhook_secrets to tests/conftest.py that
resets the process-wide Pydantic settings singleton before every test:
1. gitea_webhook_secret / plane_webhook_secret → "" (HMAC disabled by
default). Tests that deliberately test the 401 path
(test_webhook_dedup.py:268,278) override this with their own monkeypatch
which runs after autouse fixtures and wins for that test only.
2. db_path → os.environ["ORCH_DB_PATH"] (last written value after all test
modules are imported). Without this, test_webhook_dedup.py (imported
first alphabetically) seeds settings.db_path = dedup.db, while
test_webhooks.py setup_db tries to remove test_orchestrator.db — leaving
the DB dirty between tests that share a branch name and causing
get_task_by_repo_branch() to return a stale row with the wrong stage.
Per-test monkeypatches in test_webhook_dedup.setup_db still override it.
Root cause: both leaks come from the same singleton settings being read once
at import, before any per-test isolation runs. The autouse fixture is the
correct per-test reset point for process-wide singletons.
Result: pytest tests/ → 294 passed, 0 failed (was 10 failed/284 passed).
check_tests_passed did "if PASS in content" over the whole 13-test-report.md
body, so a report explicitly marked verdict: BLOCKED / status: blocked whose
prose mentioned "23 passed" / "PASS" / "All checks passed" passed the gate.
On ET-013 an unfinished feature (P1 AC-19 failed) reached Done.
Now mirrors check_reviewer_verdict (S-5) and check_deploy_status: read ONLY the
YAML frontmatter verdict:/status: fields. Positive tokens (PASS/PASSED/
READY-TO-DEPLOY/GREEN/APPROVED) -> True; negative tokens (BLOCKED/FAILED/...) are
authoritative -> False; missing/empty/no-frontmatter/bad-YAML -> False with reason;
file missing -> not found. Never raises.
Positive token set derived from REAL enduro-trails reports ET-001..ET-014
(inconsistent: PASS, ready-to-deploy+status:PASSED, stage:ready-to-deploy+status:pass,
PASS — ready-to-deploy). Validated: all 9 prior passing WIs stay True, ET-013 -> False.
ET-013: deployer writes 14-deploy-log.md and merges deploy artifacts into
main via a separate PR, so the log lands in origin/main, not the feature
branch worktree that check_deploy_status reads via _repo_path(repo, branch).
Result: every successful deploy was falsely failed (Deploy log not found)
and rolled back deploy->development.
Fix: when the log is absent in the worktree, fall back to reading it from
origin/main on the shared clone (git fetch origin main + git show
origin/main:docs/work-items/<WI>/14-deploy-log.md). Lookup order:
worktree -> origin/main -> not found. Fetch/show failures degrade to
not found (never raise). Does not touch the merge-gate in gitea.py.
Tests: origin/main SUCCESS->PASS (ET-013 case), origin/main FAILED->FAILED,
absent everywhere->not found, fetch failure->degrades no exception,
worktree log short-circuits main lookup.
edit_telegram now returns a distinguishable outcome (ok|not_modified|gone|
failed) instead of a bare bool. update_task_tracker only sends a NEW message
when the original is truly gone; not_modified and transient failures no longer
spawn duplicate trackers or orphan the live one.
render_task_tracker shows "попытка N" on an actively re-run stage (>=2 agent
runs) so the text changes between review<->development cycles. Finished (✅)
lines are unchanged.
Tests: edit_telegram classification (ok/not_modified/gone/failed via mocked
httpx), update_task_tracker (not_modified/failed -> no send, gone -> send+id),
render attempt marker.
Replace the ~15 separate Telegram messages per task (agent start/finish, stage
transition, QG-pending, tech noise) with ONE live tracker message edited in
place (editMessageText) on every stage transition. Only attention-worthy events
are still sent as SEPARATE, notifying messages: approve-gate, deploy-fail,
agent-fail, task error.
- db.py: idempotent ALTERs — tasks.tracker_message_id, tasks.title,
tasks.brd_review_started_at/ended_at, agent_runs.model. Helpers for
tracker message_id + BRD-review clock.
- usage.py: short_model_name() (strip provider/claude- prefix); parse model
from result-JSON modelUsage; record_usage persists model.
- notifications.py: render_task_tracker(task_id) (stateless render from
agent_runs), update_task_tracker (sendMessage->store id->editMessageText with
fallback to a new message, silent), edit_telegram(). Per-stage line
in↓/out↑·cost·model, ⏸️ Ревью БРД (human time), 💰 totals, finish block
(⏱️ wall/agents/yours, 🔗 PR · 📦). notify_* are now tracker-only/log-only
except the four alerts.
- stage_engine.py: stamp brd_review_ended on analysis->architecture advance.
- webhooks/plane.py: persist task title on creation.
- tests/test_telegram_tracker.py: render, short_model_name, send/edit/fallback,
separate-vs-silent alert behavior.
1. BUG 8 (second door): merge webhook no longer fake-completes a task at the
deploy stage; done is gated by the deployer verdict (check_deploy_status).
Other stages keep merge->done.
2. Token accounting: parse+persist cache_creation_input_tokens (new
idempotent agent_runs column). usage_comment / task_summary now show the
FULL input (input + cache_read + cache_creation) with a cached breakdown.
cost_usd untouched.
3. deploy->done success now forces the Plane issue to terminal Done state.
4. All agents (architect/developer/reviewer/tester/deployer) attach artifact
links to their finish comment via gitea_public_url.
Tests added for each fix; pytest 244 passed / 9 failed (off-limits HMAC group).
The analyst ready-comment used the obsolete :approved: wording (comment-based approve was removed in PR #12). Rewrite it for the status-only model: ask the stakeholder to move the issue to Approved (reject = reason comment + Rejected), and add clickable Gitea links to the analyst docs that actually exist in the worktree.
issue.updated ships only the changed fields, so name was absent and the branch slug became feature/<id>-untitled. Add fetch_issue_fields (single issue-detail GET returning name+description, reusing the endpoint/token of fetch_issue_description) and pull the name above the slug build. Empty name still falls back to untitled.
start_pipeline built the analyst .task.md with only the Title, so the analyst received a ~101-byte file and reported the business request as empty even though the description was already fetched. Append the resolved description to task_desc.
handle_verdict(rejected): the reason is now pulled from the issue latest Plane
comment (_latest_comment_reason: GET comments, newest by created_at, HTML
stripped) instead of a fixed stub. Slava writes the reason in a comment before
flipping the status to Rejected. Falls back to a fixed note when there is no
comment / the API call fails.
tests: add test_status_only_verdict.py (test_inreview_comment_does_not_revert
[bug 3 root], test_any_comment_no_pipeline_action,
test_approved_status_advances_without_inprogress_reset,
test_rejected_status_pulls_reason_from_comment) and
test_inprogress_from_needs_input_relaunches_analyst in test_status_trigger.py.
Rewrote the comment-based tests (test_verdict_status, test_plane_approved/
rejected in test_webhooks) under the status-only model: comments are no-ops,
verdicts come from status changes.
handle_verdict(approved): removed set_issue_in_progress(work_item_id) before
_try_advance_stage. _try_advance_stage -> advance_stage -> plane_notify_stage
already PATCHes the issue to the NEXT stage status, so the reset only made the
board flicker In Progress before the next stage (part of bug 3).
Status-only verdict model: comments NEVER drive the pipeline. Removed the
whole comment-based control mechanism from handle_comment (:approved: /
:rejected: / answer-to-questions) which caused bug 3 (echo self-hit): the
analyst posts its own "waiting for approval" comment, handle_comment catches
its own comment and reverts In Review -> In Progress. handle_comment is now a
pure logger with no side effects.
handle_status_start: a return to In Progress on an EXISTING task (Slava
answered the analyst questions in Needs Input) now RELAUNCHES the stage agent
instead of being a no-op. Distinguished from a duplicate In Progress webhook
via has_active_job_for_task() (new db helper): no active job => agent idle =>
relaunch; active job => busy => skip (no double launch).
ET-006 was handed to two different tasks because M-6 derives work_item_id from
the Plane sequence_id, which can collide -> the two tasks shared a branch/worktree
slug prefix and stepped on each other.
2a: ensure_unique_work_item_id() is a uniqueness-guard LAYERED ON TOP of the M-6
derive (derive is untouched): if the derived ET-NNN already exists in tasks for
the repo, it walks forward to the next free number. Applied in start_pipeline
after the derive.
2b (defense-in-depth): worktree is keyed by branch; if the resulting branch is
already owned by another task in the repo, disambiguate it with the unique
work_item_id + plane id so two tasks can never share a worktree.
Plane issue.updated (status -> In Progress) ships only changed fields, so the
webhook payload has no description and QG-0 wrongly blocked issues. start_pipeline
now pulls the full description from the Plane issue detail API (reusing the same
GET endpoint + shared token as fetch_issue_sequence_id) when the payload field is
empty/short, before QG-0 runs. Empty API -> honest QG-0 fail (truly empty ticket).
A pytest run on prod was sending REAL Telegram messages to Slava: some tests
(e.g. test_webhook_dedup advancing a stage) reach notify_stage_change ->
send_telegram, which read the live .env token/chat_id and actually POSTed.
Add an autouse fixture stubbing send_telegram to a no-op for every test. Patch
the SOURCE src.notifications.send_telegram (covers all notify_* helpers and the
many modules that do a local from .notifications import send_telegram inside
functions) AND src.stage_engine.send_telegram (module-level binding, would not be
intercepted by the source patch alone). webhooks/plane, launcher, queue_worker are
patched defensively with raising=False.
Verified: full suite run with FAKE telegram creds + an un-swallowable httpx.post
trip-wire (BaseException, so send_telegram except Exception can not hide it) shows
ZERO calls to api.telegram.org. Without the fixture the trip-wire fires, proving
the guard is real.
Feature 4. claude is now launched with --output-format json; the run-log trailing
result JSON is parsed (defensively, never fatal) for usage + total_cost_usd. New
idempotent ALTERs add input_tokens/output_tokens/cache_read_tokens/cost_usd to
agent_runs; the launcher monitor records usage per run, posts a per-agent finish
comment under that agent bot (e.g. Developer gotov · 45.2k in / 12.1k out · $0.21),
and the deployer posts an end-of-task summary (SUM over agent_runs GROUP BY agent)
on done. New src/usage.py holds parse/format/record/summary helpers; test_usage.py
covers parsing a real CLI JSON blob, NULL-on-garbage, recording, formatting, and the
per-task aggregate.
Feature 2. The issue updated dispatch (shipped with the status-trigger handler)
also routes Approved -> _try_advance_stage (== :approved: comment) and Rejected ->
_rollback_stage (== :rejected: comment). The :rejected: comment branch was
refactored into the shared _rollback_stage so both mechanisms behave identically;
a status reject passes Reason: (rejected via status, see latest comment) since no
inline reason arrives with a status change. Comments stay fully working. This
commit adds test_verdict_status.py proving both status and comment paths funnel
into the same advance/rollback logic.
Feature 1. work_item.created no longer starts the pipeline (soft QG-0 log only);
the issue stays in the backlog until moved to In Progress. The pipeline-start body
is extracted into start_pipeline(); a new issue updated handler routes a state
change to In Progress -> handle_status_start, which is idempotent: an existing task
for the plane_id is NOT re-created or restarted (protects handle_comment, which also
flips issues to In Progress). Real Plane payload: event=issue, action=updated,
data.state.id. Existing m6/plane_webhook/dedup tests updated to drive the new
trigger; new test_status_trigger.py covers created-no-op / start / idempotent.
Feature 3 + Feature 2 infra. Extend the global PLANE_STATES with the 6 new
enduro status UUIDs (architecture/development/review/testing + approved/rejected),
remap STAGE_TO_STATE so the 4 mid-pipeline stages move the issue across its own
board column instead of all sitting in In Progress, and add the
set_issue_stage_state() helper. Needs Input / In Review / Blocked keep their own
explicit setters and stay higher priority. TODO(ORCH-10): statuses are per-project;
resolve per project when more projects are onboarded.
add_comment now accepts an optional author (agent role) and POSTs under the matching Plane bot token via _headers_for(), so Plane shows the real author (Analyst/Architect/Developer/Reviewer/Tester/Deployer/Stream) instead of a single shared account. Unknown/empty roles or missing tokens fall back to the shared orchestrator token (autonomy preserved). GET/PATCH (find_issue_id, set_state) are unchanged and stay on the shared token. Call sites in stage_engine, launcher, webhooks/plane and the plane_sync notify helpers now pass author by stage role; stage transitions use stream. Adds tests/test_plane_author.py.
Add 7 optional bot-token fields (plane_bot_analyst..stream) read from the ORCH_PLANE_BOT_* env vars, default empty. Required for per-agent comment authorship; empty values fall back to the shared orchestrator token.
18 tests: happy-path advance per stage with correct agent (ORCH-4 fix),
QG-fail no-advance, reviewer REQUEST_CHANGES rollback+retry/alert, tester FAIL
rollback+retry/block, architect conflict rollback to analysis, analyst
approved-flow no-advance, and launcher+plane both delegating to the engine.
launcher._try_advance_stage and plane._try_advance_stage are now thin
wrappers over stage_engine.advance_stage. The plane webhook calls the sync
engine via asyncio.to_thread so there is exactly one implementation. The
launcher forwards finished_agent so the agent-specific rollback branches still
fire; the webhook passes None (human :approved:), matching prior behavior.
Also fixes the agent-selection bug in the launcher path: it used to enqueue
get_agent_for_stage(next_stage) (skipping a stage, e.g. analysis->architecture
launched developer instead of architect). The unified engine uses
get_agent_for_stage(current_stage), consistent with plane and gitea.
Merge the two diverged _try_advance_stage implementations (launcher sync +
plane async) into one synchronous engine. Preserves all launcher business
logic (analyst approved-flow, reviewer REQUEST_CHANGES rollback+retry, tester
FAIL rollback+retry, architect conflict rollback) and the plane
check_review_approved PR-by-branch dispatch. Unifies the QG signature
dispatch. Fixes agent selection: advancing FROM current_stage launches
get_agent_for_stage(current_stage), not next_stage.
The watchdog used to time.sleep(timeout) then immediately SIGKILL, which cut
claude off mid-write and left half-written artifacts. It now sends SIGTERM,
polls os.kill(pid, 0) for up to agent_kill_grace_seconds, and only SIGKILL if
the process is still alive; ProcessLookupError is tolerated at every step.
Timeout is now configurable via config.py: agent_timeout_seconds (default 1800),
agent_kill_grace_seconds (default 20), and agent_timeout_overrides_json for
per-agent overrides (e.g. {"reviewer": 3600}). AGENT_TIMEOUT is kept as a
backward-compatible alias. The recorded exit_code stays -9 so the ORCH-1
monitor retry/fail logic is unchanged (timeout-kills classify as permanent and
requeue within max_attempts, no retry loop).
_auto_merge_pr had zero callers (merge is handled by the deployer agent).
Removed the method; _ensure_pr (still used by the auto-advance path) is kept.
Container ORCH_CLAUDE_BIN pointed at a non-existent /usr/bin/claude while the
launcher spawns the hardcoded /opt/claude-code/bin/claude.exe. Preflight now
follows AgentLauncher.CLAUDE_BIN (the genuinely executed path), so it no longer
falsely blocks every job in production.
_finalize_job classifies the run log: transient (429/overload) -> backoff
requeue via mark_job_transient with separate transient_attempts budget honouring
Retry-After; permanent -> normal attempts<max. on_outcome callback feeds the
circuit breaker. _backoff_seconds = min(2^n*base, max) | Retry-After.
queue_worker.QueueWorker drains the queue respecting max_concurrency. main.py
lifespan: queue-recovery (requeue running jobs) after M-1 orphan-recovery, starts
worker and stops it on shutdown. New GET /queue endpoint (counts + recent jobs).
All 8 webhook launch points (plane x4, gitea x4) now enqueue a job and return
immediately instead of synchronously spawning claude in the uvicorn process.
ORCH-6: ARCHITECTURE.md gets a project-registry section; README explains
how to add a project via ORCH_PROJECTS_JSON; BUGFIXES_2026-06-03.md
records the fix and links the 2026-06-02 webhook autorun incident.
ORCH-6 / incident 2026-06-02: ignore work items from unknown Plane
projects (status=ignored) instead of funneling everything into
default_repo. Resolve repo, work-item prefix and Plane sync project from
the registry by data.project.
ORCH-6: sync functions resolve the issue PROJECT_ID via the registry
(get_project_by_repo) and accept project_id; default stays enduro so
existing ET callers keep working.
ORCH-6: get_next_work_item_id(repo, prefix="ET") numbers per (repo, prefix)
so orchestrator issues number ORCH-001 independently of the ET sequence.
Default prefix stays ET for backward compatibility.
ORCH-6: src/projects.py introduces ProjectConfig + resolvers
(get_project_by_plane_id/by_repo, known_plane_project_ids) keyed by
Plane project uuid. Source: ORCH_PROJECTS_JSON env (config.projects_json),
with a built-in default registry (enduro-trails + orchestrator) and
robust parsing (malformed JSON/entries fall back to default).
- check_reviewer_verdict reads verdict: from YAML frontmatter of 12-review.md only
- add check_tests_local: orchestrator runs make test in /repos/<repo>
- stages: development QG -> check_tests_local
- Add git fetch+checkout in agent launch cmd (ensures correct branch)
- Add git fetch+checkout in _monitor_agent before commit/push
- Post start comment in Plane when analyst launches
- Post :approved: request comment after analyst completes successfully
- Branch lookup moved before cmd construction for reuse
Title: BUG: deploy-staging петля — откат на development (self-deploy)
Description:
Симптом: на стадии deploy-staging для self-hosting orchestrator задача откатывается deploy-staging -> development и крутится по кругу.ДВЕ подтверждённые причины (ORCH-58 + ORCH-60):1. check_staging_status FAILED (ложный). deployer гоняет staging_check.py, тот падает на C9a/C9b (sandbox e2e: branch not found + analyst job in queue) с пометкой «Plane comment check skipped: bot-tokens not added to SANDBOX project». 8/10 PASS, 2 ложных FAIL из-за ненастроенных bot-токенов SANDBOX-проекта. QG check_staging_status -> FAILED -> rollback deploy-staging->development. Это НЕрегресс кода, а отсутствие sandbox-настроек.2. no changes to commit. для action-стадий (деплой = рестарт/retag, не правка кода) deployer exit0 + «no changes» тоже трактуется stage_engine как недовыполнение -> откат.Последствие: прод-деплой self-hosting репо НЕВОЗМОЖЕН автономно — ORCH-58 и ORCH-60 доводились ВРУЧНУЮ (merge PR + build-once retag + --deploy). Прямой блокер автономного внедрения (эпик ORCH-54).Fix-направления (одно или оба):(а) Настроить sandbox bot-токены в SANDBOX Plane-проект, чтобы staging_check C9a/C9b проходили честно (10/10). Тогда check_staging_status не будет ложно падать.(б) Отвязать advance deploy-стадии от git-changes для self-deploy репо: успех = exit0 + health PASS (+ опц. staging_check), а не наличие коммита.Acceptance: ORCH-задача для self-hosting orchestrator проходит deploy-staging -> deploy -> Done БЕЗ ручного вмешательства и без петли. Priority P0.
FastAPI-сервис для оркестрации мульти-агентного пайплайна разработки.
> См. [CLAUDE.md](CLAUDE.md) (паспорт проекта) и [docs/architecture/README.md](docs/architecture/README.md) (архитектура).
>
> **Витрина системы — [docs/overview/](docs/overview/README.md)**: единая точка входа в документацию
> (бизнес + тех, 7 блоков, маршруты для заказчика / менеджера / разработчика, презентация). ORCH-011.
## Что делает
FastAPI-сервис для оркестрации мульти-агентного пайплайна разработки. Принимает webhooks от Plane и Gitea, управляет жизненным циклом задач через Quality Gates, запускает Claude CLI агентов на каждой стадии.
- Принимает webhooks от **Plane** (task management) и **Gitea** (git events)
- Проверяет Quality Gates перед переходом между стадиями
| GET | `/status` | Активные задачи (stage != done) |
| GET | `/queue` | Очередь задач (ORCH-1): counts по статусам + max_concurrency + последние 10 jobs |
| POST | `/webhook/plane` | Plane webhook receiver |
| POST | `/webhook/gitea` | Gitea webhook receiver |
| POST | `/bug-fast-track/escalate?work_item=<id>` | Эскалация багфикс-задачи в полный цикл (ORCH-019): сброс`track``'bug'→'full'` → следующий переход уходит в `architecture` |
| `ORCH_RECONCILE_NOTIFY_UNBLOCK` | Telegram при разблокировке застрявшей задачи | `true` |
| `ORCH_RECONCILE_SKIP_BLOCKED_ENABLED` | F-1 Guard 2 (ORCH-060): пропуск задач в Plane-статусе Blocked / Needs Input; `false` глушит только сетевой Guard 2 (Guard 1 escalated всегда активен) | `true` |
| `ORCH_QG0_TITLE_MAX` | Верхний лимит длины заголовка QG-0 (вход `_qg0_errors`); невалидное/пустое значение → дефолт (ORCH-069) | `200` |
| `ORCH_STOP_STATUS_ENABLED` | Kill-switch отмены задачи по Plane-статусу **STOP** + закрытия дыры релонча (ORCH-090); `false` → поведение 1:1 как до ORCH-090 | `true` |
| `ORCH_STOP_STATUS_REPOS` | CSV область репо для STOP-отмены; пусто = все репо (ORCH-090) | `""` |
| `ORCH_BUG_FAST_TRACK_ENABLED` | Kill-switch багфикс-трека (ORCH-019): задача с меткой Plane `Bug` пропускает стадию `architecture`; `false` → старт и маршрут 1:1 как до ORCH-019 (нулевая регрессия) | `true` |
| `ORCH_ESTIMATOR_ENABLED` | Kill-switch оценки задачи (ORCH-020): Plane-статус «Оценка» прогнозирует стоимость/время/токены/story-points по истории; `false` → статус не обрабатывается, ничего не пишется (нулевая регрессия) | `true` |
| `ORCH_ESTIMATOR_REPOS` | CSV область репо для оценки; **пусто → self-hosting only** (`orchestrator`) — enduro не затронут (ORCH-020) | `""` |
| `ORCH_ESTIMATOR_MIN_SAMPLES` | Порог истории, ниже которого подмешивается bootstrap-дефолт прогноза (ORCH-020) | `3` |
| `ORCH_ESTIMATOR_BOOTSTRAP_TOKENS` / `_COST_USD` / `_SECONDS` | Bootstrap-прогноз при пустой истории (токены/стоимость/время; ORCH-020) | `2000000`/`3.0`/`1800` |
| `ORCH_ESTIMATOR_SP_COST_THRESHOLDS` | 4 возрастающих кат-оффа стоимости (t1,t2,t3,t5) для бакета story-points (`<=t1→1`…`<=t5→5`, иначе `8`; ORCH-020) | `0.50,2.00,5.00,12.00` |
| `ORCH_ESTIMATOR_WALL_CAP_S` / `_MAX_INFLIGHT` | Отсечка аномального wall-времени в истории / опц. семафор сглаживания массовой нагрузки (ORCH-020) | `86400`/`64` |
| `ORCH_AGENT_HOME_DIR` | ORCH-101: HOME акторских процессов + таргет маунтов `.claude`/`.ssh` + `ARG APP_HOME` (группа ORCH-040) | `/home/slin` |
Поля `name` опционально (по умолчанию = `repo`). Подробности — `docs/architecture/internals.md`.
## Ключевые механизмы
### Auto-advance
После успешного завершения агента (exit_code=0), `_try_advance_stage()` проверяет QG и автоматически продвигает задачу + запускает следующего агента.
### Review bounce
При REQUEST_CHANGES от reviewer задача откатывается в development, developer перезапускается (до 3 попыток). При исчерпании — эскалация.
### Orphan recovery (M-1)
При старте контейнера каждый run с `finished_at IS NULL` старше 35 минут помечается exit_code=-1, логируется per-run warning и отправляется Telegram-уведомление «нужна ручная проверка/перезапуск» (не молча).
### Запись task-файлов (B-1)
Task-файлы `.task-*.md` пишутся **прямой записью в смонтированный volume `/repos/<repo>/`** (без docker). При ошибке записи — RuntimeError (не молчит). В `.gitignore` проекта.
### Логи агентов (B-2)
stdout/stderr агента перенаправляются СРАЗУ в `/app/data/runs/{id}.log` на уровне ОС (без PIPE). monitor-поток делает `proc.wait()` → реальный exit_code, нет зомби.
### Watchdog
Каждый агент имеет per-role wall-clock бюджет (ORCH-109): developer 60 мин / reviewer 50 мин / прочие 30 мин дефолт (`_resolve_timeout`). При превышении — SIGTERM→grace→SIGKILL + запись exit_code=-9. Tier-3 backstop `reaper_max_running_s`=90 мин > max(timeout)+grace (ORCH-065).
- `status` → Gitea CI статус; ORCH-045: авторитетный гейт развития (`development → review`) — `check_ci_green` читает статус ветки с polling-retry (устраняет гонку «pending сразу после push»)
Реально открытые ограничения (сверено с кодом, ORCH-079):
1. **Telegram 48h** — карточки-сироты старше 48 часов неудаляемы (лимит Telegram Bot API); зачистка сирот самозалечивает только свежие (ORCH-087).
2. **Зависимости задач — только intra-repo (v1)** — `job_deps` выражают связи в пределах одного репозитория; кросс-репо зависимости пока не поддержаны (ORCH-026).
3. **Пакетный автоном — Этап 1** — per-repo serial gate сериализует задачи одного репо (ORCH-088); полный пакетный автономный прогон «10–20 задач за ночь» — в развитии (эпик ORCH-088).
### Закрыто (история)
Пункты, ранее значившиеся ограничениями, закрыты кодом — оставлены как трассировка:
# Product Vision — Автономная фабрика разработки (Orchestrator)
> Мультиагентная платформа, которая превращает идею или баг в задеплоенный на прод результат — автономно, надёжно и дёшево.
**Версия:** 1.0 · **Дата:** 2026-06-04 · **Статус:** концепция развития
> **Фактическое текущее состояние платформы** (что уже умеет, как устроена) — витрина системы
> [docs/overview/](overview/README.md) (ORCH-011). Этот документ — vision: «куда идём».
---
## 1. Зачем это (бизнес-взгляд)
### Проблема
Классическая разработка — это люди-бутылочное-горлышко на каждом шаге: аналитик, архитектор, разработчик, ревьюер, тестировщик, деплой-инженер. Каждая передача задачи между ними — потеря времени, контекста и денег. Мелкая фича или баг едут днями.
### Решение
**Orchestrator** — это конвейер из ИИ-агентов, который проводит задачу через все стадии разработки сам: от бизнес-постановки до релиза на прод. Человек ставит задачу и принимает результат. Всё между — автономно.
### Ценность
- ⚡ **Скорость:** фича проходит полный цикл (анализ → архитектура → код → ревью → тесты → деплой) за ~35 минут без ручных вмешательств.
- 💰 **Стоимость:** работа агентов в разы дешевле команды; адаптивный выбор моделей экономит на простых задачах.
- 🎯 **Автономность:** 0 ручных пинков в штатном прогоне. Человек — постановщик и приёмщик, а не оператор.
- 🛡️ **Надёжность:** многоуровневые гейты качества не пускают недоделку на прод.
- 🔁 **Масштаб:** одна платформа ведёт несколько проектов; саму платформу можно тиражировать на новые хосты.
---
## 2. Как это работает (обзор)
### Конвейер
```
created → analysis → architecture → development → review → testing → deploy → done
```
На каждом переходе стоит **quality gate** — автоматическая проверка, которая не пускает задачу дальше, пока стадия не выполнена честно:
> **Самодостаточная фабрика разработки, которая размножается, учится на ошибках, оценивает себя, бережёт бюджет и не ломает прод — превращая намерение человека в работающий продукт почти без его участия.**
---
*Документ поддерживается в репозитории orchestrator. Источник дорожной карты — задачи проекта ORCH в Plane (ORCH-7…ORCH-28).*
Инцидент 2026-06-02: Plane-вебхук слушал весь воркспейс и хардкодил `repo = settings.default_repo` (enduro-trails). Задачи ЛЮБОГО проекта сливались в один репо с одним префиксом (ET). Нужна изоляция по проектам.
## Решение
Введён реестр `src/projects.py`: `ProjectConfig` (frozen dataclass) связывает `plane_project_id` → `repo` + `work_item_prefix` + `name`. Источник правды — env `ORCH_PROJECTS_JSON`; при пустом/невалидном — встроенный дефолт (`enduro-trails`/ET, `orchestrator`/ORCH). Позволяет: фильтровать вебхуки по проекту (неизвестный → ignore), резолвить gitea-репо + префикс, роутить Plane-синк в свой проект задачи.
## Альтернативы
- Один репо на всё — отклонён (источник инцидента).
- Хардкод маппинга в коде — отклонён в пользу env-конфигурируемого реестра с безопасным дефолтом.
## Последствия
- Изоляция проектов на уровне вебхуков и роутинга.
- Парсер устойчив: битый элемент скипается, пустой результат → дефолт.
- Основа для `is_self_hosting_repo` (adr-0003).
## Связи
adr-0003 (условный гейт опирается на repo из реестра).
# adr-0002: Очередь задач вместо in-process потоков
- **Статус:** accepted
- **Дата:** 2026-06-03
- **Задача:** ORCH-1 (F-2b)
## Контекст
Ранняя версия запускала стадии конвейера в in-process daemon-потоках. Проблемы: не переживало рестарт (задачи терялись), нет контроля параллелизма, нет ретраев, нет наблюдаемости.
## Решение
Введена персистентная очередь задач (`src/queue_worker.py` + таблица `jobs` в SQLite): atomic claim задачи воркером, `max_concurrency`, ретраи при сбое, restart-safe (running-задачи реквестятся при старте), эндпоинт `GET /queue`.
## Альтернативы
- In-process потоки — отклонены (не restart-safe).
- Внешний брокер (Redis/RabbitMQ) — избыточно для текущего масштаба; SQLite-очередь проще и без новых зависимостей.
## Последствия
- Конвейер переживает рестарт контейнера.
- Контроль параллелизма и наблюдаемость через `/queue`.
- ⚠️ Очередь общая на все проекты прод-инстанса — фактор группового риска при self-hosting (см. `docs/operations/INFRA.md`).
## Связи
adr-0001 (реестр проектов), INFRA.md (общая очередь при self-hosting).
# adr-0003: Условный staging-гейт перед прод-деплоем
- **Статус:** accepted
- **Дата:** 2026-06-05
- **Задача:** ORCH-35
## Контекст
Оркестратор дорабатывает сам себя (self-hosting). Раньше стадия `deploy` имела «бумажный» вердикт: deployer-агент писал `deploy_status: SUCCESS`, но реального прогона на изолированной среде не было. Нужен предохранитель: прод-деплой орка не должен происходить, пока изменения не проверены на живой staging-среде. При этом другие проекты (enduro-trails) staging-инфры не имеют.
## Решение
Добавлена промежуточная стадия `deploy-staging` между `testing` и `deploy`: `testing → deploy-staging → deploy → done`.
- deployer гоняет `scripts/staging_check.py --base-url http://localhost:8501` и пишет `staging_status: SUCCESS|FAILED` в `15-staging-log.md`.
- **Гейт условный:** `is_self_hosting_repo(repo)` → реальная проверка только для `orchestrator`; для остальных проектов гейт = no-op `(True, "Staging gate N/A")`.
- FAILED → откат на `development`.
## Альтернативы
- Глобальный гейт для всех проектов — отклонён: у enduro нет staging-инстанса, задачи застревали бы на откате.
- Деплой реально дёргает хост-хук прямо здесь — отложен в ORCH-36 (Вариант B).
## Последствия
- Прод-деплой орка недостижим, пока staging-гейт не зелёный.
- Другие проекты не затронуты (no-op).
- Реальный docker-деплой через хук пока НЕ выполняется (вердикт «бумажный», но подкреплён прогоном сьюта). Исполняемый деплой — ORCH-36.
## Связи
adr-0001 (реестр проектов — основа `is_self_hosting_repo`), ORCH-34 (deploy-hook + rollback), ORCH-36 (исполняемый самодеплой).
# adr-0004: Поллинг с ретраем в quality-gate check_ci_green (фикс CI-race)
- **Статус:** accepted
- **Дата:** 2026-06-05
- **Задача:** ORCH-045
## Контекст
Quality-gate `check_ci_green(repo, branch)` (`src/qg/checks.py`) проверяет combined commit-status ветки через Gitea API сразу после того, как developer-агент запушил код. Реализация была **single-shot**: один `GET /repos/{owner}/{repo}/commits/{branch}/status`, чтение `data["state"]` — `success` → пропуск, иначе → сразу `False`.
Это создавало race condition. Gitea-CI после пуша 1–3 секунды держит combined state `pending`, пока не отработают чек-раннеры. Если гейт опрашивал статус в этом окне, он получал `pending` и возвращал `False`**ровно один раз** — повторного опроса не было. Combined state затем дозеленевал до `success`, но гейт уже промахнулся, и задача застревала насмерть без видимой причины.
Реальный инцидент **ORCH-017**: гейт опросил статус в 17:58:54 → `pending`; CI дозеленел в 17:58:55. Задача встала в тупик (см. `docs/history` / lessons ORCH-017).
## Решение
`check_ci_green` превращён из single-shot в **polling с ретраем**:
-`state in ("failure", "error")` → `(False, "CI state: <state>")` немедленно — CI красный, ретрай бессмыслен (терминальное состояние).
-`state == "pending"` (или `unknown` / иное не-терминальное) → `time.sleep(interval)` и опрос снова, до `N` попыток.
- После исчерпания всех попыток при всё ещё `pending` → `(False, "CI still pending after <T>s")` — **явный** провал с причиной, чтобы оператор видел тупик, а не молчаливый стол.
-`404` → `(False, "Branch ... not found or no status")` — как раньше.
- Транзиентная `httpx.HTTPError` на отдельной попытке — **не падаем сразу**: логируем и пробуем ещё в рамках лимита попыток; если все попытки — сетевая ошибка → `(False, "API error: <e>")`.
Параметры вынесены в `src/config.py` (pydantic-settings, env-prefix `ORCH_`, единый стиль с остальными настройками):
Итого по умолчанию гейт ждёт `pending` до ~2 минут (12 × 10s) перед тем как явно провалиться. Каждая не-финальная попытка логируется через существующий `logger` (`check_ci_green: attempt i/N, state=..., retrying in Ns`). `timeout=10` на каждый отдельный запрос сохранён.
Сигнатура `check_ci_green(repo, branch) -> tuple[bool, str]`**не менялась** — её зовёт stage_engine и реестр гейтов `QG_CHECKS`.
## Альтернативы
- **Оставить single-shot, опрашивать гейт повторно снаружи (на уровне stage_engine/воркера).** Отклонено: размазывает логику CI-ожидания по слоям, дублирует таймауты; гейт — естественное место знания о combined-status.
- **Webhook от Gitea на завершение CI вместо поллинга.** Отложено: требует надёжной доставки/дедупликации вебхуков именно по CI-статусу и переписывания триггера стадии; поллинг — минимальный, локализованный фикс race-а здесь и сейчас.
- **Бесконечный ретрай до зелёного.** Отклонено: задача могла бы висеть вечно при реально зависшем CI; ограниченный бюджет + явный `False`с причиной даёт оператору сигнал.
-`check_ci_green` теперь **блокирующий** до ~`max_attempts × interval` секунд при затяжном `pending` (по умолчанию ~2 мин). Это осознанный trade-off; для красного CI и success — выход немедленный, без задержки.
- Тупик больше не молчаливый: истечение попыток → `(False, "CI still pending after <T>s")`, причина видна.
- Бюджет/интервал настраиваемы через env без правки кода.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.