admin/orchestrator
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity

ORCH-1 (F-2b): persistent job queue instead of in-process daemon threads #3

Merged
admin merged 13 commits from feature/ORCH-1-job-queue into main 2026-06-03 08:09:23 +03:00
Conversation 0 Commits 13 Files Changed 14 +1785 -35
admin commented 2026-06-02 23:59:03 +03:00
Owner
Copy Link

Persistent SQLite job queue + background worker, replacing in-process Popen+daemon-thread agent spawn (8 webhook call points). Plus a resilience layer (Slava extension).

Queue core

  • jobs table + atomic claim_next_job (available_at-gated)
  • launch_job + _finalize_job (done/requeue/failed, retries)
  • 8 webhook points + 4 internal advance calls -> enqueue_job
  • queue_worker (max_concurrency) + lifespan start/stop + queue-recovery (restart-safe)
  • GET /queue observability

Resilience (ДОПОЛНЕНИЕ)

  • A. Cheap preflight (CLAUDE_BIN exists + claude --version, cached, NO tokens, no prompt-ping) gates claiming; FAIL -> jobs stay queued
  • B. 429/transient vs permanent classifier on the run-log tail + Retry-After
  • C. Exponential backoff via jobs.available_at + separate transient_attempts budget
  • D. Circuit breaker: N consecutive transient -> open (pause, no CLI, Telegram), half-open probe, closed on recovery; reflected in /queue.resilience
  • fix: preflight checks the binary the launcher actually spawns (container ORCH_CLAUDE_BIN pointed at a missing path)

Tests: 110 passed / 9 pre-existing test_webhooks failures untouched. Popen never spawned (mocked). Live-verified on the box (preflight ok, transient backoff, breaker open/half-open/closed).

B-1/B-2/M-1/ORCH-2/ORCH-6 preserved. See docs/ORCH-1_JOB_QUEUE.md.

Persistent SQLite job queue + background worker, replacing in-process Popen+daemon-thread agent spawn (8 webhook call points). Plus a resilience layer (Slava extension). **Queue core** - jobs table + atomic claim_next_job (available_at-gated) - launch_job + _finalize_job (done/requeue/failed, retries) - 8 webhook points + 4 internal advance calls -> enqueue_job - queue_worker (max_concurrency) + lifespan start/stop + queue-recovery (restart-safe) - GET /queue observability **Resilience (ДОПОЛНЕНИЕ)** - A. Cheap preflight (CLAUDE_BIN exists + claude --version, cached, NO tokens, no prompt-ping) gates claiming; FAIL -> jobs stay queued - B. 429/transient vs permanent classifier on the run-log tail + Retry-After - C. Exponential backoff via jobs.available_at + separate transient_attempts budget - D. Circuit breaker: N consecutive transient -> open (pause, no CLI, Telegram), half-open probe, closed on recovery; reflected in /queue.resilience - fix: preflight checks the binary the launcher actually spawns (container ORCH_CLAUDE_BIN pointed at a missing path) **Tests:** 110 passed / 9 pre-existing test_webhooks failures untouched. Popen never spawned (mocked). Live-verified on the box (preflight ok, transient backoff, breaker open/half-open/closed). B-1/B-2/M-1/ORCH-2/ORCH-6 preserved. See docs/ORCH-1_JOB_QUEUE.md.
admin added 6 commits 2026-06-02 23:59:03 +03:00
feat(queue): add jobs table + queue helpers and config (ORCH-1) fd3dac7d22 
Persistent SQLite job queue (F-2b): jobs table + idx, atomic claim_next_job,
enqueue/mark/count/requeue/get helpers. New settings max_concurrency
(ORCH_MAX_CONCURRENCY) and queue_poll_interval (ORCH_QUEUE_POLL_INTERVAL).
feat(launcher): launch_job + job-status finalize with retries (ORCH-1) 3345c2fa0a 
Refactor launch() into shared _spawn(); add launch_job(job) that threads job_id
through monitor/watchdog. _finalize_job marks done / requeue (attempts<max) /
failed+notify. Internal advance-chain self.launch -> enqueue_job. B-1/B-2/M-1/ORCH-2
spawn logic unchanged.
refactor(webhooks): enqueue_job instead of in-process launch (ORCH-1) 20d6556e22 
All 8 webhook launch points (plane x4, gitea x4) now enqueue a job and return
immediately instead of synchronously spawning claude in the uvicorn process.
feat(worker): background queue worker + lifespan + queue-recovery + /queue (ORCH-1) b6d4426a48 
queue_worker.QueueWorker drains the queue respecting max_concurrency. main.py
lifespan: queue-recovery (requeue running jobs) after M-1 orphan-recovery, starts
worker and stops it on shutdown. New GET /queue endpoint (counts + recent jobs).
test(queue): 19 tests for job queue lifecycle/atomicity/retry/worker (ORCH-1) 2283b8898b 
Covers enqueue->claim->mark, atomic claim (no double dispatch, 8-thread race),
retry fail->queued->failed, requeue_running_jobs, observability, worker
max_concurrency. Popen fully mocked (no real agent spawned).
docs(queue): document job queue, /queue, env vars (ORCH-1) 4be168c0ec 
ARCHITECTURE job-queue section + flow diagram, README /queue endpoint and
ORCH_MAX_CONCURRENCY/ORCH_QUEUE_POLL_INTERVAL, new docs/ORCH-1_JOB_QUEUE.md.
admin added 6 commits 2026-06-03 00:12:26 +03:00
feat(queue): resilience schema + backoff helper + config (ORCH-1) 0cd9b11fe0 
jobs.transient_attempts + available_at columns (idempotent _ensure_column
migration); claim_next_job honours available_at; mark_job_transient (backoff
requeue with separate transient budget). Config: preflight_cache_ttl,
backoff_base/max_seconds, transient_max_attempts, breaker_threshold,
breaker_pause_seconds.
feat(resilience): cheap preflight + 429/transient error classifier (ORCH-1) 4ef87a3959 
preflight.py: cached CLAUDE_BIN exists + claude --version (no tokens, no
prompt-ping). error_classifier.py: classify_log_file -> transient|permanent
from log tail + Retry-After parsing.
feat(launcher): classify failures, backoff transient retry, breaker outcome (ORCH-1) 90fdd19394 
_finalize_job classifies the run log: transient (429/overload) -> backoff
requeue via mark_job_transient with separate transient_attempts budget honouring
Retry-After; permanent -> normal attempts<max. on_outcome callback feeds the
circuit breaker. _backoff_seconds = min(2^n*base, max) | Retry-After.
feat(worker): preflight gate + circuit breaker + /queue resilience (ORCH-1) f314ae09e5 
QueueWorker gates claims behind preflight and the CircuitBreaker (open ->
pause, no CLI calls + Telegram alert; half-open probes one job; closed on
recovery). Wires launcher.on_outcome. /queue exposes resilience snapshot.
test(resilience): 34 tests for preflight/classifier/backoff/breaker (ORCH-1) a613fd8180 
Covers preflight FAIL->queued + cache, transient/permanent classifier +
Retry-After, exp backoff + available_at gating, launcher transient vs permanent
finalize, circuit breaker open/half-open/closed. test_queue worker tests stub
preflight OK. Popen never spawned.
docs(resilience): document preflight/429/backoff/breaker + env vars (ORCH-1) d0d47058b4
admin added 1 commit 2026-06-03 00:13:45 +03:00
fix(preflight): check the binary the launcher actually spawns (ORCH-1) c23f000c05 
Container ORCH_CLAUDE_BIN pointed at a non-existent /usr/bin/claude while the
launcher spawns the hardcoded /opt/claude-code/bin/claude.exe. Preflight now
follows AgentLauncher.CLAUDE_BIN (the genuinely executed path), so it no longer
falsely blocks every job in production.
admin merged commit 4e52e192e4 into main 2026-06-03 08:09:23 +03:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
admin
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/orchestrator#3
Delete Branch "feature/ORCH-1-job-queue"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?