Compare commits
9 Commits
fix/isolat
...
feature/OR
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e0c14fae5f | ||
|
|
e0b6e92b09 | ||
| e405a55f9d | |||
|
|
a6cbacb62c | ||
| 93169f16e0 | |||
|
|
94334bdd42 | ||
| 3b68a29ae1 | |||
|
|
6c1e5fff52 | ||
| d0a34249cc |
52
.env.staging.example
Normal file
52
.env.staging.example
Normal file
@@ -0,0 +1,52 @@
|
||||
# STAGING env for orchestrator-staging (port 8501).
|
||||
# Plane/Gitea tokens and sandbox project — configured in ORCH-32.
|
||||
# On Stage 1 (ORCH-31) you can copy from prod .env, changing only isolation-related keys.
|
||||
#
|
||||
# DO NOT COMMIT the real .env.staging — this file is the template only.
|
||||
# Create .env.staging on the server and fill in real values before starting staging.
|
||||
|
||||
# ── Plane ─────────────────────────────────────────────────────────────────────
|
||||
ORCH_PLANE_API_URL=http://localhost:8091
|
||||
ORCH_PLANE_API_TOKEN=<plane-api-token>
|
||||
ORCH_PLANE_WORKSPACE_SLUG=<workspace-slug>
|
||||
ORCH_PLANE_WEBHOOK_SECRET=<webhook-secret>
|
||||
|
||||
# Per-agent Plane bot tokens (authorship in Plane comments).
|
||||
# Leave empty to use ORCH_PLANE_API_TOKEN fallback.
|
||||
ORCH_PLANE_BOT_ANALYST=
|
||||
ORCH_PLANE_BOT_ARCHITECT=
|
||||
ORCH_PLANE_BOT_DEVELOPER=
|
||||
ORCH_PLANE_BOT_REVIEWER=
|
||||
ORCH_PLANE_BOT_TESTER=
|
||||
ORCH_PLANE_BOT_DEPLOYER=
|
||||
ORCH_PLANE_BOT_STREAM=
|
||||
|
||||
# ── Gitea ─────────────────────────────────────────────────────────────────────
|
||||
ORCH_GITEA_URL=http://localhost:3000
|
||||
ORCH_GITEA_PUBLIC_URL=https://git.mva154.duckdns.org
|
||||
ORCH_GITEA_TOKEN=<gitea-token>
|
||||
ORCH_GITEA_WEBHOOK_SECRET=<gitea-webhook-secret>
|
||||
|
||||
# ── Telegram ──────────────────────────────────────────────────────────────────
|
||||
ORCH_TELEGRAM_BOT_TOKEN=<telegram-bot-token>
|
||||
ORCH_TELEGRAM_CHAT_ID=<telegram-chat-id>
|
||||
|
||||
# ── Claude / repos ────────────────────────────────────────────────────────────
|
||||
ORCH_CLAUDE_BIN=/usr/bin/claude
|
||||
ORCH_REPOS_DIR=/repos
|
||||
ORCH_HOST_REPOS_DIR=/home/slin/repos
|
||||
|
||||
# ── Database (ISOLATION KEY for staging) ─────────────────────────────────────
|
||||
# The staging volume mounts ./data/staging:/app/data, so the DB physically lives
|
||||
# at ./data/staging/orchestrator.db on the host — fully isolated from prod.
|
||||
# Do NOT change this path; isolation is achieved via the volume mount, not this path.
|
||||
ORCH_DB_PATH=/app/data/orchestrator.db
|
||||
|
||||
# ── Concurrency / worker ──────────────────────────────────────────────────────
|
||||
ORCH_MAX_CONCURRENCY=1
|
||||
ORCH_QUEUE_POLL_INTERVAL=2.0
|
||||
|
||||
# ── Deploy hook ───────────────────────────────────────────────────────────────
|
||||
DEPLOY_SSH_USER=slin
|
||||
DEPLOY_SSH_HOST=127.0.0.1
|
||||
DEPLOY_HOOK_SCRIPT=/home/slin/bin/enduro-deploy-hook.sh
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -5,3 +5,7 @@ __pycache__/
|
||||
data/
|
||||
*.db
|
||||
.pytest_cache/
|
||||
# ORCH-31: staging env (secrets, not committed — see .env.staging.example)
|
||||
.env.staging
|
||||
# ORCH-31: staging DB data directory
|
||||
data/staging/
|
||||
|
||||
67
.openclaw/agents/deployer.md
Normal file
67
.openclaw/agents/deployer.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Deployer Agent
|
||||
|
||||
You are the **Deployer** agent in the orchestrator pipeline. You handle two pipeline stages:
|
||||
|
||||
## Stage: `deploy-staging` (Staging Gate — ORCH-35)
|
||||
|
||||
On stage `deploy-staging` your job is to run the staging test suite and write a machine-readable verdict.
|
||||
|
||||
### Steps:
|
||||
|
||||
1. Run the staging test suite against the live staging environment:
|
||||
```bash
|
||||
python3 scripts/staging_check.py --base-url http://localhost:8501 --mode stub
|
||||
```
|
||||
|
||||
2. Check the exit code:
|
||||
- Exit code **0** = all tests PASS → `staging_status: SUCCESS`
|
||||
- Exit code **non-zero** = tests FAILED → `staging_status: FAILED`
|
||||
|
||||
3. Write the verdict to `docs/work-items/<work_item_id>/15-staging-log.md` with YAML frontmatter:
|
||||
```markdown
|
||||
---
|
||||
staging_status: SUCCESS
|
||||
timestamp: <ISO timestamp>
|
||||
base_url: http://localhost:8501
|
||||
---
|
||||
|
||||
# Staging Gate Log
|
||||
|
||||
Staging test suite completed. All checks passed.
|
||||
```
|
||||
Or on failure:
|
||||
```markdown
|
||||
---
|
||||
staging_status: FAILED
|
||||
timestamp: <ISO timestamp>
|
||||
base_url: http://localhost:8501
|
||||
---
|
||||
|
||||
# Staging Gate Log
|
||||
|
||||
Staging test suite FAILED. See details below.
|
||||
|
||||
<paste test output here>
|
||||
```
|
||||
|
||||
4. Merge `15-staging-log.md` into `main` (commit + push, same as deploy log pattern).
|
||||
|
||||
⚠️ **CRITICAL**: The `staging_status:` field in the frontmatter MUST be exactly `SUCCESS` or `FAILED` (uppercase). This is the machine-readable verdict parsed by the `check_staging_status` quality gate. No other values are accepted.
|
||||
|
||||
---
|
||||
|
||||
## Stage: `deploy` (Production Deploy — ORCH-36, future)
|
||||
|
||||
On stage `deploy` your job is to perform (or simulate) the production deployment and write a machine-readable verdict to `docs/work-items/<work_item_id>/14-deploy-log.md` with frontmatter field `deploy_status: SUCCESS|FAILED`.
|
||||
|
||||
This stage is only reached if the staging gate (`deploy-staging`) passed with `staging_status: SUCCESS`.
|
||||
|
||||
⚠️ **CRITICAL**: Do NOT trigger real production deploys unless explicitly instructed. Real docker/SSH deploys are handled by `scripts/orchestrator-deploy-hook.sh` (ORCH-36).
|
||||
|
||||
---
|
||||
|
||||
## General Rules
|
||||
|
||||
- Always write machine-readable YAML frontmatter — the quality gates parse ONLY the frontmatter fields, never the body prose.
|
||||
- Never push directly to `main`. Always use a PR or the artifact merge pattern.
|
||||
- Never modify `.env`, `.env.staging`, `docker-compose.yml`, or production infrastructure.
|
||||
@@ -25,3 +25,39 @@ services:
|
||||
- DEPLOY_HOOK_SCRIPT=/home/slin/bin/enduro-deploy-hook.sh
|
||||
group_add:
|
||||
- "999"
|
||||
|
||||
# ORCH-31: staging instance (port 8501, isolated DB).
|
||||
# Starts ONLY with: docker compose --profile staging up -d orchestrator-staging
|
||||
# Normal "docker compose up -d" does NOT start this service.
|
||||
orchestrator-staging:
|
||||
profiles:
|
||||
- staging
|
||||
build: .
|
||||
container_name: orchestrator-staging
|
||||
restart: unless-stopped
|
||||
init: true
|
||||
network_mode: host
|
||||
command: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8501"]
|
||||
volumes:
|
||||
- ./data/staging:/app/data
|
||||
- /home/slin/repos:/repos
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- /usr/lib/node_modules/@anthropic-ai/claude-code:/opt/claude-code:ro
|
||||
- /usr/bin/node:/usr/bin/node:ro
|
||||
- /home/slin/.claude:/home/slin/.claude
|
||||
- /home/slin/.claude.json:/home/slin/.claude.json:ro
|
||||
- /home/slin/.orchestrator-ssh:/root/.ssh:ro
|
||||
env_file: .env.staging
|
||||
environment:
|
||||
- ORCH_REPOS_DIR=/repos
|
||||
- ORCH_HOST_REPOS_DIR=/home/slin/repos
|
||||
- DEPLOY_SSH_USER=slin
|
||||
- DEPLOY_SSH_HOST=127.0.0.1
|
||||
- DEPLOY_HOOK_SCRIPT=/home/slin/bin/enduro-deploy-hook.sh
|
||||
# Staging DB is isolated via ./data/staging volume mount.
|
||||
# Inside the container the path remains /app/data/orchestrator.db (same default),
|
||||
# but on the host it physically lives at ./data/staging/orchestrator.db —
|
||||
# completely separate from prod ./data/orchestrator.db.
|
||||
- ORCH_DB_PATH=/app/data/orchestrator.db
|
||||
group_add:
|
||||
- "999"
|
||||
|
||||
90
docs/DEPLOY_HOOK.md
Normal file
90
docs/DEPLOY_HOOK.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Orchestrator Deploy Hook
|
||||
|
||||
`scripts/orchestrator-deploy-hook.sh` — хост-скрипт деплоя orchestrator с health-чеком и авто-rollback.
|
||||
|
||||
## Как работает
|
||||
|
||||
### Режим `--deploy` (по умолчанию)
|
||||
|
||||
1. **Захват текущего образа** — до рестарта записывает ID образа работающего контейнера в `$PREV_IMAGE_FILE` (best-effort, не падает если сервис не запущен).
|
||||
2. **git pull** — обновляет код репозитория.
|
||||
3. **Рестарт контейнера** — `docker compose --profile $COMPOSE_PROFILE up -d --no-build $TARGET_SERVICE`.
|
||||
4. **Health-цикл** — 10 попыток × 6с = до 60с. Критерий: HTTP 200 + тело содержит `"status":"ok"`.
|
||||
- **Успех** → `exit 0`, лог "Deploy SUCCESS".
|
||||
- **Провал** → авто-rollback (шаг 5).
|
||||
5. **Авто-rollback** — восстанавливает образ из `$PREV_IMAGE_FILE`, рестарт, повторный health 5×3с.
|
||||
- Если восстановился → `exit 1` (деплой провалился, откат успешен).
|
||||
- Если и откат не помог → `exit 2` (критично).
|
||||
|
||||
### Режим `--rollback`
|
||||
|
||||
Вручную откатывает сервис на предыдущий образ из `$PREV_IMAGE_FILE`.
|
||||
|
||||
## Переменные окружения
|
||||
|
||||
| Переменная | Дефолт | Описание |
|
||||
|------------------|-----------------------------------|-----------------------------------------------|
|
||||
| `TARGET_SERVICE` | `orchestrator-staging` | Имя docker-compose сервиса |
|
||||
| `TARGET_PORT` | `8501` | Порт health-check |
|
||||
| `TARGET_IMAGE` | `orchestrator-orchestrator-staging` | Имя образа для retag при rollback |
|
||||
| `COMPOSE_PROFILE`| `staging` | Docker compose profile (пусто = без профиля) |
|
||||
| `PREV_IMAGE_FILE`| `$REPO/.deploy-prev-image-staging`| Файл для сохранения предыдущего образа |
|
||||
| `LOG` | `/var/log/orchestrator/deploy-hook.log` | Лог-файл (fallback: `$REPO/deploy-hook.log`) |
|
||||
|
||||
> ⚠️ **Дефолт — всегда STAGING**. Прод активируется только явным переопределением env.
|
||||
|
||||
## Примеры запуска
|
||||
|
||||
### Staging (дефолт, безопасно)
|
||||
|
||||
```bash
|
||||
cd /home/slin/repos/orchestrator
|
||||
bash scripts/orchestrator-deploy-hook.sh --deploy
|
||||
# или просто:
|
||||
bash scripts/orchestrator-deploy-hook.sh
|
||||
```
|
||||
|
||||
### Прод (осознанный шаг, Этап 5)
|
||||
|
||||
```bash
|
||||
TARGET_SERVICE=orchestrator \
|
||||
TARGET_PORT=8500 \
|
||||
TARGET_IMAGE=orchestrator-orchestrator \
|
||||
COMPOSE_PROFILE="" \
|
||||
PREV_IMAGE_FILE=/home/slin/repos/orchestrator/.deploy-prev-image-prod \
|
||||
bash scripts/orchestrator-deploy-hook.sh --deploy
|
||||
```
|
||||
|
||||
### Ручной rollback staging
|
||||
|
||||
```bash
|
||||
bash scripts/orchestrator-deploy-hook.sh --rollback
|
||||
```
|
||||
|
||||
## Коды выхода
|
||||
|
||||
| Код | Значение |
|
||||
|-----|------------------------------------------------------|
|
||||
| `0` | Деплой успешен, сервис здоров |
|
||||
| `1` | Деплой провалился; откат выполнен (или пропущен) |
|
||||
| `2` | Деплой провалился И откат тоже провалился (критично) |
|
||||
|
||||
## Логи
|
||||
|
||||
```
|
||||
/var/log/orchestrator/deploy-hook.log
|
||||
```
|
||||
|
||||
Каждая строка с UTC-таймстампом в формате `[2026-06-05T06:30:00Z]`.
|
||||
|
||||
## Разница с enduro-deploy-hook.sh
|
||||
|
||||
| Функция | enduro-deploy-hook.sh | orchestrator-deploy-hook.sh |
|
||||
|----------------------|-----------------------|-----------------------------|
|
||||
| Захват PREV_IMG | ✅ | ✅ |
|
||||
| git pull | ✅ | ✅ |
|
||||
| Рестарт | ✅ | ✅ |
|
||||
| Health-цикл (60с) | ❌ | ✅ 10×6с |
|
||||
| Авто-rollback | ❌ | ✅ |
|
||||
| Параметризация (env) | ❌ хардкод | ✅ дефолт=staging |
|
||||
| Compose profile | ❌ | ✅ --profile staging |
|
||||
85
docs/STAGING.md
Normal file
85
docs/STAGING.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# Staging Environment (ORCH-31)
|
||||
|
||||
Orchestrator supports a permanent **staging instance** running on port **8501** with a
|
||||
fully-isolated SQLite database. The staging instance shares the same codebase and
|
||||
Dockerfile as production but is started under the `staging` Docker Compose profile so it
|
||||
**never starts accidentally** during a normal `docker compose up -d`.
|
||||
|
||||
## Architecture
|
||||
|
||||
| | Production | Staging |
|
||||
|---|---|---|
|
||||
| Port | 8500 | 8501 |
|
||||
| Container name | `orchestrator` | `orchestrator-staging` |
|
||||
| DB (host path) | `./data/orchestrator.db` | `./data/staging/orchestrator.db` |
|
||||
| DB (container path) | `/app/data/orchestrator.db` | `/app/data/orchestrator.db` |
|
||||
| env file | `.env` | `.env.staging` |
|
||||
| Compose profile | *(default)* | `staging` |
|
||||
|
||||
DB isolation is achieved via a separate volume mount (`./data/staging:/app/data`), not by
|
||||
changing `ORCH_DB_PATH` — the container path stays identical while the host path is a
|
||||
different directory.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **`.env.staging`** — create from the template (see below). This file is **not committed**
|
||||
to the repo (it contains secrets). Copy and fill in values before first start.
|
||||
2. **`./data/staging/`** directory — created automatically on first container start.
|
||||
|
||||
### Create `.env.staging`
|
||||
|
||||
```bash
|
||||
cd /home/slin/repos/orchestrator
|
||||
cp .env.staging.example .env.staging
|
||||
# Edit .env.staging — fill in real tokens / secrets.
|
||||
# At Stage 1 (ORCH-31) you can reuse prod values; sandbox Plane project
|
||||
# and isolated Gitea webhook will be wired in ORCH-32.
|
||||
nano .env.staging
|
||||
```
|
||||
|
||||
## Starting Staging
|
||||
|
||||
```bash
|
||||
cd /home/slin/repos/orchestrator
|
||||
docker compose --profile staging up -d orchestrator-staging
|
||||
```
|
||||
|
||||
Check it is running:
|
||||
|
||||
```bash
|
||||
docker ps | grep orchestrator-staging
|
||||
curl -s http://localhost:8501/health | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Stopping Staging
|
||||
|
||||
```bash
|
||||
docker compose --profile staging stop orchestrator-staging
|
||||
# or remove the container entirely:
|
||||
docker compose --profile staging down orchestrator-staging
|
||||
```
|
||||
|
||||
## Normal `up -d` does NOT start staging
|
||||
|
||||
```bash
|
||||
# This starts ONLY the prod orchestrator (port 8500). Staging is NOT affected.
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
The `profiles: [staging]` directive in `docker-compose.yml` ensures staging is
|
||||
completely invisible to commands that do not pass `--profile staging`.
|
||||
|
||||
## Logs
|
||||
|
||||
```bash
|
||||
docker logs -f orchestrator-staging
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
|
||||
| Task | Description |
|
||||
|---|---|
|
||||
| **ORCH-31** *(this PR)* | Infra: compose service, .env template, gitignore, docs |
|
||||
| **ORCH-32** | Sandbox: isolated Plane project + Gitea repo for staging |
|
||||
| **ORCH-33** | Test suite running against staging endpoint |
|
||||
| **ORCH-34** | Deploy hook: promote `orchestrator:candidate` image to staging |
|
||||
136
docs/STAGING_CHECK.md
Normal file
136
docs/STAGING_CHECK.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# STAGING_CHECK.md — Инструкция по запуску staging check suite (ORCH-33)
|
||||
|
||||
## Что это
|
||||
|
||||
`scripts/staging_check.py` — самостоятельный скрипт проверки **живого** staging-стенда orchestrator (порт 8501). Не unit-тесты — реальные HTTP-вызовы против работающих сервисов.
|
||||
|
||||
Три блока проверок:
|
||||
|
||||
| Блок | Название | Что проверяет |
|
||||
|------|----------|---------------|
|
||||
| A | SMOKE | `/health`, `/queue`, `ORCH_STAGING=true` |
|
||||
| B | ACCESS | Plane sandbox (R), Gitea sandbox (R+push), реестр проектов |
|
||||
| C | E2E | Создать задачу → триггер конвейера → ветка + коммент → cleanup |
|
||||
|
||||
Exit code: **0** = все PASS, **non-zero** = есть FAIL.
|
||||
|
||||
---
|
||||
|
||||
## Требования к окружению
|
||||
|
||||
Скрипт читает токены/URL из env (те же переменные, что использует orchestrator):
|
||||
|
||||
| Переменная | Описание |
|
||||
|-----------|----------|
|
||||
| `ORCH_STAGING` | Должна быть `true` — защита от случайного запуска на проде |
|
||||
| `ORCH_PLANE_API_TOKEN` | Plane API token (`X-API-Key`) |
|
||||
| `ORCH_PLANE_API_URL` | Plane base URL **без** `/api/v1` (скрипт добавляет сам) |
|
||||
| `ORCH_PLANE_WORKSPACE_SLUG` | Workspace slug (`ag_proj`) |
|
||||
| `ORCH_GITEA_TOKEN` | Gitea token (`Authorization: token …`) |
|
||||
| `ORCH_GITEA_URL` | Gitea base URL (`http://localhost:3000`) |
|
||||
| `ORCH_PLANE_WEBHOOK_SECRET` | HMAC-секрет для подписи `/webhook/plane` (если пустой — без подписи) |
|
||||
|
||||
Все эти переменные **уже есть** внутри контейнера `orchestrator-staging`.
|
||||
|
||||
---
|
||||
|
||||
## Способы запуска
|
||||
|
||||
### 1. Внутри контейнера (рекомендуемый)
|
||||
|
||||
```bash
|
||||
docker exec orchestrator-staging \
|
||||
python3 /repos/orchestrator/scripts/staging_check.py --mode stub
|
||||
```
|
||||
|
||||
### 2. С хоста (если есть токены в env)
|
||||
|
||||
```bash
|
||||
export ORCH_STAGING=true
|
||||
export ORCH_PLANE_API_TOKEN=...
|
||||
# ... остальные переменные ...
|
||||
|
||||
python3 scripts/staging_check.py \
|
||||
--base-url http://localhost:8501 \
|
||||
--mode stub
|
||||
```
|
||||
|
||||
### 3. Из docker exec с передачей URL
|
||||
|
||||
```bash
|
||||
docker exec orchestrator-staging \
|
||||
python3 /repos/orchestrator/scripts/staging_check.py \
|
||||
--base-url http://localhost:8501 \
|
||||
--mode stub
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Режимы (`--mode`)
|
||||
|
||||
| Режим | Описание | Скорость |
|
||||
|-------|----------|----------|
|
||||
| `stub` (дефолт) | Проверяет **ранние артефакты** конвейера: ветка + QG-0-коммент. Создаются ДО запуска Claude CLI → быстро, детерминированно, без расхода LLM-кредитов. | ~30-90 сек |
|
||||
| `full-real` | Дополнительно ждёт реального завершения аналитика. Долго, расходует LLM-кредиты. | 5-15+ мин |
|
||||
|
||||
**Текущий дефолт: `stub`** — достаточен для проверки работоспособности стенда.
|
||||
|
||||
---
|
||||
|
||||
## Что проверяет блок C (E2E) и почему это безопасно
|
||||
|
||||
Порядок `start_pipeline` в коде orchestrator:
|
||||
1. Resolve проекта из реестра
|
||||
2. Получить name/description из Plane API (если в webhook пустые)
|
||||
3. **QG-0 гейт** (name ≥ 5 симв, description ≥ 20 симв)
|
||||
4. **Создать work_item_id + ветку в Gitea + начальные доки**
|
||||
5. **Записать строку задачи в БД**
|
||||
6. Поставить аналитика в очередь (вот тут Claude CLI)
|
||||
|
||||
Блок C проверяет **шаги 4-5**, аналитика (шаг 6) **не ждёт**.
|
||||
Тест-задача создаётся ТОЛЬКО в **SANDBOX** (`project_id 8c5a3025-...`),
|
||||
ветка создаётся ТОЛЬКО в **orchestrator-sandbox**.
|
||||
|
||||
### CLEANUP (обязателен)
|
||||
|
||||
`try/finally` гарантирует удаление тестовых артефактов:
|
||||
- Удаляет ветку из `orchestrator-sandbox`
|
||||
- Удаляет задачу из Plane SANDBOX
|
||||
|
||||
Cleanup отрабатывает даже при падении e2e.
|
||||
|
||||
---
|
||||
|
||||
## Принцип HMAC-подписи
|
||||
|
||||
Скрипт читает `ORCH_PLANE_WEBHOOK_SECRET` из env и формирует подпись:
|
||||
```python
|
||||
hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
|
||||
```
|
||||
Передаёт как заголовок `X-Plane-Signature`. Алгоритм совпадает с `verify_plane_signature` в `src/webhooks/plane.py`.
|
||||
|
||||
---
|
||||
|
||||
## Изолированность от прода
|
||||
|
||||
| Проверка | Гарантия |
|
||||
|---------|---------|
|
||||
| A3 `ORCH_STAGING=true` | При false — abort до деструктивных блоков |
|
||||
| B6 Реестр без боевых | ET/ORCH project_id absent в `known_plane_project_ids()` |
|
||||
| C: only SANDBOX project_id | Webhook payload указывает только `8c5a3025-...` |
|
||||
| C: only orchestrator-sandbox repo | Gitea operations на `admin/orchestrator-sandbox` |
|
||||
| C: cleanup в finally | Артефакты удаляются даже при ошибке |
|
||||
|
||||
---
|
||||
|
||||
## Добавление в деплой-хук
|
||||
|
||||
```bash
|
||||
# В deploy.sh, после docker-compose up -d orchestrator-staging
|
||||
docker exec orchestrator-staging \
|
||||
python3 /repos/orchestrator/scripts/staging_check.py --mode stub
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Staging check FAILED — rolling back"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
176
scripts/orchestrator-deploy-hook.sh
Executable file
176
scripts/orchestrator-deploy-hook.sh
Executable file
@@ -0,0 +1,176 @@
|
||||
#!/bin/bash
|
||||
# Deploy hook for orchestrator
|
||||
# Supports --deploy (default) and --rollback modes.
|
||||
# Adds health-check loop + automatic rollback if new deploy is unhealthy.
|
||||
#
|
||||
# Parametrised via env vars (defaults are STAGING — never prod):
|
||||
# TARGET_SERVICE - docker-compose service name (default: orchestrator-staging)
|
||||
# TARGET_PORT - health check port (default: 8501)
|
||||
# TARGET_IMAGE - image name for retag (default: orchestrator-orchestrator-staging)
|
||||
# COMPOSE_PROFILE - docker compose profile (default: staging)
|
||||
# PREV_IMAGE_FILE - path to prev-image snapshot (default: $REPO/.deploy-prev-image-staging)
|
||||
# LOG - log file path (default: /var/log/orchestrator/deploy-hook.log)
|
||||
#
|
||||
# Usage:
|
||||
# ./orchestrator-deploy-hook.sh [--deploy] # normal deploy (default)
|
||||
# ./orchestrator-deploy-hook.sh --rollback # manual rollback
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO=/home/slin/repos/orchestrator
|
||||
|
||||
# ---- Defaults (STAGING — safe) ---------------------------------------------
|
||||
TARGET_SERVICE="${TARGET_SERVICE:-orchestrator-staging}"
|
||||
TARGET_PORT="${TARGET_PORT:-8501}"
|
||||
TARGET_IMAGE="${TARGET_IMAGE:-orchestrator-orchestrator-staging}"
|
||||
COMPOSE_PROFILE="${COMPOSE_PROFILE:-staging}"
|
||||
PREV_IMAGE_FILE="${PREV_IMAGE_FILE:-$REPO/.deploy-prev-image-staging}"
|
||||
|
||||
# ---- Log setup -------------------------------------------------------------
|
||||
LOG_DIR=/var/log/orchestrator
|
||||
if mkdir -p "$LOG_DIR" 2>/dev/null; then
|
||||
LOG="${LOG:-$LOG_DIR/deploy-hook.log}"
|
||||
else
|
||||
LOG="${LOG:-$REPO/deploy-hook.log}"
|
||||
fi
|
||||
|
||||
log() {
|
||||
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $*" | tee -a "$LOG"
|
||||
}
|
||||
|
||||
log "Deploy hook called: target=$TARGET_SERVICE port=$TARGET_PORT args=$*"
|
||||
|
||||
cd "$REPO"
|
||||
|
||||
# ============================================================================
|
||||
# HEALTH CHECK helper
|
||||
# Args: max_attempts sleep_sec label
|
||||
# Returns 0 if healthy within attempts, 1 otherwise
|
||||
# ============================================================================
|
||||
health_check() {
|
||||
local max_attempts="$1"
|
||||
local sleep_sec="$2"
|
||||
local label="${3:-health-check}"
|
||||
local attempt=0
|
||||
while [[ $attempt -lt $max_attempts ]]; do
|
||||
attempt=$(( attempt + 1 ))
|
||||
log "$label: attempt $attempt/$max_attempts - GET http://localhost:$TARGET_PORT/health"
|
||||
local http_code body
|
||||
body=$(curl -s --max-time 5 "http://localhost:$TARGET_PORT/health" 2>/dev/null || true)
|
||||
http_code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://localhost:$TARGET_PORT/health" 2>/dev/null || echo "000")
|
||||
if [[ "$http_code" == "200" ]] && echo "$body" | grep -q '"status":"ok"'; then
|
||||
log "$label: OK (HTTP $http_code, body=$body)"
|
||||
return 0
|
||||
fi
|
||||
log "$label: not ready yet (HTTP $http_code, body=$body)"
|
||||
if [[ $attempt -lt $max_attempts ]]; then
|
||||
sleep "$sleep_sec"
|
||||
fi
|
||||
done
|
||||
log "$label: FAILED after $max_attempts attempts"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# ROLLBACK helper (also called for auto-rollback after bad deploy)
|
||||
# ============================================================================
|
||||
do_rollback() {
|
||||
log "ROLLBACK: checking $PREV_IMAGE_FILE"
|
||||
if [[ ! -s "$PREV_IMAGE_FILE" ]]; then
|
||||
log "ROLLBACK: no previous image recorded - rollback skipped (exit 1)"
|
||||
return 1
|
||||
fi
|
||||
local prev_img
|
||||
prev_img=$(cat "$PREV_IMAGE_FILE")
|
||||
if [[ -z "$prev_img" ]]; then
|
||||
log "ROLLBACK: PREV_IMAGE_FILE is empty - rollback skipped (exit 1)"
|
||||
return 1
|
||||
fi
|
||||
if ! docker image inspect "$prev_img" >/dev/null 2>&1; then
|
||||
log "ROLLBACK: recorded image '$prev_img' not found locally - rollback skipped (exit 1)"
|
||||
return 1
|
||||
fi
|
||||
log "ROLLBACK: retagging $prev_img -> $TARGET_IMAGE"
|
||||
docker tag "$prev_img" "$TARGET_IMAGE" >> "$LOG" 2>&1
|
||||
log "ROLLBACK: restarting $TARGET_SERVICE on previous image"
|
||||
if [[ -n "$COMPOSE_PROFILE" ]]; then
|
||||
docker compose --profile "$COMPOSE_PROFILE" up -d --no-build "$TARGET_SERVICE" >> "$LOG" 2>&1
|
||||
else
|
||||
docker compose up -d --no-build "$TARGET_SERVICE" >> "$LOG" 2>&1
|
||||
fi
|
||||
log "ROLLBACK: container restarted, running post-rollback health check (5x3s)"
|
||||
if health_check 5 3 "ROLLBACK-health"; then
|
||||
log "ROLLBACK: service is healthy on previous image ($prev_img)"
|
||||
return 0
|
||||
else
|
||||
log "ROLLBACK: ROLLBACK ALSO FAILED - service still unhealthy after restoring $prev_img"
|
||||
return 2
|
||||
fi
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# MANUAL --rollback mode
|
||||
# ============================================================================
|
||||
if [[ "${1:-}" == "--rollback" ]]; then
|
||||
log "Manual ROLLBACK requested"
|
||||
if do_rollback; then
|
||||
log "Manual ROLLBACK succeeded"
|
||||
exit 0
|
||||
else
|
||||
log "Manual ROLLBACK failed"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# NORMAL DEPLOY mode (--deploy or no argument)
|
||||
# ============================================================================
|
||||
|
||||
# 1. Capture currently running image BEFORE restart (best-effort)
|
||||
PREV_IMG=""
|
||||
SVC_CID=$(docker compose --profile "$COMPOSE_PROFILE" ps -q "$TARGET_SERVICE" 2>/dev/null || true)
|
||||
if [[ -n "$SVC_CID" ]]; then
|
||||
PREV_IMG=$(docker inspect --format '{{.Image}}' "$SVC_CID" 2>/dev/null || true)
|
||||
fi
|
||||
if [[ -n "$PREV_IMG" ]]; then
|
||||
echo "$PREV_IMG" > "$PREV_IMAGE_FILE"
|
||||
log "Saved previous image: $PREV_IMG -> $PREV_IMAGE_FILE"
|
||||
else
|
||||
log "No previous image captured (first deploy or service not running?)"
|
||||
fi
|
||||
|
||||
# 2. Pull latest code
|
||||
log "git pull origin main"
|
||||
git pull origin main >> "$LOG" 2>&1
|
||||
|
||||
# 3. Restart service
|
||||
log "Starting $TARGET_SERVICE (profile=$COMPOSE_PROFILE)"
|
||||
if [[ -n "$COMPOSE_PROFILE" ]]; then
|
||||
docker compose --profile "$COMPOSE_PROFILE" up -d --no-build "$TARGET_SERVICE" >> "$LOG" 2>&1
|
||||
else
|
||||
docker compose up -d --no-build "$TARGET_SERVICE" >> "$LOG" 2>&1
|
||||
fi
|
||||
log "$TARGET_SERVICE restarted"
|
||||
|
||||
# 4. Health-check loop: 10 attempts x 6 seconds = up to 60s
|
||||
log "Starting health-check: 10 attempts x 6s (max 60s)"
|
||||
if health_check 10 6 "deploy-health"; then
|
||||
log "Deploy SUCCESS: $TARGET_SERVICE healthy on port $TARGET_PORT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# 5. Health failed -> AUTO ROLLBACK
|
||||
log "deploy FAILED: health not ok after 60s - initiating AUTO ROLLBACK"
|
||||
rollback_rc=0
|
||||
do_rollback || rollback_rc=$?
|
||||
|
||||
if [[ $rollback_rc -eq 0 ]]; then
|
||||
log "deploy FAILED, rolled back to previous image successfully - exit 1"
|
||||
exit 1
|
||||
elif [[ $rollback_rc -eq 2 ]]; then
|
||||
log "deploy FAILED, ROLLBACK ALSO FAILED - service may be down - exit 2"
|
||||
exit 2
|
||||
else
|
||||
log "deploy FAILED, rollback skipped (no previous image) - exit 1"
|
||||
exit 1
|
||||
fi
|
||||
639
scripts/staging_check.py
Normal file
639
scripts/staging_check.py
Normal file
@@ -0,0 +1,639 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
staging_check.py — Live staging-stand health & e2e check suite (ORCH-33).
|
||||
|
||||
Checks:
|
||||
Block A — SMOKE (health/queue, correct env)
|
||||
Block B — ACCESS (read-only calls to Plane sandbox + Gitea sandbox + registry)
|
||||
Block C — E2E (create task in SANDBOX → trigger pipeline via /webhook/plane
|
||||
→ verify branch + job enqueued → CLEANUP in finally)
|
||||
|
||||
Usage (inside the container or with correct env set):
|
||||
python3 scripts/staging_check.py [--base-url http://localhost:8501] [--mode stub|full-real]
|
||||
|
||||
Exit code: 0 = all PASS, non-zero = at least one FAIL.
|
||||
|
||||
NOTE on modes:
|
||||
stub — default; checks early pipeline artifacts (branch + analyst job
|
||||
enqueued) created BEFORE Claude CLI is invoked.
|
||||
Fast, deterministic, no LLM spend.
|
||||
full-real — additionally waits for the analyst agent to finish (long, costs
|
||||
credits). Not the default.
|
||||
|
||||
NOTE on Plane comments (403):
|
||||
The orchestrator posts the "🔍 Analyst запущен" comment using per-agent bot
|
||||
tokens (ORCH_PLANE_BOT_ANALYST). These bot accounts must be added as members
|
||||
of every Plane project they comment on. In staging the sandbox project was
|
||||
created after the bots were provisioned → the bots are not yet members of
|
||||
SANDBOX → add_comment returns 403 Forbidden.
|
||||
|
||||
This is a known infrastructure limitation of the staging sandbox, NOT a bug
|
||||
in the pipeline itself. C9b therefore verifies pipeline success via the
|
||||
staging job queue (/queue → recent) instead of Plane comments: the analyst
|
||||
job is enqueued BEFORE the add_comment call and its presence in the queue
|
||||
proves the pipeline ran through correctly.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import datetime
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Colour helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
_BOLD = "\033[1m"
|
||||
_GREEN = "\033[32m"
|
||||
_RED = "\033[31m"
|
||||
_YELLOW = "\033[33m"
|
||||
_RESET = "\033[0m"
|
||||
|
||||
|
||||
def _ok(msg: str) -> str:
|
||||
return f" {_GREEN}✓ PASS{_RESET} {msg}"
|
||||
|
||||
|
||||
def _fail(msg: str) -> str:
|
||||
return f" {_RED}✗ FAIL{_RESET} {msg}"
|
||||
|
||||
|
||||
def _info(msg: str) -> str:
|
||||
return f" {_YELLOW}·{_RESET} {msg}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Low-level HTTP helpers (stdlib only — no requests/httpx in scripts/)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _http(method: str, url: str, headers: dict | None = None,
|
||||
body: bytes | None = None, timeout: int = 15) -> tuple[int, bytes]:
|
||||
"""Simple HTTP wrapper. Returns (status_code, response_body)."""
|
||||
req = urllib.request.Request(url, data=body, headers=headers or {}, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return resp.status, resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read()
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"{method} {url} → {e}") from e
|
||||
|
||||
|
||||
def _get(url: str, headers: dict | None = None, timeout: int = 15) -> tuple[int, dict]:
|
||||
status, body = _http("GET", url, headers=headers, timeout=timeout)
|
||||
try:
|
||||
data = json.loads(body)
|
||||
except Exception:
|
||||
data = {"_raw": body.decode(errors="replace")}
|
||||
return status, data
|
||||
|
||||
|
||||
def _post(url: str, headers: dict | None = None, payload: dict | None = None,
|
||||
raw_body: bytes | None = None, timeout: int = 15) -> tuple[int, dict]:
|
||||
if raw_body is not None:
|
||||
body = raw_body
|
||||
h = dict(headers or {})
|
||||
if "Content-Type" not in h:
|
||||
h["Content-Type"] = "application/json"
|
||||
else:
|
||||
body = json.dumps(payload or {}).encode()
|
||||
h = dict(headers or {})
|
||||
h["Content-Type"] = "application/json"
|
||||
status, resp_body = _http("POST", url, headers=h, body=body, timeout=timeout)
|
||||
try:
|
||||
data = json.loads(resp_body)
|
||||
except Exception:
|
||||
data = {"_raw": resp_body.decode(errors="replace")}
|
||||
return status, data
|
||||
|
||||
|
||||
def _patch(url: str, headers: dict | None = None, payload: dict | None = None,
|
||||
timeout: int = 15) -> tuple[int, dict]:
|
||||
body = json.dumps(payload or {}).encode()
|
||||
h = dict(headers or {})
|
||||
h["Content-Type"] = "application/json"
|
||||
status, resp_body = _http("PATCH", url, headers=h, body=body, timeout=timeout)
|
||||
try:
|
||||
data = json.loads(resp_body)
|
||||
except Exception:
|
||||
data = {"_raw": resp_body.decode(errors="replace")}
|
||||
return status, data
|
||||
|
||||
|
||||
def _delete(url: str, headers: dict | None = None, timeout: int = 15) -> int:
|
||||
status, _ = _http("DELETE", url, headers=headers, timeout=timeout)
|
||||
return status
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HMAC helper for /webhook/plane
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _sign_payload(secret: str, body: bytes) -> str:
|
||||
"""Compute HMAC-SHA256 signature — matches verify_plane_signature in plane.py."""
|
||||
return hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Result tracking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Results:
|
||||
def __init__(self):
|
||||
self._items: list[tuple[str, bool, str]] = [] # (label, passed, detail)
|
||||
|
||||
def add(self, label: str, passed: bool, detail: str = ""):
|
||||
self._items.append((label, passed, detail))
|
||||
line = _ok(label) if passed else _fail(label)
|
||||
if detail:
|
||||
line += f" [{detail}]"
|
||||
print(line)
|
||||
|
||||
def summary(self) -> bool:
|
||||
passed = sum(1 for _, ok, _ in self._items if ok)
|
||||
total = len(self._items)
|
||||
all_ok = passed == total
|
||||
colour = _GREEN if all_ok else _RED
|
||||
print()
|
||||
print(f"{_BOLD}{'='*60}{_RESET}")
|
||||
print(f"{colour}{_BOLD} RESULT: {passed}/{total} checks PASS{_RESET}")
|
||||
print(f"{_BOLD}{'='*60}{_RESET}")
|
||||
return all_ok
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Block A — SMOKE
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def block_a(base: str, results: Results):
|
||||
print(f"\n{_BOLD}[Block A] SMOKE{_RESET}")
|
||||
|
||||
# A1 — /health
|
||||
try:
|
||||
status, data = _get(f"{base}/health")
|
||||
ok = status == 200 and data.get("status") == "ok"
|
||||
results.add("A1 GET /health → 200 status=ok", ok,
|
||||
f"HTTP {status}, body={data}")
|
||||
except Exception as e:
|
||||
results.add("A1 GET /health → 200 status=ok", False, str(e))
|
||||
|
||||
# A2 — /queue
|
||||
try:
|
||||
status, data = _get(f"{base}/queue")
|
||||
ok = (status == 200
|
||||
and "counts" in data
|
||||
and "max_concurrency" in data
|
||||
and "resilience" in data)
|
||||
results.add("A2 GET /queue → 200 with counts/max_concurrency/resilience", ok,
|
||||
f"HTTP {status}, keys={list(data.keys())}")
|
||||
except Exception as e:
|
||||
results.add("A2 GET /queue → 200 with counts/max_concurrency/resilience", False, str(e))
|
||||
|
||||
# A3 — ORCH_STAGING=true in env (guard against hitting prod)
|
||||
staging_flag = os.environ.get("ORCH_STAGING", "").lower()
|
||||
ok = staging_flag == "true"
|
||||
results.add("A3 ORCH_STAGING=true (not prod)", ok,
|
||||
f"ORCH_STAGING={os.environ.get('ORCH_STAGING', '<unset>')}")
|
||||
if not ok:
|
||||
print(_fail(" ⛔ Safety abort: ORCH_STAGING is not 'true'. "
|
||||
"This might be prod. Skipping destructive blocks B/C."))
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Block B — ACCESS
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SANDBOX_PROJECT_ID = "8c5a3025-4f9d-4190-b79f-fa06276bb27e"
|
||||
PROD_ET_PROJECT_ID = "7a79f0a9-5278-49cd-9007-9a338f238f9c"
|
||||
PROD_ORCH_PROJECT_ID = "8da6aa25-a60e-44d6-a1e2-d8ae59aa7d6a"
|
||||
|
||||
|
||||
def block_b(results: Results):
|
||||
print(f"\n{_BOLD}[Block B] ACCESS{_RESET}")
|
||||
|
||||
plane_token = os.environ.get("ORCH_PLANE_API_TOKEN", "")
|
||||
plane_base_env = os.environ.get("ORCH_PLANE_API_URL", "http://localhost:8091")
|
||||
# env stores URL WITHOUT /api/v1 — add it ourselves
|
||||
plane_base = plane_base_env.rstrip("/") + "/api/v1"
|
||||
workspace = os.environ.get("ORCH_PLANE_WORKSPACE_SLUG", "ag_proj")
|
||||
gitea_token = os.environ.get("ORCH_GITEA_TOKEN", "")
|
||||
gitea_base = os.environ.get("ORCH_GITEA_URL", "http://localhost:3000")
|
||||
|
||||
plane_headers = {"X-API-Key": plane_token}
|
||||
gitea_headers = {"Authorization": f"token {gitea_token}"}
|
||||
|
||||
# B4 — Plane: list projects, sandbox id present
|
||||
try:
|
||||
url = f"{plane_base}/workspaces/{workspace}/projects/"
|
||||
status, data = _get(url, headers=plane_headers)
|
||||
if status == 200:
|
||||
# API may return a list or {"results": [...]}
|
||||
projects = data.get("results", data) if isinstance(data, dict) else data
|
||||
if isinstance(projects, list):
|
||||
ids = {p.get("id", "") for p in projects}
|
||||
else:
|
||||
ids = set()
|
||||
ok = SANDBOX_PROJECT_ID in ids
|
||||
results.add("B4 Plane: sandbox project accessible", ok,
|
||||
f"HTTP {status}, found {len(ids)} project(s), sandbox={'YES' if ok else 'NO'}")
|
||||
else:
|
||||
results.add("B4 Plane: sandbox project accessible", False,
|
||||
f"HTTP {status}")
|
||||
except Exception as e:
|
||||
results.add("B4 Plane: sandbox project accessible", False, str(e))
|
||||
|
||||
# B5 — Gitea: sandbox repo accessible, push=true
|
||||
try:
|
||||
url = f"{gitea_base}/api/v1/repos/admin/orchestrator-sandbox"
|
||||
status, data = _get(url, headers=gitea_headers)
|
||||
push_ok = data.get("permissions", {}).get("push", False) if status == 200 else False
|
||||
ok = status == 200 and push_ok
|
||||
results.add("B5 Gitea: orchestrator-sandbox accessible, push=true", ok,
|
||||
f"HTTP {status}, permissions={data.get('permissions')}")
|
||||
except Exception as e:
|
||||
results.add("B5 Gitea: orchestrator-sandbox accessible, push=true", False, str(e))
|
||||
|
||||
# B6 — Registry: sandbox in known IDs, prod ET/ORCH NOT in known IDs
|
||||
try:
|
||||
# Import from inside the container (script runs in /repos/orchestrator context)
|
||||
sys.path.insert(0, "/repos/orchestrator")
|
||||
# Force reload to pick up container env
|
||||
import importlib
|
||||
if "src.projects" in sys.modules:
|
||||
importlib.reload(sys.modules["src.projects"])
|
||||
from src.projects import known_plane_project_ids
|
||||
known = known_plane_project_ids()
|
||||
sandbox_present = SANDBOX_PROJECT_ID in known
|
||||
et_absent = PROD_ET_PROJECT_ID not in known
|
||||
orch_absent = PROD_ORCH_PROJECT_ID not in known
|
||||
ok = sandbox_present and et_absent and orch_absent
|
||||
detail = (
|
||||
f"sandbox={'YES' if sandbox_present else 'NO'}, "
|
||||
f"prod-ET={'NO(good)' if et_absent else 'YES(BAD!)'}, "
|
||||
f"prod-ORCH={'NO(good)' if orch_absent else 'YES(BAD!)'}"
|
||||
)
|
||||
results.add("B6 Registry: sandbox present, prod ET/ORCH absent", ok, detail)
|
||||
except Exception as e:
|
||||
results.add("B6 Registry: sandbox present, prod ET/ORCH absent", False, str(e))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Block C — E2E
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
IN_PROGRESS_STATE_ID = "b873d9eb-993c-48cd-97ac-99a9b1623967"
|
||||
|
||||
# Path to staging SQLite DB inside the container
|
||||
STAGING_DB_PATH = os.environ.get("ORCH_DB_PATH", "/app/data/orchestrator.db")
|
||||
|
||||
|
||||
def _make_webhook_payload(issue_id: str, issue_name: str, issue_desc: str) -> dict:
|
||||
"""Build the minimal webhook payload that triggers start_pipeline."""
|
||||
return {
|
||||
"event": "issue",
|
||||
"action": "updated",
|
||||
"data": {
|
||||
"id": issue_id,
|
||||
"name": issue_name,
|
||||
"description_stripped": issue_desc,
|
||||
"project": SANDBOX_PROJECT_ID,
|
||||
"state": {
|
||||
"id": IN_PROGRESS_STATE_ID,
|
||||
"name": "In Progress",
|
||||
"group": "started",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _poll(fn, timeout: int = 60, interval: int = 3, label: str = ""):
|
||||
"""Poll fn() until it returns truthy or timeout expires."""
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
result = fn()
|
||||
if result:
|
||||
return result
|
||||
if label:
|
||||
print(_info(f" waiting... ({label})"))
|
||||
time.sleep(interval)
|
||||
return None
|
||||
|
||||
|
||||
def _cleanup_staging_db(plane_issue_id: str):
|
||||
"""Delete the test task row from staging SQLite DB."""
|
||||
if not plane_issue_id:
|
||||
print(_info("CLEANUP DB: no issue_id to clean"))
|
||||
return
|
||||
try:
|
||||
import sqlite3
|
||||
conn = sqlite3.connect(STAGING_DB_PATH)
|
||||
cur = conn.execute(
|
||||
"DELETE FROM tasks WHERE plane_id = ?", (plane_issue_id,)
|
||||
)
|
||||
deleted = cur.rowcount
|
||||
conn.commit()
|
||||
conn.close()
|
||||
if deleted:
|
||||
print(_ok(f"CLEANUP DB: deleted {deleted} task row(s) for plane_id={plane_issue_id}"))
|
||||
else:
|
||||
print(_info(f"CLEANUP DB: no task row found for plane_id={plane_issue_id}"))
|
||||
except Exception as e:
|
||||
print(_fail(f"CLEANUP DB: error: {e}"))
|
||||
|
||||
|
||||
def _cleanup_staging_jobs(plane_issue_id: str):
|
||||
"""Delete job queue rows for the test task from staging SQLite DB."""
|
||||
if not plane_issue_id:
|
||||
return
|
||||
try:
|
||||
import sqlite3
|
||||
conn = sqlite3.connect(STAGING_DB_PATH)
|
||||
# Find task ids for this plane_id first
|
||||
task_rows = conn.execute(
|
||||
"SELECT id FROM tasks WHERE plane_id = ?", (plane_issue_id,)
|
||||
).fetchall()
|
||||
if task_rows:
|
||||
task_ids = [r[0] for r in task_rows]
|
||||
placeholders = ",".join("?" * len(task_ids))
|
||||
cur = conn.execute(
|
||||
f"DELETE FROM jobs WHERE task_id IN ({placeholders})", task_ids
|
||||
)
|
||||
deleted = cur.rowcount
|
||||
conn.commit()
|
||||
if deleted:
|
||||
print(_ok(f"CLEANUP DB: deleted {deleted} job row(s) for task_ids={task_ids}"))
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
print(_fail(f"CLEANUP DB jobs: error: {e}"))
|
||||
|
||||
|
||||
def _cleanup_dedup(plane_issue_id: str, wh_body_sha: str | None = None):
|
||||
"""Remove dedup event entries for the test webhook delivery."""
|
||||
if not wh_body_sha:
|
||||
return
|
||||
try:
|
||||
import sqlite3
|
||||
conn = sqlite3.connect(STAGING_DB_PATH)
|
||||
cur = conn.execute(
|
||||
"DELETE FROM events_dedup WHERE delivery_id = ?", (wh_body_sha,)
|
||||
)
|
||||
deleted = cur.rowcount
|
||||
conn.commit()
|
||||
conn.close()
|
||||
if deleted:
|
||||
print(_ok(f"CLEANUP DB: removed {deleted} dedup entry"))
|
||||
except Exception as e:
|
||||
# dedup table might not exist or different schema — not critical
|
||||
print(_info(f"CLEANUP DB dedup: {e}"))
|
||||
|
||||
|
||||
def block_c(base: str, results: Results, mode: str):
|
||||
print(f"\n{_BOLD}[Block C] E2E (mode={mode}){_RESET}")
|
||||
|
||||
plane_token = os.environ.get("ORCH_PLANE_API_TOKEN", "")
|
||||
plane_base_env = os.environ.get("ORCH_PLANE_API_URL", "http://localhost:8091")
|
||||
plane_base = plane_base_env.rstrip("/") + "/api/v1"
|
||||
workspace = os.environ.get("ORCH_PLANE_WORKSPACE_SLUG", "ag_proj")
|
||||
gitea_token = os.environ.get("ORCH_GITEA_TOKEN", "")
|
||||
gitea_base = os.environ.get("ORCH_GITEA_URL", "http://localhost:3000")
|
||||
webhook_secret = os.environ.get("ORCH_PLANE_WEBHOOK_SECRET", "")
|
||||
|
||||
plane_headers = {"X-API-Key": plane_token}
|
||||
gitea_headers = {"Authorization": f"token {gitea_token}"}
|
||||
|
||||
ts = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%S")
|
||||
issue_name = f"[staging-check] e2e {ts}"
|
||||
issue_desc = (
|
||||
"Automated e2e check created by staging_check.py. "
|
||||
"This task tests the live staging pipeline end-to-end. "
|
||||
"Safe to delete — cleanup runs in finally block."
|
||||
)
|
||||
|
||||
issue_id = None
|
||||
branch_name = None
|
||||
wh_body_bytes = None
|
||||
|
||||
try:
|
||||
# C7 — Create task in Plane SANDBOX
|
||||
print(_info(f"C7: Creating issue in SANDBOX project..."))
|
||||
url = f"{plane_base}/workspaces/{workspace}/projects/{SANDBOX_PROJECT_ID}/issues/"
|
||||
status, data = _post(url, headers=plane_headers, payload={
|
||||
"name": issue_name,
|
||||
"description_html": f"<p>{issue_desc}</p>",
|
||||
"description_stripped": issue_desc,
|
||||
})
|
||||
issue_id = data.get("id")
|
||||
ok = status in (200, 201) and bool(issue_id)
|
||||
results.add("C7 Create issue in Plane SANDBOX", ok,
|
||||
f"HTTP {status}, issue_id={issue_id}")
|
||||
if not ok:
|
||||
print(_fail(f" Cannot continue C8-C9 without issue. body={data}"))
|
||||
results.add("C8 Trigger pipeline via /webhook/plane", False, "skipped: C7 failed")
|
||||
results.add("C9a Branch appears in orchestrator-sandbox", False, "skipped")
|
||||
results.add("C9b Analyst job enqueued in staging queue", False, "skipped")
|
||||
return
|
||||
|
||||
# Small delay to let Plane finish persisting the issue
|
||||
time.sleep(2)
|
||||
|
||||
# C8 — Trigger pipeline via direct POST to /webhook/plane
|
||||
print(_info(f"C8: Triggering pipeline via POST /webhook/plane ..."))
|
||||
wh_payload = _make_webhook_payload(issue_id, issue_name, issue_desc)
|
||||
wh_body_bytes = json.dumps(wh_payload).encode()
|
||||
|
||||
wh_headers = {"Content-Type": "application/json"}
|
||||
if webhook_secret:
|
||||
sig = _sign_payload(webhook_secret, wh_body_bytes)
|
||||
wh_headers["X-Plane-Signature"] = sig
|
||||
print(_info(f" Using HMAC signature (secret len={len(webhook_secret)})"))
|
||||
else:
|
||||
print(_info(" No webhook secret configured, sending without signature"))
|
||||
|
||||
status, resp = _post(f"{base}/webhook/plane",
|
||||
headers=wh_headers,
|
||||
raw_body=wh_body_bytes)
|
||||
ok = status == 200 and resp.get("status") in ("accepted",)
|
||||
results.add("C8 Trigger pipeline via /webhook/plane", ok,
|
||||
f"HTTP {status}, resp={resp}")
|
||||
if not ok:
|
||||
print(_fail(f" Pipeline trigger failed. Cannot verify C9."))
|
||||
results.add("C9a Branch appears in orchestrator-sandbox", False, "skipped: C8 failed")
|
||||
results.add("C9b Analyst job enqueued in staging queue", False, "skipped: C8 failed")
|
||||
return
|
||||
|
||||
# C9a — Poll for branch in Gitea orchestrator-sandbox
|
||||
print(_info("C9a: Polling for branch in orchestrator-sandbox (up to 60s)..."))
|
||||
|
||||
def _check_branch():
|
||||
try:
|
||||
burl = f"{gitea_base}/api/v1/repos/admin/orchestrator-sandbox/branches"
|
||||
s, bdata = _get(burl, headers=gitea_headers)
|
||||
if s != 200:
|
||||
return None
|
||||
branches = bdata if isinstance(bdata, list) else bdata.get("results", [])
|
||||
for b in branches:
|
||||
bname = b.get("name", "")
|
||||
# Branch name: feature/SANDBOX-NNN-staging-check-...
|
||||
if "feature/" in bname and "staging-check" in bname:
|
||||
return bname
|
||||
return None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
branch_name = _poll(_check_branch, timeout=60, interval=3,
|
||||
label="waiting for branch")
|
||||
ok = bool(branch_name)
|
||||
results.add("C9a Branch appears in orchestrator-sandbox", ok,
|
||||
f"branch={branch_name or 'not found'}")
|
||||
|
||||
# C9b — Verify analyst job was enqueued via staging /queue
|
||||
# NOTE: The orchestrator posts a "🔍 Analyst запущен" comment to Plane using
|
||||
# per-agent bot tokens (ORCH_PLANE_BOT_ANALYST). In staging, the sandbox
|
||||
# project was created after the bot accounts were provisioned, so the bots are
|
||||
# not yet members of the SANDBOX project → add_comment returns 403 Forbidden.
|
||||
# This is a known staging infrastructure limitation (not a pipeline bug).
|
||||
# We therefore verify pipeline success via /queue (recent jobs): the analyst
|
||||
# job is enqueued BEFORE the add_comment call, so its presence in the queue
|
||||
# confirms the pipeline ran through to job dispatch.
|
||||
print(_info("C9b: Checking staging job queue for analyst job (up to 30s)..."))
|
||||
print(_info(" (Plane comment check skipped: bot-tokens not added to SANDBOX project)"))
|
||||
|
||||
def _check_queue():
|
||||
try:
|
||||
s, qdata = _get(f"{base}/queue")
|
||||
if s != 200:
|
||||
return None
|
||||
recent = qdata.get("recent", [])
|
||||
for job in recent:
|
||||
if (job.get("agent") == "analyst"
|
||||
and job.get("repo") == "orchestrator-sandbox"
|
||||
and issue_name in (job.get("task_content") or "")):
|
||||
return job
|
||||
return None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
analyst_job = _poll(_check_queue, timeout=30, interval=2,
|
||||
label="waiting for analyst job in queue")
|
||||
ok = bool(analyst_job)
|
||||
detail = ""
|
||||
if analyst_job:
|
||||
detail = (f"job_id={analyst_job.get('id')}, "
|
||||
f"status={analyst_job.get('status')}, "
|
||||
f"agent={analyst_job.get('agent')}")
|
||||
results.add("C9b Analyst job enqueued in staging queue", ok, detail)
|
||||
|
||||
finally:
|
||||
# C10 — CLEANUP (always runs)
|
||||
print(f"\n{_BOLD}[CLEANUP]{_RESET}")
|
||||
_cleanup(
|
||||
plane_base=plane_base,
|
||||
workspace=workspace,
|
||||
gitea_base=gitea_base,
|
||||
plane_headers=plane_headers,
|
||||
gitea_headers=gitea_headers,
|
||||
issue_id=issue_id,
|
||||
branch_name=branch_name,
|
||||
wh_body_bytes=wh_body_bytes,
|
||||
)
|
||||
|
||||
|
||||
def _cleanup(plane_base, workspace, gitea_base, plane_headers, gitea_headers,
|
||||
issue_id, branch_name, wh_body_bytes=None):
|
||||
"""Delete test branch in Gitea, test issue in Plane SANDBOX, and DB rows."""
|
||||
|
||||
# Delete branch in Gitea
|
||||
if branch_name:
|
||||
try:
|
||||
burl = (f"{gitea_base}/api/v1/repos/admin/orchestrator-sandbox"
|
||||
f"/branches/{urllib.parse.quote(branch_name, safe='')}")
|
||||
s = _delete(burl, headers=gitea_headers)
|
||||
if s in (200, 204, 404):
|
||||
print(_ok(f"CLEANUP: deleted branch {branch_name!r} (HTTP {s})"))
|
||||
else:
|
||||
print(_fail(f"CLEANUP: delete branch returned HTTP {s}"))
|
||||
except Exception as e:
|
||||
print(_fail(f"CLEANUP: delete branch error: {e}"))
|
||||
else:
|
||||
print(_info("CLEANUP: no branch to delete"))
|
||||
|
||||
# Delete issue in Plane SANDBOX
|
||||
if issue_id:
|
||||
try:
|
||||
iurl = (f"{plane_base}/workspaces/{workspace}/projects/"
|
||||
f"{SANDBOX_PROJECT_ID}/issues/{issue_id}/")
|
||||
s = _delete(iurl, headers=plane_headers)
|
||||
if s in (200, 204, 404):
|
||||
print(_ok(f"CLEANUP: deleted Plane issue {issue_id} (HTTP {s})"))
|
||||
else:
|
||||
print(_fail(f"CLEANUP: delete Plane issue returned HTTP {s}"))
|
||||
except Exception as e:
|
||||
print(_fail(f"CLEANUP: delete Plane issue error: {e}"))
|
||||
else:
|
||||
print(_info("CLEANUP: no issue to delete"))
|
||||
|
||||
# Delete task + jobs from staging DB
|
||||
if issue_id:
|
||||
_cleanup_staging_jobs(issue_id)
|
||||
_cleanup_staging_db(issue_id)
|
||||
|
||||
# Remove dedup entry so future re-runs with same body don't get "duplicate"
|
||||
if wh_body_bytes is not None:
|
||||
import hashlib as _hl
|
||||
dedup_id = "plane" + _hl.sha256(b"plane" + wh_body_bytes).hexdigest()
|
||||
_cleanup_dedup(issue_id, dedup_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Live staging-stand check suite (ORCH-33)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--base-url",
|
||||
default="http://localhost:8501",
|
||||
help="Base URL of the staging orchestrator (default: http://localhost:8501)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["stub", "full-real"],
|
||||
default="stub",
|
||||
help=(
|
||||
"stub (default): check early pipeline artifacts only (branch+job), "
|
||||
"no LLM spend. "
|
||||
"full-real: also wait for the analyst agent (slow, costs credits)."
|
||||
),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
base = args.base_url.rstrip("/")
|
||||
|
||||
print(f"{_BOLD}{'='*60}{_RESET}")
|
||||
print(f"{_BOLD} ORCH-33 Staging Check Suite{_RESET}")
|
||||
print(f" base_url : {base}")
|
||||
print(f" mode : {args.mode}")
|
||||
print(f" utc_time : {datetime.datetime.now(datetime.timezone.utc).isoformat()}")
|
||||
print(f"{_BOLD}{'='*60}{_RESET}")
|
||||
|
||||
results = Results()
|
||||
|
||||
block_a(base, results)
|
||||
block_b(results)
|
||||
block_c(base, results, args.mode)
|
||||
|
||||
all_ok = results.summary()
|
||||
sys.exit(0 if all_ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
125
src/qg/checks.py
125
src/qg/checks.py
@@ -440,6 +440,130 @@ def check_deploy_status(repo: str, work_item_id: str, branch: str | None = None)
|
||||
return False, "Deploy log not found (14-deploy-log.md)"
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Self-hosting detection: staging-infra (localhost:8501) exists ONLY for the
|
||||
# orchestrator repo itself (self-hosting). Other repos have no staging instance
|
||||
# and their deployer prompts know nothing about it -- the gate must be a no-op
|
||||
# for them. The repo value is the plain gitea repo name (ProjectConfig.repo),
|
||||
# matching what _run_qg/advance_stage pass in. See ORCH-35 / PR #31.
|
||||
# ---------------------------------------------------------------------------
|
||||
SELF_HOSTING_REPO = "orchestrator"
|
||||
|
||||
|
||||
def is_self_hosting_repo(repo: str) -> bool:
|
||||
"""Return True iff repo is the self-hosted orchestrator (has staging infra).
|
||||
|
||||
Comparison is case-insensitive and strips whitespace for safety, but in
|
||||
practice repo comes from the gitea webhook payload .repository.name which
|
||||
is always lowercase (confirmed via projects.py registry entry).
|
||||
"""
|
||||
return (repo or "").strip().lower() == SELF_HOSTING_REPO.lower()
|
||||
|
||||
|
||||
def _parse_staging_status(content: str) -> tuple[bool, str]:
|
||||
"""Parse a 15-staging-log.md body and map its `staging_status:` frontmatter to a
|
||||
quality-gate verdict. Reads ONLY the machine-readable YAML field, never prose.
|
||||
|
||||
staging_status: SUCCESS -> (True, "Staging status: SUCCESS")
|
||||
staging_status: FAILED -> (False, "Staging status: FAILED")
|
||||
missing field / no frontmatter / bad YAML -> (False, <reason>)
|
||||
"""
|
||||
import yaml
|
||||
status = None
|
||||
if content.startswith("---"):
|
||||
parts = content.split("---", 2)
|
||||
if len(parts) >= 3:
|
||||
try:
|
||||
fm = yaml.safe_load(parts[1]) or {}
|
||||
except yaml.YAMLError as e:
|
||||
return False, f"Invalid YAML frontmatter in staging log: {e}"
|
||||
status = str(fm.get("staging_status", "")).upper().strip()
|
||||
if status == "SUCCESS":
|
||||
return True, "Staging status: SUCCESS"
|
||||
if status == "FAILED":
|
||||
return False, "Staging status: FAILED"
|
||||
return False, f"No machine-readable staging_status in frontmatter (got: {status!r})"
|
||||
|
||||
|
||||
def _staging_log_from_main(repo: str, work_item_id: str) -> str | None:
|
||||
"""Best-effort read of 15-staging-log.md from origin/main on the shared clone.
|
||||
|
||||
The deployer writes 15-staging-log.md and merges the staging artifacts into main
|
||||
via a separate PR (mirroring the deploy-log pattern), so the file lands in
|
||||
origin/main, NOT in the feature branch worktree the gate normally reads.
|
||||
This recovers it from main.
|
||||
|
||||
Degrades gracefully: any git failure (no clone, network/fetch error, file
|
||||
absent in main) returns None instead of raising, so the caller falls back to
|
||||
the plain "not found" verdict. Never raises.
|
||||
"""
|
||||
repo_clone = os.path.join(settings.repos_dir, repo)
|
||||
if not os.path.isdir(os.path.join(repo_clone, ".git")):
|
||||
return None
|
||||
rel = f"docs/work-items/{work_item_id}/15-staging-log.md"
|
||||
try:
|
||||
# Refresh origin/main so we see freshly-merged staging artifacts.
|
||||
subprocess.run(
|
||||
["git", "-C", repo_clone, "fetch", "origin", "main"],
|
||||
check=False, capture_output=True, timeout=30,
|
||||
)
|
||||
show = subprocess.run(
|
||||
["git", "-C", repo_clone, "show", f"origin/main:{rel}"],
|
||||
check=False, capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("staging-log origin/main lookup failed for %s/%s: %s", repo, work_item_id, e)
|
||||
return None
|
||||
if show.returncode != 0:
|
||||
return None
|
||||
return show.stdout
|
||||
|
||||
|
||||
def check_staging_status(repo: str, work_item_id: str, branch: str | None = None) -> tuple[bool, str]:
|
||||
"""
|
||||
Gate the deploy-staging -> deploy transition on the deployer's machine-readable
|
||||
verdict in 15-staging-log.md frontmatter (staging_status: SUCCESS|FAILED).
|
||||
|
||||
ORCH-35 conditional gate (Variant A):
|
||||
- Non-self-hosting repos (anything other than "orchestrator") have no staging
|
||||
instance and no deployer knowledge of it -> gate is an immediate pass.
|
||||
- Self-hosting repo ("orchestrator") -> real check: reads ONLY the machine-
|
||||
readable staging_status: field from YAML frontmatter, never body prose.
|
||||
|
||||
Mirrors check_deploy_status (БАГ 8) for the self-hosting path.
|
||||
|
||||
Lookup order (self-hosting only): worktree -> origin/main -> not found.
|
||||
|
||||
Returns:
|
||||
(True, "Staging gate N/A for <repo>") -> non-self-hosting repo (instant pass)
|
||||
(True, ...) -> staging_status: SUCCESS (self-hosting path)
|
||||
(False, ...) -> staging_status: FAILED, missing field, or no frontmatter
|
||||
"""
|
||||
# Variant A: non-self-hosting repos have no staging infra -- skip entirely.
|
||||
if not is_self_hosting_repo(repo):
|
||||
return True, f"Staging gate N/A for {repo}"
|
||||
|
||||
# Self-hosting (orchestrator) path: real verdict check.
|
||||
repo_path = _repo_path(repo, branch)
|
||||
log_path = os.path.join(repo_path, f"docs/work-items/{work_item_id}/15-staging-log.md")
|
||||
|
||||
if os.path.isfile(log_path):
|
||||
try:
|
||||
with open(log_path, "r") as f:
|
||||
content = f.read()
|
||||
except OSError as e:
|
||||
return False, f"Error reading staging log: {e}"
|
||||
return _parse_staging_status(content)
|
||||
|
||||
# Not in the feature worktree -- the deployer may have merged it into main.
|
||||
main_content = _staging_log_from_main(repo, work_item_id)
|
||||
if main_content is not None:
|
||||
return _parse_staging_status(main_content)
|
||||
|
||||
return False, "Staging log not found (15-staging-log.md)"
|
||||
|
||||
|
||||
# Registry for dynamic lookup by name
|
||||
QG_CHECKS = {
|
||||
"check_analysis_approved": check_analysis_approved,
|
||||
@@ -451,4 +575,5 @@ QG_CHECKS = {
|
||||
"check_reviewer_verdict": check_reviewer_verdict,
|
||||
"check_tests_local": check_tests_local,
|
||||
"check_deploy_status": check_deploy_status,
|
||||
"check_staging_status": check_staging_status,
|
||||
}
|
||||
|
||||
@@ -517,6 +517,32 @@ def _handle_qg_failure_rollbacks(
|
||||
f"(job_id={new_job})"
|
||||
)
|
||||
|
||||
# ORCH-35: deployer staging verdict FAILED -> roll deploy-staging back to development.
|
||||
# Staging-провал = код плох; откат на development по образцу БАГ-8 (deploy->development).
|
||||
# НЕ трогает ветку check_deploy_status ниже.
|
||||
if agent == "deployer" and qg_name == "check_staging_status":
|
||||
update_task_stage(task_id, "development")
|
||||
notify_stage_change(task_id, current_stage, "development")
|
||||
plane_notify_stage(work_item_id, current_stage, "development")
|
||||
result.rolled_back_to = "development"
|
||||
set_issue_blocked(work_item_id)
|
||||
notify_qg_failure(task_id, "deploy-staging", "check_staging_status", reason)
|
||||
plane_add_comment(
|
||||
work_item_id,
|
||||
f"\u274c Staging gate FAILED ({reason}). Rolled back to development. "
|
||||
f"Developer \u043d\u0443\u0436\u0435\u043d \u0434\u043b\u044f \u0444\u0438\u043a\u0441\u0430.",
|
||||
author="deployer",
|
||||
)
|
||||
send_telegram(
|
||||
f"\U0001f6a8 {work_item_id}: Staging FAILED ({reason}). "
|
||||
f"Rolled back to development. Needs fix."
|
||||
)
|
||||
result.alerted = True
|
||||
logger.error(
|
||||
f"Task {task_id}: deployer staging verdict FAILED, rolled back deploy-staging -> "
|
||||
f"development ({reason})"
|
||||
)
|
||||
|
||||
# БАГ 8: deployer verdict FAILED -> roll deploy back to development.
|
||||
# The launcher's exit_code-based guard (launcher.py:475) never fires because
|
||||
# the LLM process exit code is always 0; this gate fires on the machine-readable
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Stage machine for orchestrator pipeline.
|
||||
|
||||
Stages:
|
||||
created → analysis → architecture → development → review → testing → deploy → done
|
||||
created → analysis → architecture → development → review → testing → deploy-staging → deploy → done
|
||||
|
||||
Each stage defines:
|
||||
- next: the stage to advance to
|
||||
@@ -15,8 +15,9 @@ STAGE_TRANSITIONS = {
|
||||
"architecture": {"next": "development", "agent": "developer", "qg": "check_architecture_done"},
|
||||
"development": {"next": "review", "agent": "reviewer", "qg": "check_ci_green"},
|
||||
"review": {"next": "testing", "agent": "tester", "qg": "check_reviewer_verdict"},
|
||||
"testing": {"next": "deploy", "agent": "deployer", "qg": "check_tests_passed"},
|
||||
"deploy": {"next": "done", "agent": None, "qg": "check_deploy_status"},
|
||||
"testing": {"next": "deploy-staging", "agent": "deployer", "qg": "check_tests_passed"},
|
||||
"deploy-staging": {"next": "deploy", "agent": "deployer", "qg": "check_staging_status"},
|
||||
"deploy": {"next": "done", "agent": None, "qg": "check_deploy_status"},
|
||||
"done": {"next": None, "agent": None, "qg": None},
|
||||
}
|
||||
|
||||
|
||||
183
tests/test_qg.py
183
tests/test_qg.py
@@ -19,6 +19,7 @@ from src.qg.checks import (
|
||||
check_tests_passed,
|
||||
check_tests_local,
|
||||
check_deploy_status,
|
||||
check_staging_status,
|
||||
)
|
||||
from src.stages import get_qg_for_stage
|
||||
|
||||
@@ -448,3 +449,185 @@ class TestCheckTestsLocal:
|
||||
assert "../../tests/" in cmd
|
||||
assert kwargs["cwd"] == os.path.join(str(tmp_path), "src", "api")
|
||||
|
||||
|
||||
|
||||
class TestCheckStagingStatus:
|
||||
"""ORCH-35 conditional gate (Variant A): deploy-staging gate is active ONLY for
|
||||
the self-hosting orchestrator repo (has staging infra on localhost:8501). All
|
||||
other repos pass immediately with "Staging gate N/A for <repo>".
|
||||
|
||||
Self-hosting path: reads machine-readable staging_status: from 15-staging-log.md
|
||||
frontmatter. Mirrors check_deploy_status pattern.
|
||||
"""
|
||||
|
||||
@pytest.fixture()
|
||||
def orch_dir(self, tmp_path, monkeypatch):
|
||||
"""Temp orchestrator repo dir (self-hosting)."""
|
||||
monkeypatch.setattr("src.qg.checks.settings.repos_dir", str(tmp_path))
|
||||
d = tmp_path / "orchestrator"
|
||||
d.mkdir(exist_ok=True)
|
||||
return d
|
||||
|
||||
def _write_log(self, repo_dir, content, wi="ORCH-035"):
|
||||
wi_dir = repo_dir / "docs" / "work-items" / wi
|
||||
wi_dir.mkdir(parents=True, exist_ok=True)
|
||||
(wi_dir / "15-staging-log.md").write_text(content)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Self-hosting (orchestrator) path -- real file check
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_success_verdict_passes(self, orch_dir):
|
||||
self._write_log(
|
||||
orch_dir,
|
||||
"---\nstaging_status: SUCCESS\ntimestamp: 2026-06-05T00:00:00Z\n---\n\nAll staging tests passed.\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035")
|
||||
assert passed is True
|
||||
assert "SUCCESS" in reason
|
||||
|
||||
def test_failed_verdict_fails(self, orch_dir):
|
||||
self._write_log(
|
||||
orch_dir,
|
||||
"---\nstaging_status: FAILED\ntimestamp: 2026-06-05T00:00:00Z\n---\n\n2 tests failed.\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035")
|
||||
assert passed is False
|
||||
assert "FAILED" in reason
|
||||
|
||||
def test_no_file_fails_for_self_hosting(self, orch_dir):
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035")
|
||||
assert passed is False
|
||||
assert "not found" in reason.lower()
|
||||
|
||||
def test_no_field_fails(self, orch_dir):
|
||||
# Frontmatter present but no staging_status field -> must NOT pass.
|
||||
self._write_log(
|
||||
orch_dir,
|
||||
"---\nversion: v0.0.3\n---\n\nStatus: all good (prose only).\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035")
|
||||
assert passed is False
|
||||
|
||||
def test_prose_only_no_frontmatter_fails(self, orch_dir):
|
||||
# Prose mentioning SUCCESS but no machine-readable frontmatter -> fail.
|
||||
self._write_log(
|
||||
orch_dir,
|
||||
"# Staging Log\n\nStatus: SUCCESS (prose, not frontmatter).\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035")
|
||||
assert passed is False
|
||||
|
||||
def test_origin_main_success_passes_when_absent_in_worktree(self, monkeypatch):
|
||||
# Deployer merged 15-staging-log.md into main; not in worktree -> recover from main.
|
||||
monkeypatch.setattr(
|
||||
"src.qg.checks._staging_log_from_main",
|
||||
lambda repo, wi: "---\nstaging_status: SUCCESS\n---\n\nAll good.\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035-main")
|
||||
assert passed is True
|
||||
assert "SUCCESS" in reason
|
||||
|
||||
def test_origin_main_failed_fails(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"src.qg.checks._staging_log_from_main",
|
||||
lambda repo, wi: "---\nstaging_status: FAILED\n---\n\nboom.\n",
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035-main")
|
||||
assert passed is False
|
||||
assert "FAILED" in reason
|
||||
|
||||
def test_absent_everywhere_fails(self, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"src.qg.checks._staging_log_from_main", lambda repo, wi: None
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("orchestrator", "ORCH-035-absent")
|
||||
assert passed is False
|
||||
assert "not found" in reason.lower()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Non-self-hosting path -- instant pass, no file dependency
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_non_self_hosting_passes_immediately_no_file(self, tmp_path, monkeypatch):
|
||||
"""Non-self-hosting repo: gate is N/A even without a staging log file."""
|
||||
monkeypatch.setattr("src.qg.checks.settings.repos_dir", str(tmp_path))
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("enduro-trails", "ET-035")
|
||||
assert passed is True
|
||||
assert "N/A" in reason
|
||||
assert "enduro-trails" in reason
|
||||
|
||||
def test_non_self_hosting_passes_regardless_of_file_content(self, tmp_path, monkeypatch):
|
||||
"""Even a FAILED staging log must not block a non-self-hosting repo."""
|
||||
monkeypatch.setattr("src.qg.checks.settings.repos_dir", str(tmp_path))
|
||||
et_dir = tmp_path / "enduro-trails" / "docs" / "work-items" / "ET-035"
|
||||
et_dir.mkdir(parents=True)
|
||||
(et_dir / "15-staging-log.md").write_text(
|
||||
"---\nstaging_status: FAILED\n---\nShould be ignored.\n"
|
||||
)
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("enduro-trails", "ET-035")
|
||||
assert passed is True
|
||||
assert "N/A" in reason
|
||||
|
||||
def test_unknown_repo_also_passes_immediately(self, tmp_path, monkeypatch):
|
||||
"""Any repo that is not orchestrator gets N/A gate."""
|
||||
monkeypatch.setattr("src.qg.checks.settings.repos_dir", str(tmp_path))
|
||||
from src.qg.checks import check_staging_status
|
||||
passed, reason = check_staging_status("some-other-project", "XY-001")
|
||||
assert passed is True
|
||||
assert "N/A" in reason
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# is_self_hosting_repo helper
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_is_self_hosting_true_for_orchestrator(self):
|
||||
from src.qg.checks import is_self_hosting_repo
|
||||
assert is_self_hosting_repo("orchestrator") is True
|
||||
|
||||
def test_is_self_hosting_case_insensitive(self):
|
||||
from src.qg.checks import is_self_hosting_repo
|
||||
assert is_self_hosting_repo("Orchestrator") is True
|
||||
assert is_self_hosting_repo("ORCHESTRATOR") is True
|
||||
|
||||
def test_is_self_hosting_false_for_enduro_trails(self):
|
||||
from src.qg.checks import is_self_hosting_repo
|
||||
assert is_self_hosting_repo("enduro-trails") is False
|
||||
|
||||
def test_is_self_hosting_false_for_empty(self):
|
||||
from src.qg.checks import is_self_hosting_repo
|
||||
assert is_self_hosting_repo("") is False
|
||||
assert is_self_hosting_repo(None) is False
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Stage machinery (regression: must not be broken)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_deploy_staging_qg_is_check_staging_status(self):
|
||||
assert get_qg_for_stage("deploy-staging") == "check_staging_status"
|
||||
|
||||
def test_registered_in_qg_checks(self):
|
||||
from src.qg.checks import QG_CHECKS, check_staging_status
|
||||
assert QG_CHECKS.get("check_staging_status") is check_staging_status
|
||||
|
||||
def test_deploy_stage_qg_still_check_deploy_status(self):
|
||||
"""Regression: existing deploy QG must not be broken."""
|
||||
assert get_qg_for_stage("deploy") == "check_deploy_status"
|
||||
|
||||
def test_stage_chain(self):
|
||||
"""Full chain: testing->deploy-staging->deploy->done."""
|
||||
from src.stages import get_next_stage
|
||||
assert get_next_stage("testing") == "deploy-staging"
|
||||
assert get_next_stage("deploy-staging") == "deploy"
|
||||
assert get_next_stage("deploy") == "done"
|
||||
|
||||
|
||||
@@ -136,7 +136,7 @@ class TestHappyPathAgentSelection:
|
||||
("architecture", "development", "developer"),
|
||||
("development", "review", "reviewer"),
|
||||
("review", "testing", "tester"),
|
||||
("testing", "deploy", "deployer"),
|
||||
("testing", "deploy-staging", "deployer"),
|
||||
],
|
||||
)
|
||||
def test_advance_launches_current_stage_agent(
|
||||
@@ -507,6 +507,120 @@ class TestAnalysisApprovedFlow:
|
||||
flow.assert_called_once()
|
||||
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ORCH-35: deploy-staging gate — rollback on staging failure
|
||||
# ---------------------------------------------------------------------------
|
||||
class TestStagingGate:
|
||||
"""deploy-staging -> deploy must be gated on check_staging_status.
|
||||
FAILED verdict rolls back to development (same as deploy БАГ-8 pattern:
|
||||
staging failure = code is bad, needs developer fix)."""
|
||||
|
||||
def test_staging_success_advances_to_deploy(self, monkeypatch):
|
||||
"""Happy path: staging SUCCESS -> advance to deploy (no agent launched)."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS, "check_staging_status": _pass},
|
||||
)
|
||||
task_id = _make_task("deploy-staging")
|
||||
res = advance_stage(
|
||||
task_id, "deploy-staging", "enduro-trails", "ET-035",
|
||||
"feature/ET-035-x", finished_agent="deployer",
|
||||
)
|
||||
assert res.advanced is True
|
||||
assert res.to_stage == "deploy"
|
||||
assert _stage(task_id) == "deploy"
|
||||
# deploy-staging has agent=deployer, so deployer is enqueued for deploy stage
|
||||
assert res.enqueued_agent == "deployer"
|
||||
jobs = _jobs()
|
||||
assert len(jobs) == 1
|
||||
assert jobs[0]["agent"] == "deployer"
|
||||
|
||||
def test_staging_failed_rolls_back_to_development(self, monkeypatch):
|
||||
"""ORCH-35: staging FAILED -> roll back to development, not to testing."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS,
|
||||
"check_staging_status": _fail("Staging status: FAILED")},
|
||||
)
|
||||
task_id = _make_task("deploy-staging")
|
||||
res = advance_stage(
|
||||
task_id, "deploy-staging", "enduro-trails", "ET-035",
|
||||
"feature/ET-035-x", finished_agent="deployer",
|
||||
)
|
||||
assert res.advanced is False
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development" # NOT deploy, NOT testing
|
||||
assert res.alerted is True
|
||||
assert stage_engine.set_issue_blocked.called
|
||||
assert stage_engine.send_telegram.called
|
||||
|
||||
def test_staging_failed_does_not_reach_deploy(self, monkeypatch):
|
||||
"""Prod deploy is unreachable if staging gate is not green."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS,
|
||||
"check_staging_status": _fail("Staging log not found")},
|
||||
)
|
||||
task_id = _make_task("deploy-staging")
|
||||
res = advance_stage(
|
||||
task_id, "deploy-staging", "enduro-trails", "ET-035",
|
||||
"feature/ET-035-x", finished_agent="deployer",
|
||||
)
|
||||
assert res.advanced is False
|
||||
# Task must NOT be in deploy stage
|
||||
assert _stage(task_id) != "deploy"
|
||||
|
||||
def test_staging_missing_log_rolls_back(self, monkeypatch):
|
||||
"""Missing 15-staging-log.md -> gate fails -> rollback to development."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS,
|
||||
"check_staging_status": _fail("Staging log not found (15-staging-log.md)")},
|
||||
)
|
||||
task_id = _make_task("deploy-staging")
|
||||
res = advance_stage(
|
||||
task_id, "deploy-staging", "enduro-trails", "ET-035",
|
||||
"feature/ET-035-x", finished_agent="deployer",
|
||||
)
|
||||
assert res.advanced is False
|
||||
assert _stage(task_id) == "development"
|
||||
|
||||
def test_testing_to_deploy_staging_advance(self, monkeypatch):
|
||||
"""testing -> deploy-staging: deployer is enqueued (ORCH-35 chain check)."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS, "check_tests_passed": _pass},
|
||||
)
|
||||
task_id = _make_task("testing")
|
||||
res = advance_stage(
|
||||
task_id, "testing", "enduro-trails", "ET-035",
|
||||
"feature/ET-035-x", finished_agent="tester",
|
||||
)
|
||||
assert res.advanced is True
|
||||
assert res.to_stage == "deploy-staging"
|
||||
assert _stage(task_id) == "deploy-staging"
|
||||
assert res.enqueued_agent == "deployer"
|
||||
|
||||
def test_deploy_still_rolls_back_on_check_deploy_status_fail(self, monkeypatch):
|
||||
"""Existing БАГ-8 rollback must still work for deploy stage (regression guard)."""
|
||||
monkeypatch.setattr(
|
||||
stage_engine, "QG_CHECKS",
|
||||
{**stage_engine.QG_CHECKS,
|
||||
"check_deploy_status": _fail("Deploy status: FAILED")},
|
||||
)
|
||||
task_id = _make_task("deploy")
|
||||
res = advance_stage(
|
||||
task_id, "deploy", "enduro-trails", "ET-011",
|
||||
"feature/ET-011-x", finished_agent="deployer",
|
||||
)
|
||||
assert res.advanced is False
|
||||
assert res.rolled_back_to == "development"
|
||||
assert _stage(task_id) == "development"
|
||||
assert res.alerted is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# launcher + plane both delegate to the engine
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user