docs(ORCH-058): staging gate re-run — staging_status FAILED (8/10, C9a/C9b)
E2E pipeline not triggered on staging webhook ("no pipeline action" on
state b873d9eb...); reproduces prior FAILED. Rolls task back to development.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
---
|
||||
staging_status: FAILED
|
||||
timestamp: 2026-06-07T10:14:30Z
|
||||
timestamp: 2026-06-07T10:39:15Z
|
||||
base_url: http://localhost:8501
|
||||
---
|
||||
|
||||
@@ -8,10 +8,12 @@ base_url: http://localhost:8501
|
||||
|
||||
Staging test suite ran against the live staging environment and **FAILED** (exit code `1`,
|
||||
**8/10 checks PASS**). The two end-to-end (Block C) checks failed: the pipeline was **not
|
||||
triggered** on the freshly-built staging image, so no task / branch / analyst job was created.
|
||||
triggered** on the staging image, so no task / branch / analyst job was created.
|
||||
|
||||
Per the staging-gate contract this is a machine verdict `FAILED` → the task rolls back to
|
||||
`development`. The verdict reflects the real suite exit code, not an LLM declaration.
|
||||
`development`. The verdict reflects the real suite exit code, not an LLM declaration. This run
|
||||
**reproduces** the regression already recorded in the previous re-run of this file (same 8/10,
|
||||
same `C9a`/`C9b` failure, same `no pipeline action` classification).
|
||||
|
||||
## Execution
|
||||
- Canonical `docker exec` into `orchestrator-staging` (ORCH-048, ADR-001), invoked via the
|
||||
@@ -26,32 +28,27 @@ Per the staging-gate contract this is a machine verdict `FAILED` → the task ro
|
||||
|
||||
## Root cause (actionable for development rollback)
|
||||
The E2E flow (`staging_check.py` Block C) creates a SANDBOX Plane issue (C7 ✓), then POSTs a
|
||||
signed `/webhook/plane` payload with state `IN_PROGRESS_STATE_ID` (name `"In Progress"`,
|
||||
group `"started"`) to start the pipeline (C8 ✓ — HTTP 200 `{"status":"accepted"}`). However the
|
||||
staging instance logged:
|
||||
signed `/webhook/plane` payload to start the pipeline (C8 ✓ — HTTP 200 `{"status":"accepted"}`).
|
||||
However the staging instance logged:
|
||||
|
||||
```
|
||||
2026-06-07 10:14:09 [INFO] orchestrator.webhooks.plane: issue ed5db89e-657d-4728-9179-901d2404be85
|
||||
2026-06-07 10:39:17,333 [INFO] orchestrator.webhooks.plane: issue 990c99b5-6a1d-4e63-a59a-9a11716e07b9
|
||||
updated to state b873d9eb..., no pipeline action
|
||||
```
|
||||
|
||||
→ **"no pipeline action"**: the `In Progress` / `started` webhook did NOT start the pipeline,
|
||||
so no `tasks` row, no Gitea branch (C9a FAIL — branch never appeared after 60s), and no analyst
|
||||
job enqueued (C9b FAIL — queue had no new job; latest job is id=8 from 2026-06-06). Cleanup
|
||||
confirmed `no task row found for plane_id=ed5db89e...` and `no branch to delete`.
|
||||
→ **"no pipeline action"**: the webhook transition did NOT start the pipeline, so no `tasks`
|
||||
row, no Gitea branch (C9a FAIL — branch never appeared after 60s), and no analyst job enqueued
|
||||
(C9b FAIL — queue had no new job after 30s). Cleanup confirmed `no task row found for
|
||||
plane_id=990c99b5...` and `no branch to delete`.
|
||||
|
||||
This is a **deterministic regression in the validated artifact**, not a timing flake (the
|
||||
webhook was explicitly classified as no-op, not a poll timeout):
|
||||
- The **same** `staging_check.py` against the **same** SANDBOX config passed **10/10** at
|
||||
09:31 UTC on the pre-rebuild image (see git history of this file).
|
||||
- The staging image was **freshly rebuilt** at 10:13:29 UTC (revision label
|
||||
`org.opencontainers.image.revision=094b5e2f960f696216f8661ff9c27b0d4706f219`, container
|
||||
recreated 10:13:36 UTC) — consistent with ORCH-058 Strategy A rebuilding 8501 from the
|
||||
validated commit. The new image now exposes the `reconcile` key in `/queue` (ORCH-053),
|
||||
absent at 09:31, confirming the image changed between the two runs.
|
||||
- Net: the artifact about to be promoted to prod no longer starts the pipeline on a Plane
|
||||
`In Progress` (group `started`) transition. **Investigate `handle_status_start` /
|
||||
webhook start-state matching in `src/webhooks/plane.py`** against the validated commit.
|
||||
webhook was explicitly classified as a no-op, not a poll timeout):
|
||||
- The **same** `staging_check.py` against the **same** SANDBOX config passed **10/10** on an
|
||||
earlier pre-rebuild image (see git history of this file).
|
||||
- The state id `b873d9eb...` from the webhook payload is not matched as a pipeline-start
|
||||
(`group="started"`) transition by the staging instance. **Investigate `handle_status_start`
|
||||
/ webhook start-state matching in `src/webhooks/plane.py`** against the validated commit, and
|
||||
confirm the staging start-state id wiring used by `staging_check.py`.
|
||||
|
||||
Smoke (A1–A3) and access (B4–B6) all passed, including B6 registry isolation
|
||||
(sandbox present; prod ET/ORCH absent) — confirming the check ran inside the staging
|
||||
@@ -64,7 +61,7 @@ instance's own process-env, so there is no false-FAIL / spurious-rollback risk f
|
||||
ORCH-33 Staging Check Suite
|
||||
base_url : http://localhost:8501
|
||||
mode : stub
|
||||
utc_time : 2026-06-07T10:14:07.188198+00:00
|
||||
utc_time : 2026-06-07T10:39:15.004026+00:00
|
||||
============================================================
|
||||
|
||||
[Block A] SMOKE
|
||||
@@ -79,7 +76,7 @@ instance's own process-env, so there is no false-FAIL / spurious-rollback risk f
|
||||
|
||||
[Block C] E2E (mode=stub)
|
||||
· C7: Creating issue in SANDBOX project...
|
||||
✓ PASS C7 Create issue in Plane SANDBOX [HTTP 201, issue_id=ed5db89e-657d-4728-9179-901d2404be85]
|
||||
✓ PASS C7 Create issue in Plane SANDBOX [HTTP 201, issue_id=990c99b5-6a1d-4e63-a59a-9a11716e07b9]
|
||||
· C8: Triggering pipeline via POST /webhook/plane ...
|
||||
· Using HMAC signature (secret len=40)
|
||||
✓ PASS C8 Trigger pipeline via /webhook/plane [HTTP 200, resp={'status': 'accepted'}]
|
||||
@@ -93,8 +90,8 @@ instance's own process-env, so there is no false-FAIL / spurious-rollback risk f
|
||||
|
||||
[CLEANUP]
|
||||
· CLEANUP: no branch to delete
|
||||
✓ PASS CLEANUP: deleted Plane issue ed5db89e-657d-4728-9179-901d2404be85 (HTTP 204)
|
||||
· CLEANUP DB: no task row found for plane_id=ed5db89e-657d-4728-9179-901d2404be85
|
||||
✓ PASS CLEANUP: deleted Plane issue 990c99b5-6a1d-4e63-a59a-9a11716e07b9 (HTTP 204)
|
||||
· CLEANUP DB: no task row found for plane_id=990c99b5-6a1d-4e63-a59a-9a11716e07b9
|
||||
· CLEANUP DB dedup: no such table: events_dedup
|
||||
|
||||
============================================================
|
||||
|
||||
Reference in New Issue
Block a user