diff --git a/docs/work-items/ORCH-016/13-test-report.md b/docs/work-items/ORCH-016/13-test-report.md new file mode 100644 index 0000000..68d5b8d --- /dev/null +++ b/docs/work-items/ORCH-016/13-test-report.md @@ -0,0 +1,159 @@ +--- +type: test-report +work_item_id: ORCH-016 +verdict: PASS +result: PASS +version: 1 +--- + +# Test Report — ORCH-016 + +## Окружение +- Python: 3.12.13 +- pytest: 8.3.3 +- Worktree: `/repos/_wt/orchestrator/feature_ORCH-016-plane` +- Ветка: `feature/ORCH-016-plane` @ `1778d8f` (reviewer auto-commit) +- Дата: 2026-06-05 +- Prod-инстанс orchestrator: `/health` → `{"status":"ok"}` (не трогался) + +## Команды + +```bash +# Полный регресс из worktree +pytest tests/ -v --tb=short + +# ORCH-016 целевой набор +pytest tests/test_status_comment_format.py \ + tests/test_post_usage_comments_integration.py \ + tests/test_status_comment_authorship.py \ + tests/test_status_comment_dedup_regression.py \ + tests/test_status_comment_duration_db_fallback.py \ + tests/test_fmt_duration.py \ + tests/test_qg_registry_snapshot.py \ + tests/test_analyst_comment.py \ + tests/test_analyst_comment_regression.py \ + tests/test_analyst_status_only_regression.py \ + tests/test_notify_done_regression.py -v +``` + +## Сводка + +| Прогон | Passed | Failed | Skipped | +|--------|-------:|-------:|--------:| +| Полный (`tests/`) | **392** | **4** | 6 | +| ORCH-016 целевой (62 теста) | **62** | **0** | 0 | + +## Smoke test API + +| Endpoint | HTTP | Ответ | +|----------|------|-------| +| `GET /health` | 200 | `{"status":"ok","service":"orchestrator"}` | +| `GET /status` | 200 | JSON, активна задача `ORCH-016` (stage `testing`) | +| `GET /queue` | 200 | JSON, `counts={queued:0,running:1,done:36,failed:0}`, breaker `closed`, preflight OK | + +## Покрытие плана тестов (`04-test-plan.yaml`) + +| TC | Модуль | AC | Результат | +|----|--------|----|-----------| +| TC-01 | `test_status_comment_format.py::test_tc01_architect_comment` | AC-1 | PASS | +| TC-02 | `test_status_comment_format.py::test_tc02_developer_comment_links_branch_and_pr` | AC-2 | PASS | +| TC-03 | `test_status_comment_format.py::test_tc03_reviewer_verdict_approve` | AC-3 | PASS | +| TC-04 | `test_status_comment_format.py::test_tc04_reviewer_verdict_request_changes` | AC-3 | PASS | +| TC-05 | `test_status_comment_format.py::test_tc05_reviewer_missing_artifact_graceful` | AC-3, AC-8 | PASS | +| TC-06 | `test_status_comment_format.py::test_tc06_tester_pass` | AC-4 | PASS | +| TC-07 | `test_status_comment_format.py::test_tc07_tester_fail` + `test_tc07b_tester_falls_back_to_status_key` | AC-4 | PASS | +| TC-08 | `test_status_comment_format.py::test_tc08_deployer_deploy_status_success` + `test_deployer_status_failed_drives_status_line` | AC-5 | PASS | +| TC-09 | `test_status_comment_format.py::test_tc09_deployer_staging_status_success` | AC-5 | PASS | +| TC-10 | `test_status_comment_format.py::test_tc10_url_fallback_to_gitea_url` | AC-9 | PASS | +| TC-11 | `test_analyst_comment_regression.py::test_tc11_analyst_text_preserved_with_links` + `test_tc11_analyst_includes_duration_when_db_has_run` | AC-6 | PASS | +| TC-12 | `test_status_comment_format.py::test_tc12_frontmatter_*` (×4 кейса) | AC-8 | PASS | +| TC-13 | `test_post_usage_comments_integration.py::test_tc13_reviewer_posts_one_status_comment` | AC-3, AC-7 | PASS | +| TC-14 | `test_post_usage_comments_integration.py::test_tc14_tester_posts_one_status_comment` | AC-4, AC-7 | PASS | +| TC-15 | `test_post_usage_comments_integration.py::test_tc15_deployer_posts_status_then_summary` + `test_deployer_staging_picks_15_log` | AC-5, AC-7 | PASS | +| TC-16 | `test_analyst_status_only_regression.py::test_tc16_analyst_goes_to_in_review_no_advance` | AC-6 | PASS | +| TC-17 | `test_status_comment_dedup_regression.py::test_tc17_*` (×4) | AC-7 | PASS | +| TC-18 | `test_notify_done_regression.py::test_notify_done_*` + `test_orch016_does_not_steal_done_signal` (×4) | AC-10 | PASS | +| TC-19 | `test_status_comment_authorship.py::test_tc19_*` (×7) | AC-7 | PASS | +| TC-20 | `test_qg_registry_snapshot.py::test_tc20_qg_registry_unchanged` + `test_tc20_qg_callables_unchanged` + `test_tc20_stage_transitions_unchanged` | AC-11 | PASS | +| TC-21 | `test_fmt_duration.py::test_fmt_duration_boundary_table` | AC-13 | PASS | +| TC-22 | `test_fmt_duration.py::test_fmt_duration_none_returns_empty` + `test_fmt_duration_negative_returns_empty` + `test_fmt_duration_garbage_returns_empty` | AC-13 | PASS | +| TC-23 | `test_status_comment_format.py::test_tc23_no_duration_no_line` | AC-13, AC-14 | PASS | +| TC-24 | `test_status_comment_duration_db_fallback.py::test_tc24_*` (×5) + `test_explicit_duration_wins_over_db_fallback` | AC-14 | PASS | +| TC-25 | `test_status_comment_duration_db_fallback.py::test_tc25_db_read_failure_no_raise` | AC-14 | PASS | + +**Итого: 25/25 TC = PASS** (на 25 ID плана приходится 62 фактических теста; все зелёные.) + +## Сопоставление с критериями (`03-acceptance-criteria.md`) + +| AC | Покрытие | Результат | +|----|----------|-----------| +| AC-1 Architect comment | TC-01 + `test_ac1_architect_header_literal` | PASS | +| AC-2 Developer comment | TC-02 | PASS | +| AC-3 Reviewer verdict | TC-03, TC-04, TC-05, TC-13 | PASS | +| AC-4 Tester verdict | TC-06, TC-07, TC-14 | PASS | +| AC-5 Deployer status | TC-08, TC-09 + `test_ac5_deployer_deploy_description` + `test_ac5_deployer_staging_description` + TC-15 | PASS | +| AC-6 Analyst no regression | TC-11, TC-16 | PASS | +| AC-7 Один коммент на агента | TC-13, TC-14, TC-15, TC-17, TC-19 | PASS | +| AC-8 Graceful fallback артефакта | TC-05, TC-12 | PASS | +| AC-9 `gitea_public_url` | TC-10 | PASS | +| AC-10 Зелёные существующие тесты | Регрессии нет (см. ниже) | PASS | +| AC-11 QG / STAGE_TRANSITIONS неизменны | TC-20 (×3) | PASS | +| AC-12 Документация обновлена | Reviewer верифицировал в `12-review.md` (CHANGELOG, architecture/README, ADR-001) | PASS | +| AC-13 `fmt_duration` формат | TC-21, TC-22, TC-23 | PASS | +| AC-14 Длительность fallback | TC-24, TC-25 | PASS | + +**AC-1…AC-14 = PASS.** + +## Анализ 4 фейлов в полном прогоне (AC-10) + +``` +FAILED tests/test_m6_sequence.py::test_created_uses_plane_sequence_id +FAILED tests/test_m6_sequence.py::test_created_falls_back_to_db_when_plane_down +FAILED tests/test_plane_webhook.py::test_orchestrator_project_routes_to_orchestrator_repo +FAILED tests/test_plane_webhook.py::test_prefixes_independent_per_project +``` + +Эти 4 фейла — **предсуществующая регрессия на `main`**, не индуцированная ORCH-016. Проверка: + +``` +$ git clone -b main /repos/orchestrator /tmp/orch-main-check +$ cd /tmp/orch-main-check +$ pytest tests/test_m6_sequence.py tests/test_plane_webhook.py +… +==================== 4 failed, 7 passed, 1 warning in 0.80s ==================== +FAILED tests/test_m6_sequence.py::test_created_uses_plane_sequence_id +FAILED tests/test_m6_sequence.py::test_created_falls_back_to_db_when_plane_down +FAILED tests/test_plane_webhook.py::test_orchestrator_project_routes_to_orchestrator_repo +FAILED tests/test_plane_webhook.py::test_prefixes_independent_per_project +``` + +На свежем клоне `main` те же 4 теста падают с идентичными сообщениями (`assert None is not None`, `KeyError: 'o1'`). ORCH-016 не трогает `src/webhooks/plane.py`, `src/plane_sync.py::fetch_issue_sequence_id`, `src/projects.py` — то есть участки, ответственные за эти кейсы. Reviewer ранее зафиксировал тот же факт в `12-review.md`. **Регрессий, индуцированных ORCH-016 = 0** → AC-10 PASS. + +Эти 4 фейла должны быть подняты отдельной задачей (вне scope ORCH-016). + +## Вывод pytest (хвост полного прогона) + +``` +=========================== short test summary info ============================ +FAILED tests/test_m6_sequence.py::test_created_uses_plane_sequence_id - asser... +FAILED tests/test_m6_sequence.py::test_created_falls_back_to_db_when_plane_down +FAILED tests/test_plane_webhook.py::test_orchestrator_project_routes_to_orchestrator_repo +FAILED tests/test_plane_webhook.py::test_prefixes_independent_per_project - K... +============ 4 failed, 392 passed, 6 skipped, 13 warnings in 7.44s ============= +``` + +## Self-hosting + +Прод-контейнер `orchestrator` (порт 8500) во время прогонов не перезапускался, не ронялся: `/health` → ok, `/queue` → breaker closed, текущая задача `ORCH-016` (running) в очереди. Тесты выполнялись в worktree-копии `feature_ORCH-016-plane`, не затрагивая прод-БД. + +## Итог + +**PASS.** + +- Все 25 TC из `04-test-plan.yaml` = PASS (62 фактических теста зелёные). +- Все 14 AC из `03-acceptance-criteria.md` = PASS. +- Регрессий относительно `main` нет (4 хронических фейла предсуществуют, см. выше). +- Smoke test API зелёный. +- Прод-инстанс не задет. + +Задача готова к стадии `deploy-staging`.