fix(tests): hermetic ORCH-41 model/effort tests vs host env (unblock merge-gate)
Some checks failed
CI / test (push) Has been cancelled
CI / test (pull_request) Successful in 55s

Merge-gate re-test runs under the orchestrator's prod env, where the
operator legitimately set ORCH_AGENT_FALLBACK_MODEL and changed
ORCH_AGENT_MODEL_DEFAULT / ORCH_AGENT_EFFORT_*. Two ORCH-41-era tests
asserted SHIPPED defaults through the env-backed settings singleton and
failed 3/3 there, while Gitea CI (clean env) stayed green. Branch
ORCH-009 touches neither src/ nor these tests - latent non-hermetic
landmine on main, detonated by the prod env change.

- test_resolve_agent_effort.py: autouse fixture now mirrors the sibling
  model-file baseline (pins shipped model/fallback fields) so the
  flag-assembly tests are env-independent.
- test_resolve_agent_model.py: fixture also resets agent_fallback_model;
  test_fallback_model_disabled_by_default now asserts the CLASS field
  default (the actual ORCH-074 ADR-001 G4 invariant: shipped default
  is ""), never-break is_valid_model asserts unchanged byte-for-byte.

Clean-env behaviour is byte-equivalent (fixtures pin exactly what an
empty env yields). Full suite: 1713 passed (was 2 failed / 1711).

Refs: ORCH-009

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-10 16:17:54 +03:00
parent dd09e3da89
commit e9038182a1
3 changed files with 19 additions and 2 deletions

View File

@@ -42,6 +42,15 @@ def _clean_settings(monkeypatch):
monkeypatch.setattr(settings, "agent_effort_default", "high")
for a, e in CANON_EFFORT.items():
monkeypatch.setattr(settings, f"agent_effort_{a}", e)
# Hermeticity (mirrors test_resolve_agent_model's baseline): the flag-assembly
# tests below also read the MODEL/fallback fields, and the host env (prod .env;
# the merge-gate re-test runs under it) may legitimately set
# ORCH_AGENT_MODEL_* / ORCH_AGENT_FALLBACK_MODEL. These tests assert
# shipped-default behaviour, not the host config -> pin the shipped defaults.
monkeypatch.setattr(settings, "agent_model_default", "claude-opus-4-8")
for a in CANON_EFFORT:
monkeypatch.setattr(settings, f"agent_model_{a}", "")
monkeypatch.setattr(settings, "agent_fallback_model", "")
monkeypatch.setattr(P.settings, "projects_json", "")
reload_projects()
yield

View File

@@ -41,6 +41,9 @@ def _clean_settings(monkeypatch):
monkeypatch.setattr(settings, "agent_model_default", "claude-opus-4-8")
for a in ("analyst", "architect", "developer", "reviewer", "tester", "deployer"):
monkeypatch.setattr(settings, f"agent_model_{a}", "")
# Hermeticity: the host env (prod .env; merge-gate re-test runs under it)
# may legitimately set ORCH_AGENT_FALLBACK_MODEL -> reset to shipped default.
monkeypatch.setattr(settings, "agent_fallback_model", "")
# default registry (no per-project overrides)
monkeypatch.setattr(P.settings, "projects_json", "")
reload_projects()
@@ -233,8 +236,12 @@ def test_valid_per_project_override_unchanged(monkeypatch):
# ---- TC-09 / TC-11: G4 fallback is OFF (ADR-001 decision 3) ------------------
def test_fallback_model_disabled_by_default():
# G4 not enabled: agent_fallback_model stays "" -> no --fallback-model flag.
assert settings.agent_fallback_model == ""
# G4 not enabled (ORCH-074 ADR-001, Решение 3): the SHIPPED default of
# agent_fallback_model is "" -> no --fallback-model flag out of the box.
# Assert the CLASS field default, not the runtime singleton: the host env
# may legitimately enable the fallback via ORCH_AGENT_FALLBACK_MODEL, and
# this test must stay hermetic (the merge-gate re-test runs under that env).
assert type(settings).model_fields["agent_fallback_model"].default == ""
# never-break: the SAME predicate guards the inline fallback read in _spawn,
# so a typo there would be rejected exactly like a model name.
assert is_valid_model("claude-bad typo") is False