fix(tests): hermetic ORCH-41 model/effort tests vs host env (unblock merge-gate)
Merge-gate re-test runs under the orchestrator's prod env, where the operator legitimately set ORCH_AGENT_FALLBACK_MODEL and changed ORCH_AGENT_MODEL_DEFAULT / ORCH_AGENT_EFFORT_*. Two ORCH-41-era tests asserted SHIPPED defaults through the env-backed settings singleton and failed 3/3 there, while Gitea CI (clean env) stayed green. Branch ORCH-009 touches neither src/ nor these tests - latent non-hermetic landmine on main, detonated by the prod env change. - test_resolve_agent_effort.py: autouse fixture now mirrors the sibling model-file baseline (pins shipped model/fallback fields) so the flag-assembly tests are env-independent. - test_resolve_agent_model.py: fixture also resets agent_fallback_model; test_fallback_model_disabled_by_default now asserts the CLASS field default (the actual ORCH-074 ADR-001 G4 invariant: shipped default is ""), never-break is_valid_model asserts unchanged byte-for-byte. Clean-env behaviour is byte-equivalent (fixtures pin exactly what an empty env yields). Full suite: 1713 passed (was 2 failed / 1711). Refs: ORCH-009 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -42,6 +42,15 @@ def _clean_settings(monkeypatch):
|
||||
monkeypatch.setattr(settings, "agent_effort_default", "high")
|
||||
for a, e in CANON_EFFORT.items():
|
||||
monkeypatch.setattr(settings, f"agent_effort_{a}", e)
|
||||
# Hermeticity (mirrors test_resolve_agent_model's baseline): the flag-assembly
|
||||
# tests below also read the MODEL/fallback fields, and the host env (prod .env;
|
||||
# the merge-gate re-test runs under it) may legitimately set
|
||||
# ORCH_AGENT_MODEL_* / ORCH_AGENT_FALLBACK_MODEL. These tests assert
|
||||
# shipped-default behaviour, not the host config -> pin the shipped defaults.
|
||||
monkeypatch.setattr(settings, "agent_model_default", "claude-opus-4-8")
|
||||
for a in CANON_EFFORT:
|
||||
monkeypatch.setattr(settings, f"agent_model_{a}", "")
|
||||
monkeypatch.setattr(settings, "agent_fallback_model", "")
|
||||
monkeypatch.setattr(P.settings, "projects_json", "")
|
||||
reload_projects()
|
||||
yield
|
||||
|
||||
@@ -41,6 +41,9 @@ def _clean_settings(monkeypatch):
|
||||
monkeypatch.setattr(settings, "agent_model_default", "claude-opus-4-8")
|
||||
for a in ("analyst", "architect", "developer", "reviewer", "tester", "deployer"):
|
||||
monkeypatch.setattr(settings, f"agent_model_{a}", "")
|
||||
# Hermeticity: the host env (prod .env; merge-gate re-test runs under it)
|
||||
# may legitimately set ORCH_AGENT_FALLBACK_MODEL -> reset to shipped default.
|
||||
monkeypatch.setattr(settings, "agent_fallback_model", "")
|
||||
# default registry (no per-project overrides)
|
||||
monkeypatch.setattr(P.settings, "projects_json", "")
|
||||
reload_projects()
|
||||
@@ -233,8 +236,12 @@ def test_valid_per_project_override_unchanged(monkeypatch):
|
||||
|
||||
# ---- TC-09 / TC-11: G4 fallback is OFF (ADR-001 decision 3) ------------------
|
||||
def test_fallback_model_disabled_by_default():
|
||||
# G4 not enabled: agent_fallback_model stays "" -> no --fallback-model flag.
|
||||
assert settings.agent_fallback_model == ""
|
||||
# G4 not enabled (ORCH-074 ADR-001, Решение 3): the SHIPPED default of
|
||||
# agent_fallback_model is "" -> no --fallback-model flag out of the box.
|
||||
# Assert the CLASS field default, not the runtime singleton: the host env
|
||||
# may legitimately enable the fallback via ORCH_AGENT_FALLBACK_MODEL, and
|
||||
# this test must stay hermetic (the merge-gate re-test runs under that env).
|
||||
assert type(settings).model_fields["agent_fallback_model"].default == ""
|
||||
# never-break: the SAME predicate guards the inline fallback read in _spawn,
|
||||
# so a typo there would be rejected exactly like a model name.
|
||||
assert is_valid_model("claude-bad typo") is False
|
||||
|
||||
Reference in New Issue
Block a user