Files
C.E.L_Slide_test2/tests/test_phase_z2_ai_fallback_config.py
kyeongmin 896f273ffa feat(#92): IMP-92 u1~u5 AI fallback config validation (model ping + operational error classification)
Replaces #84 UI-noise removal plan with positive operational-alert contract.
Five-axis stack lands together: (1) default model literal moved to current
Opus-family ID, (2) Anthropic SDK error classifier mapping exceptions to
quota/billing/auth/other, (3) api_error_kind plumbed through ai_repair_status
summary + per-record retention, (4) Step 0 preflight ping gated under
ai_fallback_enabled (default OFF preserved) with fail-fast on invalid
model/key, (5) frontend formatter rewritten to surface only operational
quota/billing/auth toasts (non-operational paths return null per
feedback_auto_pipeline_first silent-pipeline policy).

u1 - default model literal claude-opus-4-6-20250415 -> claude-opus-4-7
     (src/config.py + tests/test_phase_z2_ai_fallback_config.py lock mirror)
u2 - classify_operational_error type+status_code dispatch + Step 12
     api_error_kind stamp on except path (src/phase_z2_ai_fallback/client.py
     + src/phase_z2_ai_fallback/step12.py + tests/phase_z2_ai_fallback/test_step12.py)
u3 - _summarize_ai_repair_status aggregates api_error_kinds {quota,billing,
     auth,other}; error_records[i].api_error_kind retained per-record
     (src/phase_z2_pipeline.py + tests/test_imp47b_failure_surface.py)
u4 - _run_step0_ai_preflight + Step0PreflightError; preflight only fires
     when ai_fallback_enabled=true; one-token ping; invalid key/model =>
     setup failure before Step 1 (src/phase_z2_pipeline.py +
     tests/phase_z2/test_pipeline_step0_preflight.py NEW)
u5 - AiRepairStatus.api_error_kinds? interface + formatAiRepairHumanReview
     Message rewritten: operational quota/billing/auth -> Korean copy
     verbatim from issue body (tie-break quota -> billing -> auth);
     validation/coverage_violated/unsupported_kind/generic-other/legacy
     payload -> null (Front/client/src/services/designAgentApi.ts +
     Front/client/tests/imp47b_human_review_toast.test.tsx)

Guardrails respected:
- feedback_demo_env_toggle_policy: default OFF preserved; preflight skipped
  when ai_fallback_enabled=false (test_preflight_skipped_when_disabled
  asserts anthropic.Anthropic() not called).
- feedback_auto_pipeline_first: non-operational AI failures stay silent;
  only quota/billing/auth reach user toast.
- feedback_ai_isolation_contract: AI remains fallback-only; no normal-path
  migration; MDX preserved.
- project_imp46_carveout_caveat: cache_key/fingerprints fields untouched on
  every record; no overlap with #62 cache region.
- feedback_no_hardcoding: zero MDX-sample-specific literals; classifier
  dispatch by SDK type, not by string parsing.
- feedback_artifact_status_naming: operational toast scoped to alert axis,
  not overall PASS signal.

Tests:
- Targeted u1+u2+u3+u4: 63 passed
- u5 vitest (Front/): 10/10 passed
- tests/phase_z2_ai_fallback dir regression: 240 passed
- tests/phase_z2 dir regression: 323 passed
- IMP-92-adjacent (-k "imp47b or ai_fallback or preflight or step12 or step0"): 299 passed (808 deselected)
- u1 baseline lock (test_client_mock.py): 8 passed
Zero failures, zero regressions outside scope.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 22:07:25 +09:00

87 lines
3.4 KiB
Python

"""IMP-33 u1 — AI fallback Settings defaults (locked).
These defaults are the binding contract from Stage 2 plan (per-unit u1):
- ai_fallback_enabled = False (master flag OFF; fallback path only)
- ai_fallback_model = "claude-opus-4-7"
- ai_fallback_timeout_s = 60.0
- ai_fallback_max_retries = 3
- ai_fallback_backoff_base_s = 1.0
- ai_fallback_backoff_cap_s = 8.0
- ai_fallback_backoff_jitter = 0.3
- ai_fallback_budget_per_run = 10
- ai_fallback_circuit_breaker_threshold = 5
Downstream u4 (client) MUST source timeout/retry/backoff/budget/circuit from
Settings; inline literals are forbidden by Stage 2 plan.
"""
from __future__ import annotations
from src.config import Settings
def test_ai_fallback_master_flag_default_off() -> None:
s = Settings()
assert s.ai_fallback_enabled is False, (
"AI fallback master flag MUST default OFF (normal path AI=0 contract)."
)
def test_ai_fallback_model_default_locked() -> None:
s = Settings()
assert s.ai_fallback_model == "claude-opus-4-7"
def test_ai_fallback_retry_timeout_backoff_defaults_locked() -> None:
s = Settings()
assert s.ai_fallback_timeout_s == 60.0
assert s.ai_fallback_max_retries == 3
assert s.ai_fallback_backoff_base_s == 1.0
assert s.ai_fallback_backoff_cap_s == 8.0
assert s.ai_fallback_backoff_jitter == 0.3
def test_ai_fallback_budget_and_circuit_defaults_locked() -> None:
s = Settings()
assert s.ai_fallback_budget_per_run == 10
assert s.ai_fallback_circuit_breaker_threshold == 5
# IMP-46 u5 — auto-cache opt-in setting default lock.
# The CLI flag ``--auto-cache`` in src/phase_z2_pipeline.py mutates this
# setting at parse time. The default MUST stay OFF so the dual-gate
# contract (visual_check_passed AND user_approved) survives without an
# explicit operator opt-in.
def test_ai_fallback_auto_cache_default_off() -> None:
s = Settings()
assert s.ai_fallback_auto_cache is False, (
"IMP-46 u5 auto-cache MUST default OFF; the dual-gate contract "
"(visual_check_passed AND user_approved) survives without an "
"explicit --auto-cache opt-in."
)
# IMP-47B u1 — reject route hint policy correction.
# Prior to 2026-05-21 the reject V4 label routed to ``design_reference_only``
# (no AI). The user policy correction (issue #76) reroutes reject to
# ``ai_adaptation_required`` so the rank-1 reject frame is kept and the AI
# re-maps MDX content into its declared slots. Activation remains gated by
# ``ai_fallback_enabled`` (default OFF preserves the normal-path AI=0
# contract — see test_ai_fallback_master_flag_default_off above).
def test_reject_route_hint_routes_to_ai_adaptation() -> None:
from src.phase_z2_pipeline import _IMP05_ROUTE_HINTS, _imp05_route_hint
assert _IMP05_ROUTE_HINTS["reject"] == "ai_adaptation_required", (
"IMP-47B u1: reject must route to ai_adaptation_required so the "
"rank-1 reject frame is retained and AI re-maps MDX content into "
"its slots (frame auto-swap forbidden)."
)
assert _imp05_route_hint("reject") == "ai_adaptation_required"
# Sibling routes unchanged — guardrail against accidental drift.
assert _imp05_route_hint("use_as_is") == "direct_render"
assert _imp05_route_hint("light_edit") == "deterministic_minor_adjustment"
assert _imp05_route_hint("restructure") == "ai_adaptation_required"