Files

minsung e28a029704 runner: engine-bridge sidecar integration (#10 )

At the end of each scenario playback the runner now fetches a state
snapshot from the engine-bridge HTTP server and diffs it against an
approved engine-state baseline. This closes the engine-bridge v3 loop
and adds a semantic-state axis to the golden-file regression strategy —
diffs can now catch "camera moved" or "wrong selection after command"
even when the main saved-file diff still passes.

New:
  src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
    IEngineStateSnapshotClient.TryCapture() -> string? (never throws)
    HttpEngineStateSnapshotClient — GETs /scene /camera /selection off a
    base URL (default http://localhost:38080), composes them into
      { "scene": {...}, "camera": {...}, "selection": {...} }
    with stable ordering so the downstream differ stays friendly.
    Runner is Generic tier, so this client carries zero HmEG knowledge;
    it forwards raw JSON strings.

  TestRunner.RunAll now takes an optional IEngineStateSnapshotClient.
  After engine.Run() completes, CaptureAndDiffSidecar():
    - null client           -> SidecarStatus = "skipped"
    - TryCapture null/throw -> "unavailable" (main result still evaluated)
    - success               -> writes engine-state.received.json
    - baseline found        -> normalize + diff + "pass"/"fail"
    - baseline missing      -> "missing_baseline" (first run convention)
  A sidecar "fail" promotes the overall scenario Status to "fail" so
  exit code reflects semantic divergence even when the save-file diff
  agrees.

  ScenarioResult: SidecarCaptured / SidecarHunks / SidecarStatus.
  Markdown report grows Sidecar + Sidecar Hunks columns; JSON report
  picks up the new fields automatically via camelCase serialization.

  Program.cs: --sidecar-url <url> (default localhost:38080) and
  --no-sidecar. Default behaviour is sidecar-on so that a loaded
  bridge plugin is picked up automatically; when the plugin is not
  deployed the client silently reports "unavailable" and CI still runs.

Baseline lookup (new):
  <baselinesDir>/<scenario>.engine-state.approved.json
  <baselinesDir>/<scenario>.engine-state.json

Tests (Recordingtest.Runner.Tests, +6):
  - Sidecar_NullClient_SkippedStatus
  - Sidecar_ClientReturnsNull_UnavailableStatus
  - Sidecar_Throws_UnavailableStatus_MainStillPasses
  - Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile
  - Sidecar_Captured_BaselineIdentical_PassPass
  - Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail

Full suite 126 -> 132, 0 failures.

Follow-ups (PLAN.md):
  - Live loop: first run writes received, user approves, rerun passes.
  - Normalizer profile for engine-state (float epsilon for camera
    coords, selected_ids sort, document_path masking). Currently
    runs through the default profile as identity, so false fails are
    possible for sensitive camera moves until this lands.

Ref: #10 engine-bridge v3 final integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-09 11:56:15 +09:00

2.3 KiB

Raw Blame History

PLAN.md — recordingtest

다음에 할 일의 우선순위 큐. 에이전트는 상단부터 집어든다. 비자명한 항목은 /contract <name> → docs/contracts/<name>.md 로 Sprint Contract 먼저 작성.

P0 — 지금 바로

훅 동작 검증 — SessionStart/Stop/Guard 3개 shell 스크립트를 실제로 트리거시켜 확인
- 의존: jq 설치 여부 확인

P1 — 다음 통합 단계

Runner sidecar 라이브 검증 — dotnet run --project src\Recordingtest.Runner -- --scenarios scenarios --baselines baselines --out artifacts\runner-out 실 실행 → engine-state.received.json 생성 확인 → 사용자가 approve 해서 <scenario>.engine-state.approved.json 베이스라인 승격 → 재실행 시 sidecar diff pass 확인.
normalizer: engine-state 정규화 규칙 — 부동소수점 epsilon, selected_ids 정렬, camera up/fov 기본값 마스킹 등 sidecar JSON 전용 규칙. 현재는 identity 정규화로 지나감.
~~recorder Gap I-1~~ — deferred. UIA poller PoC 결과 본질적 한계 확인 (AutomationPeer 부재 컨트롤은 못 봄). generic WPF DLL injection 또는 AutomationPeer AI 부착 PoC가 선결.

Follow-ups (non-blocking)

sut-prober snake_case JSON — JsonNamingPolicy.SnakeCaseLower 적용. Evaluator가 pass 처리했지만 contract 엄격 준수를 위해 권장.

P2 — 통합

test-runner — 시나리오 일괄 실행 + 실패 triage
- 의존: recorder + player + normalizer + diff-reporter 전부
engine-bridge 탐색 — HmEG PDB 리플렉션으로 카메라/선택 상태 접근 가능 여부 확인
- Sprint Contract 필요
AutomationPeer PR 전략 PoC — SUT fork에 AI로 AutomationPeer 자동 부착 샘플

P3 — 상시

Scaffolding review (PoC 3개마다)
.claude/ 비계 감사 — 기사 원칙: 불필요한 비계 제거
history 파일 누락 체크 — Stop hook이 경고

규칙

항목을 집을 때 PROGRESS.md의 "In progress"에 기록하고 본인(에이전트명/세션) 명시.
비자명한 P0/P1/P2 항목은 반드시 Sprint Contract 먼저.
Generator와 Evaluator는 같은 세션이 겸하지 않는다.
완료 시: evaluator pass → PROGRESS.md Done 이동 → PLAN.md 제거 → history 작성 → /handoff.

2.3 KiB Raw Blame History

PLAN.md — recordingtest

P0 — 지금 바로

P1 — 다음 통합 단계

Follow-ups (non-blocking)

P2 — 통합

P3 — 상시

규칙

2.3 KiB

Raw Blame History