e28a029704
At the end of each scenario playback the runner now fetches a state
snapshot from the engine-bridge HTTP server and diffs it against an
approved engine-state baseline. This closes the engine-bridge v3 loop
and adds a semantic-state axis to the golden-file regression strategy —
diffs can now catch "camera moved" or "wrong selection after command"
even when the main saved-file diff still passes.
New:
src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
IEngineStateSnapshotClient.TryCapture() -> string? (never throws)
HttpEngineStateSnapshotClient — GETs /scene /camera /selection off a
base URL (default http://localhost:38080), composes them into
{ "scene": {...}, "camera": {...}, "selection": {...} }
with stable ordering so the downstream differ stays friendly.
Runner is Generic tier, so this client carries zero HmEG knowledge;
it forwards raw JSON strings.
TestRunner.RunAll now takes an optional IEngineStateSnapshotClient.
After engine.Run() completes, CaptureAndDiffSidecar():
- null client -> SidecarStatus = "skipped"
- TryCapture null/throw -> "unavailable" (main result still evaluated)
- success -> writes engine-state.received.json
- baseline found -> normalize + diff + "pass"/"fail"
- baseline missing -> "missing_baseline" (first run convention)
A sidecar "fail" promotes the overall scenario Status to "fail" so
exit code reflects semantic divergence even when the save-file diff
agrees.
ScenarioResult: SidecarCaptured / SidecarHunks / SidecarStatus.
Markdown report grows Sidecar + Sidecar Hunks columns; JSON report
picks up the new fields automatically via camelCase serialization.
Program.cs: --sidecar-url <url> (default localhost:38080) and
--no-sidecar. Default behaviour is sidecar-on so that a loaded
bridge plugin is picked up automatically; when the plugin is not
deployed the client silently reports "unavailable" and CI still runs.
Baseline lookup (new):
<baselinesDir>/<scenario>.engine-state.approved.json
<baselinesDir>/<scenario>.engine-state.json
Tests (Recordingtest.Runner.Tests, +6):
- Sidecar_NullClient_SkippedStatus
- Sidecar_ClientReturnsNull_UnavailableStatus
- Sidecar_Throws_UnavailableStatus_MainStillPasses
- Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile
- Sidecar_Captured_BaselineIdentical_PassPass
- Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail
Full suite 126 -> 132, 0 failures.
Follow-ups (PLAN.md):
- Live loop: first run writes received, user approves, rerun passes.
- Normalizer profile for engine-state (float epsilon for camera
coords, selected_ids sort, document_path masking). Currently
runs through the default profile as identity, so false fails are
possible for sensitive camera moves until this lands.
Ref: #10 engine-bridge v3 final integration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
41 lines
2.3 KiB
Markdown
41 lines
2.3 KiB
Markdown
# PLAN.md — recordingtest
|
|
|
|
> 다음에 할 일의 *우선순위 큐*. 에이전트는 상단부터 집어든다.
|
|
> 비자명한 항목은 `/contract <name>` → `docs/contracts/<name>.md` 로 Sprint Contract 먼저 작성.
|
|
|
|
## P0 — 지금 바로
|
|
|
|
1. **훅 동작 검증** — SessionStart/Stop/Guard 3개 shell 스크립트를 실제로 트리거시켜 확인
|
|
- 의존: jq 설치 여부 확인
|
|
|
|
## P1 — 다음 통합 단계
|
|
|
|
4. **Runner sidecar 라이브 검증** — `dotnet run --project src\Recordingtest.Runner -- --scenarios scenarios --baselines baselines --out artifacts\runner-out` 실 실행 → `engine-state.received.json` 생성 확인 → 사용자가 approve 해서 `<scenario>.engine-state.approved.json` 베이스라인 승격 → 재실행 시 sidecar diff pass 확인.
|
|
5. **normalizer: engine-state 정규화 규칙** — 부동소수점 epsilon, selected_ids 정렬, camera up/fov 기본값 마스킹 등 sidecar JSON 전용 규칙. 현재는 identity 정규화로 지나감.
|
|
6. ~~recorder Gap I-1~~ — **deferred**. UIA poller PoC 결과 본질적 한계 확인 (AutomationPeer 부재 컨트롤은 못 봄). generic WPF DLL injection 또는 AutomationPeer AI 부착 PoC가 선결.
|
|
|
|
## Follow-ups (non-blocking)
|
|
|
|
- **sut-prober snake_case JSON** — `JsonNamingPolicy.SnakeCaseLower` 적용. Evaluator가 pass 처리했지만 contract 엄격 준수를 위해 권장.
|
|
|
|
## P2 — 통합
|
|
|
|
7. **test-runner** — 시나리오 일괄 실행 + 실패 triage
|
|
- 의존: recorder + player + normalizer + diff-reporter 전부
|
|
8. **engine-bridge 탐색** — HmEG PDB 리플렉션으로 카메라/선택 상태 접근 가능 여부 확인
|
|
- Sprint Contract 필요
|
|
9. **AutomationPeer PR 전략 PoC** — SUT fork에 AI로 AutomationPeer 자동 부착 샘플
|
|
|
|
## P3 — 상시
|
|
|
|
- **Scaffolding review** (PoC 3개마다)
|
|
- **.claude/ 비계 감사** — 기사 원칙: 불필요한 비계 제거
|
|
- **history 파일 누락 체크** — Stop hook이 경고
|
|
|
|
## 규칙
|
|
|
|
- 항목을 집을 때 PROGRESS.md의 "In progress"에 기록하고 본인(에이전트명/세션) 명시.
|
|
- 비자명한 P0/P1/P2 항목은 반드시 Sprint Contract 먼저.
|
|
- Generator와 Evaluator는 같은 세션이 겸하지 않는다.
|
|
- 완료 시: evaluator pass → PROGRESS.md Done 이동 → PLAN.md 제거 → history 작성 → `/handoff`.
|