kimminsung/recordingtest
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

runner: engine-bridge sidecar integration (#10)

Browse Source 
At the end of each scenario playback the runner now fetches a state
snapshot from the engine-bridge HTTP server and diffs it against an
approved engine-state baseline. This closes the engine-bridge v3 loop
and adds a semantic-state axis to the golden-file regression strategy —
diffs can now catch "camera moved" or "wrong selection after command"
even when the main saved-file diff still passes.

New:
  src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
    IEngineStateSnapshotClient.TryCapture() -> string? (never throws)
    HttpEngineStateSnapshotClient — GETs /scene /camera /selection off a
    base URL (default http://localhost:38080), composes them into
      { "scene": {...}, "camera": {...}, "selection": {...} }
    with stable ordering so the downstream differ stays friendly.
    Runner is Generic tier, so this client carries zero HmEG knowledge;
    it forwards raw JSON strings.

  TestRunner.RunAll now takes an optional IEngineStateSnapshotClient.
  After engine.Run() completes, CaptureAndDiffSidecar():
    - null client           -> SidecarStatus = "skipped"
    - TryCapture null/throw -> "unavailable" (main result still evaluated)
    - success               -> writes engine-state.received.json
    - baseline found        -> normalize + diff + "pass"/"fail"
    - baseline missing      -> "missing_baseline" (first run convention)
  A sidecar "fail" promotes the overall scenario Status to "fail" so
  exit code reflects semantic divergence even when the save-file diff
  agrees.

  ScenarioResult: SidecarCaptured / SidecarHunks / SidecarStatus.
  Markdown report grows Sidecar + Sidecar Hunks columns; JSON report
  picks up the new fields automatically via camelCase serialization.

  Program.cs: --sidecar-url <url> (default localhost:38080) and
  --no-sidecar. Default behaviour is sidecar-on so that a loaded
  bridge plugin is picked up automatically; when the plugin is not
  deployed the client silently reports "unavailable" and CI still runs.

Baseline lookup (new):
  <baselinesDir>/<scenario>.engine-state.approved.json
  <baselinesDir>/<scenario>.engine-state.json

Tests (Recordingtest.Runner.Tests, +6):
  - Sidecar_NullClient_SkippedStatus
  - Sidecar_ClientReturnsNull_UnavailableStatus
  - Sidecar_Throws_UnavailableStatus_MainStillPasses
  - Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile
  - Sidecar_Captured_BaselineIdentical_PassPass
  - Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail

Full suite 126 -> 132, 0 failures.

Follow-ups (PLAN.md):
  - Live loop: first run writes received, user approves, rerun passes.
  - Normalizer profile for engine-state (float epsilon for camera
    coords, selected_ids sort, document_path masking). Currently
    runs through the default profile as identity, so false fails are
    possible for sensitive camera moves until this lands.

Ref: #10 engine-bridge v3 final integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
minsung
2026-04-09 11:56:15 +09:00
parent 062a285462
commit e28a029704
8 changed files with 510 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

117
docs/history/2026-04-09_runner-sidecar-integration.md Normal file
View File

@@ -0,0 +1,117 @@
# 2026-04-09 — Runner ↔ engine-bridge sidecar 통합
**이슈**: #10 follow-up (engine-bridge v3 final integration)
**소요 시간**: ~35분
**Context 사용량**: input ~60k / output ~12k tokens (Opus 4.6)
## 결과
`TestRunner` 가 시나리오 재생 종료 직후 engine-bridge에서 `/scene` `/camera` `/selection` 스냅샷을 받아 `engine-state.received.json` 으로 기록하고, `<scenario>.engine-state.approved.json` 베이스라인과 별도 diff 패스를 돌린다. 이로써 golden-file 회귀 테스트의 **의미적(semantic) 차원**이 정식으로 파이프라인에 편입됐다.
## 구현
### 1. `IEngineStateSnapshotClient` (Generic tier)
경로: `src/Recordingtest.Runner/IEngineStateSnapshotClient.cs`
```csharp
public interface IEngineStateSnapshotClient
{
string? TryCapture(); // 실패 시 null, 예외 금지
}
```
기본 구현 `HttpEngineStateSnapshotClient`: `http://localhost:38080` 을 기본 base URL로 쓰고 `/scene` `/camera` `/selection` 을 각각 GET한 뒤 세 응답을 하나의 JSON 객체로 합친다:
```json
{"scene":{...},"camera":{...},"selection":{...}}
```
고정 순서(scene → camera → selection)로 diff 친화적. 타임아웃 기본 2초. `HttpClient` 소유권 지원 (외부 주입 가능).
Runner는 **Generic tier** 라서 HmEG 응답 shape를 모르고 raw JSON 문자열만 전달한다. 응답 해석 / 정규화는 하위 Normalizer가 담당.
### 2. `TestRunner.RunAll(..., IEngineStateSnapshotClient? sidecarClient = null)`
- 새 옵셔널 파라미터. null이면 기존 동작(sidecar skip).
- `engine.Run` 직후 `CaptureAndDiffSidecar` 호출. 순서 중요: 재생이 끝났지만 host/SUT가 아직 살아있을 때 상태를 찍어야 의미있는 스냅샷.
- 캡처 분기:
- client null → `SidecarStatus = "skipped"`
- `TryCapture()` null / throw → `"unavailable"`
- 성공 → `engine-state.received.json` 기록
- 베이스라인 조회:
- `<baselinesDir>/<scenario>.engine-state.approved.json`
- `<baselinesDir>/<scenario>.engine-state.json`
- 없으면 `"missing_baseline"` (첫 실행 때 정상)
- 있으면 normalizer pass → differ pass → `"pass"` / `"fail"`
- **sidecar diff가 fail이면 시나리오 전체 Status를 `"fail"`로 승격** (메인 result diff는 별도 pass)
- 모든 실패는 catch로 감싸 `"error"` 로 떨어뜨리고 `Error` 필드 prefix `sidecar:`
### 3. `ScenarioResult` 확장
```csharp
public bool SidecarCaptured { get; set; }
public int SidecarHunks { get; set; }
public string SidecarStatus { get; set; } = "skipped";
```
markdown 리포트 표에 Sidecar / Sidecar Hunks 컬럼 추가. JSON 리포트는 camelCase로 자동 직렬화.
### 4. `Program.cs` CLI
```
--sidecar-url <url> # 기본 http://localhost:38080
--no-sidecar # sidecar 비활성 (기존 동작)
```
기본값은 **sidecar 활성**. 브릿지 플러그인이 로드돼 있으면 자동으로 잡힌다. 로드 안 돼 있어도 `unavailable` 상태로 기록되고 main result 는 계속 평가되므로 CI 안전.
### 5. 테스트 (Runner.Tests)
6개 신규:
1. `Sidecar_NullClient_SkippedStatus` — null 클라이언트는 "skipped"
2. `Sidecar_ClientReturnsNull_UnavailableStatus` — 빈 응답은 "unavailable"
3. `Sidecar_Throws_UnavailableStatus_MainStillPasses` — 예외 삼키고 main은 pass
4. `Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile` — 첫 실행 시 received만 쓰고 missing_baseline
5. `Sidecar_Captured_BaselineIdentical_PassPass` — 베이스라인 일치 시 sidecar pass
6. `Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail` — 불일치 시 시나리오를 fail로 승격
`FakeSidecarClient` 로 `string?` payload / throw 스위치 제어.
**총 132 tests pass (126 → 132, +6).**
## 다음 단계 (라이브 검증 + 정규화 규칙)
### P1-A — 라이브 루프 검증
```
dotnet run --project src\Recordingtest.Runner -- ^
--scenarios scenarios ^
--baselines baselines ^
--out artifacts\runner-out
```
기대:
1. 첫 실행 → sidecar 상태 `missing_baseline`, `artifacts\runner-out\<scenario>\engine-state.received.json` 생성
2. 사용자가 파일을 베이스라인 폴더로 복사(approve) → `<scenario>.engine-state.approved.json`
3. 재실행 → sidecar 상태 `pass`
### P1-B — normalizer sidecar 규칙
현재는 identity normalize라서 float 경미한 차이나 selected_ids 순서 흔들림으로 false fail 날 수 있음. 필요 규칙:
- **float epsilon** — camera eye/target 좌표. 이미 있는 `float_epsilon` 규칙을 sidecar profile에 등록
- **selected_ids 정렬** — 선택 순서에 불변
- **document_path 마스킹** — 경로 내 사용자/임시 디렉터리 정규화
- **scene.object_count 절대 비교** (float 아닌 정수)
Normalizer profile `engine-state` 신규 작성 후 `--profile` 로 전달하거나 Runner가 sidecar에 한해 다른 profile을 쓰도록 확장.
## 관련
- `src/Recordingtest.Runner/IEngineStateSnapshotClient.cs` (신규)
- `src/Recordingtest.Runner/TestRunner.cs` (CaptureAndDiffSidecar, FindSidecarBaseline)
- `src/Recordingtest.Runner/RunReport.cs` (SidecarCaptured/Hunks/Status)
- `src/Recordingtest.Runner/Program.cs` (CLI 옵션)
- `tests/Recordingtest.Runner.Tests/TestRunnerTests.cs` (6 신규 테스트)