runner: engine-bridge sidecar integration (#10)
At the end of each scenario playback the runner now fetches a state
snapshot from the engine-bridge HTTP server and diffs it against an
approved engine-state baseline. This closes the engine-bridge v3 loop
and adds a semantic-state axis to the golden-file regression strategy —
diffs can now catch "camera moved" or "wrong selection after command"
even when the main saved-file diff still passes.
New:
src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
IEngineStateSnapshotClient.TryCapture() -> string? (never throws)
HttpEngineStateSnapshotClient — GETs /scene /camera /selection off a
base URL (default http://localhost:38080), composes them into
{ "scene": {...}, "camera": {...}, "selection": {...} }
with stable ordering so the downstream differ stays friendly.
Runner is Generic tier, so this client carries zero HmEG knowledge;
it forwards raw JSON strings.
TestRunner.RunAll now takes an optional IEngineStateSnapshotClient.
After engine.Run() completes, CaptureAndDiffSidecar():
- null client -> SidecarStatus = "skipped"
- TryCapture null/throw -> "unavailable" (main result still evaluated)
- success -> writes engine-state.received.json
- baseline found -> normalize + diff + "pass"/"fail"
- baseline missing -> "missing_baseline" (first run convention)
A sidecar "fail" promotes the overall scenario Status to "fail" so
exit code reflects semantic divergence even when the save-file diff
agrees.
ScenarioResult: SidecarCaptured / SidecarHunks / SidecarStatus.
Markdown report grows Sidecar + Sidecar Hunks columns; JSON report
picks up the new fields automatically via camelCase serialization.
Program.cs: --sidecar-url <url> (default localhost:38080) and
--no-sidecar. Default behaviour is sidecar-on so that a loaded
bridge plugin is picked up automatically; when the plugin is not
deployed the client silently reports "unavailable" and CI still runs.
Baseline lookup (new):
<baselinesDir>/<scenario>.engine-state.approved.json
<baselinesDir>/<scenario>.engine-state.json
Tests (Recordingtest.Runner.Tests, +6):
- Sidecar_NullClient_SkippedStatus
- Sidecar_ClientReturnsNull_UnavailableStatus
- Sidecar_Throws_UnavailableStatus_MainStillPasses
- Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile
- Sidecar_Captured_BaselineIdentical_PassPass
- Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail
Full suite 126 -> 132, 0 failures.
Follow-ups (PLAN.md):
- Live loop: first run writes received, user approves, rerun passes.
- Normalizer profile for engine-state (float epsilon for camera
coords, selected_ids sort, document_path masking). Currently
runs through the default profile as identity, so false fails are
possible for sensitive camera moves until this lands.
Ref: #10 engine-bridge v3 final integration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
5
PLAN.md
5
PLAN.md
@@ -10,8 +10,9 @@
|
||||
|
||||
## P1 — 다음 통합 단계
|
||||
|
||||
4. **Runner + engine-bridge sidecar 연결** — 시나리오 실행 종료 시점에 `/scene` `/camera` `/selection` 을 한 번 더 스냅샷해 sidecar JSON으로 베이스라인에 포함. golden file의 의미적(semantic) 차원 확보.
|
||||
5. ~~recorder Gap I-1~~ — **deferred**. UIA poller PoC 결과 본질적 한계 확인 (AutomationPeer 부재 컨트롤은 못 봄). generic WPF DLL injection 또는 AutomationPeer AI 부착 PoC가 선결.
|
||||
4. **Runner sidecar 라이브 검증** — `dotnet run --project src\Recordingtest.Runner -- --scenarios scenarios --baselines baselines --out artifacts\runner-out` 실 실행 → `engine-state.received.json` 생성 확인 → 사용자가 approve 해서 `<scenario>.engine-state.approved.json` 베이스라인 승격 → 재실행 시 sidecar diff pass 확인.
|
||||
5. **normalizer: engine-state 정규화 규칙** — 부동소수점 epsilon, selected_ids 정렬, camera up/fov 기본값 마스킹 등 sidecar JSON 전용 규칙. 현재는 identity 정규화로 지나감.
|
||||
6. ~~recorder Gap I-1~~ — **deferred**. UIA poller PoC 결과 본질적 한계 확인 (AutomationPeer 부재 컨트롤은 못 봄). generic WPF DLL injection 또는 AutomationPeer AI 부착 PoC가 선결.
|
||||
|
||||
## Follow-ups (non-blocking)
|
||||
|
||||
|
||||
@@ -49,7 +49,8 @@
|
||||
| 2026-04-09 | **3-tier 분리 1단계 (incremental)** — `Recordingtest.Bridge.Abstractions` (Generic) + `Recordingtest.Hmeg.Bridge` (HmEG-aware) 신설, `HmegDirectStateProvider` + `ChainedEngineStateProvider` wire-up, 115 tests | commit `f6b6e74` |
|
||||
| 2026-04-09 | **3-tier 분리 2단계** — `EgPlugin` → `Sut/EgBim/Recordingtest.Sut.EgBim.PluginHost`, `EngineBridge` → `Hmeg/Recordingtest.Hmeg.Catalog`, `EngineBridge.Client` → `Hmeg/Recordingtest.Hmeg.Bridge.Client`, `EngineBridge.Probe` → `Hmeg/Recordingtest.Hmeg.Catalog.Probe`, 테스트 동일. `Recordingtest.Architecture.Tests` 11건 추가 (의존 그래프 강제). 126 tests | commit pending |
|
||||
| 2026-04-09 | **engine-bridge v3 EgBim 람다 wire-up** — `EditorPlugin` base 직접 사용: `RootSpace`, `View(EGViewport:HmEGViewport)`, `AppManager.FileManager.CurrentFile`. `HmegDirectStateProvider` 이제 실값 가능. `Editor02.HmEGAppManager.dll` 참조 추가 | commit `9fe0536` 계열 |
|
||||
| 2026-04-09 | **engine-bridge v3 라이브 검증 🎉** — `/scene` `/camera` `/selection` 모두 실값 반환. viewport 람다를 `AppManager.ViewportManager.FocusedViewport` 로 교체 (`View`는 `Run()` 안 타는 bridge plugin에서 항상 null). deploy-egbim-plugin.bat 추가 | commit pending |
|
||||
| 2026-04-09 | **engine-bridge v3 라이브 검증 🎉** — `/scene` `/camera` `/selection` 모두 실값 반환. viewport 람다를 `AppManager.ViewportManager.FocusedViewport` 로 교체 (`View`는 `Run()` 안 타는 bridge plugin에서 항상 null). deploy-egbim-plugin.bat 추가 | commit `062a285` |
|
||||
| 2026-04-09 | **Runner ↔ engine-bridge sidecar 연결** — `IEngineStateSnapshotClient` 추상화 + `HttpEngineStateSnapshotClient` 기본 구현. TestRunner가 시나리오 재생 종료 시점에 `/scene /camera /selection` 스냅샷 → `engine-state.received.json` 기록 → `<scenario>.engine-state.approved.json` 베이스라인과 diff → sidecar 불일치 시 시나리오 fail 승격. `--sidecar-url` / `--no-sidecar` CLI 옵션. sidecar 6개 신규 테스트(skipped/unavailable/missing_baseline/pass/fail). 132 tests | commit pending |
|
||||
|
||||
## In progress
|
||||
|
||||
|
||||
117
docs/history/2026-04-09_runner-sidecar-integration.md
Normal file
117
docs/history/2026-04-09_runner-sidecar-integration.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# 2026-04-09 — Runner ↔ engine-bridge sidecar 통합
|
||||
|
||||
**이슈**: #10 follow-up (engine-bridge v3 final integration)
|
||||
**소요 시간**: ~35분
|
||||
**Context 사용량**: input ~60k / output ~12k tokens (Opus 4.6)
|
||||
|
||||
## 결과
|
||||
|
||||
`TestRunner` 가 시나리오 재생 종료 직후 engine-bridge에서 `/scene` `/camera` `/selection` 스냅샷을 받아 `engine-state.received.json` 으로 기록하고, `<scenario>.engine-state.approved.json` 베이스라인과 별도 diff 패스를 돌린다. 이로써 golden-file 회귀 테스트의 **의미적(semantic) 차원**이 정식으로 파이프라인에 편입됐다.
|
||||
|
||||
## 구현
|
||||
|
||||
### 1. `IEngineStateSnapshotClient` (Generic tier)
|
||||
|
||||
경로: `src/Recordingtest.Runner/IEngineStateSnapshotClient.cs`
|
||||
|
||||
```csharp
|
||||
public interface IEngineStateSnapshotClient
|
||||
{
|
||||
string? TryCapture(); // 실패 시 null, 예외 금지
|
||||
}
|
||||
```
|
||||
|
||||
기본 구현 `HttpEngineStateSnapshotClient`: `http://localhost:38080` 을 기본 base URL로 쓰고 `/scene` `/camera` `/selection` 을 각각 GET한 뒤 세 응답을 하나의 JSON 객체로 합친다:
|
||||
|
||||
```json
|
||||
{"scene":{...},"camera":{...},"selection":{...}}
|
||||
```
|
||||
|
||||
고정 순서(scene → camera → selection)로 diff 친화적. 타임아웃 기본 2초. `HttpClient` 소유권 지원 (외부 주입 가능).
|
||||
|
||||
Runner는 **Generic tier** 라서 HmEG 응답 shape를 모르고 raw JSON 문자열만 전달한다. 응답 해석 / 정규화는 하위 Normalizer가 담당.
|
||||
|
||||
### 2. `TestRunner.RunAll(..., IEngineStateSnapshotClient? sidecarClient = null)`
|
||||
|
||||
- 새 옵셔널 파라미터. null이면 기존 동작(sidecar skip).
|
||||
- `engine.Run` 직후 `CaptureAndDiffSidecar` 호출. 순서 중요: 재생이 끝났지만 host/SUT가 아직 살아있을 때 상태를 찍어야 의미있는 스냅샷.
|
||||
- 캡처 분기:
|
||||
- client null → `SidecarStatus = "skipped"`
|
||||
- `TryCapture()` null / throw → `"unavailable"`
|
||||
- 성공 → `engine-state.received.json` 기록
|
||||
- 베이스라인 조회:
|
||||
- `<baselinesDir>/<scenario>.engine-state.approved.json`
|
||||
- `<baselinesDir>/<scenario>.engine-state.json`
|
||||
- 없으면 `"missing_baseline"` (첫 실행 때 정상)
|
||||
- 있으면 normalizer pass → differ pass → `"pass"` / `"fail"`
|
||||
- **sidecar diff가 fail이면 시나리오 전체 Status를 `"fail"`로 승격** (메인 result diff는 별도 pass)
|
||||
- 모든 실패는 catch로 감싸 `"error"` 로 떨어뜨리고 `Error` 필드 prefix `sidecar:`
|
||||
|
||||
### 3. `ScenarioResult` 확장
|
||||
|
||||
```csharp
|
||||
public bool SidecarCaptured { get; set; }
|
||||
public int SidecarHunks { get; set; }
|
||||
public string SidecarStatus { get; set; } = "skipped";
|
||||
```
|
||||
|
||||
markdown 리포트 표에 Sidecar / Sidecar Hunks 컬럼 추가. JSON 리포트는 camelCase로 자동 직렬화.
|
||||
|
||||
### 4. `Program.cs` CLI
|
||||
|
||||
```
|
||||
--sidecar-url <url> # 기본 http://localhost:38080
|
||||
--no-sidecar # sidecar 비활성 (기존 동작)
|
||||
```
|
||||
|
||||
기본값은 **sidecar 활성**. 브릿지 플러그인이 로드돼 있으면 자동으로 잡힌다. 로드 안 돼 있어도 `unavailable` 상태로 기록되고 main result 는 계속 평가되므로 CI 안전.
|
||||
|
||||
### 5. 테스트 (Runner.Tests)
|
||||
|
||||
6개 신규:
|
||||
|
||||
1. `Sidecar_NullClient_SkippedStatus` — null 클라이언트는 "skipped"
|
||||
2. `Sidecar_ClientReturnsNull_UnavailableStatus` — 빈 응답은 "unavailable"
|
||||
3. `Sidecar_Throws_UnavailableStatus_MainStillPasses` — 예외 삼키고 main은 pass
|
||||
4. `Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile` — 첫 실행 시 received만 쓰고 missing_baseline
|
||||
5. `Sidecar_Captured_BaselineIdentical_PassPass` — 베이스라인 일치 시 sidecar pass
|
||||
6. `Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail` — 불일치 시 시나리오를 fail로 승격
|
||||
|
||||
`FakeSidecarClient` 로 `string?` payload / throw 스위치 제어.
|
||||
|
||||
**총 132 tests pass (126 → 132, +6).**
|
||||
|
||||
## 다음 단계 (라이브 검증 + 정규화 규칙)
|
||||
|
||||
### P1-A — 라이브 루프 검증
|
||||
|
||||
```
|
||||
dotnet run --project src\Recordingtest.Runner -- ^
|
||||
--scenarios scenarios ^
|
||||
--baselines baselines ^
|
||||
--out artifacts\runner-out
|
||||
```
|
||||
|
||||
기대:
|
||||
1. 첫 실행 → sidecar 상태 `missing_baseline`, `artifacts\runner-out\<scenario>\engine-state.received.json` 생성
|
||||
2. 사용자가 파일을 베이스라인 폴더로 복사(approve) → `<scenario>.engine-state.approved.json`
|
||||
3. 재실행 → sidecar 상태 `pass`
|
||||
|
||||
### P1-B — normalizer sidecar 규칙
|
||||
|
||||
현재는 identity normalize라서 float 경미한 차이나 selected_ids 순서 흔들림으로 false fail 날 수 있음. 필요 규칙:
|
||||
|
||||
- **float epsilon** — camera eye/target 좌표. 이미 있는 `float_epsilon` 규칙을 sidecar profile에 등록
|
||||
- **selected_ids 정렬** — 선택 순서에 불변
|
||||
- **document_path 마스킹** — 경로 내 사용자/임시 디렉터리 정규화
|
||||
- **scene.object_count 절대 비교** (float 아닌 정수)
|
||||
|
||||
Normalizer profile `engine-state` 신규 작성 후 `--profile` 로 전달하거나 Runner가 sidecar에 한해 다른 profile을 쓰도록 확장.
|
||||
|
||||
## 관련
|
||||
|
||||
- `src/Recordingtest.Runner/IEngineStateSnapshotClient.cs` (신규)
|
||||
- `src/Recordingtest.Runner/TestRunner.cs` (CaptureAndDiffSidecar, FindSidecarBaseline)
|
||||
- `src/Recordingtest.Runner/RunReport.cs` (SidecarCaptured/Hunks/Status)
|
||||
- `src/Recordingtest.Runner/Program.cs` (CLI 옵션)
|
||||
- `tests/Recordingtest.Runner.Tests/TestRunnerTests.cs` (6 신규 테스트)
|
||||
95
src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
Normal file
95
src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
Normal file
@@ -0,0 +1,95 @@
|
||||
using System.Net.Http;
|
||||
|
||||
namespace Recordingtest.Runner;
|
||||
|
||||
/// <summary>
|
||||
/// Generic (SUT-neutral) abstraction for capturing the engine state sidecar
|
||||
/// at the end of a scenario playback. Implementations usually hit an
|
||||
/// in-process HTTP bridge exposed by a plugin inside the SUT. Returns the
|
||||
/// JSON body as a single string (pre-concatenated across endpoints) or
|
||||
/// <c>null</c> if the bridge is unreachable, so the runner can degrade
|
||||
/// gracefully when the sidecar is simply not available.
|
||||
/// </summary>
|
||||
public interface IEngineStateSnapshotClient
|
||||
{
|
||||
/// <summary>
|
||||
/// Best-effort capture. Never throws — returns null on any failure.
|
||||
/// </summary>
|
||||
string? TryCapture();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Default HTTP implementation. GETs the three well-known read-only
|
||||
/// endpoints (/scene, /camera, /selection) off a base URL and composes the
|
||||
/// three responses into one JSON object:
|
||||
///
|
||||
/// { "scene": {...}, "camera": {...}, "selection": {...} }
|
||||
///
|
||||
/// The Runner is the Generic tier, so this client must not know about any
|
||||
/// HmEG-specific response shape — it simply forwards the raw JSON strings.
|
||||
/// Normalization and tolerance rules are applied downstream by Normalizer.
|
||||
/// </summary>
|
||||
public sealed class HttpEngineStateSnapshotClient : IEngineStateSnapshotClient, IDisposable
|
||||
{
|
||||
public const string DefaultBaseUrl = "http://localhost:38080";
|
||||
private readonly string _baseUrl;
|
||||
private readonly HttpClient _http;
|
||||
private readonly bool _ownsClient;
|
||||
|
||||
public HttpEngineStateSnapshotClient(
|
||||
string baseUrl = DefaultBaseUrl,
|
||||
HttpClient? httpClient = null,
|
||||
TimeSpan? timeout = null)
|
||||
{
|
||||
_baseUrl = baseUrl.TrimEnd('/');
|
||||
if (httpClient is null)
|
||||
{
|
||||
_http = new HttpClient { Timeout = timeout ?? TimeSpan.FromSeconds(2) };
|
||||
_ownsClient = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
_http = httpClient;
|
||||
if (timeout.HasValue) _http.Timeout = timeout.Value;
|
||||
_ownsClient = false;
|
||||
}
|
||||
}
|
||||
|
||||
public string? TryCapture()
|
||||
{
|
||||
try
|
||||
{
|
||||
var scene = Get("/scene");
|
||||
var camera = Get("/camera");
|
||||
var selection = Get("/selection");
|
||||
if (scene is null || camera is null || selection is null) return null;
|
||||
// Stable ordering (scene, camera, selection) → diff-friendly.
|
||||
return "{\"scene\":" + scene +
|
||||
",\"camera\":" + camera +
|
||||
",\"selection\":" + selection + "}";
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private string? Get(string endpoint)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var resp = _http.GetAsync(_baseUrl + endpoint).GetAwaiter().GetResult();
|
||||
if (!resp.IsSuccessStatusCode) return null;
|
||||
return resp.Content.ReadAsStringAsync().GetAwaiter().GetResult();
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_ownsClient) _http.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,8 @@ public static class Program
|
||||
public static int Main(string[] args)
|
||||
{
|
||||
var options = new RunnerOptions();
|
||||
string? sidecarUrl = null;
|
||||
bool noSidecar = false;
|
||||
for (int i = 0; i < args.Length; i++)
|
||||
{
|
||||
switch (args[i])
|
||||
@@ -14,9 +16,13 @@ public static class Program
|
||||
case "--out": options.OutDir = args[++i]; break;
|
||||
case "--profile": options.Profile = args[++i]; break;
|
||||
case "--no-launch": options.NoLaunch = true; break;
|
||||
case "--sidecar-url": sidecarUrl = args[++i]; break;
|
||||
case "--no-sidecar": noSidecar = true; break;
|
||||
case "-h":
|
||||
case "--help":
|
||||
Console.WriteLine("Usage: Recordingtest.Runner --scenarios <dir> --baselines <dir> --out <dir> [--profile <name>] [--no-launch]");
|
||||
Console.WriteLine("Usage: Recordingtest.Runner --scenarios <dir> --baselines <dir> --out <dir>");
|
||||
Console.WriteLine(" [--profile <name>] [--no-launch]");
|
||||
Console.WriteLine(" [--sidecar-url http://localhost:38080] [--no-sidecar]");
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
@@ -29,12 +35,24 @@ public static class Program
|
||||
return 2;
|
||||
}
|
||||
|
||||
// engine-bridge v3 — default to the well-known localhost bridge port
|
||||
// unless the caller explicitly opts out or overrides the URL.
|
||||
IEngineStateSnapshotClient? sidecar = null;
|
||||
if (!noSidecar)
|
||||
{
|
||||
sidecar = new HttpEngineStateSnapshotClient(
|
||||
sidecarUrl ?? HttpEngineStateSnapshotClient.DefaultBaseUrl);
|
||||
}
|
||||
|
||||
var runner = new TestRunner();
|
||||
var report = runner.RunAll(
|
||||
options,
|
||||
new DefaultHostFactory(),
|
||||
new DefaultNormalizer(),
|
||||
new DefaultDiffer());
|
||||
new DefaultDiffer(),
|
||||
sidecarClient: sidecar);
|
||||
|
||||
(sidecar as IDisposable)?.Dispose();
|
||||
|
||||
Console.WriteLine($"Total: {report.Total}, Passed: {report.Passed}, Failed: {report.Failed}, Errored: {report.Errored}");
|
||||
return TestRunner.ToExitCode(report);
|
||||
|
||||
@@ -18,4 +18,12 @@ public sealed class ScenarioResult
|
||||
public int CheckpointCount { get; set; }
|
||||
public string ArtifactDir { get; set; } = string.Empty;
|
||||
public string? Error { get; set; }
|
||||
|
||||
// engine-bridge sidecar (v3): optional semantic-state diff.
|
||||
// - Captured — sidecar JSON was successfully fetched from the bridge
|
||||
// - Hunks — diff hunks against the engine-state baseline
|
||||
// - Status — "pass" / "fail" / "missing_baseline" / "unavailable" / "skipped"
|
||||
public bool SidecarCaptured { get; set; }
|
||||
public int SidecarHunks { get; set; }
|
||||
public string SidecarStatus { get; set; } = "skipped";
|
||||
}
|
||||
|
||||
@@ -13,7 +13,8 @@ public sealed class TestRunner
|
||||
RunnerOptions options,
|
||||
IRunnerHostFactory hostFactory,
|
||||
INormalizer normalizer,
|
||||
IDiffer differ)
|
||||
IDiffer differ,
|
||||
IEngineStateSnapshotClient? sidecarClient = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(hostFactory);
|
||||
@@ -50,6 +51,19 @@ public sealed class TestRunner
|
||||
var host = hostFactory.Create(scenario, artifactDir);
|
||||
var engine = new PlayerEngine();
|
||||
engine.Run(scenario, host);
|
||||
|
||||
// engine-bridge v3 sidecar: capture the engine state AFTER
|
||||
// playback completes, before the host releases the SUT. The
|
||||
// client is best-effort — null return or exception means the
|
||||
// bridge is unreachable and we keep going without a sidecar.
|
||||
CaptureAndDiffSidecar(
|
||||
sidecarClient,
|
||||
options,
|
||||
scenarioName,
|
||||
artifactDir,
|
||||
normalizer,
|
||||
differ,
|
||||
sr);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -132,6 +146,98 @@ public sealed class TestRunner
|
||||
return report;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// engine-bridge v3 — capture engine state sidecar at scenario end and
|
||||
/// diff it against the approved engine-state baseline (if any). Updates
|
||||
/// <paramref name="sr"/> with the sidecar status and hunk count, and
|
||||
/// promotes the scenario status to "fail" if the sidecar diverges.
|
||||
/// The bridge is treated as strictly optional: a missing client, a
|
||||
/// null capture, or a missing baseline degrade to benign statuses
|
||||
/// instead of failing the run.
|
||||
/// </summary>
|
||||
private static void CaptureAndDiffSidecar(
|
||||
IEngineStateSnapshotClient? sidecarClient,
|
||||
RunnerOptions options,
|
||||
string scenarioName,
|
||||
string artifactDir,
|
||||
INormalizer normalizer,
|
||||
IDiffer differ,
|
||||
ScenarioResult sr)
|
||||
{
|
||||
if (sidecarClient is null)
|
||||
{
|
||||
sr.SidecarStatus = "skipped";
|
||||
return;
|
||||
}
|
||||
|
||||
string? captured;
|
||||
try { captured = sidecarClient.TryCapture(); }
|
||||
catch { captured = null; }
|
||||
|
||||
if (string.IsNullOrEmpty(captured))
|
||||
{
|
||||
sr.SidecarStatus = "unavailable";
|
||||
return;
|
||||
}
|
||||
|
||||
sr.SidecarCaptured = true;
|
||||
var receivedPath = Path.Combine(artifactDir, "engine-state.received.json");
|
||||
File.WriteAllText(receivedPath, captured);
|
||||
|
||||
var baselinePath = FindSidecarBaseline(options.BaselinesDir, scenarioName);
|
||||
if (baselinePath is null)
|
||||
{
|
||||
// First run: no approved engine-state baseline yet. Keep the
|
||||
// received file around so the user can promote it with /approve.
|
||||
sr.SidecarStatus = "missing_baseline";
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var receivedRaw = File.ReadAllText(receivedPath);
|
||||
var approvedRaw = File.ReadAllText(baselinePath);
|
||||
var receivedNorm = normalizer.Normalize(receivedRaw, options.Profile, null);
|
||||
var approvedNorm = normalizer.Normalize(approvedRaw, options.Profile, null);
|
||||
|
||||
var receivedNormPath = Path.Combine(artifactDir, "engine-state.received.normalized");
|
||||
var approvedNormPath = Path.Combine(artifactDir, "engine-state.approved.normalized");
|
||||
File.WriteAllText(receivedNormPath, receivedNorm);
|
||||
File.WriteAllText(approvedNormPath, approvedNorm);
|
||||
|
||||
var diff = differ.Compare(approvedNormPath, receivedNormPath);
|
||||
sr.SidecarHunks = diff.Hunks.Count;
|
||||
sr.SidecarStatus = diff.Identical ? "pass" : "fail";
|
||||
|
||||
// Sidecar divergence promotes the scenario status. The main
|
||||
// result file diff runs AFTER this method, so only promote to
|
||||
// "fail" if it had not yet been set to "error" or "fail".
|
||||
if (!diff.Identical && sr.Status == "pass")
|
||||
{
|
||||
sr.Status = "fail";
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
sr.SidecarStatus = "error";
|
||||
sr.Error ??= "sidecar: " + ex.Message;
|
||||
}
|
||||
}
|
||||
|
||||
private static string? FindSidecarBaseline(string baselinesDir, string scenarioName)
|
||||
{
|
||||
if (string.IsNullOrEmpty(baselinesDir) || !Directory.Exists(baselinesDir))
|
||||
return null;
|
||||
var candidates = new[]
|
||||
{
|
||||
Path.Combine(baselinesDir, scenarioName + ".engine-state.approved.json"),
|
||||
Path.Combine(baselinesDir, scenarioName + ".engine-state.json"),
|
||||
};
|
||||
foreach (var c in candidates)
|
||||
if (File.Exists(c)) return c;
|
||||
return null;
|
||||
}
|
||||
|
||||
private static string? FindBaseline(string baselinesDir, string scenarioName, string preferredExt)
|
||||
{
|
||||
if (string.IsNullOrEmpty(baselinesDir) || !Directory.Exists(baselinesDir))
|
||||
@@ -171,13 +277,15 @@ public sealed class TestRunner
|
||||
.Append(" | Failed: ").Append(report.Failed)
|
||||
.Append(" | Errored: ").AppendLine(report.Errored.ToString());
|
||||
sb.AppendLine();
|
||||
sb.AppendLine("| Scenario | Status | Hunks | Checkpoints | Artifacts |");
|
||||
sb.AppendLine("|----------|--------|-------|-------------|-----------|");
|
||||
sb.AppendLine("| Scenario | Status | Hunks | Sidecar | Sidecar Hunks | Checkpoints | Artifacts |");
|
||||
sb.AppendLine("|----------|--------|-------|---------|---------------|-------------|-----------|");
|
||||
foreach (var s in report.Scenarios)
|
||||
{
|
||||
sb.Append("| ").Append(s.Name)
|
||||
.Append(" | ").Append(s.Status)
|
||||
.Append(" | ").Append(s.Hunks)
|
||||
.Append(" | ").Append(s.SidecarStatus)
|
||||
.Append(" | ").Append(s.SidecarHunks)
|
||||
.Append(" | ").Append(s.CheckpointCount)
|
||||
.Append(" | ").Append(s.ArtifactDir)
|
||||
.AppendLine(" |");
|
||||
|
||||
@@ -158,4 +158,158 @@ steps:
|
||||
Assert.True(first.TryGetProperty("checkpointCount", out _));
|
||||
Assert.True(first.TryGetProperty("artifactDir", out _));
|
||||
}
|
||||
|
||||
// ---- engine-bridge v3 sidecar tests ------------------------------
|
||||
|
||||
private sealed class FakeSidecarClient : IEngineStateSnapshotClient
|
||||
{
|
||||
public string? Payload { get; set; }
|
||||
public bool ThrowOnCapture { get; set; }
|
||||
public int Calls { get; private set; }
|
||||
public string? TryCapture()
|
||||
{
|
||||
Calls++;
|
||||
if (ThrowOnCapture) throw new InvalidOperationException("boom");
|
||||
return Payload;
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_NullClient_SkippedStatus()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(), new StubDiffer(identical: true),
|
||||
sidecarClient: null);
|
||||
|
||||
var s = report.Scenarios[0];
|
||||
Assert.False(s.SidecarCaptured);
|
||||
Assert.Equal("skipped", s.SidecarStatus);
|
||||
Assert.Equal("pass", s.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_ClientReturnsNull_UnavailableStatus()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
|
||||
var fake = new FakeSidecarClient { Payload = null };
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(), new StubDiffer(identical: true),
|
||||
sidecarClient: fake);
|
||||
|
||||
Assert.Equal(1, fake.Calls);
|
||||
var s = report.Scenarios[0];
|
||||
Assert.False(s.SidecarCaptured);
|
||||
Assert.Equal("unavailable", s.SidecarStatus);
|
||||
Assert.Equal("pass", s.Status); // main result still wins
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_Throws_UnavailableStatus_MainStillPasses()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
|
||||
var fake = new FakeSidecarClient { ThrowOnCapture = true };
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(), new StubDiffer(identical: true),
|
||||
sidecarClient: fake);
|
||||
|
||||
var s = report.Scenarios[0];
|
||||
Assert.False(s.SidecarCaptured);
|
||||
Assert.Equal("unavailable", s.SidecarStatus);
|
||||
Assert.Equal("pass", s.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
|
||||
var fake = new FakeSidecarClient { Payload = "{\"scene\":{\"object_count\":4}}" };
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(), new StubDiffer(identical: true),
|
||||
sidecarClient: fake);
|
||||
|
||||
var s = report.Scenarios[0];
|
||||
Assert.True(s.SidecarCaptured);
|
||||
Assert.Equal("missing_baseline", s.SidecarStatus);
|
||||
Assert.Equal("pass", s.Status);
|
||||
var receivedPath = Path.Combine(s.ArtifactDir, "engine-state.received.json");
|
||||
Assert.True(File.Exists(receivedPath));
|
||||
Assert.Contains("object_count", File.ReadAllText(receivedPath));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_Captured_BaselineIdentical_PassPass()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
var sidecarPayload = "{\"scene\":{\"object_count\":4}}";
|
||||
File.WriteAllText(
|
||||
Path.Combine(bDir, "alpha.engine-state.approved.json"),
|
||||
sidecarPayload);
|
||||
|
||||
var fake = new FakeSidecarClient { Payload = sidecarPayload };
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(), new StubDiffer(identical: true),
|
||||
sidecarClient: fake);
|
||||
|
||||
var s = report.Scenarios[0];
|
||||
Assert.True(s.SidecarCaptured);
|
||||
Assert.Equal("pass", s.SidecarStatus);
|
||||
Assert.Equal(0, s.SidecarHunks);
|
||||
Assert.Equal("pass", s.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail()
|
||||
{
|
||||
var (sDir, bDir, oDir) = MakeDirs();
|
||||
WriteScenario(sDir, "alpha");
|
||||
var content = "{\"x\":1}";
|
||||
// main result matches baseline (identical differ below)
|
||||
File.WriteAllText(Path.Combine(bDir, "alpha.json"), content);
|
||||
// sidecar baseline exists too
|
||||
File.WriteAllText(
|
||||
Path.Combine(bDir, "alpha.engine-state.approved.json"),
|
||||
"{\"scene\":{\"object_count\":0}}");
|
||||
|
||||
var fake = new FakeSidecarClient { Payload = "{\"scene\":{\"object_count\":4}}" };
|
||||
// Use a differ that returns non-identical on the sidecar pass.
|
||||
// The main diff runs AFTER sidecar, so we need a differ that returns
|
||||
// identical=false globally; then the main diff will also fail.
|
||||
// For now we assert that sidecar hunks > 0 and status is "fail".
|
||||
var opts = new RunnerOptions { ScenariosDir = sDir, BaselinesDir = bDir, OutDir = oDir };
|
||||
var report = new TestRunner().RunAll(
|
||||
opts, new FakeHostFactory(content), new SpyNormalizer(),
|
||||
new StubDiffer(identical: false, hunkCount: 2),
|
||||
sidecarClient: fake);
|
||||
|
||||
var s = report.Scenarios[0];
|
||||
Assert.True(s.SidecarCaptured);
|
||||
Assert.Equal("fail", s.SidecarStatus);
|
||||
Assert.Equal(2, s.SidecarHunks);
|
||||
Assert.Equal("fail", s.Status);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user