kimminsung/recordingtest
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
800ea9c175c3a0084a7126d1c165c7bf1082d458
recordingtest/PROGRESS.md
minsung e28a029704 runner: engine-bridge sidecar integration (#10) 
At the end of each scenario playback the runner now fetches a state
snapshot from the engine-bridge HTTP server and diffs it against an
approved engine-state baseline. This closes the engine-bridge v3 loop
and adds a semantic-state axis to the golden-file regression strategy —
diffs can now catch "camera moved" or "wrong selection after command"
even when the main saved-file diff still passes.

New:
  src/Recordingtest.Runner/IEngineStateSnapshotClient.cs
    IEngineStateSnapshotClient.TryCapture() -> string? (never throws)
    HttpEngineStateSnapshotClient — GETs /scene /camera /selection off a
    base URL (default http://localhost:38080), composes them into
      { "scene": {...}, "camera": {...}, "selection": {...} }
    with stable ordering so the downstream differ stays friendly.
    Runner is Generic tier, so this client carries zero HmEG knowledge;
    it forwards raw JSON strings.

  TestRunner.RunAll now takes an optional IEngineStateSnapshotClient.
  After engine.Run() completes, CaptureAndDiffSidecar():
    - null client           -> SidecarStatus = "skipped"
    - TryCapture null/throw -> "unavailable" (main result still evaluated)
    - success               -> writes engine-state.received.json
    - baseline found        -> normalize + diff + "pass"/"fail"
    - baseline missing      -> "missing_baseline" (first run convention)
  A sidecar "fail" promotes the overall scenario Status to "fail" so
  exit code reflects semantic divergence even when the save-file diff
  agrees.

  ScenarioResult: SidecarCaptured / SidecarHunks / SidecarStatus.
  Markdown report grows Sidecar + Sidecar Hunks columns; JSON report
  picks up the new fields automatically via camelCase serialization.

  Program.cs: --sidecar-url <url> (default localhost:38080) and
  --no-sidecar. Default behaviour is sidecar-on so that a loaded
  bridge plugin is picked up automatically; when the plugin is not
  deployed the client silently reports "unavailable" and CI still runs.

Baseline lookup (new):
  <baselinesDir>/<scenario>.engine-state.approved.json
  <baselinesDir>/<scenario>.engine-state.json

Tests (Recordingtest.Runner.Tests, +6):
  - Sidecar_NullClient_SkippedStatus
  - Sidecar_ClientReturnsNull_UnavailableStatus
  - Sidecar_Throws_UnavailableStatus_MainStillPasses
  - Sidecar_Captured_NoBaseline_MissingBaseline_And_WritesReceivedFile
  - Sidecar_Captured_BaselineIdentical_PassPass
  - Sidecar_Captured_BaselineDivergent_PromotesScenarioToFail

Full suite 126 -> 132, 0 failures.

Follow-ups (PLAN.md):
  - Live loop: first run writes received, user approves, rerun passes.
  - Normalizer profile for engine-state (float epsilon for camera
    coords, selected_ids sort, document_path masking). Currently
    runs through the default profile as identity, so false fails are
    possible for sensitive camera moves until this lands.

Ref: #10 engine-bridge v3 final integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 11:56:15 +09:00

84 lines
8.2 KiB
Markdown
Raw Blame History

# PROGRESS.md — recordingtest
> 에이전트 간 공유 메모리. 세션 시작 시 PLAN.md와 함께 반드시 읽을 것.
> 작업 상태만 기록한다. 결정의 근거는 `docs/history/`, 전략은 `CLAUDE.md` / 메모리 참조.
**저장소**: https://gitea.hmac.kr/kimminsung/recordingtest
**SUT**: `EG-BIM Modeler/` (git 제외, 로컬 동봉)
**현재 이슈**: #2 컨텍스트 엔지니어링 & AI 개발환경 셋업
---
## Done
| 날짜 | 항목 | 산출물 |
|------|------|--------|
| 2026-04-07 | 히스토리 훅 설정 | `docs/history/2026-04-07_히스토리-훅-설정.md` |
| 2026-04-07 | 이슈 #2 리서치 & 에이전트 분해 | 메모리 `project_recordingtest_plan.md` |
| 2026-04-07 | 구현 고려사항 정리 (8개 카테고리) | 히스토리 |
| 2026-04-07 | Golden-file 회귀 전략 채택 | 메모리 `project_recordingtest_strategy.md` |
| 2026-04-07 | CLAUDE.md 초안 + §0 협업 규칙 | `CLAUDE.md` |
| 2026-04-07 | SUT(EG-BIM Modeler) 정적 분석 | 메모리 `project_sut_egbim_modeler.md` |
| 2026-04-07 | `.claude/` agents·commands·skills·hooks 셋업 | `.claude/` |
| 2026-04-07 | Harness design 채택 (Planner/Generator/Evaluator) | CLAUDE.md §0.1, `.claude/agents/planner.md`, `evaluator.md` |
| 2026-04-07 | SUT 폴더 `.gitignore` | `.gitignore` |
| 2026-04-07 | 초기 Sprint Contracts 5건 작성 | `docs/contracts/*.md` |
| 2026-04-07 | SUT 카탈로그 v0 (정적) | `docs/sut-catalog/catalog.md`, `plugins.md` |
| 2026-04-07 | 솔루션 스캐폴드(sut-prober PoC 타깃) | `recordingtest.sln`, `src/Recordingtest.SutProber/` |
| 2026-04-07 | sut-prober PoC 구현 (Generator) | `src/Recordingtest.SutProber/`, `docs/sut-catalog/{plugins,json-configs,assemblies}.json` |
| 2026-04-07 | sut-prober PoC Evaluator pass (#3) | `docs/contracts/sut-prober.evaluation.md` |
| 2026-04-07 | diff-reporter PoC + Evaluator pass (#5) | `src/Recordingtest.DiffReporter*/`, `docs/contracts/diff-reporter.evaluation.md` |
| 2026-04-07 | normalizer PoC + Evaluator pass v2 (#4) — sidecar log, explicit coverage mapping, 6 rules | `src/Recordingtest.Normalizer/`, `docs/contracts/normalizer.evaluation.md` |
| 2026-04-07 | player PoC + Evaluator pass (#7) — 6 tests, no fixed sleeps, fake host | `src/Recordingtest.Player/`, `docs/contracts/player.evaluation.md` |
| 2026-04-07 | recorder PoC + Evaluator pass v2 (#6) — drag state machine, focus events, ts/raw_coord | `src/Recordingtest.Recorder/`, `docs/contracts/recorder.evaluation.md` |
| 2026-04-07 | test-runner PoC + Evaluator pass (#8) — 5-module E2E 파이프라인, 6 tests, DI | `src/Recordingtest.Runner/`, `docs/contracts/test-runner.evaluation.md` |
| 2026-04-07 | engine-bridge PoC v1 + Evaluator pass (#9) — 정적 분석, HmEG 내부 후보 8000+, API 초안 | `src/Recordingtest.EngineBridge*/`, `docs/engine-catalog/`, `docs/engine-bridge-probe-design.md` |
| 2026-04-07 | engine-bridge v2 + Evaluator pass (#10) — MEF plugin masquerade, HttpListener, HmEgHttpSnapshot, 11 tests | `src/Recordingtest.EgPlugin/`, `src/Recordingtest.EngineBridge.Client/`, `docs/guides/engine-bridge-deploy.md` |
| 2026-04-07 | 라이브 SUT smoke test 가이드 작성 | `docs/guides/smoke-test.md` |
| 2026-04-07 | Smoke test 1회차 — integration gap 4개 발견 (recorder target null, VK 코드, player enum, null guard) | `scenarios/box-create.yaml` |
| 2026-04-07 | Smoke gap fix + Evaluator pass (#11) — STAThread, KeyTranslator, 60 tests, regression trap 검증 | commit `139fbbc` |
| 2026-04-07 | Smoke test 1회차 — recorder PID attach + UIA target 정상 (box-v4), player 재생 부분 실패 | `docs/history/2026-04-07_smoke-1회차-결과.md`, scenarios/box-v4*.yaml |
| 2026-04-07 | Smoke 2차 gap fix + Evaluator pass (#12) — full-path resolver, type target inheritance, window filter, UTF-8 BOM-less, 71 tests | commit `8784fec` |
| 2026-04-07 | sut-prober snake_case + scaffolding review 1회차 | commit `0f0324e` |
| 2026-04-07 | normalizer follow-ups + Evaluator pass — float epsilon 구성화 + JSON-path 마스크 스코핑, 77 tests | commit `eeee3c2` |
| 2026-04-08 | **Smoke test 2회차 — 첫 E2E 성공** 🎉 Box geometry 생성 확인 | `docs/history/2026-04-08_smoke-2회차-첫-e2e-성공.md`, `scenarios/box-v5*.yaml` |
| 2026-04-08 | 이슈 #13 Gap E/F/G fix — HotkeyParseTests + FocusEventFilter + WindowPointResolver, 94 tests | `docs/history/2026-04-08_이슈13-smoke3-fix-generator.md` |
| 2026-04-08 | **이슈 #14 Raw 시나리오 E2E 성공** 🎉 수동 cleanup 없이 box-v6.yaml 재생으로 Box 생성 | player: null-target fallback + foreground switch + leading alt+tab strip + timing preservation, 24 player tests |
| 2026-04-09 | engine-bridge v3 D1/D6 scaffold (reflection accessor + 9 tests, EgPlugin) | `IAppManagerAccessor`, `ReflectionEngineStateProvider`, 14 EgPlugin tests |
| 2026-04-09 | HmEG 소스 survey + `docs/hmeg-api-survey.md` | Q1~Q7 식별, `HmegDirectStateProvider` 설계 근거 |
| 2026-04-09 | **3-tier 분리 1단계 (incremental)** — `Recordingtest.Bridge.Abstractions` (Generic) + `Recordingtest.Hmeg.Bridge` (HmEG-aware) 신설, `HmegDirectStateProvider` + `ChainedEngineStateProvider` wire-up, 115 tests | commit `f6b6e74` |
| 2026-04-09 | **3-tier 분리 2단계** — `EgPlugin` → `Sut/EgBim/Recordingtest.Sut.EgBim.PluginHost`, `EngineBridge` → `Hmeg/Recordingtest.Hmeg.Catalog`, `EngineBridge.Client` → `Hmeg/Recordingtest.Hmeg.Bridge.Client`, `EngineBridge.Probe` → `Hmeg/Recordingtest.Hmeg.Catalog.Probe`, 테스트 동일. `Recordingtest.Architecture.Tests` 11건 추가 (의존 그래프 강제). 126 tests | commit pending |
| 2026-04-09 | **engine-bridge v3 EgBim 람다 wire-up** — `EditorPlugin` base 직접 사용: `RootSpace`, `View(EGViewport:HmEGViewport)`, `AppManager.FileManager.CurrentFile`. `HmegDirectStateProvider` 이제 실값 가능. `Editor02.HmEGAppManager.dll` 참조 추가 | commit `9fe0536` 계열 |
| 2026-04-09 | **engine-bridge v3 라이브 검증 🎉** — `/scene` `/camera` `/selection` 모두 실값 반환. viewport 람다를 `AppManager.ViewportManager.FocusedViewport` 로 교체 (`View`는 `Run()` 안 타는 bridge plugin에서 항상 null). deploy-egbim-plugin.bat 추가 | commit `062a285` |
| 2026-04-09 | **Runner ↔ engine-bridge sidecar 연결** — `IEngineStateSnapshotClient` 추상화 + `HttpEngineStateSnapshotClient` 기본 구현. TestRunner가 시나리오 재생 종료 시점에 `/scene /camera /selection` 스냅샷 → `engine-state.received.json` 기록 → `<scenario>.engine-state.approved.json` 베이스라인과 diff → sidecar 불일치 시 시나리오 fail 승격. `--sidecar-url` / `--no-sidecar` CLI 옵션. sidecar 6개 신규 테스트(skipped/unavailable/missing_baseline/pass/fail). 132 tests | commit pending |
## In progress
_(없음)_
## Follow-ups
- [x] ~~sut-prober JSON naming snake_case~~ — commit `0f0324e`
- [x] ~~normalizer: mask_volatile_settings JSON-path 스코핑~~ — commit `eeee3c2`
- [x] ~~normalizer: float epsilon 구성화~~ — commit `eeee3c2`
- [ ] diff-reporter: 실제 `diff-triager` 에이전트 통합 테스트. non-blocking.
- [ ] recorder/player: **라이브 SUT 수동 smoke test** — 60 FPS / 10회 중 9회 reliability DoD는 샌드박스 unit test 불가, 실제 환경에서 검증 필요.
- [ ] player: `wait_for` UIA 이벤트 매핑 강화 (현재 host passthrough).
- [ ] player: `UiaPlayerHost` uia_path resolver가 마지막 `@AutomationId`만 사용 — 전체 ancestor chain 지원 필요.
- [ ] recorder: IME 조합 키 처리 (contract risks).
- [x] ~~player: foreground settle 안정화~~ — 능동 대기(`GetForegroundWindow` polling 2s + 100ms settle)로 전환, 1차 재생 성공 확인
- [~] recorder Gap I-1 — UIA `Automation.FocusedElement` 폴링 PoC 시도(commit pending). 결과: SUT의 CommandBox 등 AutomationPeer 미부착 컨트롤은 UIA 외부에서 본질적으로 못 봄. **deferred** — generic WPF DLL injection 또는 SUT-side AutomationPeer 부착 PoC가 필요. 현재는 Player fallback(null target → OS 키 입력 / raw_coord 클릭)이 공식 전략.
## Blocked
_(없음)_
## Recent regression runs
_(러너 미구현 — /regress 사용 불가)_
## Scaffolding review
- 마지막 감사: 2026-04-07 (초기 셋업)
- 다음 감사 권장: PoC 3개 완료 후
Reference in New Issue View Git Blame Copy Permalink