Orchestrate engine-bridge PoC v1 evaluation (#9)

- Static HmEG catalog via MetadataLoadContext, 13 assemblies, 11k+ candidates
- IEngineSnapshot API draft + probe design doc (plugin masquerade recommended)
- All DoD pass on first iteration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
minsung
2026-04-07 15:53:10 +09:00
parent 2a4f1d3fa4
commit 4cee3c2d86
5 changed files with 137 additions and 4 deletions

View File

@@ -8,13 +8,14 @@
1. **훅 동작 검증** — SessionStart/Stop/Guard 3개 shell 스크립트를 실제로 트리거시켜 확인
- 의존: jq 설치 여부 확인
## P1 — 라이브 검증
## P1 — 라이브 검증 & 런타임 엔진 접근
4. **라이브 SUT smoke test** — 사용자 환경에서 recorder/player/runner 실제 검증 (E2E)
- 의존: 없음 (test-runner까지 PoC 완료)
- 가이드: `docs/guides/smoke-test.md` (작성 필요)
5. **engine-bridge 탐색** — HmEG PDB 리플렉션 스파이크
- 의존: 없음
- 가이드: `docs/guides/smoke-test.md` (작성 필요)
5. **engine-bridge v2** — MEF plugin masquerade 구현 (design doc 권고)
- 의존: engine-bridge v1 (완료)
- 범위: 커스텀 MEF plugin이 HmEG 상태를 로컬 HTTP/named pipe로 노출 → recordingtest가 수집
## Follow-ups (non-blocking)

View File

@@ -32,6 +32,7 @@
| 2026-04-07 | player PoC + Evaluator pass (#7) — 6 tests, no fixed sleeps, fake host | `src/Recordingtest.Player/`, `docs/contracts/player.evaluation.md` |
| 2026-04-07 | recorder PoC + Evaluator pass v2 (#6) — drag state machine, focus events, ts/raw_coord | `src/Recordingtest.Recorder/`, `docs/contracts/recorder.evaluation.md` |
| 2026-04-07 | test-runner PoC + Evaluator pass (#8) — 5-module E2E 파이프라인, 6 tests, DI | `src/Recordingtest.Runner/`, `docs/contracts/test-runner.evaluation.md` |
| 2026-04-07 | engine-bridge PoC v1 + Evaluator pass (#9) — 정적 분석, HmEG 내부 후보 8000+, API 초안 | `src/Recordingtest.EngineBridge*/`, `docs/engine-catalog/`, `docs/engine-bridge-probe-design.md` |
## In progress

View File

@@ -0,0 +1,60 @@
# engine-bridge PoC v1 — Evaluation
- Contract: `docs/contracts/engine-bridge.md`
- Generator commit: `2a4f1d3`
- Evaluator date: 2026-04-07
- Issue: #9
## Verdict: **PASS**
All DoD items satisfied. Build green, 6/6 tests pass, probe exits 0, catalog output is deterministic across two runs, all 4 candidate categories present with non-trivial counts, skeleton API throws NotImplementedException, probe design doc contains 5 options + render-signal table + explicit recommendation. No runtime invocation or SUT writes found in `src/Recordingtest.EngineBridge*`.
## DoD table
| # | DoD item | Result | Evidence |
|---|---|---|---|
| 1 | `Recordingtest.EngineBridge` lib + `Recordingtest.EngineBridge.Probe` exe | PASS | `src/Recordingtest.EngineBridge/Recordingtest.EngineBridge.csproj`, `src/Recordingtest.EngineBridge.Probe/Recordingtest.EngineBridge.Probe.csproj` |
| 2 | Static analysis via `MetadataLoadContext` + `PathAssemblyResolver(sutRoot, runtimeDir)` | PASS | `MetadataLoader.cs` uses `MetadataLoadContext` + `PathAssemblyResolver` with both SUT root and `RuntimeEnvironment.GetRuntimeDirectory()`; only `LoadFromAssemblyPath` is called |
| 3 | Candidate identification (select/camera/scene/render keywords) | PASS | `CandidateFinder.cs` Categories array: Selection/Selected/Picked/Pick, Camera/Viewport/EyePoint/LookAt/View, Scene/Document/World/Root, Render/Draw/Frame/Dirty |
| 4 | Outputs `hmeg-types.json` + `hmeg-candidates.json`, sorted, deterministic | PASS | `CatalogWriter.cs` sorts by Ordinal, `WriteIndented=true`, forward-slash paths, `\r\n``\n` normalization; two probe runs produce byte-identical output (`diff -q` empty) |
| 5 | `IEngineSnapshot` interface + DTOs | PASS | `IEngineSnapshot.cs` matches contract shape verbatim: `SelectedObjectIds`, `Camera`, `Scene`, `IsRenderComplete`; `CameraState(double[] EyePoint, double[] Target, double[] Up, double Fov)`, `SceneSummary(int ObjectCount, string? DocumentPath)` |
| 6 | `HmEgSnapshot` skeleton with reflection hint constants | PASS | `HmEgSnapshot.cs` all members throw `NotImplementedException`; constants `HmEgAssemblyHint="HmEG"`, `EditorManagerTypeHint="HmEGAppManager"` (+ Selection/Camera/Scene/Render hints); catalog check: `HmEGAppManager` appears 8 times in `hmeg-candidates.json` TypeNames and is its own assembly `Editor02.HmEGAppManager` |
| 7 | Probe design doc with 3 injection options + render signal + recommendation | PASS | `docs/engine-bridge-probe-design.md` contains 5 options (ALC side-load, CLR profiler, managed injector, MEF plugin masquerade, AutomationPeer baseline), render-complete latency table (poll / event / `CompositionTarget.Rendering`), explicit recommendation: **plugin masquerade (option 4)** with CLR profiler as fallback |
| 8 | xUnit tests ≥ 5 (all 5 named cases) | PASS | `EngineBridgeTests.cs` contains all 5 required tests plus bonus `HmEgSnapshot_Constants_MatchCatalog`; 6/6 pass |
| 9 | `dotnet build` green + `dotnet test` all pass | PASS | `dotnet build recordingtest.sln` → 0 warnings, 0 errors; `dotnet test tests/Recordingtest.EngineBridge.Tests` → 6 passed, 0 failed |
| 10 | No SUT write access | PASS | Grep of `src/Recordingtest.EngineBridge*/**/*.cs` for `File.Write`/`File.Create`/`StreamWriter` returns 0 hits; `CatalogWriter.WriteJson` writes only to caller-provided `--out` path; `"EG-BIM Modeler"` literal does not appear inside any EngineBridge source file |
| 11 | Execution path `dotnet run --project src/Recordingtest.EngineBridge.Probe -- --sut "EG-BIM Modeler" --out ...` | PASS | `Program.cs` parses `--sut`/`--out`, returns 2 on missing SUT path, 0 on success; live run produced 13 assemblies loaded (HmEG 2285 / HmGeometry 532 / HmGeometry.V2 1669 / HmTriangle 113 / EditorCore 416 / Editor01..07 various / Editor.AI01.HttpConnector 15), candidates select=726 camera=4226 scene=3081 render=3602 |
## Static-only guarantee
Grep across `src/Recordingtest.EngineBridge*/**/*.cs` for runtime-invocation patterns returned zero matches:
- `Activator.CreateInstance` — 0
- `Assembly.Load(` (non-path) — 0
- `RunClassConstructor` — 0
- `DllImport` / `LibraryImport` (P/Invoke) — 0
`MetadataLoadContext` by design never runs type initializers or user code; the test `MetadataLoader_LoadsHmegAssembly_WithoutExecution` is the observable check.
## Determinism evidence
Two back-to-back probe runs against `EG-BIM Modeler` with different `--out` directories:
```
diff -q hmeg-types.json -> (empty)
diff -q hmeg-candidates.json -> (empty)
```
File sizes: `hmeg-types.json` ≈ 895 KB, `hmeg-candidates.json` ≈ 3.24 MB; both parse as valid JSON.
## Catalog content spot-check
- Categories: `['camera','render','scene','select']` all present; total candidates 11,635.
- `EditorManagerTypeHint = "HmEGAppManager"` appears in 8 candidate TypeNames (also the dedicated assembly `Editor02.HmEGAppManager`), satisfying the "constant cross-checked against catalog" DoD.
- `HmEgAssemblyHint = "HmEG"` is a prefix of `HmEG`, `HmGeometry`, `HmGeometry.V2`, `HmTriangle` — matches.
## Notes / non-blocking observations
- `CandidateFinder` enumerates with `BindingFlags.DeclaredOnly` (avoids double-counting inherited members) and dedupes before sort — good hygiene for byte-identical runs.
- The probe's default assembly patterns include `Editor*.dll` wildcard which picked up all 7 Editor-prefixed assemblies automatically; future sprints can rely on this.
- Probe design doc already answers the "AutomationPeer comparison" DoD item by explicitly scoping it out to sut-prober/recorder.

View File

@@ -0,0 +1,40 @@
# 이슈 #9 — engine-bridge PoC v1 Evaluator
- 날짜: 2026-04-07
- 이슈: #9
- 역할: Evaluator (Generator와 독립, 엄격 채점)
- 대상 커밋: `2a4f1d3`
- 소요 시간: 약 10분
- Context 사용량: 약 55k / 1M 토큰 (추정)
## 수행
1. `docs/contracts/engine-bridge.md` 계약 재확인
2. `IEngineSnapshot.cs`, `HmEgSnapshot.cs`, `MetadataLoader.cs`, `CandidateFinder.cs`, `CatalogWriter.cs`, `Program.cs`, `EngineBridgeTests.cs`, `docs/engine-bridge-probe-design.md` 전수 정독
3. `dotnet build recordingtest.sln` — 0 경고 0 오류
4. `dotnet test tests/Recordingtest.EngineBridge.Tests` — 6 통과 / 0 실패
5. 금지 패턴 grep (`Activator.CreateInstance`, `Assembly.Load(` non-path, `RunClassConstructor`, P/Invoke) — 전부 0건
6. SUT 쓰기 검사 (`File.Write*`, `StreamWriter`, `"EG-BIM Modeler"` 리터럴) — EngineBridge src 내 0건
7. 프로브 2회 실행 (`/tmp/engine-catalog-eval`, `/tmp/engine-catalog-eval2`) — exit 0, `diff -q` 양 쌍 모두 비어있음
8. `docs/engine-catalog/hmeg-candidates.json` 검증 — 4 카테고리 모두 present, `HmEGAppManager` 8회 등장
9. 평가 리포트 `docs/contracts/engine-bridge.evaluation.md` 작성
10. 본 히스토리 파일 작성
## 결과
**VERDICT: PASS** — 모든 DoD 항목 통과.
- 13 어셈블리 로드: HmEG(2285) / HmGeometry(532) / HmGeometry.V2(1669) / HmTriangle(113) / EditorCore(416) / Editor01..07 + Editor.AI01.HttpConnector
- 카테고리 분포: select=726, camera=4226, scene=3081, render=3602 (총 11,635)
- 결정성 확인: 두 번 실행 byte-identical
- Probe 설계 문서: 5개 옵션 비교 + 렌더 신호 지연 표 + 명시적 권고 (plugin masquerade)
## 산출물
- `docs/contracts/engine-bridge.evaluation.md`
- `docs/history/2026-04-07_이슈9-engine-bridge-evaluator.md`
## 비고
- Generator 코드 및 `PROGRESS.md`는 수정하지 않음.
- `MetadataLoadContext` 는 타입 이니셜라이저를 실행하지 않는다는 CLR 보장 덕에 "정적 전용" 요건이 구조적으로 만족됨. 테스트 `MetadataLoader_LoadsHmegAssembly_WithoutExecution` 가 관찰 가능한 증거.

View File

@@ -0,0 +1,31 @@
# 2026-04-07 이슈 #9 — engine-bridge PoC v1 오케스트레이션
- **이슈**: #9 (engine-bridge v1)
- **소요 시간**: ~12분 (1회 사이클)
- **Context 사용량**: ~265k tokens (orchestrator 누적)
## 사이클
1. Planner 역할로 `docs/contracts/engine-bridge.md` 작성
2. 이슈 #9 생성
3. Generator 백그라운드 → commit `2a4f1d3` (6/6 tests, 13 assemblies, 결정적 카탈로그)
4. Evaluator 백그라운드 → **pass** (12/12 DoD)
5. PROGRESS/PLAN 갱신, 이슈 #9 close
## 핵심 발견
HmEG 내부에 selection/camera/scene/render 관련 멤버가 **수천 개** 존재 (obfuscation 없음). 플러그인 masquerade 경로가 매우 유력.
- select 후보: 726
- camera 후보: 4226
- scene 후보: 3081
- render 후보: 3602
- `HmEGAppManager` 타입이 다수 확인됨 → MEF entry point 후보
## 비용
Generator ~66k + Evaluator ~40k + Orchestrator ~15k = **~121k**
## 다음 단계
**engine-bridge v2** — MEF plugin masquerade 구현. 단, 뷰포트 selection 문제 해결의 1순위 경로이긴 하나, v1까지로 Sprint Contract 만족. v2는 SUT 팀 협조/plugin 로드 세이프티 이슈가 있어 별도 이슈로 분리 권장.