836afea5ee
- recorder v1 (fail) → v2 (pass): drag state machine, focus events, ts/raw_coord - player pass with caveats: reliability untestable in sandbox - PROGRESS.md Done rows + follow-ups for live SUT smoke test - PLAN.md P1 pivoted to test-runner + live smoke test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
43 lines
4.9 KiB
Markdown
43 lines
4.9 KiB
Markdown
# Recorder — Evaluation
|
|
|
|
- Generator commit: `d486cbb`
|
|
- Build: `dotnet build recordingtest.sln` → green (0 warnings, 0 errors)
|
|
- Tests: `dotnet test tests/Recordingtest.Recorder.Tests` → 5 passed / 0 failed / 0 skipped
|
|
- Evaluator: independent reading of source + test artifacts
|
|
|
|
## Verdict table
|
|
|
|
| # | DoD item | Verdict | Evidence |
|
|
|---|---|---|---|
|
|
| 1 | Console attach to SUT + 입력 캡처 시작 | partial | `Program.TryAttach` uses `Application.Attach(pid)` or window-title scan; never `Launch()`. `LowLevelHook` installs WH_KEYBOARD_LL + WH_MOUSE_LL on dedicated STA thread. Wired but cannot be exercised in this sandbox (no SUT). |
|
|
| 2 | 캡처 이벤트: 키 down/up, 클릭/드래그/휠, 포커스 변경 | partial | `LowLevelHook.KeyboardProc` emits `key_down`/`key_up`; `MouseProc` emits L/R/M down+up, `wheel`, `move`. Drag is NOT collapsed into a single drag step (only down/up are recorded; `Program.IsInterestingForStep` only keeps `mouse_down_l/r` and `key_down`). Focus-change events are NOT captured (no UIA focus listener). |
|
|
| 3 | Event shape `{ts, kind, uia_path, offset_norm, raw_coord, value}` | partial | `RawEvent` carries `TimestampMs, Kind, X, Y, Code, WheelDelta`; `ScenarioStep`/`ScenarioTarget` carry `kind, uia_path, offset, value`. There is no persistent per-event log with all six fields — `raw_coord` is consumed for resolution but not stored on the emitted step. |
|
|
| 4 | 3D viewport `offset_norm ∈ [0..1]` | pass | `OffsetNormalizer.Normalize` divides by width/height, clamps each axis to `[0,1]`, returns `(0,0)` for zero-sized rects. Unit test `OffsetNormalizer_ClicksInsideElement_ReturnsZeroToOne` covers center, top-left, and out-of-bounds clamp. |
|
|
| 5 | Yaml schema 준수 (`name, description, sut{exe, startup_timeout_ms}, steps[{kind, target{uia_path, offset}, value, wait_for}]`) | pass | `Scenario.cs` matches the schema; `ScenarioWriter` uses `UnderscoredNamingConvention` so casing matches contract (`startup_timeout_ms`, `uia_path`, `wait_for`). Test `YamlSerializer_RoundtripsScenario` round-trips both a click and a masked-type step. |
|
|
| 6 | 비밀번호/토큰 마스킹 (PasswordBox → `<MASKED>`) | pass | `MaskPolicy.Apply` returns `<MASKED>` when `IsPassword` or `ClassName == "PasswordBox"`. `Program.ConsumeAsync` sets `step.Value = MaskPolicy.MaskedValue` on masked targets. Test `FocusedElementIsPassword_ReturnsMasked` covers masked + plain paths. |
|
|
| 7 | 60 FPS 영향 없음 | untestable | Requires running SUT + perf measurement; not possible in sandbox. Architecture (separate STA hook thread + Channel) is consistent with the requirement. |
|
|
| 8 | 종료 시 요약(이벤트 수, 소요 시간, 미결 건수) | pass (source-only) | `Program.Run` writes `[recorder] done. events={count} elapsed={sw.Elapsed} unresolved_paths={unresolved}` on Ctrl+C exit. |
|
|
|
|
Additional checks:
|
|
- `Program.ParseArgs` returns null when `--attach` is missing → `Main` prints usage to stderr and returns `2`. Verified by `Cli_MissingAttach_ExitTwo`.
|
|
- `ElementPathBuilder.Build` produces `ClassName[@AutomationId='...']/...` walking from topmost ancestor down, falling back to `@Name` and then bare `ClassName`. Verified by `ElementPathBuilder_WithNestedElements_ReturnsFullPath`.
|
|
- IME (한글 조합) handling: not implemented (acknowledged in generator notes; listed as a Risk in the contract, not a DoD item).
|
|
|
|
## Gaps / required follow-ups
|
|
|
|
1. **Drag collapse** — `mouse_down_l` + movement + `mouse_up_l` should produce a single `kind: drag` step with start/end offsets. Today the recorder records only the down event as `click`. Blocks the contract evaluation step "Box 생성 드래그".
|
|
2. **Focus-change events** — No UIA `FocusChangedEventHandler` registration. Required by DoD #2.
|
|
3. **Per-event log shape** — Steps drop `ts` and `raw_coord`; the contract requires every event to be recorded in the `{ts, kind, uia_path, offset_norm, raw_coord, value}` shape. Either keep a sidecar event log or extend `ScenarioStep` with these fields.
|
|
4. **Manual SUT verification** — DoD #1 and the perf check (#7) require attaching to EG-BIM Modeler on a real workstation. This evaluator cannot perform that step.
|
|
|
|
## Overall verdict
|
|
|
|
**fail — blocks release until manual SUT run + drag/focus implementation.**
|
|
|
|
Rationale (per CLAUDE.md): overall `pass` requires every DoD item `pass`. Items 1 and 2 are concretely incomplete (drag collapse + focus events missing; not merely untestable). Item 7 is structurally untestable in the sandbox and is treated as partial. Items 3 is partial because `ts`/`raw_coord` are not persisted in the output. The honest call is `fail` with the following release gates:
|
|
|
|
- Implement drag collapse and focus-change capture, add unit tests for the drag state machine.
|
|
- Persist `ts` and `raw_coord` on each emitted step (or sidecar log).
|
|
- Manual smoke on EG-BIM Modeler: attach by pid, click Box command, drag a box, type into a PasswordBox, Ctrl+C, verify yaml + summary.
|
|
- Re-run evaluator after the above.
|