kimminsung/recordingtest
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Orchestrate P1 evaluations and update progress (#3, #4, #5)

Browse Source 
- sut-prober evaluation (pass)
- diff-reporter evaluation (pass with 1 partial follow-up)
- normalizer evaluations v1 (fail) + v2 (pass)
- PROGRESS.md Done rows for #3, #4, #5 + Follow-ups
- PLAN.md P0 reduced to hook verification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
minsung
2026-04-07 14:20:55 +09:00
parent 05c7a3f388
commit e3d2ff6c77
11 changed files with 267 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
PLAN.md
View File

@@ -5,23 +5,19 @@
## P0 — 지금 바로 ## P0 — 지금 바로
1. **sut-prober PoC 구현** — Sprint Contract: [docs/contracts/sut-prober.md](docs/contracts/sut-prober.md) 1. **훅 동작 검증** — SessionStart/Stop/Guard 3개 shell 스크립트를 실제로 트리거시켜 확인
- Generator: 일반 세션
- Evaluator: `/evaluate sut-prober`
- 의존: 없음 (독립 실행)
2. **PROGRESS.md / PLAN.md / CLAUDE.md 훅 동작 검증** — SessionStart/Stop/Guard 3개 shell 스크립트를 실제로 트리거시켜 확인
- 의존: jq 설치 여부 확인 - 의존: jq 설치 여부 확인
## P1 — PoC 1단계 ## P1 — UI 자동화 의존
3. **normalizer PoC** — Sprint Contract: [docs/contracts/normalizer.md](docs/contracts/normalizer.md)
- 의존: sut-prober의 Json 카탈로그
4. **recorder PoC (element-aware)** — Sprint Contract: [docs/contracts/recorder.md](docs/contracts/recorder.md) 4. **recorder PoC (element-aware)** — Sprint Contract: [docs/contracts/recorder.md](docs/contracts/recorder.md)
- 의존: FlaUI 패키지 승인 - 의존: FlaUI 패키지 승인 (사용자 확인 필요)
5. **player PoC** — Sprint Contract: [docs/contracts/player.md](docs/contracts/player.md) 5. **player PoC** — Sprint Contract: [docs/contracts/player.md](docs/contracts/player.md)
- 의존: recorder 산출물 포맷 확정 - 의존: recorder 산출물 포맷 확정
6. **diff-reporter PoC** — Sprint Contract: [docs/contracts/diff-reporter.md](docs/contracts/diff-reporter.md)
- 의존: normalizer 규칙 1개 이상 ## Follow-ups (non-blocking)
- **sut-prober snake_case JSON** — `JsonNamingPolicy.SnakeCaseLower` 적용. Evaluator가 pass 처리했지만 contract 엄격 준수를 위해 권장.
## P2 — 통합 ## P2 — 통합

12
PROGRESS.md
View File

@@ -26,10 +26,20 @@
| 2026-04-07 | SUT 카탈로그 v0 (정적) | `docs/sut-catalog/catalog.md`, `plugins.md` | | 2026-04-07 | SUT 카탈로그 v0 (정적) | `docs/sut-catalog/catalog.md`, `plugins.md` |
| 2026-04-07 | 솔루션 스캐폴드(sut-prober PoC 타깃) | `recordingtest.sln`, `src/Recordingtest.SutProber/` | | 2026-04-07 | 솔루션 스캐폴드(sut-prober PoC 타깃) | `recordingtest.sln`, `src/Recordingtest.SutProber/` |
| 2026-04-07 | sut-prober PoC 구현 (Generator) | `src/Recordingtest.SutProber/`, `docs/sut-catalog/{plugins,json-configs,assemblies}.json` | | 2026-04-07 | sut-prober PoC 구현 (Generator) | `src/Recordingtest.SutProber/`, `docs/sut-catalog/{plugins,json-configs,assemblies}.json` |
| 2026-04-07 | sut-prober PoC Evaluator pass (#3) | `docs/contracts/sut-prober.evaluation.md` |
| 2026-04-07 | diff-reporter PoC + Evaluator pass (#5) | `src/Recordingtest.DiffReporter*/`, `docs/contracts/diff-reporter.evaluation.md` |
| 2026-04-07 | normalizer PoC + Evaluator pass v2 (#4) — sidecar log, explicit coverage mapping, 6 rules | `src/Recordingtest.Normalizer/`, `docs/contracts/normalizer.evaluation.md` |
## In progress ## In progress
- sut-prober PoC — Evaluator 채점 대기 (Generator 측 빌드/실행/결정성 확인 완료) _(없음)_
## Follow-ups
- [ ] sut-prober JSON naming을 `JsonNamingPolicy.SnakeCaseLower`로 변경 (contract 엄격 준수). non-blocking.
- [ ] diff-reporter: 실제 `diff-triager` 에이전트 통합 테스트 (현재 schema 단위 테스트로 대체, DoD #8 partial). non-blocking.
- [ ] normalizer: `mask_volatile_settings` 규칙을 JSON-path 스코핑으로 제한 (현재는 필드명 전역 매칭). non-blocking risk.
- [ ] normalizer: float epsilon 구성화 (현재 6 decimals 하드코딩). contract risks 섹션.
## Blocked ## Blocked

21
docs/contracts/diff-reporter.evaluation.md Normal file
View File

@@ -0,0 +1,21 @@
# Evaluation — diff-reporter (2026-04-07 15:00)
Verdict: **pass**
| # | DoD item | Score | Evidence |
|---|----------|-------|----------|
| 1 | `Recordingtest.DiffReporter` 라이브러리 + CLI | pass | `src/Recordingtest.DiffReporter/`, `src/Recordingtest.DiffReporter.Cli/` 존재; `dotnet build recordingtest.sln` 0 warning / 0 error |
| 2 | CLI 입력 `--approved --received --out` | pass | `Program.cs` 인자 파서 + 누락 시 exit 2 |
| 3 | JSON/텍스트 의미 diff, 바이너리 hex 요약 | pass | `JsonDiffer.cs` (path-flatten 후 path별 hunk), `LineDiffer.cs`, `BinaryDiffer.cs` 모두 존재; `Differ.Compare`가 타입별 분기 |
| 4 | 출력 `diff.json`, `diff.md` (`diff.html`은 옵션) | pass | CLI가 `diff.json` + `diff.md` 작성 확인 (`/tmp/dr/diff/`); html은 contract상 옵션 |
| 5 | `diff.json` 스키마 `{file, hunks[], summary{added,removed,changed}}` | pass | 샘플 출력 일치: `file`, `identical`, `hunks`, `summary{added,removed,changed}` |
| 6 | 동일 파일 → identical, exit 0 | pass | `a.json` vs `b.json` 동일 → stdout `identical`, EXIT=0, `identical:true` |
| 7 | 차이 존재 → exit 1 + 1줄 요약 | pass | `a.json` vs `c.json` → stdout `diff: +0 -0 ~1 in c.json`, EXIT=1 |
| 8 | diff-triager 통합 테스트 1개 | partial | DiffReporter 단위 테스트 5/5 통과(`DifferTests.cs`)이나 `diff-triager` 에이전트 통합 케이스 별도 확인 불가 — diff.json 스키마는 triager가 읽기 좋은 평탄 구조라 충족 가능, 외부 에이전트 의존이라 본 평가에선 partial 처리 |
## Notes
- Library API `Differ.Compare(approvedPath, receivedPath) → DiffResult{File, Identical, Hunks, Summary}` 계약과 일치 (계약은 `Identical` 명시 안 했으나 추가 필드는 호환).
- JSON differ는 객체/배열을 path로 flatten 후 path별 hunk를 발행 — 1필드만 다른 케이스에서 hunks.length=1 검증됨.
- `diff.html`은 contract상 옵션이라 평가 기준에서 제외.
- DoD #8 통합 테스트 미존재는 partial이지만 전체 verdict는 pass(다른 모든 항목 통과 + 계약 평가 plan의 1~3 모두 충족). 후속 작업으로 triager 통합 테스트 1건 추가 권장.
- 테스트 결과: `통과 5, 실패 0, 건너뜀 0`.

32
docs/contracts/normalizer.evaluation.md Normal file
View File

@@ -0,0 +1,32 @@
# Evaluation — normalizer (v2, 2026-04-07)
Verdict: **pass**
Generator iteration: commit `05c7a3f`.
| # | DoD item | Score | Evidence |
|---|----------|-------|----------|
| 1 | `Normalize(input, profile)` API | pass | `src/Recordingtest.Normalizer/Normalizer.cs` exposes `Normalize(string, string)` and overload `Normalize(string, string, string?)`. Build green. |
| 2 | Default profile with >=5 rules | pass | `src/Recordingtest.Normalizer/profiles/default.yaml` lists 6 rules: `strip_timestamps`, `mask_guids`, `normalize_paths`, `round_floats`, `mask_volatile_settings`, `sort_json_keys`. All implemented in `Rules.cs`. |
| 3 | Profiles as `profiles/*.yaml`, code-free addition | pass | `Profile.Load` reads YAML by name. |
| 4 | Per-rule before/after sample test | pass | `RuleTests.cs` covers each rule plus `Normalize_AppliesAllDefaultRules` (asserts 6 entries in log including `mask_volatile_settings`). |
| 5 | Idempotent | pass | `RuleTests.Normalize_IsIdempotent`. |
| 6 | Sidecar log `normalization.log` | pass | `Normalizer.cs` lines 150-176: when `sidecarPath` supplied, writes file containing `{RuleId}\tcount={Count}` lines sorted by RuleId (Ordinal) and final `total=` line. Accepts either a file path or directory (in which case it writes `normalization.log` inside). Two real-temp-file tests: `Normalize_WritesSidecarLogFile` and `Normalize_SidecarPath_AcceptsDirectory` — both assert file existence and content (sorted order, total line, per-rule lines). |
| 7 | `json-configs.json` suspected fields fully covered | pass | `CoverageTests.cs` now declares an explicit `Dictionary<string,string> FieldRuleMap` (18 entries, `StringComparer.Ordinal`) with no `|| true` and no catch-all. Path-bearing fields → `normalize_paths`; volatile boolean/scalar/color fields → `mask_volatile_settings`. Test fails if any suspected field is unmapped or if its mapped rule is missing from `default.yaml`. |
| 8 | All Normalizer tests pass | pass | `dotnet test tests/Recordingtest.Normalizer.Tests`: **10 passed, 0 failed, 0 skipped** (167 ms). |
## Notes
- `dotnet build recordingtest.sln`: 0 warnings, 0 errors.
- Test count grew from 8 → 10 (added two sidecar tests). Coverage test rewritten in place.
- New rule `mask_volatile_settings` (`Rules.cs` lines 172-224) is fully implemented (not a stub): allowlist `HashSet` of 16 known volatile field names, walks `JsonNode` recursively, replaces matching values with `"<VOLATILE>"` and counts replacements. Idempotent because the placeholder string itself is not in the allowlist's value space.
- **Risk (non-blocking)**: the volatile-field allowlist is keyed on local field name only (no JSON path scoping). A real bug that incidentally toggles a field named e.g. `GridSnap` in a structurally unrelated subtree would be masked and silently hidden by golden-file diffs. Allowlist is currently 16 names — narrow enough to be acceptable, but should be revisited if the catalog grows. Recommend documenting this allowlist scope in `normalizer.md` in a follow-up (does not block this iteration).
- Coverage test no longer accepts catch-all to `sort_json_keys`; mapping is strict and explicit per the contract's field→rule requirement.
- Sidecar format matches the spec exactly: tab-separated `ruleId\tcount=N`, ordinal-sorted, terminated by `total=N`.
## Artifacts
- `src/Recordingtest.Normalizer/Normalizer.cs`
- `src/Recordingtest.Normalizer/Rules.cs`
- `src/Recordingtest.Normalizer/profiles/default.yaml`
- `tests/Recordingtest.Normalizer.Tests/RuleTests.cs`
- `tests/Recordingtest.Normalizer.Tests/CoverageTests.cs`
- Previous report: `docs/contracts/normalizer.evaluation.v1.md`

23
docs/contracts/normalizer.evaluation.v1.md Normal file
View File

@@ -0,0 +1,23 @@
# Evaluation — normalizer (2026-04-07 14:30)
Verdict: **fail**
| # | DoD item | Score | Evidence |
|---|----------|-------|----------|
| 1 | `Normalize(input, profile)` API in `Recordingtest.Normalizer` | pass | `src/Recordingtest.Normalizer/Normalizer.cs` exposes `Normalizer.Normalize(string, string)` returning `NormalizeResult`. Build green via `dotnet build recordingtest.sln`. |
| 2 | Default profile with >=5 rules (timestamps, GUIDs, paths, floats epsilon 1e-6, key sort) | pass | `src/Recordingtest.Normalizer/profiles/default.yaml` lists 5 rules; `Rules.cs` implements all five (TimestampRegex, GuidRegex, NormalizePaths with `<REPO>`/`<USER>`, RoundFloatsInNode at 6 decimals, SortJsonKeys recursive). |
| 3 | Profiles declared as `profiles/*.yaml`, code-free addition | pass | `Profile.Load` reads YAML; adding a YAML file in `profiles/` registers a new profile without code change. |
| 4 | Per-rule before/after sample test | pass | `tests/Recordingtest.Normalizer.Tests/RuleTests.cs` has one test per rule (StripTimestamps, MaskGuids, NormalizePaths, RoundFloats, SortJsonKeys) plus `Normalize_AppliesAllDefaultRules`. |
| 5 | Idempotent (same bytes on second pass) | pass | `RuleTests.Normalize_IsIdempotent` asserts `first.Output == second.Output`. |
| 6 | Sidecar log `normalization.log` | **fail** | `Normalizer.cs` only returns `NormalizeResult(Output, Log)` in-memory. No file is written; no `normalization.log` artifact exists anywhere in the repo. Generator self-flagged this. |
| 7 | `json-configs.json` suspected fields fully covered by default profile (per-field mapping) | **partial** | `CoverageTests.cs` builds the field set then short-circuits with `|| true` claiming `sort_json_keys` covers any scalar. There is no per-field mapping table; the assertion is vacuous beyond the `IsPathField` heuristic. Per the contract, this is `partial` (catch-all via generic rule). |
| 8 | All Normalizer tests pass | pass | `dotnet test tests/Recordingtest.Normalizer.Tests`: **8 passed, 0 failed, 0 skipped** (129 ms). |
## Notes
- Build: 0 warnings / 0 errors.
- Test count: 8 (7 in `RuleTests.cs`, 1 in `CoverageTests.cs`).
- Verdict is **fail** because DoD #6 (sidecar log) is unimplemented and DoD #7 (suspected-field coverage) is only catch-all. Both must be addressed before PROGRESS.md flips to done.
- Suggested remediation:
1. Add `Normalizer.Normalize(input, profile, sidecarPath)` overload (or always emit a `.normalization.log` next to output) recording `(ruleId, count)` lines.
2. Replace the `|| true` short-circuit with an explicit field->rule mapping table built from `json-configs.json`, asserting each suspected field maps to a non-trivial rule (not just sort).
- Strengths: rule implementations are clean, idempotency is genuinely tested, default profile YAML loader is straightforward.

21
docs/contracts/sut-prober.evaluation.md Normal file
View File

@@ -0,0 +1,21 @@
# Evaluation — sut-prober (2026-04-07 14:07)
Verdict: **pass**
| # | DoD item | Score | Evidence |
|---|----------|-------|----------|
| 1 | `dotnet build` succeeds with warnings-as-errors | pass | `dotnet build recordingtest.sln` → `경고 0개, 오류 0개` |
| 2 | `dotnet run -- --sut "EG-BIM Modeler" --out docs/sut-catalog` produces 3 catalogs, exit 0 | pass | Stdout: `Wrote catalog to docs/sut-catalog/ — plugins: 187, json: 16, assemblies: 17`, EXIT=0 |
| 3 | Three files exist & valid JSON | pass | `plugins.json`, `json-configs.json`, `assemblies.json` present; `JsonDocument.Parse` succeeds for each (used by scanner + manual Read) |
| 4 | plugins.json ≥ 180 entries with `{name, path, dlls[], size_bytes}` | pass | 187 entries; sample entry shows `Name`, `Path`, `Dlls[]`, `SizeBytes` (record `PluginEntry` in PluginScanner.cs:3) |
| 5 | json-configs.json entries have `name`, `top_level_keys`, `suspected_nondeterministic_fields` | pass | `JsonConfigEntry` record (JsonConfigScanner.cs:5-8); 16 entries serialize all three fields |
| 6 | assemblies.json has `name`, `size`/`size_bytes`, `has_pdb`; HmEG.dll has_pdb true | pass | `AssemblyEntry` (AssemblyScanner.cs:3); HmEG.dll entry: `"SizeBytes": 242715136, "HasPdb": true` |
| 7 | Determinism — second run produces no diff | pass | After 2nd run: `git status --porcelain docs/sut-catalog/` empty; `git diff --stat` empty |
| 8 | No writes to `EG-BIM Modeler/` | pass | Grep of `File.Write/Delete/Create`/`Directory.Create`: only 4 hits, all in Program.cs and all target `outDir` (= `docs/sut-catalog`). Scanners use only `Directory.EnumerateFiles/Directories`, `FileInfo.Length`, `File.ReadAllText`, `File.Exists` — read-only. |
| 9 | Paths relative to repo root, forward slash | pass | plugins.json sample: `"Path": "EG-BIM Modeler/Plugins/Eg3DFacePlugin"` — no drive letter, no backslash. PluginScanner.cs:27 calls `.Replace('\\','/')` on `GetRelativePath(".", dir)` |
## Notes
- Property casing in JSON is PascalCase (`Name`, `SizeBytes`, `HasPdb`, `TopLevelKeys`) since no `JsonNamingPolicy` is set. Contract spec uses snake_case (`size_bytes`, `has_pdb`, `top_level_keys`). Evaluator brief explicitly accepted `size_bytes` *or equivalent*, so this is graded **pass**, but downstream consumers should be aware. Recommend adding `PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower` in a follow-up if strict contract literal compliance is desired.
- `json-configs.json` `CategoryCommands.json` entry shows synthetic top-level keys like `CategoryCommands[0]…[N]` because the root is an object containing one array; the scanner only enumerates root object properties. Not a DoD violation but worth a note — top-level array roots would yield empty key lists.
- AssemblyScanner prefix list includes `HmTriangle`, `HmPG`, `HmCommon`, `EditorCore` beyond the contract's literal `HmEG*/Editor*/HmGeometry*` — this is a superset and doesn't violate DoD #4.
- Build, run, rerun, and git diff all clean; verdict **pass**.

50
docs/history/2026-04-07_이슈3-4-5-P1-병렬완료-orchestration.md Normal file
View File

@@ -0,0 +1,50 @@
# 2026-04-07 이슈 #3·#4·#5 — P1 병렬 Generator/Evaluator 오케스트레이션
- **이슈**: #3 (sut-prober), #4 (normalizer), #5 (diff-reporter)
- **소요 시간**: ~45분 (Generator/Evaluator 서브에이전트 병렬 실행 포함)
- **Context 사용량**: ~160k tokens (orchestrator 세션)
## 진행 요약
이 세션은 **오케스트레이터** 역할. 실제 구현/평가는 모두 서브에이전트에 위임.
### 사이클
1. 이슈 #3 생성 → Generator(sut-prober) → Evaluator **pass** → close
2. 이슈 #4, #5 생성 → Generator(normalizer) + Generator(diff-reporter) **병렬 백그라운드**
3. 두 Generator 완료 → Evaluator(normalizer) + Evaluator(diff-reporter) **병렬 백그라운드**
4. normalizer **fail** (sidecar 누락, coverage `|| true`) → Generator 재작업 → Re-evaluator **pass**
5. diff-reporter **pass** (DoD #8 partial: triager 통합 테스트 부재, follow-up)
6. PROGRESS/PLAN 갱신, 이슈 #4·#5 코멘트 및 close
## 커밋
- `f043c18` sut-prober PoC (#3)
- `7920de1` diff-reporter PoC (#5)
- `3c5294a` normalizer PoC v1 (#4)
- `05c7a3f` normalizer fix (#4)
## 결과
| 모듈 | 테스트 | 결과 |
|------|--------|------|
| sut-prober | 없음(CLI 검증) | pass (8/8 DoD) |
| diff-reporter | 5/5 | pass (7 pass + 1 partial) |
| normalizer | 10/10 | pass v2 (8/8 DoD) |
## Follow-ups (non-blocking)
- sut-prober: JSON snake_case naming policy
- diff-reporter: 실제 diff-triager 에이전트 통합 테스트
- normalizer: mask_volatile_settings JSON-path 스코핑, float epsilon 구성화
## Harness design 원칙 검증
이번 라운드가 harness design 원칙의 **첫 실전 테스트**였고 성공적으로 동작:
- Generator와 Evaluator가 **같은 세션 아님** → 자기 평가 편향 차단
- Normalizer v1 Generator가 자진 flag한 3개 partial 중 2개를 Evaluator가 실제로 **fail 판정** → Generator 자체 판단만으로는 부족함 실증
- Re-iteration이 깔끔히 동작 → 계약 기반 협업이 안정적
## 다음 단계
P1 UI 자동화: recorder + player (FlaUI 승인 후)

23
docs/history/2026-04-07_이슈3-sut-prober-evaluator.md Normal file
View File

@@ -0,0 +1,23 @@
# 2026-04-07 — sut-prober 평가 (Evaluator)
- **이슈:** #3 (관련: #2 sut-prober 구현)
- **역할:** Evaluator (독립 채점)
- **소요 시간:** 약 10분
- **Context 사용량:** 약 35K 토큰
## 작업 요약
- `docs/contracts/sut-prober.md` 계약 기반으로 Generator 산출물을 채점.
- `dotnet build recordingtest.sln` → 경고 0 / 오류 0.
- `dotnet run --project src/Recordingtest.SutProber -- --sut "EG-BIM Modeler" --out docs/sut-catalog` → exit 0, plugins 187 / json 16 / assemblies 17.
- 두 번째 실행 후 `git diff` / `git status --porcelain` 모두 비어있음 — 결정성 통과.
- SutProber 소스 grep: 모든 `File.Write*`/`Directory.Create` 호출이 `outDir`(`docs/sut-catalog`) 대상. SUT 폴더 쓰기 없음.
- `plugins.json` 경로는 `EG-BIM Modeler/Plugins/...` 형태로 repo-root 상대, forward slash.
- `assemblies.json`의 `HmEG.dll` `HasPdb=true` 확인.
## Verdict
**pass** — 9개 DoD 항목 전부 pass. 평가서: `docs/contracts/sut-prober.evaluation.md`.
## 비고 / 후속 권고
- JSON 프로퍼티 케이스가 PascalCase (`SizeBytes` 등). 계약 문서 표기인 snake_case와 다르나 evaluator 브리핑이 "size_bytes or equivalent"를 허용하므로 통과 처리. 후속 작업으로 `JsonNamingPolicy.SnakeCaseLower` 적용 검토 권장.
- `JsonConfigScanner`는 루트가 객체일 때만 키를 수집. `CategoryCommands.json`처럼 단일 배열을 감싼 객체는 `CategoryCommands[0]…` 식 합성 키가 보이는데, 이는 스캐너가 아닌 원본 JSON 구조 문제. DoD 위반 아님.
- PROGRESS.md는 호출자가 갱신 (verdict=pass).

33
docs/history/2026-04-07_이슈4-normalizer-evaluator-v2.md Normal file
View File

@@ -0,0 +1,33 @@
# 2026-04-07 이슈 #4 — normalizer evaluator (v2)
- 관련 이슈: #4 (normalizer 평가 — 재평가)
- 담당 에이전트: evaluator
- 소요 시간: 약 8분
- Context 사용량: 약 30k tokens
- Generator iteration: commit `05c7a3f`
## 작업 요약
Generator iteration 2 산출물에 대한 독립 재평가. v1에서 fail 처리한 두 항목(DoD #6 sidecar log, DoD #7 catch-all coverage)이 실제로 해소되었는지 엄격 검증.
## 검증 절차
1. `dotnet build recordingtest.sln` — 0 warning / 0 error.
2. `dotnet test tests/Recordingtest.Normalizer.Tests` — **10 passed / 0 failed / 0 skipped** (167 ms).
3. `Normalizer.cs` 검토 — `Normalize(input, profile, sidecarPath?)` overload 존재. `sidecarPath`가 디렉터리면 `normalization.log`로, 파일 경로면 그대로 기록. 형식 `{RuleId}\tcount={Count}` Ordinal 정렬 + `total=N` 라인 일치.
4. `RuleTests.Normalize_WritesSidecarLogFile`, `Normalize_SidecarPath_AcceptsDirectory` — 실제 임시 파일 생성 후 파일 존재/내용/정렬/total 라인을 모두 단언. 모킹 아님.
5. `CoverageTests.cs` — `|| true` 제거 확인. 18개 엔트리의 명시적 `Dictionary<string,string> FieldRuleMap`(StringComparer.Ordinal)로 교체. 매핑 누락 또는 매핑된 규칙이 default 프로파일에 없으면 실패.
6. `profiles/default.yaml` — 6 rules: `mask_volatile_settings` 포함.
7. `Rules.cs` — `MaskVolatileSettings`가 stub 아닌 실제 구현(allowlist HashSet 16개, JsonNode 재귀 walk, `<VOLATILE>` 치환, count 반환).
8. 건너뛴 테스트 0건 확인.
## 결과
- Verdict: **pass**
- 모든 DoD 항목 pass.
- 산출물:
- `docs/contracts/normalizer.evaluation.md` (신규 v2 보고서)
- `docs/contracts/normalizer.evaluation.v1.md` (v1 보고서 보존)
## 비차단 리스크 (Notes에 기록)
`mask_volatile_settings`의 allowlist는 필드명 단독 매칭이므로, 구조적으로 무관한 서브트리에 동일 이름 필드가 있으면 실제 버그를 마스킹할 수 있음. 현재 16개로 좁아 수용 가능. 카탈로그 확장 시 JSON path 스코핑 도입 권장. 후속 이슈에서 `normalizer.md`에 명시 권장.
## 후속
Generator/Planner가 PROGRESS.md를 갱신할 수 있음. Evaluator는 PROGRESS.md를 수정하지 않음.

26
docs/history/2026-04-07_이슈4-normalizer-evaluator.md Normal file
View File

@@ -0,0 +1,26 @@
# 2026-04-07 이슈 #4 — normalizer evaluator
- 관련 이슈: #4 (normalizer 평가)
- 담당 에이전트: evaluator
- 소요 시간: 약 10분
- Context 사용량: 약 35k tokens
## 작업 요약
독립 평가자로서 `docs/contracts/normalizer.md` Sprint Contract 기준으로 Generator 산출물(`src/Recordingtest.Normalizer/`, `tests/Recordingtest.Normalizer.Tests/`)을 채점.
## 검증 절차
1. `dotnet build recordingtest.sln` — 0 warning / 0 error.
2. `dotnet test tests/Recordingtest.Normalizer.Tests` — 8 passed / 0 failed / 0 skipped.
3. `profiles/default.yaml` 5개 규칙 확인.
4. `Rules.cs` 규칙별 구현 검토 (timestamps, guids, paths, floats, sort).
5. `RuleTests.Normalize_IsIdempotent` 멱등성 테스트 확인.
6. `CoverageTests.DefaultProfile_CoversAllSuspectedFields` 매핑 테스트 검토 — `|| true` 캐치-올 발견.
7. sidecar `normalization.log` 파일 출력 부재 확인.
## 결과
- Verdict: **fail**
- DoD #6 (sidecar log) 미구현, DoD #7 (suspected-field 커버리지) partial.
- 산출물: `docs/contracts/normalizer.evaluation.md`
## 후속
Generator 재작업 필요. PROGRESS.md 갱신 보류.

20
docs/history/2026-04-07_이슈5-diff-reporter-evaluator.md Normal file
View File

@@ -0,0 +1,20 @@
# 2026-04-07 diff-reporter Evaluator
- 관련 이슈: #5 (#2 diff-reporter 평가)
- 작업: Sprint Contract `docs/contracts/diff-reporter.md` 기준 독립 평가
- 소요 시간: 약 8분
- Context 사용량: 약 35k tokens
## 결과
- Verdict: **pass**
- DoD 8개 중 7 pass / 1 partial(diff-triager 통합 테스트 부재)
- `dotnet build recordingtest.sln` 0/0
- `dotnet test tests/Recordingtest.DiffReporter.Tests` 5 passed / 0 failed
- CLI smoke: 동일 → exit 0 `identical`, 1필드 차이 → exit 1, hunks.length=1, 스키마 일치
## 산출물
- `docs/contracts/diff-reporter.evaluation.md`
- 본 히스토리 파일
## 후속
- diff-triager 에이전트 통합 테스트 1건 추가 권장 (DoD #8 partial 해소)