e3d2ff6c77
- sut-prober evaluation (pass) - diff-reporter evaluation (pass with 1 partial follow-up) - normalizer evaluations v1 (fail) + v2 (pass) - PROGRESS.md Done rows for #3, #4, #5 + Follow-ups - PLAN.md P0 reduced to hook verification Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
33 lines
3.6 KiB
Markdown
33 lines
3.6 KiB
Markdown
# Evaluation — normalizer (v2, 2026-04-07)
|
|
|
|
Verdict: **pass**
|
|
|
|
Generator iteration: commit `05c7a3f`.
|
|
|
|
| # | DoD item | Score | Evidence |
|
|
|---|----------|-------|----------|
|
|
| 1 | `Normalize(input, profile)` API | pass | `src/Recordingtest.Normalizer/Normalizer.cs` exposes `Normalize(string, string)` and overload `Normalize(string, string, string?)`. Build green. |
|
|
| 2 | Default profile with >=5 rules | pass | `src/Recordingtest.Normalizer/profiles/default.yaml` lists 6 rules: `strip_timestamps`, `mask_guids`, `normalize_paths`, `round_floats`, `mask_volatile_settings`, `sort_json_keys`. All implemented in `Rules.cs`. |
|
|
| 3 | Profiles as `profiles/*.yaml`, code-free addition | pass | `Profile.Load` reads YAML by name. |
|
|
| 4 | Per-rule before/after sample test | pass | `RuleTests.cs` covers each rule plus `Normalize_AppliesAllDefaultRules` (asserts 6 entries in log including `mask_volatile_settings`). |
|
|
| 5 | Idempotent | pass | `RuleTests.Normalize_IsIdempotent`. |
|
|
| 6 | Sidecar log `normalization.log` | pass | `Normalizer.cs` lines 150-176: when `sidecarPath` supplied, writes file containing `{RuleId}\tcount={Count}` lines sorted by RuleId (Ordinal) and final `total=` line. Accepts either a file path or directory (in which case it writes `normalization.log` inside). Two real-temp-file tests: `Normalize_WritesSidecarLogFile` and `Normalize_SidecarPath_AcceptsDirectory` — both assert file existence and content (sorted order, total line, per-rule lines). |
|
|
| 7 | `json-configs.json` suspected fields fully covered | pass | `CoverageTests.cs` now declares an explicit `Dictionary<string,string> FieldRuleMap` (18 entries, `StringComparer.Ordinal`) with no `|| true` and no catch-all. Path-bearing fields → `normalize_paths`; volatile boolean/scalar/color fields → `mask_volatile_settings`. Test fails if any suspected field is unmapped or if its mapped rule is missing from `default.yaml`. |
|
|
| 8 | All Normalizer tests pass | pass | `dotnet test tests/Recordingtest.Normalizer.Tests`: **10 passed, 0 failed, 0 skipped** (167 ms). |
|
|
|
|
## Notes
|
|
- `dotnet build recordingtest.sln`: 0 warnings, 0 errors.
|
|
- Test count grew from 8 → 10 (added two sidecar tests). Coverage test rewritten in place.
|
|
- New rule `mask_volatile_settings` (`Rules.cs` lines 172-224) is fully implemented (not a stub): allowlist `HashSet` of 16 known volatile field names, walks `JsonNode` recursively, replaces matching values with `"<VOLATILE>"` and counts replacements. Idempotent because the placeholder string itself is not in the allowlist's value space.
|
|
- **Risk (non-blocking)**: the volatile-field allowlist is keyed on local field name only (no JSON path scoping). A real bug that incidentally toggles a field named e.g. `GridSnap` in a structurally unrelated subtree would be masked and silently hidden by golden-file diffs. Allowlist is currently 16 names — narrow enough to be acceptable, but should be revisited if the catalog grows. Recommend documenting this allowlist scope in `normalizer.md` in a follow-up (does not block this iteration).
|
|
- Coverage test no longer accepts catch-all to `sort_json_keys`; mapping is strict and explicit per the contract's field→rule requirement.
|
|
- Sidecar format matches the spec exactly: tab-separated `ruleId\tcount=N`, ordinal-sorted, terminated by `total=N`.
|
|
|
|
## Artifacts
|
|
- `src/Recordingtest.Normalizer/Normalizer.cs`
|
|
- `src/Recordingtest.Normalizer/Rules.cs`
|
|
- `src/Recordingtest.Normalizer/profiles/default.yaml`
|
|
- `tests/Recordingtest.Normalizer.Tests/RuleTests.cs`
|
|
- `tests/Recordingtest.Normalizer.Tests/CoverageTests.cs`
|
|
- Previous report: `docs/contracts/normalizer.evaluation.v1.md`
|