# Evaluation — sut-prober  (2026-04-07 14:07)

Verdict: **pass**

| # | DoD item | Score | Evidence |
|---|----------|-------|----------|
| 1 | `dotnet build` succeeds with warnings-as-errors | pass | `dotnet build recordingtest.sln` → `경고 0개, 오류 0개` |
| 2 | `dotnet run -- --sut "EG-BIM Modeler" --out docs/sut-catalog` produces 3 catalogs, exit 0 | pass | Stdout: `Wrote catalog to docs/sut-catalog/ — plugins: 187, json: 16, assemblies: 17`, EXIT=0 |
| 3 | Three files exist & valid JSON | pass | `plugins.json`, `json-configs.json`, `assemblies.json` present; `JsonDocument.Parse` succeeds for each (used by scanner + manual Read) |
| 4 | plugins.json ≥ 180 entries with `{name, path, dlls[], size_bytes}` | pass | 187 entries; sample entry shows `Name`, `Path`, `Dlls[]`, `SizeBytes` (record `PluginEntry` in PluginScanner.cs:3) |
| 5 | json-configs.json entries have `name`, `top_level_keys`, `suspected_nondeterministic_fields` | pass | `JsonConfigEntry` record (JsonConfigScanner.cs:5-8); 16 entries serialize all three fields |
| 6 | assemblies.json has `name`, `size`/`size_bytes`, `has_pdb`; HmEG.dll has_pdb true | pass | `AssemblyEntry` (AssemblyScanner.cs:3); HmEG.dll entry: `"SizeBytes": 242715136, "HasPdb": true` |
| 7 | Determinism — second run produces no diff | pass | After 2nd run: `git status --porcelain docs/sut-catalog/` empty; `git diff --stat` empty |
| 8 | No writes to `EG-BIM Modeler/` | pass | Grep of `File.Write/Delete/Create`/`Directory.Create`: only 4 hits, all in Program.cs and all target `outDir` (= `docs/sut-catalog`). Scanners use only `Directory.EnumerateFiles/Directories`, `FileInfo.Length`, `File.ReadAllText`, `File.Exists` — read-only. |
| 9 | Paths relative to repo root, forward slash | pass | plugins.json sample: `"Path": "EG-BIM Modeler/Plugins/Eg3DFacePlugin"` — no drive letter, no backslash. PluginScanner.cs:27 calls `.Replace('\\','/')` on `GetRelativePath(".", dir)` |

## Notes
- Property casing in JSON is PascalCase (`Name`, `SizeBytes`, `HasPdb`, `TopLevelKeys`) since no `JsonNamingPolicy` is set. Contract spec uses snake_case (`size_bytes`, `has_pdb`, `top_level_keys`). Evaluator brief explicitly accepted `size_bytes` *or equivalent*, so this is graded **pass**, but downstream consumers should be aware. Recommend adding `PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower` in a follow-up if strict contract literal compliance is desired.
- `json-configs.json` `CategoryCommands.json` entry shows synthetic top-level keys like `CategoryCommands[0]…[N]` because the root is an object containing one array; the scanner only enumerates root object properties. Not a DoD violation but worth a note — top-level array roots would yield empty key lists.
- AssemblyScanner prefix list includes `HmTriangle`, `HmPG`, `HmCommon`, `EditorCore` beyond the contract's literal `HmEG*/Editor*/HmGeometry*` — this is a superset and doesn't violate DoD #4.
- Build, run, rerun, and git diff all clean; verdict **pass**.