recordingtest/docs/contracts/sut-prober.evaluation.md

Evaluation — sut-prober (2026-04-07 14:07)

Verdict: pass

#	DoD item	Score	Evidence
1	dotnet build succeeds with warnings-as-errors	pass	dotnet build recordingtest.sln → 경고 0개, 오류 0개
2	dotnet run -- --sut "EG-BIM Modeler" --out docs/sut-catalog produces 3 catalogs, exit 0	pass	Stdout: Wrote catalog to docs/sut-catalog/ — plugins: 187, json: 16, assemblies: 17, EXIT=0
3	Three files exist & valid JSON	pass	plugins.json, json-configs.json, assemblies.json present; JsonDocument.Parse succeeds for each (used by scanner + manual Read)
4	plugins.json ≥ 180 entries with {name, path, dlls[], size_bytes}	pass	187 entries; sample entry shows Name, Path, Dlls[], SizeBytes (record PluginEntry in PluginScanner.cs:3)
5	json-configs.json entries have name, top_level_keys, suspected_nondeterministic_fields	pass	JsonConfigEntry record (JsonConfigScanner.cs:5-8); 16 entries serialize all three fields
6	assemblies.json has name, size/size_bytes, has_pdb; HmEG.dll has_pdb true	pass	AssemblyEntry (AssemblyScanner.cs:3); HmEG.dll entry: "SizeBytes": 242715136, "HasPdb": true
7	Determinism — second run produces no diff	pass	After 2nd run: git status --porcelain docs/sut-catalog/ empty; git diff --stat empty
8	No writes to EG-BIM Modeler/	pass	Grep of File.Write/Delete/Create/Directory.Create: only 4 hits, all in Program.cs and all target outDir (= docs/sut-catalog). Scanners use only Directory.EnumerateFiles/Directories, FileInfo.Length, File.ReadAllText, File.Exists — read-only.
9	Paths relative to repo root, forward slash	pass	plugins.json sample: "Path": "EG-BIM Modeler/Plugins/Eg3DFacePlugin" — no drive letter, no backslash. PluginScanner.cs:27 calls .Replace('\\','/') on GetRelativePath(".", dir)

Notes

• Property casing in JSON is PascalCase (Name, SizeBytes, HasPdb, TopLevelKeys) since no JsonNamingPolicy is set. Contract spec uses snake_case (size_bytes, has_pdb, top_level_keys). Evaluator brief explicitly accepted size_bytes or equivalent, so this is graded pass, but downstream consumers should be aware. Recommend adding PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower in a follow-up if strict contract literal compliance is desired.
• json-configs.json CategoryCommands.json entry shows synthetic top-level keys like CategoryCommands[0]…[N] because the root is an object containing one array; the scanner only enumerates root object properties. Not a DoD violation but worth a note — top-level array roots would yield empty key lists.
• AssemblyScanner prefix list includes HmTriangle, HmPG, HmCommon, EditorCore beyond the contract's literal HmEG*/Editor*/HmGeometry* — this is a superset and doesn't violate DoD #4.
• Build, run, rerun, and git diff all clean; verdict pass.