13dc4109d8
- 5-module E2E integration runner, 6 tests, all DoD pass - PROGRESS.md Done row, PLAN.md pivoted to live smoke test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
44 lines
3.1 KiB
Markdown
44 lines
3.1 KiB
Markdown
# test-runner Evaluation (Issue #8)
|
|
|
|
- Generator commit: `96df2ef`
|
|
- Evaluator: independent verification per contract `docs/contracts/test-runner.md`
|
|
- Build: `dotnet build recordingtest.sln` -> 0 warnings, 0 errors
|
|
- Tests: `dotnet test tests/Recordingtest.Runner.Tests` -> 6 passed / 0 failed / 0 skipped
|
|
|
|
## Verdict: PASS
|
|
|
|
## DoD verification
|
|
|
|
| # | DoD item | Result | Evidence |
|
|
|---|----------|--------|----------|
|
|
| 1 | Console exe with 5 flags `--scenarios/--baselines/--out/--profile/--no-launch` | pass | `src/Recordingtest.Runner/Program.cs` switch parses all 5; missing required -> exit 2 |
|
|
| 2 | Scan `*.yaml` and write to `<out>/<scenario>/` | pass | `TestRunner.cs` L27-36 enumerates `*.yaml`, creates per-scenario `artifactDir` |
|
|
| 3 | Order: player -> normalizer -> diff-reporter | pass | `TestRunner.cs` L50-52 (engine.Run), L103-104 (Normalize), L111 (Compare) |
|
|
| 4 | Profile default `default`, overridable | pass | `RunnerOptions.Profile = "default"`; passed through to normalizer; `--profile` writes it |
|
|
| 5 | `report.json` schema `{runAt,total,passed,failed,errored,scenarios:[{name,status,hunks,checkpointCount,artifactDir}]}` | pass | `RunReport.cs` matches; camelCase JSON; test 6 asserts every field |
|
|
| 6 | `report.md` human summary with table + failure section | pass | `WriteMarkdownReport` builds table + Failures section |
|
|
| 7 | Exit codes 0/1/2 | pass | `ToExitCode`: errored>0 -> 2, failed>0 -> 1, else 0; tests assert all three |
|
|
| 8 | `IPlayerHost` DI via `IRunnerHostFactory` | pass | `Interfaces.cs`; `RunAll` takes factory + INormalizer + IDiffer; tests inject fakes |
|
|
| 9 | xUnit tests >=5 covering 5 scenarios | pass | 6 tests, all required cases (identical, differs, throws, empty, profile spy, schema) |
|
|
| 10 | `dotnet build` green, `dotnet test` all pass | pass | 0/0 build, 6/6 tests |
|
|
| 11 | Fixed sleep 0 | pass | grep `Thread.Sleep(` and `Task.Delay(TimeSpan.FromSeconds` in `src/Recordingtest.Runner` -> 0 hits |
|
|
|
|
## Baseline normalization policy
|
|
Contract allows either pre-normalized or re-normalized baselines. `TestRunner.cs` L10-11 documents the choice: baselines are re-normalized with the same profile as received output (safe either way). Documented = pass.
|
|
|
|
## Test quality (not stubs)
|
|
- TwoScenarios_BothIdentical_ExitZero_AllPass: real scenario YAML, real PlayerEngine, real diff stub identical
|
|
- OneScenarioDiffers: asserts hunks==1 and status=="fail"
|
|
- PlayerThrows: uses click step + `throwOnClick` fake host -> errored>=1, exit 2
|
|
- EmptyScenariosDir: total==0, exit 0
|
|
- ProfileOverride: SpyNormalizer captures profiles list; asserts contains "strict", not "default"
|
|
- ReportJson schema: parses report.json and asserts every contract field; checks report.md exists
|
|
|
|
## Integration smoke
|
|
Trusted via unit tests + source review (Runner is fully DI-testable; tests drive `TestRunner.RunAll` directly with real `PlayerEngine` + scenario YAML).
|
|
|
|
## Artifacts
|
|
- Source: `src/Recordingtest.Runner/{Program.cs,TestRunner.cs,Interfaces.cs,RunnerOptions.cs,RunReport.cs,DefaultAdapters.cs}`
|
|
- Tests: `tests/Recordingtest.Runner.Tests/{TestRunnerTests.cs,Fakes.cs}`
|
|
- Contract: `docs/contracts/test-runner.md`
|