feat(baseline): add --baseline-output option for the disk baseline provider

[mpownby]

The disk baseline provider always writes the baseline to a hard-coded "StrykerOutput" folder under the project path. Storing the baseline inside the test project's directory is problematic: without a non-obvious workaround, Stryker flags it as a change to the test project and ignores the baseline.

The new optional --baseline-output argument (config: baseline.output) lets the baseline be stored and loaded from a configurable directory. Relative paths resolve against the project path; absolute paths are used as-is. It defaults to "StrykerOutput", so existing behavior is unchanged unless the option is supplied. Only the disk provider is affected; the Azure and S3 providers are untouched.

#3615

[rouke-broersma]

@mpownby I don't think a new option is warranted here, instead stryker should follow the outputpath parameter for disk baseline provider, it is then auto-ignored because it will be covered by a gitignore.

[mpownby]

Reproduction

The disk baseline provider writes its report to StrykerOutput in the current working directory and ignores the --output parameter. These steps reproduce that, plus the downstream incremental-run breakage, using the released tool against a public sample repo.

1. Clone the repo and restore the Stryker tool

git clone https://github.com/RulecityLLC/DotNetMutationTestingSandbox.git
cd DotNetMutationTestingSandbox
dotnet tool restore

Reproduces on any recent 4.x and on current main — the path is hard-coded in DiskBaselineProvider.

2. Move into the test project

The documented baseline workflow runs from the test project directory, so this is the realistic case:

cd RestApi.Unit.Test

3. Seed a baseline, explicitly redirecting Stryker's output elsewhere

dotnet stryker --with-baseline:89b7eb28a64d1fa3f418faf99760e6d2a4bf63e5 --version main --output ../redirected-stryker-output

89b7eb2… is the repo's initial commit, so everything counts as "new" and a full baseline is written.

Inspect the result — two things to notice:

• ../redirected-stryker-output/ received the reports/logs and an auto-generated .gitignore containing *.
• But the baseline was written to RestApi.Unit.Test/StrykerOutput/baseline/main/stryker-report.json — i.e. --output was ignored for the baseline, which landed in the current directory instead.
• From the repo root, git status lists RestApi.Unit.Test/StrykerOutput/ as untracked. Compare the two folders: the redirected output folder has an auto-.gitignore; the baseline folder does not, and StrykerOutput is not in the repo's .gitignore either. The baseline is the one Stryker artifact that isn't auto-ignored.

4. Run again incrementally — the baseline should be reused, but isn't

dotnet stryker --with-baseline:main --version main --output ../redirected-stryker-output

The log shows Found baseline report for current branch main, but then it re-tests every mutant with a message like "Non-CSharp files in test project were changed" — because the untracked baseline JSON inside the test project shows up in the git diff as a changed test file. The baseline is found yet provides no benefit, and there's no documented, supported way to relocate or suppress it.

The problem and proposed solution

The disk baseline provider currently ignores --output and always writes StrykerOutput into the current working directory, and it's the only Stryker artifact not covered by the auto-generated .gitignore (which is only written into the timestamped report folder, never the baseline folder). So the baseline surfaces as an un-ignored file in the current working directory — which a) may surprise users as this behavior seems to be undocumented and b) breaks incremental runs, since it lands in the git diff and forces a full re-test.

--baseline-output lets users redirect it (the behavior they'd reach for --output to get) without changing anything for existing users. Two reasons a new opt-in option may make sense rather than making the baseline follow --output:

1. The default output path is per-run timestamped. When --output isn't supplied, the output path is StrykerOutput/<timestamp>, a new folder every run — so a baseline written there would never be found on the next run. "Just follow --output" can't work as-is.
2. It may silently break existing users. Anyone who already passes --output (for reports) would have their baseline relocate from StrykerOutput to the output directory, so the next run wouldn't find the existing baseline. A new opt-in option keeps current behavior untouched and lets users move the baseline only when they ask to.

[mpownby]

@mpownby I don't think a new option is warranted here, instead stryker should follow the outputpath parameter for disk baseline provider, it is then auto-ignored because it will be covered by a gitignore.

My first pass at this PR was to have disk baseline provider following --output if the user explicitly set it (or ignore it otherwise) but I was worried about breaking any existing users. If you'd rather me go back to this solution, I can do that.

[rouke-broersma]

Baseline and custom output directory are currently functionally at odds as you justifiably reported. Since we've never had this report before, it's unlikely it is actually being used together much in practice. I would class this as an oversight/bug when custom output directory was introduced.

The two options are simply incompatible right now, and this must be addressed. It would also be acceptable to me if disk baseline provider places it's own .gitignore in the baseline directory instead of following the output directory, however I expect that users expect that output directory affects all files generated by stryker anyway.

Beside that we could also additionally support a new config parameter to explicitly set a baseline output path, that is also a valid ask.

[mpownby]

Baseline and custom output directory are currently functionally at odds as you justifiably reported. Since we've never had this report before, it's unlikely it is actually being used together much in practice. I would class this as an oversight/bug when custom output directory was introduced.

The two options are simply incompatible right now, and this must be addressed. It would also be acceptable to me if disk baseline provider places it's own .gitignore in the baseline directory instead of following the output directory, however I expect that users expect that output directory affects all files generated by stryker anyway.

Beside that we could also additionally support a new config parameter to explicitly set a baseline output path, that is also a valid ask.

If disk baseline follows the custom output directory, then another flag needs to be added to indicate whether the user set a custom output directory, because by the time the disk baseline runs, the output directory will be set to either an auto-generated one (with timestamp) or a user-supplied one, and it will fail if it writes to an auto-generated timestamp path.

I think if few are relying on this feature, then the 'right' solution is to make it follow the output path if the user explicitly sets it.

If you agree, I can rework this PR to go down that path.

[rouke-broersma]

I'm considering whether or not we should move full output path generation including timestamp back to stryker.core and should only pass the working directory or custom output directory from stryker.cli, then output directory can be used directly in baseline. I am not yet sure if that is worth it.

[richardwerkman]

I'm also not in favor of adding a new option for this. Especially when setting the path in the settings file it could cause unexpected behavior when the relative path can not be resolved on another system. It just seems like a bug to me that this file is not outputted in the output directory.

I suggest we place the file in the root of the StrykerOutput dir, not in a run specific sub-dir. Because the baseline feature needs to be able to find the file later. It would be impractical to search all sub-dirs for the latest baseline file.

[rouke-broersma]

@richardwerkman yes but we don't have the 'root' available in core because it's computed in Stryker cli that's why I'm considering moving the computing back up to core

[rouke-broersma]

@mpownby Thanks for your patience and allowing us to think about this

We talked it over and I'd still like to avoid a new parameter for this. I also don't think moving the output directory computation back to Core is the way to go. The CLI is the correct location. I consider it a bug that the disk baseline ignores --output and isn't covered by the gitignore. I propose keeping with my initial suggestion: make the disk baseline follow the output path, but make sure that location is gitignored.

As you said the default output path is per-run timestamped, so the baseline can't live there or it'd never be found next run. My suggestion is to introduce the notion of a stable "output root" with the StrykerOutput directory before the timestamp and place the gitignore here. Also compute the BaselineOutput based on the StrykerOutput directory in Stryker.CLI and pass it as a new input BaselineOutputInput to Stryker.Core.

Basically almost identical to your current change except without the new param. This would technically be a breaking change as the baseline output is now covered by a gitignore and would follow --output but functionally probably what people want.

WDYT?

[mpownby]

@mpownby Thanks for your patience and allowing us to think about this

We talked it over and I'd still like to avoid a new parameter for this. I also don't think moving the output directory computation back to Core is the way to go. The CLI is the correct location. I consider it a bug that the disk baseline ignores --output and isn't covered by the gitignore. I propose keeping with my initial suggestion: make the disk baseline follow the output path, but make sure that location is gitignored.

As you said the default output path is per-run timestamped, so the baseline can't live there or it'd never be found next run. My suggestion is to introduce the notion of a stable "output root" with the StrykerOutput directory before the timestamp and place the gitignore here. Also compute the BaselineOutput based on the StrykerOutput directory in Stryker.CLI and pass it as a new input BaselineOutputInput to Stryker.Core.

Basically almost identical to your current change except without the new param. This would technically be a breaking change as the baseline output is now covered by a gitignore and would follow --output but functionally probably what people want.

WDYT?

(just got back from vacation, apologies for the delay) I'll give implementation a go. Are you saying that you want the baseline to appear one level up from the timestamp directory (so the timestamp directory would be a child within the output directory instead of being the output directory) ?

UPDATE: I misspoke in my above comment and I think I understand what you mean now. If --output is specified, the baseline folder will be put in the output directory, as a sibling of 'reports' and .gitignore . And if it's not specified, a StrykerOutput directory will be created which will contain the .gitignore, the baesline folder and a timestamp folder which will contain the reports folder.

[rouke-broersma]

If --output is specified, the baseline folder will be put in the output directory, as a sibling of 'reports' and .gitignore . And if it's not specified, a StrykerOutput directory will be created which will contain the .gitignore, the baesline folder and a timestamp folder which will contain the reports folder.

Yes correct, so make Baseline follow the output param and move the gitignore up one level so it covers everything in output. Stryker.Core should no longer try to compute the BaselineDirectory, it should take this verbatim from the input as computed by Stryker.CLI. I think you were already most of the way there with the param implementation.

I think it's fine to always place the gitignore even if output is specified by the user. If they user wants to include some files in the output directory into git they can modify the gitignore afterwards since we only place it if the file doesn't exist.

[rouke-broersma]

Not sure if we should still path combine with the project path, I think BaselineOutputPath should always be a full path now. Maybe we should add a validation in the input validation to make sure it's a full path. Makes our lives easier in the rest of the code.

[mpownby]

ok I think I got this sorted. I ran another manual test and it still seems to be working properly.