mtp runner lacks on option to test each mutant in its own process

[gem4511]

Describe the bug

With "test-runner": "mtp", mutants that are provably equivalent (no test can possibly detect them) are intermittently reported Killed when the code-under-test holds process-global state — e.g. any CSLA BusinessBase, whose validation rules are registered once per process and kept in a static cache.

Two independent levers each drive the false-kills to zero:

• running the same solution at --concurrency 1, and
• removing the process-global state — an otherwise-identical stateless codebase never misattributes, even at ~8,600 mutants / 17 min.

This looks like a per-mutant isolation problem: the mtp shared test-server is reused across concurrently-evaluated mutants, so a genuine failure produced by one mutant is attributed to another — here, an equivalent one → false "Killed". Consequences: the mutation score is inflated (equivalent mutants counted as detected) and the per-mutant survivor set is not reproducible between identical runs.

Minimal repro: https://github.com/gem4511/stryker-mtp-isolation-repro — two runnable solutions (a CSLA case and a stateless control):

solution	process-global state	concurrency	provably-equivalent "canary" mutants reported Killed
CSLA (100 business objects)	CSLA per-type rule cache	default	~9–18 of 200 (varies run-to-run)
CSLA	same	--concurrency 1	0 of 200
stateless control (100 functions)	none	default	0 of 200
stateless, scaled ~8,600 mutants / 17 min	none	default	0 of ~1,700

Each "canary" is a covered-but-unobservable mutation. In the CSLA case it is the removal of context.AddSuccessResult(true) from a single-rule success branch — equivalent because CSLA clears that rule's prior broken result on every re-run regardless. The README includes a one-command proof (delete every canary line, dotnet test → all 300 pass), so the canaries are demonstrably unkillable and any Killed count > 0 is a false positive.

Notes that rule out the obvious alternative explanations:

• Tests are stateless and run with [assembly: CollectionBehavior(DisableTestParallelization = true)], so this is not intra-suite test parallelism.
• The repro sets "additional-timeout": 30000, and the misattributed mutants are reported Killed, not Timeout — so this is not the "timeouts are scored as killed" behavior.

Expected behavior

Each mutant is evaluated in isolation. A mutant that no test can detect (equivalent) must be reported Survived, deterministically, regardless of --concurrency.

Logs

The repro reproduces in ~30 s, so running it directly may be easiest — but I'm happy to attach -L logs from a run if useful; just let me know.

Desktop (please complete the following information):

• OS: Windows 11
• Type of project: core (net10.0)
• Framework Version: .NET SDK 10.0.301; xunit.v3 3.2.2; Microsoft.NET.Test.Sdk 18.5.1; runner mtp
• Stryker Version: 4.16.0

Additional context

The CSLA half of the mechanism is documented, not guessed. CSLA registers each business-object type's rules once per process and caches them in static state — Rockford Lhotka: "Validation rules are loaded once per-AppDomain per business object type. They are kept in a static cache … they are never unloaded — that is intentional" (performance). In source, BusinessRuleManager holds a static Lazy<ConcurrentDictionary<…>> keyed by Type, and BusinessBase.InitializeBusinessRules() calls AddBusinessRules() exactly once per type under an Initialized-guarded double-checked lock. So CSLA's caching is intentional and thread-safe for registration; the isolation gap appears to be on the runner side — a shared test-server reused across concurrently-evaluated mutants. (Consistent with mtp not yet doing per-test coverage, i.e. the server being heavily shared.)

The precise leak across the shared server is the one inferred step; the CSLA static cache and the two levers (stateless → 0, --concurrency 1 → 0) are directly demonstrated in the repro. Happy to run any diagnostic you suggest.

Possible mitigation: per-mutant process isolation (avoid reusing one test-server across concurrently-evaluated mutants). CSLA exposes no public API to reset its rule cache, so cache-clearing is not a clean option. --concurrency 1 restores correct results today, at a large wall-clock cost.

[gem4511]

Added a -L log from a reproducing run: logs/stryker-log.txt (absolute paths scrubbed; run details in logs/README.md).

That run (default concurrency) scored 41.36% and reported 6 provably-equivalent "canary" mutants as Killed — Statement removals / true→false flips of context.AddSuccessResult(true) on // CANARY lines (unkillable per the repo's one-command proof):

mutant id	line	mutator	reported
284	505	Boolean false	Killed
379	673	Statement ;	Killed
715	1261	Statement ;	Killed
780	1373	Boolean false	Killed
1403	2465	Statement ;	Killed
1548	2717	Boolean false	Killed

(The exact ids/lines vary run-to-run since the misattribution is nondeterministic; at --concurrency 1 the canary-Killed count is 0.)

The same run also produced ~430 MB of per-worker MTP RPC traces (test-servers/rpc-CslaRepro.Tests-MtpRunner-0 … -11 — ≈12 concurrent workers), omitted from the repo for size but available on request in whole or in any slice you'd find useful. That ≈12-concurrent-test-server condition is exactly where the false-kills appear.

[rouke-broersma]

Probably #3695

[gem4511]

Followed the pointer to #3695 — built it and measured. It resolves that PR's case, but not this one.

Thanks for the redirect. I built #3695 (@npnelson's branch fix/mtp-session-result-isolation, commit b2fd312) from source on .NET SDK 10.0.301 and ran two repros through the built Stryker.CLI.dll — @npnelson's own gist and this issue's repro.

@npnelson's fix works for its target. ✅ Same --test-runner mtp --concurrency 1 their script uses:

Stryker	@npnelson's gist (mutates a static field)
4.16.0	100.00% — id 3 falsely Killed (BUG)
#3695 branch	75.00% — id 3 correctly Survived

Confirmed live and correct for a mutation of a static member baking into the reused host.

This issue (#3742) is unchanged by it. Public repro: https://github.com/gem4511/stryker-mtp-isolation-repro

Runner	Concurrency	Equivalent "canary" mutants falsely Killed
4.16.0	default	17 / 200
#3695 branch	default	16, then 18 / 200 (unchanged)
#3695 branch	--concurrency 1	0 / 200

Coverage capture reports 0 static mutations on this repro, so #3695's gate — mutant.IsStaticValue || mutant.MustBeTestedInIsolation (SingleMicrosoftTestPlatformRunner.cs:109) — never fires. These are non-static mutants whose effect leaks through the code-under-test's process-global static state (a CSLA per-type rule cache, cached once per process by design) across concurrently-evaluated mutants on the reused host.

	#3695	#3742 (this issue)
trigger	mutation of a static member bakes into the reused host	non-static mutation whose effect leaks via the code-under-test's static cache
reproduces at --concurrency 1?	yes (single reused process)	no (needs concurrent mutants)
caught by #3695's gate?	yes	no (0 static mutations)

Same root theme — a reused MTP test-host sharing process-global state across mutant sessions — but two distinct triggers, and #3695's mutant-classification gate only isolates the first. This facet still needs per-mutant process isolation (or --concurrency 1, at the usual wall-clock cost). Happy to run any diagnostic against either branch.

@npnelson — flagging you directly so you can see the full measurement: your PR is confirmed working within its scope here. We simply want to inform you of our "flavor" of our common concern, as you may possibly come up against our situation in the future.

[dupdob]

@gem4511
Disclaimer, I am not familiar with the architecture of the MTP Runner. Slowing getting there, but not there yet. That being said, I have significant expertise in static mutant related issues.
I also ran the repro project, but digging into it requires too much work it is still a large project.

Solution

You should enable disable-mix-mutants in the Stryker configuration. I think this option is not available via the command line and must be set in the config file.
Caveat: looking a the code, I do not think the MTP runner properly respect this settings. So this may be the actual issue, MTP runner not respecting disable-mix-mutant . This option should ensure mutants are tested in isolated processes.

This is not a workaround, i.e. one should not expect Stryker to address this situation without any hint provided.

'Static mutants'

'Static mutants' is the unofficial term we use to describe mutations that relates to static data. These mutations can happen within static methods/class constructor as well as within instance methods. The presence of static data often relates to the infamous singleton pattern.
From a testing perspective, having one or more singleton implies it is gonna be very difficult to have isolated tests, as in tests that cannot alter the behavior of other tests.
As a consequence, tests which behavior depends on some mutable static values must be viewed as unreliable.

In practice, this is rarely an issue: the test base should be green. Problems arise when some code change results in some tests being red. It can result in tests failing while they have no clear relation with the modified code. Worse, in some extreme circumstances, the code change may still result in a green test base while some tests were expected to fail.

How does Stryker deal with that?

In theory, this situation should be rare; singletons are frowned upon for reasons. But they are sometimes unavoidable; when this happens, the best strategy is to secure its value during tests; which is often cumbersome. Also, you may have no control about said singletons, as in the case of CSLA.
As a result, we have introduced several strategies to detect this situation in Stryker. We use both code parsing and coverage analysis to identify mutations happening in such situations. Stryker then flags these mutations and the test runner is supposed to test them in dedicated processes to prevent any unwanted side effects. That is what PR #3695 provides for MTP.

it does not fix your problems and why it never will?

When the static is located in a dependency, Stryker cannot identify it and will not be flagged during mutation generation (since it is not part of the project source code) and it will not be flagged during coverage analysis since no mutation has been introduced.

Hence, Stryker cannot work secure the execution of impacted mutations (since it cannot detect which ones are impacted). So, the only valid solution is to mark the whole run as `disable-mix-mutants' or even disable coverage analysis, as it may sometimes disrupt coverage analysis.

Other remarks

• I have no idea why disabling concurrency fixes the issue. It should not, but I am not familiar enough with the MTP runner design to elaborate on why it does.
• I am not convinced agents will be able to work on addressing the lack of coverage revealed by mutation testing.

[gem4511]

@dupdob — thank you, this is exactly the clarity I needed. Your explanation of why it can't be auto-detected — the static lives in a dependency (CSLA), so it's invisible to both mutation generation and coverage analysis — makes complete sense and settles that question for me.

I tested your suggestion. With disable-mix-mutants set, Stryker acknowledges it (Capture mutant coverage using 'CoverageBasedTest, DisableMixMutants' mode) — but the provably-equivalent "canary" false-kills persist:

config (4.16.0, mtp runner)	equivalent canaries reported Killed
baseline	~17 / 200
disable-mix-mutants	21 / 200
disable-mix-mutants + coverage-analysis: off	27 / 200
--concurrency 1	0 / 200

So your suspicion looks right: the MTP runner recognizes disable-mix-mutants but doesn't actually isolate — with or without coverage analysis. --concurrency 1 remains the only thing that zeroes the misattribution (I can't explain that one either; glad to help dig if it's useful).

It reproduces in one line — add "disable-mix-mutants": true to csla/CslaRepro.Tests/stryker-config.json and re-run. If making the MTP runner honor disable-mix-mutants becomes a candidate fix, I'm happy to test any branch against the repro — I have it and a local build standing by.

Thanks again for digging in.

[npnelson]

I think the effect of --concurrency 1 in this reproduction is to make the state-contaminated execution order reproducible, not correct. It makes the original false-kill symptom disappear, but the comparison below demonstrates a false survivor instead.

A fresh test-host process for each mutant seems to be the reliable general mitigation, but perhaps this can be done in a performant way or some other clever mechanism.

This can be reproduced without a helper script:

git clone https://github.com/gem4511/stryker-mtp-isolation-repro.git
cd stryker-mtp-isolation-repro

# Pin the repro revision verified with dotnet-stryker 4.16.0,
# the latest release at the time of this issue.
git checkout --detach 905c7834ba694760c0cb58fde254a42c37d7e3a0

cd csla/CslaRepro.Tests

# Restore the manifest-pinned local tool rather than using any global installation.
dotnet tool restore
dotnet tool run dotnet-stryker -- --version

The version command should report 4.16.0.

First, run only the target mutant:

dotnet tool run dotnet-stryker -- --concurrency 1 --mutate "**/Answers.cs{954..1018}" --reporter cleartexttree --verbosity error | more

The relevant result is:

[Killed] Object initializer mutation on line 18
[-] new OkRule { PrimaryProperty = ValueProperty }
[+] new OkRule {}

Now select the same target plus one earlier mutant:

dotnet tool run dotnet-stryker -- --concurrency 1 --mutate "**/Answers.cs{898..939}{954..1018}" --reporter cleartexttree --verbosity error | more

The relevant results are:

[Survived] Statement mutation on line 17
[-] base.AddBusinessRules();
[+] ;

[Survived] Object initializer mutation on line 18
[-] new OkRule { PrimaryProperty = ValueProperty }
[+] new OkRule {}

The target-only invocation establishes the target's clean-process result as Killed. Selecting another mutant must not change that target's independent result, so the Survived verdict in the second invocation is a false survivor.

cleartexttree also lists ignored mutants, which is why the output is piped through more. The relevant mutations are near the beginning of the tree. The source spans are intentionally wide enough to work with either LF or CRLF checkouts.

[gem4511]

Confirming @npnelson's result — independently reproduced on the pinned revision (905c783) with dotnet-stryker 4.16.0, --concurrency 1 throughout:

• --mutate "**/Answers.cs{954..1018}" (target alone): new OkRule { PrimaryProperty = ValueProperty } → new OkRule {} is Killed.
• --mutate "**/Answers.cs{898..939}{954..1018}" (target + one earlier mutant): the same mutant is Survived, together with base.AddBusinessRules(); → ;.

Same mutant, opposite verdict, determined solely by what else is co-selected — a false survivor, exactly as described. Worth noting there's no timeout anywhere in this reproduction, so it's cleanly separate from the timeout-scoring symptom.

I think the concrete vector is nameable, since both contaminating mutations live in AddBusinessRules(). CSLA registers a type's business rules into a process-wide static store (BusinessRuleManager, keyed by type), populated once on first use of the type; later instances reuse it and AddBusinessRules() is not called again. Across mutants that share a test-host process, a mutant evaluated earlier can populate that cache, and every later mutant's mutation inside AddBusinessRules is then inert:

• target alone → fresh process, the mutated AddBusinessRules fills the cache with the rule stripped of PrimaryProperty; the missing validation is observed → Killed (correct).
• target after a neighbor → the cache already holds the un-mutated rule from the earlier evaluation, AddBusinessRules is skipped, the mutation never takes effect → Survived (false). The mirror case — an earlier mutant poisoning the cache for a later one — yields the false kill.

So this is the CSLA-specific face of the shared-state contamination, and it matches the observed order/selection dependence: the leaked state outlives a single mutant's evaluation. It reinforces the conclusion that a fresh test-host process per mutant is the reliable mitigation — --concurrency 1 only makes one contaminated order deterministic, it doesn't restore isolation. Any framework with process-lifetime static state (one-time rule/metadata caches, static singletons, lazy initializers) is exposed the same way.

[gem4511]

Follow-up — I ran the obvious in-process counter-test, and it pins the carrier down to that rule cache specifically.

If I clear CSLA's process-wide per-type rule cache before every test — Csla.Rules.BusinessRuleManager._perTypeRules (a static Lazy<ConcurrentDictionary<RuleSetKey, …>>, reached by reflection since it's private) — so the next BO instantiation re-runs AddBusinessRules under whatever mutant is currently active, the equivalent-canary false-kills disappear at default concurrency:

default concurrency (~22s/run)	run 1	run 2	run 3
baseline (unmodified)	23	12	15
clear rule cache per test	0	0	0

The reset is just an xUnit constructor (runs before each test), clearing that static dictionary:

public AnswersTests() => RuleCacheReset.Clear();   // empties BusinessRuleManager._perTypeRules

So the shared-process state that leaks between concurrently-evaluated mutants is, concretely, that per-type rule cache: once it is populated (by any earlier mutant, or the initial run), CSLA does not re-run AddBusinessRules for that type, so a later mutant's mutation inside the rule is evaluated against a neighbor's already-cached registration. Forcing fresh registration under the active mutant makes the misattribution vanish — and it holds at full concurrency, so this isn't a timing/serialization artifact.

To be clear this is a diagnostic + stopgap, not a fix: it reaches a private field by reflection (brittle across CSLA versions), and it only clears the one static I can name. Which is exactly the argument for per-mutant process isolation as the general answer — a fresh process resets every process-global static in the SUT and its dependencies with nobody having to enumerate them; a hand-cleared cache only covers the statics you already know about. But for anyone mutation-testing a CSLA codebase today, a per-test rule-cache clear appears to restore trustworthy verdicts at default concurrency in the meantime.

[gem4511]

One more data point — this time from a real production CSLA codebase, not the minimal repro. The per-test rule-cache clear holds up, and the effect on reproducibility is dramatic.

I wired the same reset (clear BusinessRuleManager._perTypeRules before every test) into a real app's engine/BO mutation suite — xunit.v3, mtp runner, additional-timeout: 30000 — and measured a scoped run (~390 mutants over one rule-heavy module) at default concurrency:

default concurrency	mutation score	Killed↔Survived flips between two identical runs	survivor-set churn
baseline (no reset), 2 runs	69.25% / 64.86%	97	68%
rule-cache clear per test, 3 runs	49.10% ×3	0	0%

With the reset, three back-to-back runs were byte-identical — same score, same 190 killed / 163 survived, same survivor set. Without it, ~97 mutants flipped verdict between two runs and the survivor set churned 68%. No timeouts in either case, so this isn't the timeout-scoring path — it's the cross-mutant rule-cache leak.

The score dropping (69→49%) is the point, not a regression: the baseline was inflated by equivalent mutants being false-killed; clearing the cache lets them correctly survive. You trade a noisy inflated number for an honest, reproducible one plus a stable survivor set you can actually triage.

Same caveats as the repro version — it's a reflection hack over a private field, and it only addresses the one static I can name (a full app run mutates plenty of non-rule code; other process-global statics in a framework could still leak). So it's a stopgap that happens to fully cover the rule-mediated case, not a substitute for per-mutant process isolation. But for a CSLA suite today it turns mtp mutation results from unusable-run-to-run into reproducible.

[dupdob]

@npnelson good point on the effect of removing concurrency: a sequential behavior is constant, hence would hide any varation.

@npnelson & @gem4511 : bear in mind that per test isolation will come with a performance penalty, marginal but I expect it to be significant (it actually depends on the total run time of the test suite). Note that it will also remove benefits for future improvements.

Overall, the root cause is a fragility in the test base; resetting cache before unit test is always important. Improving disable-mix-mutants will provide a workaround, not a fix, and one that is worse that clearing the cache.

[gem4511]

Thanks @dupdob — that settles it for us, and we landed on exactly that path: clear the process-global cache before each test. On a real CSLA codebase it fully restored reproducibility — same mtp runner, default concurrency:

• Before (no reset): a broad module of business objects churned ~92% of its survivor set between two identical runs — ~36 mutants flipping Killed↔Survived, zero timeouts.
• After (clear CSLA's per-type rule cache before each test): byte-identical runs — 0 flips, 0% churn — plus an honest de-inflation as the equivalents that were being false-killed now correctly survive.

So we're aligned that clearing the cache is the mitigation, not a runner change — and agreed on the isolation tradeoff: a fresh process per mutant would carry the perf cost you mention for a strictly worse result than resetting the one static we can name. Thanks to you and @npnelson for digging in.

[dupdob]

It looks like opening an issue/PR on CLSA (I think it's OSS) to have some official support for this is the next move.

[gem4511]

Already on it, @dupdob — opened a CSLA Discussion (Ideas) proposing exactly that: a supported way to reset the per-type rule cache for test isolation, MarimerLLC/csla#4875 . There's already an internal CleanupRulesForType, so it's a small surface to expose; happy to turn it into a PR if the maintainers are receptive. Thanks again for the steer throughout this.