Test project recognised multiple times

[richardwerkman]

Describe the bug
Test projects are duplicated. The project is built multiple times and the tests are also recognized and executed multiple times, slowing stryker down a lot.

Before:

After:

Reproduction:

This was found when running against the Polly.Specs project with MTP testrunner. But will probably happen on other projects too (not verified yet)

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: MacOS
• Type of project: Core
• Framework Version Core 6.0
• Stryker Version: current master (not released yet)

Additional context
This seems to have been introduced in #3751

[richardwerkman]

@dupdob could you take a look at this?

[dupdob]

I am on it. I got a spotty connection which does not help, but I reproduce.

[dupdob]

This is linked to multi targeting, this project probably reveals an error in resolution logic

[dupdob]

It happens because Polly.Specs targets net 8, net 9 and net 10 and all of them refer to Polly net6.0
Just to clarify, this behavior makes sense: if the test project targets multiple framework for the same version there may be good reason for it, there may be behavior differences between these frameworks hence it could make sense to run all versions.

That being said, users would be surprised if it was the default behavior.

Anyway, this means I need to define when, and how to perform this filtering

[richardwerkman]

What was our previous behavior? I suppose we took the highest framework version and used that?

[dupdob]

the previous behavior was more of a hack than that: it was pick the first net framework or netcore available, depending on which one we wanted. There is no 'highest' framework logic anywhere. The design problem to solve is that Stryker now works at the specific target (framework) level, not the project one, ensuring coherency regarding framework versions. The designed assumed a one to one match, or one to zero, never one to many. So either I filter early, keeping the design as is, but forcing me to apply some default filtering logic that may be suboptimal, or I keep the one to many links and filter them at a later stage, ensuring better decision logic but incurring a bit more work in terms of code to produce. I just need some time to figure which one is better.

[dupdob]

Humm, as an experiment, I just added a filter at the build stage to make sure test projects are built once.
Stryker fails afterward without a message. I suspect this is linked to MTP:

[11:57:43 INF] Building test project /Users/dupdob/Downloads/Polly-main/test/Polly.Specs/Polly.Specs.csproj (1/1)
[11:57:44 INF] Building project Polly.Specs.csproj using dotnet build Polly.Specs.csproj -c Debug --property:Platform=AnyCPU (directory /Users/dupdob/Downloads/Polly-main/test/Polly.Specs.)
[11:59:50 INF] Number of tests found: 2115 for project /Users/dupdob/Downloads/Polly-main/src/Polly/Polly.csproj. Initial test run started.
[12:01:22 INF] Time Elapsed 00:04:00.3851194
Stryker.NET failed to mutate your project. For more information see the logs below:


No test result reported. Make sure your test project contains test and is compatible with VsTest.

At the very least, this error message needs to be fixed :-)

[richardwerkman]

Are you running against polly main branch? That runs vstest not mpt. So that might explain the no test results issue if you are running with the mtp testrunner

[dupdob]

I assumed Polly was using MTP and that I needed to specify it. I am debugging it. This is new to me: the test session seems to run (significant run time), but does not return any events....
debugging it.

Update: well, I enabled logging. Test session appears to be running fine. Absolutely no idea why test runner appears to receive nothing. My guess is on some accidental regression

ReUpdate: debugging showed that the test session ran fine, returned results to Stryker, but Stryker failed to process it. git history show little change in related code for the past two years.
I will dig into this later, I have to leave for now

[dupdob]

Update: the initial test problem is linked to having duplicate test projects. For polly, we have Polly.Specs for net8, net9 and net10 detected as test projects, hence the test runner expect three results per test, which fails for some reason.

As properly dealing with this situation require some redesign work and some thinking, I pushed a quick workaround that simply discards duplicate (see #3785)

[dupdob]

This is an intermediate fix in order unblock Stryker's release. A proper handling for this requires:

• Detecting and tracking all quasi duplicates
• Select the best version to keep, staying consistent with net framework selection behavior

It would be great if we could alose:

• Accepts duplicate (enabled via explicit option)
• Handle said duplicate properly when running tests (my personal tests showed issues were present)