Full testing report

[dcwhite]

List your testing hardware here, along with compilers/libraries used in the build:

@dcwhite

• Macbook Air M4 / macOS 15.7.4 / Apple clang 17.0.0 / Qt 6.10.2
• PC Windows 11 / Visual Studio 2019 / Qt 5.15

[dcwhite]

Round 1

Mac

viewscene-shortcuts branch (should be equivalent to master other than import test hiding)

99% tests passed, 1 tests failed out of 789

Total Test time (real) = 798.98 sec

The following tests did not run:
	394 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.MDSplus (Disabled)
	400 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Streamlines-Scalar-Interp (Disabled)
	404 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Cutting (Disabled)
	405 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Viz (Disabled)
	429 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.DistributedParticles (Disabled)
	432 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.SizingFieldMedialAxisAndBoundaries (Disabled)
	779 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-example (Disabled)
	780 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-gage (Disabled)
	781 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Samples.tooth1 (Disabled)

The following tests FAILED:
	279 - .Test.ExampleNetwork.exportMatlabMatrices_srn (Failed)

Rerun to test flakiness:

	➜  SCIRun git:(viewscene-shortcuts) ctest --rerun-failed
	Test project /Users/danwhite/dev/SCIRun/bin/SCIRun
	    Start 279: .Test.ExampleNetwork.exportMatlabMatrices_srn
	1/1 Test #279: .Test.ExampleNetwork.exportMatlabMatrices_srn ...   Passed    1.17 sec

	100% tests passed, 0 tests failed out of 1

	Total Test time (real) =   1.22 sec

refactor branch

Actually, it's the fix-python-test branch off of refactor. I'll get that ready as a mergeable thing by tomorrow. So on refactor proper, expect different results.

	98% tests passed, 12 tests failed out of 798

Total Test time (real) = 1353.02 sec

The following tests FAILED:
	310 - .Test.ExampleNetwork.generateProbeWithWidget_srn (SEGFAULT)
	394 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.MDSplus (Failed)
	400 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Streamlines-Scalar-Interp (Failed)
	404 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Cutting (Failed)
	405 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Viz (Failed)
	429 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.DistributedParticles (Failed)
	432 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.SizingFieldMedialAxisAndBoundaries (Failed)
	660 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.Modules.Visualization.ExtractIsosurface-typetriinterp (Timeout)
	758 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Modules.Unu.Unu3op-add (Timeout)
	779 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-example (Failed)
	780 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-gage (Failed)
	781 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Samples.tooth1 (Failed)

Rerun:

	25% tests passed, 9 tests failed out of 12

	Total Test time (real) =   8.09 sec

	The following tests FAILED:
		394 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.MDSplus (Failed)
		400 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Streamlines-Scalar-Interp (Failed)
		404 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Cutting (Failed)
		405 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Viz (Failed)
		429 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.DistributedParticles (Failed)
		432 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.SizingFieldMedialAxisAndBoundaries (Failed)
		779 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-example (Failed)
		780 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-gage (Failed)
		781 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Samples.tooth1 (Failed)

Good, only the skipped import tests.

[dcwhite]

Round 2, turned off import tests

Mac

viewscene-shortcuts branch

99% tests passed, 1 tests failed out of 364

Total Test time (real) = 455.36 sec

The following tests FAILED:
	299 - .Test.ExampleNetwork.additionalTypesOfVSObjectsSaved_srn (Subprocess aborted)

Rerun:

	➜  SCIRun git:(viewscene-shortcuts) ctest --rerun-failed
\Test project /Users/danwhite/dev/SCIRun/bin/SCIRun
    Start 299: .Test.ExampleNetwork.additionalTypesOfVSObjectsSaved_srn
1/1 Test #299: .Test.ExampleNetwork.additionalTypesOfVSObjectsSaved_srn ...   Passed    2.12 sec

100% tests passed, 0 tests failed out of 1

Did a second run a bit later to compare times:

 100% tests passed, 0 tests failed out of 364

Total Test time (real) = 432.92 sec

refactor/fix-python-test

100% tests passed, 0 tests failed out of 364

Total Test time (real) = 431.85 sec

That is the goal!

[dcwhite]

Windows

master

First time running this in a while, it's a big mess:

4% tests passed, 765 tests failed out of 798
1>
1>Total Test time (real) =  52.21 sec
1>
1>The following tests FAILED:
1>	  3 - Modules_Python_Tests (Exit code 0xc0000135
1>)
1>	  4 - Modules_BrainStimulator_Tests (Exit code 0xc0000135
1>)
1>	  5 - Modules_Teem_Converters_Tests (Exit code 0xc0000135
1>)
1>	  6 - Modules_Teem_Misc_Tests (Exit code 0xc0000135
1>)
1>	  7 - Modules_Teem_DataIO_Tests (Exit code 0xc0000135
1>)
1>	  8 - Modules_Legacy_Matlab_DataIO_Tests (Exit code 0xc0000135
1>)
1>	  9 - Modules_Bundle_Tests (Exit code 0xc0000135
1>)
1>	 11 - Modules_Visualization_Tests (Exit code 0xc0000135
1>)
1>	 14 - Modules_Math_Tests (Exit code 0xc0000135
1>)
1>	 15 - Modules_Fields_Tests (Exit code 0xc0000135
1>)
1>	 16 - Modules_Factory_Tests (Exit code 0xc0000135
1>)
1>	 17 - Modules_DataIO_Tests (Exit code 0xc0000135
1>)
1>	 19 - Core_Serialization_Network_Tests (Exit code 0xc0000135
1>)
1>	 21 - Engine_Python_Tests (Exit code 0xc0000135
1>)
1>	 22 - Engine_Scheduler_Tests (Exit code 0xc0000135
1>)
1>	 23 - Engine_Network_Tests (Exit code 0xc0000135
1>)
1>	 25 - Core_Python_Tests (Exit code 0xc0000135
1>)
1>	 37 - Core_Datatypes_Mesh_Tests (Failed)
1>	 45 - Core_Application_Tests (Exit code 0xc0000135
1>)
1>	 51 - Algorithms_Field_Tests (Failed)
1>	 54 - Interface_Modules_Render_Tests (Exit code 0xc0000135
1>)
1>	 55 - Modules_Dialog_Factory_Tests (Exit code 0xc0000135

Plus all the regression tests and all the import tests, since the executable doesn't run without the manual Qt dll copy step.
TODO:

• Add the qtdeploy step after all these years
• Claude will investigate all the failing unit tests one by one

[basisunus]

MacBook M1 Pro / Tahoe 26.3.1 (a) / clang version 17.0.0 / Qt 6.10.2

97% tests passed, 20 tests failed out of 798

Total Test time (real) = 3375.34 sec

The following tests FAILED:
21 - Engine_Python_Tests (Subprocess aborted)
38 - Core_Datatypes_Tests (Subprocess aborted)
49 - Algorithms_DataIO_Tests (Failed)
51 - Algorithms_Field_Tests (Failed)
199 - .Test.ExampleNetwork.moreThan10ProbesBug_srn (SEGFAULT)
279 - .Test.ExampleNetwork.exportMatlabMatrices_srn (Failed)
314 - .Test.ExampleNetwork.negative_tensors_srn (Subprocess aborted)
315 - .Test.ExampleNetwork.negative_tensors_normals_deubg_srn (Subprocess aborted)
347 - .Test.ExampleNetwork.tensorField_srn (Subprocess aborted)
348 - .Test.ExampleNetwork.tensorOrders_srn (Subprocess aborted)
394 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.MDSplus (Failed)
400 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Streamlines-Scalar-Interp (Failed)
404 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Cutting (Failed)
405 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Viz (Failed)
429 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.DistributedParticles (Failed)
432 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.SizingFieldMedialAxisAndBoundaries (Failed)
779 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-example (Failed)
780 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-gage (Failed)
781 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Samples.tooth1 (Failed)
788 - .Test.ImportNetwork.Other.v4nets.testing.empty (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /Users/basisunus/Documents/SCIRun/build/SCIRun/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[dcwhite]

Same with the regression tests, they might pass on a less-resource-constrained test run.

[dcwhite]

And lastly the import tests, those are expected to fail since they include unconverted modules. I put those in a skip list, also in PR #2513

[dcwhite]

One other note, the run took very long. Was anything else running on your system?

[basisunus]

Nothing else is running. I guess that's just the M1 speed? I started at about the same time on Windows. It's still building now. So no complaint on the MacBook's speed from me.

[dcwhite]

I'll whip up a spreadsheet so we can compare test running times. Would be good to know if there's a particular slowdown or if everything is slower. M1 can't be that much slower than M4!

[basisunus]

Intel Core i7-5930K computer / Windows 10 / Visual Studio 2022 / Qt 6.10.1

6% tests passed, 750 tests failed out of 799

Total Test time (real) = 253.07 sec

The following tests FAILED:
1 - Unit.testSCIRunAddModule.TestSCIRunAddModule.test_scirun_module_ids.test_scirun_module_ids (Not Run)
22 - Engine_Python_Tests (Failed)
26 - Core_Python_Tests (Failed)
38 - Core_Datatypes_Mesh_Tests (Failed)
39 - Core_Datatypes_Tests (Failed)
50 - Algorithms_DataIO_Tests (Failed)
52 - Algorithms_Field_Tests (Failed)
57 - .Test.ExampleNetwork.AddKnownstoLinearSystem_srn (Not Run)
58 - .Test.ExampleNetwork.AppendMatrixDynamicInputs_srn (Not Run)
59 - .Test.ExampleNetwork.BooleanMatrix_Example1_srn (Not Run)
... The next 700 or so didn't run.

[dcwhite]

As expected. We can chat about this tomorrow, since I can fix all of these, but I don't want to hold up the merge any longer.

[dcwhite]

I'll be able to test out Linux tomorrow morning.

[basisunus]

Mac Mini Intel Core i7 / Sequoia 15.7.7 / clang 17.0.0 / Qt 6.11.1

98% tests passed, 13 tests failed out of 798

Total Test time (real) = 1866.84 sec

The following tests FAILED:
21 - Engine_Python_Tests (Subprocess aborted)
51 - Algorithms_Field_Tests (Failed)
261 - .Test.ExampleNetwork.downstreamOnlyTest_srn (SEGFAULT)
279 - .Test.ExampleNetwork.exportMatlabMatrices_srn (Failed)
394 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.MDSplus (Failed)
400 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Streamlines-Scalar-Interp (Failed)
404 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Cutting (Failed)
405 - .Test.ImportNetwork.Other.v4nets.example_nets.FusionPSE.Volume-Viz (Failed)
429 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.DistributedParticles (Failed)
432 - .Test.ImportNetwork.Other.v4nets.example_nets.SCIRun.MeshingPipeline.SizingFieldMedialAxisAndBoundaries (Failed)
779 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-example (Failed)
780 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Optional.volume-colored-gage (Failed)
781 - .Test.ImportNetwork.Other.v4nets.example_nets.Teem.Samples.tooth1 (Failed)
Errors while running CTest
Output from these tests are in: /Users/fluorender/Documents/SCIRun/build/SCIRun/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Faster than the M1 MacBook Pro. Maybe there was something going on with the MacBook.