Merge CI testintegration failures - NFS

[pwalczysko]

Issues

After the move to NFS this month, we are experiencing new test failures, cause typically by 2 problems, see e.g. https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/16/testReport/

• pyramids are not being built fast enough (ca 80% of failing tests)
• mismatches of expected numbers of objects on OmeroWeb tests (ca 20% of failing tests)

Solutions - pyramids: polling

• Introduced newly: pyramids can be polled for in the test for "readiness" by polling in a loop for
  • for the presence of the file on disk, see e.g. https://github.com/ome/openmicroscopy/pull/6467/changes (the wait_for_pyramid_file method)
  • for the "renderableness" of the pyramid, see e.g. 2607cba)

Present polling/waiting methods - pyramids

• the present method wait_for_pyramid in omero-py is already used in many of the problematic tests and proven insufficient

Solutions - OmeroWeb tests

• only suggestions were made, no direct coding - I believe @will-moore is going to get into the problematics
  • do not assume a clean state of the DB when the test starts
  • watch out for race conditions, e.g. between the chgrp server-side process and the method in json API which hastily fetches the "new" state inside the test

Caveats of polling

• even the polling loop has to have a default timeout - usually I was setting 2 mins:
  • the fixed tests are running longer (but actually this is much less of an issue when watching average times of the whole suite)
  • The timeout comes into play, because even 2 mins are (often) not enough to build the pyramid, see eg. https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/18/testReport/OmeroPy.test.integration.test_thumbs/TestThumbs/testThumbnailVersion_one_/

Alternative solutions to polling - pyramids

• marking the tests which are problematic as "local only" (there is a precedens). I think this is a very attractive solution, as the pyramid code is deprecated ?
• making smaller images to make pyramids from
  • needs changes in omero-py as the methods live there
  • possibly also omero-server would need to be reconfigured for lower "large image" limit, with undesirable consequences for other tests
• splitting the tests
  • still not sure how this will help and mainly how this is to be achieved. Note that OmeroPy tests are enough per se to create the race conditions and they do not "need" the preceding OmeroJava tests to create the unclean states, races, timeouts
  • the pyramid tests could be rather marked as local-only, which would be more logical (see above)

Handy setup for running OmeroPy suite only via Jenkins job:

1. Run omero-push and omero-build jobs
2. Check that snoopycrimecop repo has on "merge-ci" branch the recent changes (merged open PRs from openmicroscopy repo)
3. Edit the Config of the OMERO-test-integration job as follows

...
echo Running the integration tests with -Dtestng.useDefaultListeners=true

# commented out by PW 21 July 2026 to spped up the investigation of the OmeroPy suite
# $SRC/build.py -f components/tools/OmeroJava/build.xml -Dtestng.useDefaultListeners=true -Dtestreports.dir=target/reports/integration integration
$SRC/build.py -f components/tools/OmeroPy/build.xml integration  -Dtestreports.dir=target/reports/integration
# $SRC/build.py -f components/tools/OmeroFS/build.xml integration -Dtestreports.dir=target/reports/integration
$SRC/build.py -f components/tools/OmeroWeb/build.xml integration -Dtestreports.dir=target/reports/integration

## Broken
#$SRC/build.py -f components/tools/OmeroPy/build.xml integration -DMARK=broken -Dtestreports.dir=target/reports/broken
#$SRC/build.py -f components/tools/OmeroJava/build.xml -Dtestng.useDefaultListeners=true -Dtestreports.dir=target/reports/broken broken

deactivate

4. Run the OMERO-test-integration job - this will take just 2.5hours instead of ~6h.

cc @jburel @sbesson

[sbesson]

Thanks for all the work but into resurrecting this critical infrastructure and investigating the failures in the integration tests.

For the pyramid failures, I would suggest a different approach than the pyramid polling proposed here.

The current strategy for testing pyramidal images for most tests is to import a single resolution image exceeding the 3Kx3K plane threshold, let the PixelData service generate an OMERO pyramid. As shown above, all tests based on binary access i.e. via the pixels or thumbnail service are blocked on the completion of this background server task which suffers from two shortcomings:

• the underlying infrastructure, notable in terms of storage, which can lead to race conditions
• the fact the PixelData service has been deprecated for more than four years, at least since OMERO 5.6.4

Would it be possible to:

• review the pyramidal integration tests and identify those that are specifically testing OMERO pyramids
• update all the other pyramidal tests to import multi-resolution synthetic images similarly to

  openmicroscopy/components/tools/OmeroWeb/test/integration/test_marshal.py

  Lines 142 to 144 in bf5e0bd

  	images = self.import_fake_file(
  	client=self.client, sizeX=20000, sizeY=20000, sizeC=3, rgb=3,
  	resolutions=5)

[pwalczysko]

Would it be possible to:

* review the pyramidal integration tests  and identify those that are specifically testing OMERO pyramids

* update all the other pyramidal tests to import multi-resolution synthetic images similarly to

I have started on it in #6471 - maybe is this the right direction @sbesson ? I still think it might not be exhaustive, as some tests will not be in that test_pyramids.py file and still fail, but as a direction is that ok ? Thanks

[sbesson]

As mentioned during a call with @pwalczysko and @jburel, a quick identification strategy would be to set omero config set omero.server.nodedescriptors master:Blitz-0,Indexer-0,Processor-0,Storm,Tables-0 in the configuration of the integration test job to disable the pyramid generation server-side and cause all related tests to fail.

[pwalczysko]

As mentioned during a call with @pwalczysko and @jburel, a quick identification strategy would be to set omero config set omero.server.nodedescriptors master:Blitz-0,Indexer-0,Processor-0,Storm,Tables-0 in the configuration of the integration test job to disable the pyramid generation server-side and cause all related tests to fail.

Thank you @sbesson , added this ^^^ config to the build which is due to run in few minutes from now, see what happens.
I am wondering though whether there are some DropBox tests which will complain too (as DropBox is not listed in your nodedescriptors list)

[pwalczysko]

I have added the DropBox services, meaning omero config set omero.server.nodedescriptors master:Blitz-0,Processor-0,Tables-0,Indexer-0,DropBox,MonitorServer,FileServer,Storm and am running the tests now. First result (only the OmeroPy suite on Jenkins) are showing 10 failing tests:

Edit:

I have now OmeroPy and OmeroWeb suites sampled with the Pixel-0 service off, I have 2 repeats, they solidly show the following 13 tests:

No.	Component	Test path	Result	Fix available
1	OmeroPy	test/integration/clitest/test_pyramids.py::TestRemovePyramidsFullAdmin::test_remove_pyramids_little_endian	❌ FAILED	No
2	OmeroPy	test/integration/clitest/test_pyramids.py::TestRemovePyramidsFullAdmin::test_remove__pre_fs_pyramids	❌ FAILED	No
3	OmeroPy	test/integration/clitest/test_pyramids.py::TestRemovePyramidsFullAdmin::test_remove_pyramids_big_endian	❌ FAILED	No
4	OmeroPy	test/integration/clitest/test_pyramids.py::TestRemovePyramidsFullAdmin::test_remove_pyramids	❌ FAILED	No
5	OmeroPy	test/integration/clitest/test_pyramids.py::TestRemovePyramidsFullAdmin::test_remove_pyramids_check_thumbnails	❌ FAILED	Yes
6	OmeroPy	test/integration/test_rawpixelsstore.py::TestRPS::testRomioToPyramid	❌ FAILED	No
7	OmeroPy	test/integration/test_rawpixelsstore.py::TestRPS::test2RomioToPyramidWithNegOne	❌ FAILED	No
8	OmeroPy	test/integration/test_rawpixelsstore.py::TestRPS::testPyramidConcurrentAccess	❌ FAILED	No
9	OmeroPy	test/integration/test_thumbs.py::TestThumbs::testThumbnailVersion[one]	❌ FAILED	No
10	OmeroPy	test/integration/test_thumbs.py::TestThumbs::testThumbnailVersion[set]	❌ FAILED	No
11	OmeroWeb	test/integration/test_rendering.py::TestRenderImageRegion::test_render_image_region_tile_params_big_image	❌ FAILED	No
12	OmeroWeb	test/integration/test_rendering.py::TestRenderImageRegion::test_render_image_region_region_params_big_image	❌ FAILED	No
13	OmeroWeb	test/integration/test_rendering.py::TestRenderImageRegion::test_render_image_region_big_image_resolution	❌ FAILED	No

[sbesson]

Agreed, the critical issue is not about the usage of the fake image but the fact is the images imported in the current failing tests are single-resolution and thus rely on the legacy pyramid generation infrastructure - https://github.com/ome/omero-py/blob/91d9fe26eb1887700c71c30f62c75bd6d359e350/src/omero/testlib/__init__.py#L673-L688.

[pwalczysko]

As for the OmeroJava test suite, with Pixel-0 service off, we have either 74 or 75 failures. I think that these failures are rather at the "preparation of the test framework" then the test failing per se.

See run 30 and run 31.

As the Test NG cited above shows, these are mainly from the integration.thumbnail package. The test which failed only once out of the two runs (in run 30) is the only test NOT from integration.thumbnail but from integration.
testIndexObjectbyRestrictedSystemUser, see stacktrace

[sbesson]

@pwalczysko for the OMERO.java thumbnail integration tests, I believe the place that needs to be adjusted is

openmicroscopy/components/tools/OmeroJava/test/integration/thumbnail/SkipThumbnailsPermissionsTest.java

Line 691 in 81c671c

File f = File.createTempFile("bigImageFake&sizeX=3500&sizeY=3500&little=false", ".fake");

to set a resolutions key/value pair