feat(search): add skipCount parameter to skip expensive COUNT queries#8325
Merged
Conversation
…ueries
Add an optional `skipCount` boolean query parameter to 6 v3 search
endpoints. When set to true, the storage layer skips the COUNT(*)
query and returns count=0 in the response. This can significantly
improve performance for large datasets when callers don't need total
pagination count.
Affected endpoints:
- GET/POST /search/artifacts
- GET/POST /search/versions
- GET /groups/{groupId}/artifacts
- GET /groups/{groupId}/artifacts/{artifactId}/versions
Fixes #4723
github-actions
Bot
added
lifecycle/ready-for-review
|
PR auto-accepted (trusted author). Full test suite will run. A maintainer can use |
|
The test suite was cancelled for commit 8fdb5e1. See the workflow run. Use |
Regenerate Go SDK to include the new skipCount query parameter added to search endpoints. Relates to #4723
github-actions
Bot
added
the
lifecycle/tested
|
Verify — ✅ passed (run)
Change detectionjava: |
github-actions
Bot
added
lifecycle/tested
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
skipCountboolean query parameter to v3 search endpoints, allowing callers to skip the expensiveCOUNT(*)query when they don't need total pagination count. WhenskipCount=true, the response returnscount=0instead of executing the count query. Fixes #4723Root Cause
The
COUNT(*)query insearchArtifacts()andsearchVersions()accounts for ~10% of all queries but takes ~95% of database time on large datasets. Callers who only need search results (not the total count for pagination) pay this cost unnecessarily with no way to opt out.Changes
common/src/main/resources/META-INF/openapi.json): AddedskipCountboolean query parameter (defaultfalse) to 6 endpoints:searchArtifacts,searchArtifactsByContent,searchVersions,searchVersionsByContent,listArtifactsInGroup,listArtifactVersionsRegistryStorage.java): Addedboolean skipCountparameter tosearchArtifacts()andsearchVersions()SqlSearchRepository.java): Conditionally skips COUNT query construction and execution whenskipCount=trueElasticsearchSearchService.java): DisablestrackTotalHitswhenskipCount=trueReadOnlyDelegatingStorage,ElasticsearchSearchDecorator,AbstractPollingRegistryStorage,AbstractSqlRegistryStorageto thread the parameter throughSearchResourceImpl.java,GroupsResourceImpl.java): Updated method signatures to accept and pass the new parameterfalse(preserving existing behavior) inSearchResourceImpl(v2),GroupsResourceImpl(v2),WellKnownResourceImpl,IcebergApiResourceImpl,SubjectsResourceImpl,SchemasResourceImplAbstractRegistryStorageTest,RegistryStoragePerformanceTest,ReadOnlyRegistryStorageTest,TestInMemoryRegistryStorageTest plan
./mvnw clean install -DskipTests -pl common,app -am)./mvnw checkstyle:check -pl app)skipCount=falseskipCount=trueverifyingcount=0in responseskipCount=falseverifying correct count in responseGET /search/artifacts?skipCount=truereturns results withcount=0