Fix: curl manifest futilely includes external LungMAP files (#8027)#8030
Fix: curl manifest futilely includes external LungMAP files (#8027)#8030nadove-ucsc wants to merge 9 commits into
Conversation
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
from
a70d2de to
9e2cb1f
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
3 times, most recently
from
3194880 to
e012e86
Compare
|
coverage: 84.932% (+0.001%) from 84.931% — issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files into develop |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
2 times, most recently
from
4648e95 to
ace1dd9
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8030 +/- ##
===========================================
+ Coverage 84.85% 84.87% +0.01%
===========================================
Files 165 165
Lines 24177 24206 +29
===========================================
+ Hits 20515 20544 +29
Misses 3662 3662 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
|
||
|
|
||
| @attr.s(auto_attribs=True, kw_only=True, frozen=True) | ||
| @attrs.frozen(auto_attribs=True, frozen=True) |
There was a problem hiding this comment.
It seems frozen is redundant and kw_only was dropped.
| @attrs.frozen(auto_attribs=True, frozen=True) | |
| @attrs.frozen(auto_attribs=True, kw_only=True) |
| config.catalogs[catalog].atlas == 'lungmap' | ||
| and drs_uri.startswith('drs://dg.4503:') | ||
| ): | ||
| # Lungmap contains external files hosted on BioDataCatalyst. |
There was a problem hiding this comment.
| # Lungmap contains external files hosted on BioDataCatalyst. | |
| # LungMAP contains external files hosted on BioDataCatalyst. |
| @cache | ||
| def mirror_service(self, catalog: CatalogName) -> MirrorService: | ||
| return MirrorService(catalog=catalog) | ||
| @attrs.frozen(kw_only=True) |
There was a problem hiding this comment.
Like the sibling class definitions in query_service.py?
| @attrs.frozen(kw_only=True) | |
| @attrs.frozen(auto_attribs=True, kw_only=True) |
Confirmed. The affected fields in the response are synthesized by the service and are not present in the index. |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
2 times, most recently
from
4fb3f6f to
275c9bc
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
Please remove the FileUrlService refactoring and only fix the issue at hand, plus the test. Leave the attrs conversion, but make sure it is done correctly (see comment below).
Please clear these types of refactorings with me before embarking on them. Feel free to create an issue for the refactoring and motivate the refactoring in the description of that issue.
| @cache | ||
| def mirror_service(self, catalog: CatalogName) -> MirrorService: | ||
| return MirrorService(catalog=catalog) | ||
| @attrs.frozen(auto_attribs=True, kw_only=True) |
There was a problem hiding this comment.
| @attrs.frozen(auto_attribs=True, kw_only=True) | |
| @attrs.frozen(kw_only=True) |
The new API infers it automatically.
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
from
40836f1 to
71ca313
Compare
hannes-ucsc
added
3 reviews
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
from
bd29c06 to
26cfd2d
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
The new test is a bit lazy. It indexes two bundles, even though only one file in one bundle is patched. The same bundles were indexed already in a different test. The need for extracting a superclass stems from the fact that these test cases index their bundles during class setup, so the patching needs to be done at class setup, and therefore a new class is needed.
We'd get more bang for the buck if actually canned two LungMAP bundles, one from a project with compact DRS UIRs, and one from a project with host-based DRS URIS. We wouldn't extend the test duration but we would get more meaningful coverage.
Also, I'd like to the main commit to be split and pushed individually so that I can see the test fail.
| } | ||
|
|
||
| external_file_uuid = '27fc1a2e-d70e-47ee-a4b7-92bf57e5b7a6' | ||
| # Compact identifier for BioDataCatalyst |
There was a problem hiding this comment.
| # Compact identifier for BioDataCatalyst |
There was a problem hiding this comment.
The need for extracting a superclass stems from the fact that these test cases...
The need for extracting a superclass stems from the fact that the new test uses a different catalog configuration due to requiring the Config.Catalog.atlas property to be set to 'lungmap' instead of 'hca', regardless of how/which bundles are indexed.
There was a problem hiding this comment.
This bundle naturally contains a mixture of host-based and compact DRS URIs, so I just canned that one.
There was a problem hiding this comment.
The need for extracting a superclass stems from the fact that these test cases...
The need for extracting a superclass stems from the fact that the new test uses a different catalog configuration due to requiring the
Config.Catalog.atlasproperty to be set to 'lungmap' instead of 'hca', regardless of how/which bundles are indexed.
I don't see how you would be able to avoid the superclass and new subclass, even if you were able to patch the catalog per test method. I think both reasons are valid and I didn't claim that the reason I gave was exclusive.
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
2 times, most recently
from
f9c1dee to
0699599
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
The bundle you canned is pretty large. Make sure you can the smallest possible bundle. I don't mind indexing two bundles if you can't find a small bundle that has both types of DRS URIs. Document your process of finding the bundles to can.
Index: test/service/test_response.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/test/service/test_response.py b/test/service/test_response.py
--- a/test/service/test_response.py (revision 5fd60826db7275ef02493d217ca1fdc2b95287ae)
+++ b/test/service/test_response.py (date 1780014578969)
@@ -3772,7 +3772,7 @@
return one(response.json()['files'])
-class TestResponseWithDCP2Cans(DCP2ResponseTestCase):
+class TestResponseWithHCADCP2Cans(DCP2ResponseTestCase):
@classmethod
def bundles(cls) -> list[SourcedBundleFQID]:
@@ -3864,7 +3864,7 @@
self.assertEqual(expected_tree, project['contributedAnalyses'])
-class TestExternalLungmapFiles(LungmapTestCase, DCP2ResponseTestCase):
+class TestResponseWithLungmapCans(LungmapTestCase, DCP2ResponseTestCase):
@classmethod
def bundles(cls) -> list[SourcedBundleFQID]:
@@ -3873,20 +3873,19 @@
version='2024-06-25T15:12:06.304736Z'),
]
- def test_file_url(self):
- with self.subTest('host-based DRS URI'):
- file = self.get_file('005f71ed-fcba-4026-bc97-a8c9707f26ee')
- self.assertEqual('drs://jade.datarepo-dev.broadinstitute.org/v2_85c93a64-a9d0-4331-ad36-53a6498e57fc',
- file['drs_uri'])
- expected_file_url = str(self.base_url.set(
- path='/repository/files/005f71ed-fcba-4026-bc97-a8c9707f26ee',
- args=dict(catalog=self.catalog,
- version='2023-10-31T16:42:42.419707Z')
- ))
- self.assertEqual(expected_file_url, file['azul_url'])
+ def test_file_with_host_based_drs_uri(self):
+ file = self.get_file('005f71ed-fcba-4026-bc97-a8c9707f26ee')
+ self.assertEqual('drs://jade.datarepo-dev.broadinstitute.org/v2_85c93a64-a9d0-4331-ad36-53a6498e57fc',
+ file['drs_uri'])
+ expected_file_url = str(self.base_url.set(
+ path='/repository/files/005f71ed-fcba-4026-bc97-a8c9707f26ee',
+ args=dict(catalog=self.catalog,
+ version='2023-10-31T16:42:42.419707Z')
+ ))
+ self.assertEqual(expected_file_url, file['azul_url'])
- with self.subTest('compact DRS URI'):
- file = self.get_file('002ebcd6-722d-434d-b21b-13a06e659a67')
- self.assertEqual('drs://dg.4503:696f302b-ecd9-4bcc-a750-56c22b496f02',
- file['drs_uri'])
- self.assertIsNone(file['azul_url'])
+ def test_file_with_compact_drs_uri(self):
+ file = self.get_file('002ebcd6-722d-434d-b21b-13a06e659a67')
+ self.assertEqual('drs://dg.4503:696f302b-ecd9-4bcc-a750-56c22b496f02',
+ file['drs_uri'])
+ self.assertIsNone(file['azul_url'])| config.catalogs[self.catalog].atlas == 'lungmap' | ||
| and isinstance(DRSURI.parse(drs_uri), CompactDRSURI) | ||
| ): | ||
| # LungMAP contains external files hosted on BioDataCatalyst. |
There was a problem hiding this comment.
| # LungMAP contains external files hosted on BioDataCatalyst. | |
| # LungMAP contains files not hosted on TDR. |
… and remove other mentions of BDC you may have added.
| } | ||
|
|
||
| external_file_uuid = '27fc1a2e-d70e-47ee-a4b7-92bf57e5b7a6' | ||
| # Compact identifier for BioDataCatalyst |
There was a problem hiding this comment.
The need for extracting a superclass stems from the fact that these test cases...
The need for extracting a superclass stems from the fact that the new test uses a different catalog configuration due to requiring the
Config.Catalog.atlasproperty to be set to 'lungmap' instead of 'hca', regardless of how/which bundles are indexed.
I don't see how you would be able to avoid the superclass and new subclass, even if you were able to patch the catalog per test method. I think both reasons are valid and I didn't claim that the reason I gave was exclusive.
There is only one bundle on I lack the permissions needed to can bundles from I have manually removed most of the files from the canned bundle to reduce it to a more reasonable size. |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/8027-curl-manifest-includes-external-lungmap-files
branch
from
5fd6082 to
be1ff25
Compare
hannes-ucsc
left a comment
hannes-ucsc
left a comment
There was a problem hiding this comment.
Please can production bundles for the test. You observed that you lack the permissions to do so. This can be remedied easily. You should have permissions to read the metadata in BigQuery. If not, please ping me on Slack. Once you identified suitable bundle(s), ask the operator for help to actually can them. Don't forget to document your process/reasoning of identifying suitable bundles.
5 participants
Linked issues: #8027
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all linked issuespartiallabelAuthor (reindex)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAuthor (mirror)
mirror:devor the changes introduced by it will not require mirroring ofdevmirror:anvildevor the changes introduced by it will not require mirroring ofanvildevmirror:anvilprodor the changes introduced by it will not require mirroring ofanvilprodmirror:prodor the changes introduced by it will not require mirroring ofprodmirror:partialand its description documents the specific mirroring procedure fordev,anvildev,anvilprodandprodor requires a full mirroring or carries none of the labelsmirror:dev,mirror:anvildev,mirror:anvilprodandmirror:prodAuthor (API changes)
APIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues linked to this PRAuthor (before every review)
develop, squashed fixups from prior reviewsmake requirements_updateor this PR does not modifyDockerfile,environment,requirements*.txt,common.mk,Makefileorenvironment.bootRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
Note that after requesting changes, the PR must be assigned to only the author.
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator
reindex:…labels andrcommit title tagmirror:…labelsno demodevelopOperator (deploy
.sharedand.gitlabcomponents)_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator (post-deploy of
.gitlabcomponent)dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (deploy runner image)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerOperator (sandbox build)
sandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices insandboxanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvilboxsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledmirror:devanvilboxor this PR is not labeledmirror:anvildevsandboxor this PR is not labeledmirror:devanvilboxor this PR is not labeledmirror:anvildevOperator (merge the branch)
pif the PR is also labeledpartialOperator (main build)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator (mirroring)
devor this PR is not labelledmirror:devanvildevor this PR is not labelledmirror:anvildevdevor this PR is not labelledmirror:devanvildevor this PR is not labelledmirror:anvildevdevor this PR is not labelledmirror:devanvildevor this PR is not labelledmirror:anvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprod,reindex:prod,mirror:partial,mirror:anvilprodandmirror:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprod,reindex:prod,mirror:partial,mirror:anvilprodandmirror:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem