Add deacon and deacon DM #7473
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| categories: | ||
| - Data Managers | ||
| - Metagenomics | ||
| homepage_url: https://github.com/bede/deacon | ||
| description: Data manager for Deacon index files | ||
| long_description: Data manager for Deacon index files | ||
| name: deacon_build_database | ||
| owner: iuc | ||
| remote_repository_url: https://github.com/galaxyproject/tools-iuc/tree/main/data_managers/data_manager_deacon | ||
| type: unrestricted |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| <tool id="deacon_build_database" name="Deacon" tool_type="manage_data" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="@PROFILE@"> | ||
| <description>database builder</description> | ||
| <macros> | ||
| <!-- on update run a local test setting `test` to something else than "true" --> | ||
| <token name="@TOOL_VERSION@">0.12.0</token> | ||
| <token name="@VERSION_SUFFIX@">0</token> | ||
| <token name="@PROFILE@">26.0</token> | ||
| </macros> | ||
| <requirements> | ||
| <requirement type="package" version="@TOOL_VERSION@">deacon</requirement> | ||
| </requirements> | ||
| <command detect_errors="exit_code"><![CDATA[ | ||
| mkdir -p '$out_file.extra_files_path' && | ||
| #if $test != "true" | ||
| #if $input.is_select == "prebuild" | ||
| #if $download == "human" | ||
| wget -P '$out_file.extra_files_path' 'https://zenodo.org/records/17288185/files/panhuman-1.k31w15.idx' && | ||
| #else | ||
| wget -P '$out_file.extra_files_path' 'https://objectstorage.uk-london-1.oraclecloud.com/n/lrbvkel2wjot/b/human-genome-bucket/o/deacon/3/panmouse-1.k31w15.idx' && | ||
| #end if | ||
| #else if $input.is_select == "url" | ||
| wget -P '$out_file.extra_files_path' '$link' && | ||
| #else | ||
| wget '$link' && | ||
| deacon index build '${link.split('/')[-1]}' -o '${file_name}.idx' && | ||
| mv '${file_name}.idx' '$out_file.extra_files_path' && | ||
| #end if | ||
| #else | ||
| touch '$out_file.extra_files_path'/test.idx && | ||
| #end if | ||
| cp '$dmjson' '$out_file' | ||
| ]]></command> | ||
| <configfiles> | ||
| <configfile name="dmjson"><![CDATA[ | ||
| #from datetime import datetime | ||
| #set time=datetime.now().strftime("%Y-%m-%d") | ||
| { | ||
| "data_tables":{ | ||
| "deacon":[ | ||
| { | ||
| #if $input.is_select == "prebuild" | ||
| #if $download == "human" | ||
| "path":"panhuman-1.k31w15.idx", | ||
| #else | ||
| "path":"panmouse-1.k31w15.idx", | ||
| #end if | ||
| #else if $input.is_select == "url" | ||
| "path":"${link.split('/')[-1]}", | ||
| #else | ||
| "path":"${file_name}.idx", | ||
| #end if | ||
| "dbkey":"", | ||
| #if $input.is_select == "prebuild" | ||
| #if $download == "human" | ||
| "name":"panhuman-1 (k=31, w=15)", | ||
| #else | ||
| "name":"panmouse-1 (k=31, w=15, e=0.5)", | ||
| #end if | ||
| #else | ||
| "name":"$name", | ||
| #end if | ||
| #if $input.is_select == "prebuild" | ||
| "version":"@TOOL_VERSION@", | ||
| #else | ||
| "version":"$version", | ||
| #end if | ||
| #if $input.is_select == "prebuild" | ||
| "value":"prebuilt-${time}", | ||
| #else | ||
| "value":"custom-${time}", | ||
| #end if | ||
| #if $input.is_select == "prebuild" | ||
| "format_version":"3", | ||
| #else | ||
| "format_version":"$format_version", | ||
| #end if | ||
| #if $input.is_select == "prebuild" | ||
| "note":"Prebuilt index files from the devs of deacon" | ||
|
Contributor
There was a problem hiding this comment. Replace |
||
| #else | ||
| "note":"$note" | ||
| #end if | ||
| } | ||
| ] | ||
| } | ||
| }]]> | ||
| </configfile> | ||
| </configfiles> | ||
| <inputs> | ||
| <conditional name="input"> | ||
| <param name="is_select" type="select" label="Choose how to add data to the DM"> | ||
| <option value="url">Copy an index file from an URL</option> | ||
| <option value="prebuild" selected="true">Download a prebuilt file</option> | ||
| <option value="fasta">Download a fasta file and build index file from it</option> | ||
| </param> | ||
| <when value="prebuild"> | ||
| <param name="download" type="select" label="Select which prebuilt should be downloaded" help="See help section for more information"> | ||
| <option value="human">panhuman-1 (k=31, w=15)</option> | ||
| <option value="mouse">panmouse-1 (k=31, w=15, e=0.5)</option> | ||
| </param> | ||
| </when> | ||
| <when value="url"> | ||
|
Contributor
There was a problem hiding this comment. Do we need the url option?
Contributor
There was a problem hiding this comment.
Is there so much pre-built data out there anyway? If so, we should also store the origin URL in the loc file? If the number of available pre-built files is small, I would prefer to have them hardcoded in the DM.
Contributor
Author
There was a problem hiding this comment. Im not sure i just add if there might be a lot of usage later on. Currently only the 2 hardcoded pre-builds are out so this was also the reason i did hardcode them. I can add the URL/Zenodo record ID in the loc file/data table if wanted |
||
| <param name="link" type="text" label="Input the URL to download a index file"/> | ||
| <param name="name" type="text" label="Set the name for the entry in the DM" help="For an example look in the name of the prebuilt files. Also add in brackets the values used for building the file!"/> | ||
| <param name="version" type="text" label="Which version was used to build the file to copy" help="State the tool version used to build the index file"/> | ||
| <param name="format_version" type="text" label="Set the index format version" help="The current index format version for Deacon v.@TOOL_VERSION@ is 3"/> | ||
| <param name="note" type="text" label="Add a note" help="Here some notes can be set for example from where the data comes, who created the data and so on"/> | ||
| </when> | ||
| <when value="fasta"> | ||
| <param name="file_name" type="text" label="Enter filename" help="Please enter the name of the file which will be downloaded without the extension" /> | ||
|
Contributor
There was a problem hiding this comment. We should maybe consider the fasta data table here as input as well.
Contributor
Author
There was a problem hiding this comment. So just to check if im right this should work with the current fasta data table or a new data table?
Contributor
There was a problem hiding this comment. The all_fasta data table |
||
| <param name="link" type="text" label="Input the URL to download a index file"/> | ||
| <param name="name" type="text" label="Set the name for the entry in the DM" help="For an example look in the name of the prebuilt files. Also add in brackets the values used for building the file!"/> | ||
| <param name="version" type="text" label="Which version was used to build the file to copy" help="State the tool version used to build the index file"/> | ||
| <param name="format_version" type="text" label="Set the index format version" help="The current index format version for Deacon v.@TOOL_VERSION@ is 3"/> | ||
|
Contributor
There was a problem hiding this comment. The DM should set this information and not the DM user.
Contributor
There was a problem hiding this comment. This referred to the fasta input case. Your change seems to be regarding url input (which should be removed?) |
||
| <param name="note" type="text" label="Add a note" help="Here some notes can be set for example from where the data comes, who created the data and so on"/> | ||
| </when> | ||
| </conditional> | ||
| <param name="test" type="hidden"/> | ||
| </inputs> | ||
| <outputs> | ||
| <data name="out_file" format="data_manager_json" /> | ||
| </outputs> | ||
| <tests> | ||
| <test expect_num_outputs="1"> | ||
| <conditional name="input"> | ||
| <param name="is_select" value="prebuild"/> | ||
| <param name="download" value="human"/> | ||
| </conditional> | ||
| <param name="test" value="true"/> | ||
| <output name="out_file"> | ||
| <assert_contents> | ||
| <has_text text='"format_version":"3"'/> | ||
| <has_text text='"name":"panhuman-1 (k=31, w=15)"'/> | ||
| </assert_contents> | ||
| </output> | ||
| </test> | ||
| <test expect_num_outputs="1"> | ||
| <conditional name="input"> | ||
| <param name="is_select" value="url"/> | ||
| <param name="link" value="https://zenodo.org/records/17288185/files/panhuman-1.k31w15.idx"/> | ||
| <param name="name" value="panhuman-1 (k=31, w=15)"/> | ||
| <param name="version" value="0.12.0"/> | ||
| <param name="format_version" value="3"/> | ||
| <param name="note" value="test"/> | ||
| </conditional> | ||
| <param name="test" value="true"/> | ||
| <output name="out_file"> | ||
| <assert_contents> | ||
| <has_text text='"format_version":"3"'/> | ||
| <has_text text='"name":"panhuman-1 (k=31, w=15)"'/> | ||
| </assert_contents> | ||
| </output> | ||
| </test> | ||
| <test expect_num_outputs="1"> | ||
| <conditional name="input"> | ||
| <param name="is_select" value="fasta"/> | ||
| <param name="file_name" value="panhuman-1.k31w15_test"/> | ||
| <param name="link" value="https://zenodo.org/records/17288185/files/panhuman-1.k31w15.fasta"/> | ||
| <param name="name" value="panhuman-1 (k=31, w=15)"/> | ||
| <param name="version" value="0.12.0"/> | ||
| <param name="format_version" value="3"/> | ||
| <param name="note" value="test"/> | ||
| </conditional> | ||
| <param name="test" value="true"/> | ||
| <output name="out_file"> | ||
| <assert_contents> | ||
| <has_text text='"format_version":"3"'/> | ||
| <has_text text='"name":"panhuman-1 (k=31, w=15)"'/> | ||
| </assert_contents> | ||
| </output> | ||
| </test> | ||
| </tests> | ||
| <help><![CDATA[ | ||
| Download prebuilt index files for deacon or download other index files made for deacon via url. | ||
| ]]></help> | ||
| <citations> | ||
| <citation type="doi">10.1101/2025.06.09.658732</citation> | ||
| </citations> | ||
| </tool> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| <data_managers> | ||
| <data_manager tool_file="data_manager/deacon_datamanager.xml" id="deacon_build_database"> | ||
| <data_table name="deacon"> | ||
| <output> | ||
| <column name="value"/> | ||
| <column name="dbkey"/> | ||
| <column name="name"/> | ||
| <column name="version"/> | ||
| <column name="path" output_ref="out_file"> | ||
| <move type="file"> | ||
| <source>${path}</source> | ||
| <target base="${GALAXY_DATA_MANAGER_DATA_PATH}">deacon/${value}/${path}</target> | ||
| </move> | ||
| <value_translation>${GALAXY_DATA_MANAGER_DATA_PATH}/deacon/${value}/${path}</value_translation> | ||
| <value_translation type="function">abspath</value_translation> | ||
| </column> | ||
| <column name="format_version"/> | ||
| <column name="note"/> | ||
| </output> | ||
| </data_table> | ||
| </data_manager> | ||
| </data_managers> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
|
|
||
| db_download_xxxx-xx-xx deacon human index db 0.12.0 /tmp/tmpf_hplx2a/galaxy-dev/tool-data/deacon/0.12.0/test.idx 3 Testing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| #This is a sample file distributed with Galaxy that enables tools | ||
| #to use a the deacon dabase. | ||
| # | ||
| #<unique_build_id> <dbkey> <display_name> <version> <file_base_path> <index_format_version> <note_like_who_did_create_the_db> | ||
|
|
||
| #The <version> column indicates the deacon version that generated the database | ||
|
|
||
| # | ||
| #deacon_db Deacon_database 0.12.0 /mnt/galaxyIndices/deacon_database/test.idx 3 just for the test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| <tables> | ||
| <table name="deacon" comment_char="#" allow_duplicate_entries="False"> | ||
| <columns>value, dbkey, name, version, path, format_version, note</columns> | ||
| <file path="tool-data/deacon.loc" /> | ||
| </table> | ||
| </tables> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| <tables> | ||
| <!-- Location of deacon indexes for testing --> | ||
| <table name="deacon" comment_char="#" allow_duplicate_entries="False"> | ||
| <columns>value, dbkey, name, version, path, format_version, note</columns> | ||
| <file path="${__HERE__}/test-data/deacon.loc" /> | ||
| </table> | ||
| </tables> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| name: deacon | ||
| owner: iuc | ||
| description: filters DNA sequences in FASTA/Q files and streams using accelerated minimizer comparison | ||
| homepage_url: https://github.com/bede/deacon | ||
| long_description: | | ||
| Filter sequences using accelerated minimizer comparison with query sequence(s), | ||
| emitting either matching sequences (search mode), or sequences without matches (deplete mode). | ||
| Sequences match when they share enough distinct minimizers with the indexed query to exceed chosen | ||
| absolute and relative thresholds. | ||
| remote_repository_url: https://github.com/galaxyproject/tools-iuc/tree/main/tools/deacon | ||
| type: unrestricted | ||
| categories: | ||
| - Metagenomics |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do they provide md5 hashes for checking? I'm worried about data corruption.
Also stange that the do not gz?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont know but i can ask upstream if they have or if they are willing to upload it to zenodo. And no they dont use gz since the index files are small. The human index file only has around 4GB
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Asking does not hurt.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi there, Deacon author here.
panhuman-1index is deposited on Zenodo, butpanmouse-1is not. I am happy to putpanmouse-1onto Zenodo in a manner consistent with panhuman-1 if desired. Please note that Zenodo downloads are much slower (at least in the UK) than the S3 bucket downloads, which is why Deacon defaults to using object storage withdeacon index fetch <name>, added in 0.13.0There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe just .md5 files? The intention is only to verify the integrity of the download.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@SantaMcCloud could you try to implement checksum test? Or should we skip it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can try yes if this is wanted :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Give it a shot if easy to implement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
did make a commit which should work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also did see that the tool had some new releases. I will make an upadate RP after this tool got merge so it is usable since it was requested