Add Stanza NLP tool and data manager

[ksuderman]

Summary

• Adds Stanford Stanza NLP tool supporting 80+ languages
• Includes data manager for downloading Stanza language models from HuggingFace
• Provides tokenization, POS tagging, lemmatization, dependency parsing, NER
• Supports sentiment analysis and constituency parsing for select languages
• Multiple output formats: JSON, CoNLL-U, tabular, text

Stanza Tool Features

• Memory-optimized nocharlm models for better performance
• Comprehensive language support with standardized Universal Dependencies
• Docker containerization for consistent execution environment
• Integration with Galaxy data tables for model selection

Data Manager Features

• Downloads models directly from HuggingFace using default_fast package
• Multi-select installation interface for language packages
• Automatic registration in Galaxy data tables
• Duplicate prevention when re-run

Test plan

• Tool passes planemo lint validation
• Data manager passes planemo lint validation
• Comprehensive test data included
• README documentation provided for both components
• .shed.yml configured for IUC submission

🤖 Generated with Claude Code

[RZ9082]

This one could also use a cleanup, please remove all duplicate files

[RZ9082]

Are the any small mock models which can be used for testing?
If not, maybe you can add a test option to the python script and download the models on the fly?

[ksuderman]

Are the any small mock models which can be used for testing? If not, maybe you can add a test option to the python script and download the models on the fly?

I have been looking for smaller models, but have not found any yet. Even the small models are large and my local tests have been timing out when I try to download them on the fly. In the worst case I will look into training my own test models. I am running into the same problem with the Stanford CoreNLP tools.

[ksuderman]

CI is now green. Summary of changes since the last review:

Reviewer comments

• Removed the duplicate files (the nested galaxy_tools_stanza/ copy, the stray data_manager_stanza file, and the redundant macros.xml). The tool_data_table_conf.xml.sample / .loc.sample that remain identical between the tool and data manager are intentional — each is an independently-installable repo.
• Reset the version suffix to +galaxy0, conda stanza 1.12.0 requirement, removed the manual HOME workaround, citation DOIs, ftype="json" on the test output, and replaced the hardcoded stanza_models.loc path with ${__HERE__}.

Data manager

• Added the missing data_manager_conf.xml and moved the tool into a data_manager/ subdirectory; without this Galaxy couldn't load the data manager and crash-looped at startup.
• Models are now installed with stanza.download(package="default_fast"); the old code fetched a default_fast.zip that doesn't exist on HuggingFace. Added a stanza requirement.
• Download into the job's extra_files_path and <move> into ${GALAXY_DATA_MANAGER_DATA_PATH}, since tool-data is read-only in the biocontainer.

Tool

• Set USER in the command: PyTorch (loaded by Stanza) calls getpass.getuser(), which falls back to getpwuid() and fails for the container's unmapped uid. HOME is provided by Galaxy.

All tool and data manager tests now pass in biocontainers.

[anuprulez]

The datamanager tool fails with the following error. Do I need to make some configuration changes to make it run locally?

Once it is running, it will install the data manages (local models) that can be directly used with the Stanza NLP Annotators tool ?

File "/tmp/tmpwygi5u0c/galaxy-dev/lib/galaxy/jobs/init.py", line 2269, in finish
task_wrapper = self.tool.exec_after_process(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/tmpwygi5u0c/galaxy-dev/lib/galaxy/tools/init.py", line 3691, in exec_after_process
assert (
AssertionError: Invalid data manager (data_manager_stanza_models) requested. It may have been removed before the job completed.

[anuprulez]

Suggested change

	<tool id="stanza_nlp" name="Stanza NLP Annotators" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="24.1">
	<tool id="stanza_nlp" name="Stanza NLP Annotators" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="26.1">

[ksuderman]

Is there a reason for the profile bump? The tool and data manager should both be compatible with v24.1 Galaxy instances. There is nothing that would make it 25.x or 26.x specific.

[anuprulez]

I think we try to use later Galaxy profiles, matching the profiles of data manager and the stanza tool would be important.

[anuprulez]

The test coverage should be enhanced for all different types of annotators

[ksuderman]

I have added another test for the pos tagger. The other annotators require large models to be installed which are problematic in CI. This is best effort until I can find, or train, smaller models.

[anuprulez]

I have added more tests (see my last comment). pretrain and depparse models are large (~100MBs) but others are smaller I think. Don't these models get deleted automatically by the CI once the testing finishes? Performance wise, they did not take a lot of time locally.

[anuprulez]

I could run the tool with by setting the hidden parameter to "true" for downloading the models but not by invoking the data manager.

[ksuderman]

The datamanager tool fails with the following error. Do I need to make some configuration changes to make it run locally?

The AssertionError is raised in exec_after_process when Galaxy finishes the job and looks the tool up in its data manager registry and doesn't find it. In other words, the manage_data tool ran, but that Galaxy instance never registered it as a data manager. So it's a configuration/registration gap on the running instance.

A normal Tool Shed install wires this up automatically, because the repo ships its own registration file: data_managers/data_manager_stanza_models/data_manager_conf.xml. If you're running it from a source checkout instead, you need your Galaxy's data_manager_config_file to include that file otherwise the tool loads into the tool panel but isn't in the DM registry, which produces exactly this error.

Two ways to run it locally that do register it:

• planemo (this is what CI does): planemo test data_managers/data_manager_stanza_models/data_manager/data_manager_stanza_models.xml. planemo derives the registration from the sibling ../data_manager_conf.xml automatically.
• A local Galaxy from source: add the repo's data_manager_conf.xml to data_manager_config_file in galaxy.yml, restart, then run it from Admin → Local Data. I reproduced your setup on a local Galaxy 26.0 this way and the log shows Loaded Data Manager: data_manager_stanza_models and the job registers/populates the stanza_models table with no assertion error.

Once it is running, it will install the data manages (local models) that can be directly used with the Stanza NLP Annotators tool ?

Yes. Once the data manager runs, it installs the language models into the stanza_models data table, and the Stanza NLP Annotators tool reads them directly from that table (the language dropdown). No re-download happens at annotation time; that's exactly what the hidden download_models parameter was a stand-in for during CI testing.

[anuprulez]

thanks @ksuderman

I tested it locally based on your suggestions and the tool works for me (tested ENG and DE).

The following tests also worked for me that tests other annotators like ner, parse, sentiment and constituency

<test>
            <param name="input" value="input.txt"/>
            <param name="language_model" value="en"/>
            <param name="annotators" value="ner"/>
            <param name="format" value="text"/>
            <param name="download_models" value="true"/>
            <output name="output_file" ftype="txt">
                <assert_contents>
                    <has_text text="Named Entities"/>
                    <has_text text="John Smith went to Walmart"/>
                </assert_contents>
            </output>
        </test>
        <test>
            <param name="input" value="input.txt"/>
            <param name="language_model" value="en"/>
            <param name="annotators" value="parse"/>
            <param name="format" value="text"/>
            <param name="download_models" value="true"/>
            <output name="output_file" ftype="txt">
                <assert_contents>
                    <has_text text="lemma=John"/>
                    <has_text text="then he went to the theater"/>
                </assert_contents>
            </output>
        </test>
        <test>
            <param name="input" value="input.txt"/>
            <param name="language_model" value="en"/>
            <param name="annotators" value="sentiment"/>
            <param name="format" value="text"/>
            <param name="download_models" value="true"/>
            <output name="output_file" ftype="txt">
                <assert_contents>
                    <has_text text="Sentiment: neutral"/>
                    <has_text text="then he went to the theater"/>
                </assert_contents>
            </output>
        </test>
        <test>
            <param name="input" value="input.txt"/>
            <param name="language_model" value="en"/>
            <param name="annotators" value="constituency"/>
            <param name="format" value="text"/>
            <param name="download_models" value="true"/>
            <output name="output_file" ftype="txt">
                <assert_contents>
                    <has_text text="lemma=None"/>
                    <has_text text="then he went to the theater"/>
                </assert_contents>
            </output>
        </test>