Add auto-generated Configuration schema from Pydantic model

[davelopez]

Follow up to #16514

I was manually creating the model when I realized we already have the config_schema.yml. It would be much easier to maintain if we could update the model whenever we introduce a new config parameter. As wisely suggested by Marius, I will approach this by first generating the initial basic Pydantic Model from the config schema, and then, the idea is to use the model as the single source of truth and generate the schema from it.

TODO

• Generate initial Pydantic model from current config_schema.yml
• Manually curate the model:
  • Include any missing fields that are serialized in the API but not in the YAML schema (computed fields), etc.
  • Create different views of the Configuration model, initially:
    • ComputedGalaxyConfig: contains common Galaxy configuration options computed at runtime (not defined in the YAML config schema).
    • UserExposableComputedGalaxyConfig / AdminExposableComputedGalaxyConfig: mainly used for discriminating admin/non-admin models, but can be used to extend the common ComputedGalaxyConfig with admin-only options.
    • AdminExposableGalaxyConfig/UserExposableGalaxyConfig: contains (admin-only or common) configuration options that are exposed through the API.
    • FullGalaxyConfig: contains all the options in the YAML configuration file. This model will be used to generate the YAML schema. These options are not exposed to the API.
  • Add stricter types using literal values instead of strings
  • Include Fix: serialize tool_shed_urls directly from the API #16561 in model
  • Mark deprecated options as deprecated.
  • Take by_host dynamic config options into account Allow various configuration parameters to be set per host. #12328
  • Annotate path_resolves_to options
  • Annotate reloadable options
  • Figure out why converter tests are failing
    • After some digging, the problem was caused by use_remote_user getting a string (email) instead of a boolean as a value assigned from the single_user option in some cases. I decided to fix it by casting to the bool value which is more explicit c9a6794
  • Fix API schema linting with custom annotations (per_host, path_resolves_to, etc...)
• Generate config_schema.yml from the Full model

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  • TBD

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[mvdbeek]

Opening PR to gather feedback on this approach.

This is amazing, I love it. How about we turn this around and make the pydantic model the source of truth from which we generate the other stuff ... it should be much more expressive than what we have currently, and the model_validators would let us for instance easily flag stuff that can't be combined.

And we can easily define what can and can't be exposed through the API inline.

[davelopez]

Thanks so much @mvdbeek! That is exactly what I needed to hear 😄 👍

[mvdbeek]

Discussed this in the backend channel, we're all on board with having the pydantic models as the source of truth, and that if necessary, we can replace the tooling built on pykwalify.