6180: persist DCAT-US3 catalog metadata during harvest

[Aguiardavidm]

Pull Request

Related to GSA/data.gov#6180

Supersedes #795, rewritten now that datagov_data_access has been deprecated (GSA/data.gov#6209) and its models/migrations vendored directly into this repo. Also folds in the standalone DB migration from datagov_data_access#10, which targeted the now-defunct package and can no longer land there.

About

Adds support for retaining DCAT-US 3.0 catalog-level metadata during a harvest, per the ticket's sketch:

• Adds an Alembic migration for the new dcatus_catalog JSONB column on harvest_job, chained onto this repo's current migration head.
• acquire_minimum_external_data now captures the full downloaded catalog for dcatus3.0 sources, recursively extracts every dataset (including datasets nested in sub-catalogs, however deep), and persists a stripped copy of the catalog metadata (with dataset/service/record removed, recursively) onto the job via update_harvest_job.
• New helpers in harvester/utils/general_utils.py: strip_dcatus3_catalog_objects and extract_dcatus3_catalog_datasets.
• New fixture (dcatus3_0_nested_catalog.json) and tests covering both the flat and nested-catalog cases, plus integration tests for add_harvest_job/update_harvest_job with dcatus_catalog.

No datagov-catalog changes are needed: catalog deliberately has no HarvestJob model (GSA/data.gov#6211, catalog never queries harvest_job), and its FK-satisfying fixture stand-in is checked against a static snapshot that predates this column.

Ran the full unit/integration suite locally; all passing tests remain green (12 pre-existing failures in test_auth_logging.py and test_load_manager.py reproduce identically on unmodified main in this environment and are unrelated to this change).

PR TASKS

• Code well documented
• Tests written, run and passed
• Files linted

[Aguiardavidm]

Merged in dev: https://github.com/GSA/datagov-harvester/actions/runs/31033409801

[jbrown-xentity]

This all looks great, but a schema change in DCAT-US3.0 means we need one more item cleaned/removed, and then this is good to go! See below comment with reference.

[jbrown-xentity]

So unfortunately this changed last minute @Aguiardavidm ; it should also exclude datasetSeries, see https://github.com/GSA/dcat-us/pull/162/changes#diff-a418ae196beadca489f75ac5098c2e3a211223057f86bcad24bde5014e81dee5R56

[jbrown-xentity]

LGTM