Skip to content

feat: add openbb-government-ca extension — Statistics Canada SDMX catalog (Phase 1)#7548

Draft
rohanjethwani17 wants to merge 2 commits into
OpenBB-finance:v5from
rohanjethwani17:feature/v5-government-ca
Draft

feat: add openbb-government-ca extension — Statistics Canada SDMX catalog (Phase 1)#7548
rohanjethwani17 wants to merge 2 commits into
OpenBB-finance:v5from
rohanjethwani17:feature/v5-government-ca

Conversation

@rohanjethwani17

Copy link
Copy Markdown

What this is

First phase of the openbb-government-ca extension discussed in #7490. This PR delivers the V5 package scaffold plus the Statistics Canada (StatsCan) SDMX catalog infrastructure, following the Hatchling formula from the openbb-oecd refactor (#7413):

  • Hatchling custom build hook that materializes a compressed JSON catalog at build time, shelling out to the generator by path (no openbb_core import in the build env), gated by OPENBB_GOVERNMENT_CA_FORCE_CACHE_REBUILD, asset gitignored at the repo root alongside the OECD/IMF entries
  • Catalog generator (stdlib + requests only) that digests per-PID SDMX-ML structure messages into a normalized catalog
  • Runtime catalog reader (no network) and one available_indicators discovery endpoint reading from the shipped catalog
  • 100% line coverage of the openbb_government_ca package, with a CI workflow mirroring test-unit-openbb-oecd.yml (ruff + ty + pytest, --cov-fail-under=100, across Linux/macOS/Windows on Python 3.10/3.11/3.13)

This is opened as a draft — it's Phase 1 (catalog infrastructure). Phases to follow: StatsCan data fetchers + router (obb.statscan.*), Bank of Canada Valet datasets (obb.boc.* — yield curves, open market operations, balance sheet), and topical surfacing (obb.economy.calendar, obb.economy.available_indicators).

A few decisions I made that I'd value your read on

1. Curated catalog scope. StatsCan's full cube population is ~8,213 tables, which compresses to roughly 50–130 MB — too large to ship as an asset. I scoped the catalog to a curated seed of the 36 distinct table PIDs found in ind-econ.json (results.indicators[].source), which produces a 47 KB compressed catalog (~317 KB decompressed) covering the headline economic indicators. Happy to widen the seed set if you'd prefer broader coverage.

2. StatsCan diverges from the OECD SDMX assumptions. Worth flagging what the recon turned up, since it shaped the design:

  • No dataflow-listing, standalone codelist, or availability endpoint exist (all 404). Everything is fetched per-PID from structure/Data_Structure_{pid}.
  • The DSD skeleton is uniform across cubes (9 attributes, single OBS_VALUE measure, common codelists), so one generic parser handles every table.
  • Frequency is not in the SDMX TimeDimension (the TextFormat element is empty in real responses) — it's carried as a freq annotation on the DataStructure. The parser reads it from there.

3. The discovery endpoint is model-less. available_indicators returns a plain list[dict] rather than flowing through the standard fetcher/OBBject/provider-choices machinery, since it's pure catalog metadata with no provider query. Core supports this and the route registers cleanly, but it's a different idiom than the model-backed discovery endpoints in CFTC/OECD/economy. If you'd prefer it go through the standard OBBject path, easy to change — wanted your call before building the rest of the endpoints on this pattern.

Notes

  • The curated set includes one irregular-frequency table (PID 32100007, grain stocks at three fixed annual reference dates) that legitimately doesn't map to monthly/quarterly/annual; it's recorded as undetermined rather than guessed.
  • Branched feature/v5-government-ca off v5 per your guidance. I know a direct PR to v5 may hit branch protection — flagging in case you need to adjust anything upstream, as you mentioned.
  • Generated the catalog against the live StatsCan API end to end: 36/36 PIDs parse clean, deterministic output, well under the 1 MiB ceiling.

…ture)

Signed-off-by: rohanjethwani17 <rohanjethwani17@gmail.com>
@CLAassistant

CLAassistant commented Jun 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants