feat: add group inspection to compatibility endpoint (#186)

Add support for hierarchically organized datasets in the compatibility endpoint

Co-authored-by: Chuck Daniels <chuck@developmentseed.org>

Author: hrodmn

.gitignore

@@ -106,6 +106,7 @@ node_modules
 
 notebooks/
 .envrc
+issue-drafts/
 
 # Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode
 # Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode

README.md

@@ -22,6 +22,7 @@ See also the [API documentation](https://staging.openveda.cloud/api/titiler-cmr/
 
 - Render tiles from assets discovered via queries to [NASA's CMR](https://cmr.earthdata.nasa.gov/search)
 - Two backends: **xarray** (`/xarray`) for NetCDF/HDF5 datasets and **rasterio** (`/rasterio`) for GeoTIFF/COG assets
+- `/compatibility` helps inspect sample granules, including optional `group` selection and lightweight group discovery for hierarchical HDF5 and NetCDF assets
 - Timeseries endpoints for generating GIFs, statistics, and tilejsons across a temporal range
 - Queries CMR directly via the [granule search API](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html)
 - Built on top of [titiler](https://github.com/developmentseed/titiler)

docs/api/compatibility_api_example.ipynb

@@ -2,74 +2,74 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "cdeef4c6-75b0-44d9-90d4-c850b5d8908a",
+   "id": "intro",
    "metadata": {},
    "source": [
     "# How to use the Compatibility API\n",
     "\n",
-    "The `/compatibility` endpoint displays information about the collection and returns some details about a sample granule. The output is helpful for understanding the structure of the collection and the granules so that you can craft the right set of parameters for visualization or statistics requests.\n",
+    "The `/compatibility` endpoint helps you understand how to call TiTiler-CMR for a collection. For hierarchical HDF5 and NetCDF datasets it can return a lightweight list of candidate `group` paths from the root request, then let you inspect variables by making a follow-up request with `group=...`.\n",
     "\n",
-    "This notebook demonstrates how to use the compatibility endpoint for th [GHRSST Level 4 MUR Global Foundation Sea Surface Temperature Analysis (v4.1)](https://cmr.earthdata.nasa.gov/search/concepts/C1996881146-POCLOUD) dataset.\n",
+    "This notebook demonstrates that workflow for the [NISAR Beta Geocoded Polarimetric Covariance (GCOV)](https://www.earthdata.nasa.gov/data/catalog/asf-nisar-l2-gcov-beta-v1-1) collection, which is a known example of a hierarchical HDF5 dataset that needs a `group` parameter for xarray requests.\n",
     "\n",
     "## Setup"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7d015182-5347-437a-8b66-d7d62212f0e3",
+   "id": "setup",
    "metadata": {},
    "outputs": [],
    "source": [
+    "import json\n",
     "import os\n",
     "\n",
-    "import json\n",
     "import earthaccess\n",
     "import httpx2 as httpx\n",
     "\n",
     "titiler_endpoint = os.getenv(\n",
-    "    \"TITILER_CMR_ENDPOINT\", \"https://openveda.cloud/api/titiler-cmr\"\n",
+    "    \"TITILER_CMR_ENDPOINT\", \"https://staging.openveda.cloud/api/titiler-cmr\"\n",
     ")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d375b5b7-9322-4f1e-8859-000ef8ac4898",
+   "id": "identify-dataset",
    "metadata": {},
    "source": [
     "## Identify the dataset\n",
     "\n",
-    "You can find the MUR SST dataset using the `earthaccess.search_datasets` function."
+    "You can find the collection with `earthaccess.search_datasets`."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f3fcc9cd-6105-42fc-98bf-2de40910a79c",
+   "id": "search-dataset",
    "metadata": {},
    "outputs": [],
    "source": [
-    "datasets = earthaccess.search_datasets(concept_id=\"C1996881146-POCLOUD\")\n",
+    "datasets = earthaccess.search_datasets(concept_id=\"C3622214170-ASF\")\n",
     "ds = datasets[0]\n",
     "\n",
     "collection_concept_id = ds[\"meta\"][\"concept-id\"]\n",
-    "print(\"CollectionConcept-Id: \", collection_concept_id)\n",
-    "\n",
-    "print(\"Abstract: \", ds[\"umm\"][\"Abstract\"])"
+    "print(\"CollectionConcept-Id:\", collection_concept_id)\n",
+    "print(\"Short name:\", ds[\"umm\"][\"ShortName\"])\n",
+    "print(\"Version:\", ds[\"umm\"][\"Version\"])"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2a4cffa6-0059-4033-a708-db60d743f0e3",
+   "id": "explore-collection",
    "metadata": {},
    "source": [
-    "## Explore the collection using the `/compatibility` endpoint"
+    "## Explore the collection using `/compatibility`"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1bde609a-26df-4f35-b7e1-9e1922e87808",
+   "id": "compatibility-request",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -84,12 +84,105 @@
   },
   {
    "cell_type": "markdown",
-   "id": "04014a32-9c11-4b75-b40a-e5ad4efd686b",
+   "id": "interpret-response",
+   "metadata": {},
+   "source": [
+    "## Interpret the response\n",
+    "\n",
+    "For hierarchical datasets, a root `/compatibility` request stays lightweight. The most important field is:\n",
+    "\n",
+    "- `compatible_groups`: candidate group paths to try in a follow-up request\n",
+    "\n",
+    "At this stage the response may not include variable metadata or xarray links yet, because TiTiler-CMR has not inspected a nested group."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "summarize-response",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "compatible_groups = compatibility_response.get(\"compatible_groups\", [])\n",
+    "variables = list((compatibility_response.get(\"variables\") or {}).keys())\n",
+    "tilejson_link = next(\n",
+    "    (\n",
+    "        link[\"href\"]\n",
+    "        for link in compatibility_response.get(\"links\", [])\n",
+    "        if link[\"rel\"] == \"tilejson\"\n",
+    "    ),\n",
+    "    None,\n",
+    ")\n",
+    "\n",
+    "print(\"compatible_groups:\")\n",
+    "for group in compatible_groups:\n",
+    "    print(\" -\", group)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "inspect-explicit-group",
+   "metadata": {},
+   "source": [
+    "## Inspect a specific group\n",
+    "\n",
+    "Once you choose a group from `compatible_groups`, make a follow-up `/compatibility` request with `group=...` to inspect variables and get xarray links."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "compatibility-request-with-group",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "selected_group = compatible_groups[1]\n",
+    "\n",
+    "grouped_compatibility_response = httpx.get(\n",
+    "    f\"{titiler_endpoint}/compatibility\",\n",
+    "    params={\n",
+    "        \"collection_concept_id\": collection_concept_id,\n",
+    "        \"group\": selected_group,\n",
+    "    },\n",
+    "    timeout=30,\n",
+    ").json()\n",
+    "\n",
+    "print(json.dumps(grouped_compatibility_response, indent=2))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "summarize-grouped-response",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "group_variables = list((grouped_compatibility_response.get(\"variables\") or {}).keys())\n",
+    "group_tilejson_link = next(\n",
+    "    link[\"href\"]\n",
+    "    for link in grouped_compatibility_response.get(\"links\", [])\n",
+    "    if link[\"rel\"] == \"tilejson\"\n",
+    ")\n",
+    "\n",
+    "print(\"selected group:\", selected_group)\n",
+    "print(\"variables in selected group (first 10):\", group_variables[:10])\n",
+    "print(\"tilejson link for selected group:\", group_tilejson_link)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "what-to-do-next",
    "metadata": {},
    "source": [
-    "The details from the sample granule show that it is a NetCDF file with four variables (`analysed_sst`, `analysis_error`, `mask`, and `sea_ice_fraction`) and each contains an array with a single time coordinate. The `datetime` key shows the reported temporal range from CMR which indicates that the dataset has granules from `2008-07-23` to present. For each variable several summary statistics are available to help you craft min/max values for the `rescale` parameter.\n",
+    "For a grouped dataset like NISAR GCOV, the workflow is:\n",
+    "\n",
+    "1. Call `/compatibility` without `group` to get candidate `compatible_groups`.\n",
+    "2. Choose one of those group paths.\n",
+    "3. Call `/compatibility` again with `group=...` to inspect variables and dimensions for that group.\n",
+    "4. Pick a variable name from `variables`.\n",
+    "5. Reuse the returned xarray links from the grouped response.\n",
     "\n",
-    "This information is handy for generating tiles via the tiles API, which you can learn more about in [the Tiles API Documentation](../tiling_api_xarray_backend_example)."
+    "That keeps the root compatibility response lightweight while still giving you a practical path to the next xarray request."
    ]
   }
  ],