Skip to content

Node: describe datasets so other people's catalogues can read them #2

Description

@oesteban

A node has to describe what it holds in formats that other people's tools already understand, rather than in something invented here. Two established open standards do that work.

A Frictionless Data Package is a widely used open format that pairs a set of tables with a small datapackage.json file listing their columns and types. Any Frictionless-compatible tool can then read the collection. DCAT-AP is the dataset-description format that European open-data portals use: it records a dataset's title, publisher, formats, and where to fetch it. A node that emits a DCAT-AP record can be listed by an existing open-data portal automatically, with no code written on either side.

So every published collection carries a manifest that is also a valid Data Package descriptor, and every node exposes its holdings as a DCAT-AP record. An existing catalogue can index a federation without adopting anything of ours. Both outputs are checked on every change against each standard's own validator, so anyone can confirm they conform.

Outcomes

  • Manifest format v1 specified and published: versioned, schema-aware, content-addressed, with a JSON Schema and a validator
  • Every published collection emits a valid datapackage.json descriptor, validated by the upstream Frictionless validator in CI
  • DCAT-AP catalogue export of a node's holdings, in the two standard text encodings (JSON-LD and Turtle), checked against the official DCAT-AP validation rules (its SHACL shapes) on every change
  • Additive schema evolution tolerated by older peers, with the interoperability surface documented

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort: mediumEstimated medium effort taskenhancementNew feature or requestimpact: highEstimated high impact task

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions