Node: read a federated dataset without downloading the internet

[oesteban]

A user opens the federated collection in pandas, PyArrow or DuckDB and it behaves like a local dataset. This task builds the read path: fetching data into a local cache while verifying every byte against its hash, and transfers that resume after a peer or a laptop drops off the network.

Then it attempts the hard part. Parquet, the standard file format for large tabular datasets, is laid out so a reader can jump straight to just the parts of a file it needs, guided by an index stored at the end of the file (its "footer"). The goal is to serve only those byte ranges over the peer-to-peer transport, so a large dataset can be queried over an ordinary home connection without downloading the whole file first. Whether Parquet's footer-first pattern can be matched to verified range requests (fetching one slice of a file and still checking it is authentic) is genuinely unproven. That outcome is therefore written to deliver either the working implementation or the measured findings.

Outcomes

• Whole-file verified fetch into a local content-addressed cache; a federated collection opens unchanged in PyArrow, pandas and DuckDB
• Resumable, incremental synchronisation surviving restarts and intermittently available peers, with a test that kills and restarts a transfer mid-flight
• Implement or report on Parquet footer-first partial reads over verified range requests: either a shipped range-read path with a measured reduction in bytes fetched, or a published engineering report characterising the mismatch together with an issue filed upstream with the transport maintainers
• Discovery wiring: a node joins a federation from a single ticket and collection announcements propagate, with an operator-configurable bootstrap list and no undeclared default network
• Client documentation and a runnable quickstart notebook