Central data management for Machine Learning for Trading, 3rd Edition.
Each dataset has its own directory with a download script, loader, config, and exploration notebook. All loaders return Polars DataFrames with a consistent API.
# 1. Set data path in repository root .env file
ML4T_DATA_PATH=/path/to/your/data
# 2. Download free datasets (no API keys needed)
uv run python data/download_all.py --free-only
# 3. Use in notebooks
from data import load_etfs
df = load_etfs()Organized by asset class and data type. "Type" column maps each dataset to its place in the Ch2/Ch4 taxonomy: market (OHLCV / microstructure / options), fundamentals (accounting + regulatory filings), positioning (positions / insider activity), onchain (crypto-native fundamentals), or cross-asset (factors, macro, prediction markets, news, text).
| Dataset | Asset Class | Type | Frequency | Symbols | Coverage | Source | Access |
|---|---|---|---|---|---|---|---|
| ETF Universe | Equity | Market | Daily | 100 | 2006-2025 | Yahoo Finance | No |
| US Equities | Equity | Market | Daily | 3,199 | 1962-2018 | NASDAQ DL | Free |
| S&P 500 Bars | Equity | Market | Daily | ~638 | 2017-2021 | AlgoSeek | Soon |
| S&P 500 Options | Equity | Market | Daily | ~500 | 2017-2021 | AlgoSeek | Soon |
| NASDAQ-100 Bars | Equity | Market | Minute | ~100 | 2020-2021 | AlgoSeek | Soon |
| TAQ Tick | Equity | Market | Tick | 1 | Mar 2020 | AlgoSeek | Soon |
| MBO Tick | Equity | Market | Tick | 1 | Nov 2024 | Databento | Manual |
| NASDAQ ITCH | Equity | Market | Tick | all | varies | NASDAQ FTP | No |
| IEX DEEP/TOPS | Equity | Market | Tick | all | varies | IEX public | No |
| SEC XBRL Fundamentals | Equity | Fundamentals | Quarterly | 20 | 2022-2024 | SEC EDGAR | No |
| SEC 10-K (SP100) | Equity | Fundamentals | Annual | ~100 | 2020-2025 | SEC EDGAR | No |
| SEC 10-Q MD&A (SP500) | Equity | Fundamentals | Quarterly | ~600 | 2017-2021 | SEC EDGAR | No |
| SEC 8-K (SP100) | Equity | Fundamentals | Event | ~100 | 2024-2025 | SEC EDGAR | No |
| 13F Institutional | Equity | Positioning | Quarterly | 10 inst | rolling | SEC EDGAR | No |
| Form 4 Insider | Equity | Positioning | Event | varies | varies | SEC EDGAR | No |
| Firm Characteristics | Equity | Packaged | Monthly | anon | 1967-2016 | GitHub | No |
| CME Futures | Futures | Market | Daily/Hourly | 30 | 2011-2025 | Databento | Paid |
| CFTC Commitment of Traders | Futures | Positioning | Weekly | 25+ | 2020-2025 | CFTC public | No |
| Crypto Perps | Crypto | Market | 1h | 19 | 2020-2025 | Binance Public | No |
| Crypto Premium | Crypto | Market | 8h | 19 | 2020-2025 | Binance Public | No |
| DefiLlama TVL | Crypto | Onchain | Daily | chains | varies | DefiLlama | No |
| CoinGecko OHLCV | Crypto | Onchain | Daily | varies | 365 days | CoinGecko | No |
| FX Pairs | Currency | Market | 4h/Daily | 20 | 2011-2025 | OANDA | Free |
| FF Factors | Cross-asset | Factors | Monthly | 5 | 1926-now | Ken French | No |
| AQR Factors | Cross-asset | Factors | Monthly | 8 | varies | AQR | No |
| FRED Macro | Cross-asset | Macro | Various | 40 | 2000-2025 | FRED | Free |
| Kalshi events | Cross-asset | Prediction | Daily | varies | 2021-2025 | Kalshi public | No |
| Polymarket events | Cross-asset | Prediction | Daily | varies | 2020-2025 | Polymarket public | No |
| FNSPID news | Cross-asset | News | Daily | 4,775 | 1999-2023 | HuggingFace | No |
| Bloomberg news archive | Cross-asset | News | Daily | mixed | 2006-2013 | HuggingFace | No |
| Financial Phrasebank | Cross-asset | Text | Static | — | n/a | HuggingFace | No |
Access legend. No — included with the repo or fetched by an
unauthenticated script. Free — script-download, free API key required.
Paid — script-download, billed API (see per-dataset estimates).
Manual — reader downloads from a hosted URL or provider portal and
places the files under $ML4T_DATA_PATH (no script); the DataBento MBO
one-off has step-by-step instructions below. Soon — the reduced
reader-facing AlgoSeek datasets are being prepared for hosting; the
download URL and instructions will be published before launch.
Times below are rough, indicative estimates only. Actual duration depends on your bandwidth, the providers' current rate limits, and disk speed — treat them as ballpark, not guarantees.
# All free datasets at once (includes the ~1.5 GB firm-characteristics
# dataset; add --skip-firm-characteristics to leave it out)
uv run python data/download_all.py --free-only
# Individual datasets (from repo root)
uv run python data/etfs/market/download.py # ~30s
uv run python data/crypto/market/download.py # ~10-15 min (see note)
uv run python data/factors/ff_download.py # ~5s
uv run python data/factors/aqr_download.py # ~5s
uv run python data/equities/firm_characteristics/download.py # ~1.5 GB, largest free dataset; downloads + converts (minutes, bandwidth-dependent)
uv run python data/futures/positioning/cot_download.py # ~2-3 min (CFTC CoT)Note on crypto download time: The Binance public API returns max 1,500 rows per request with ~1s server response time. Downloading 5 years of hourly data for 19 symbols requires ~700 API calls. Downloads run in parallel (5 concurrent), but the total still takes 10-15 minutes. This is a Binance server-side rate limit, not a bug.
# FRED macro indicators
uv run python data/macro/download.py
# US Equities (NASDAQ Data Link — frozen, ends 2018)
uv run python data/equities/market/us_equities/download.py
# FX pairs (OANDA)
uv run python data/fx/market/download.py # 4-hourly (default)
uv run python data/fx/market/download.py --daily # Daily# CME Futures — ALWAYS estimate cost first!
uv run python data/futures/market/download.py --estimate-only
uv run python data/futures/market/download.pyThe Chapter 3 MBO slice (NVDA, 10 trading days in November 2024) is best obtained as a one-off download from the Databento Download Center — total cost is under $10 and the files stay available for 30 days.
See data/equities/market/microstructure/MBO_DOWNLOAD.md for step-by-step
instructions. An API-based alternative (mbo_download.py) is available
for users who already have a DATABENTO_API_KEY.
Extend datasets beyond the default end date:
uv run python data/download_all.py --updateAll loaders are importable from data and return Polars DataFrames:
from data import (
load_etfs,
load_crypto_perps,
load_crypto_premium,
load_cme_futures,
load_cot,
load_fx_pairs,
load_macro,
load_us_equities,
load_ff_factors,
load_aqr_factors,
load_firm_characteristics,
load_nasdaq100_bars,
load_sp500_daily_bars,
load_sp500_options,
load_sp500_options_eda,
load_sp500_options_straddles_raw,
load_sp500_options_surface,
load_sp500_options_straddles,
load_nasdaq100_taq,
load_mbo_data,
load_nasdaq_itch,
load_iex_hist,
)
# All loaders support filtering
df = load_etfs(symbols=["SPY", "QQQ"], start_date="2020-01-01")
# Futures use 'products' instead of 'symbols'
futures = load_cme_futures(products=["ES", "NQ"], start_date="2020-01-01")
# Test mode: limit to N random symbols (seed-deterministic)
df = load_etfs(max_symbols=15)When data is missing, loaders raise DataNotFoundError with download instructions.
| Provider | Variable | Sign Up |
|---|---|---|
| FRED | FRED_API_KEY |
https://fred.stlouisfed.org/docs/api/api_key.html |
| NASDAQ Data Link | QUANDL_API_KEY |
https://data.nasdaq.com/sign-up |
| OANDA | OANDA_API_KEY |
https://www.oanda.com/ |
| Provider | Variable | Cost |
|---|---|---|
| Databento | DATABENTO_API_KEY |
$125 free credit |
Create .env in repository root:
ML4T_DATA_PATH=/path/to/your/data
# Free API keys
FRED_API_KEY=your-fred-key
QUANDL_API_KEY=your-nasdaq-key
OANDA_API_KEY=your-oanda-key
# Paid
DATABENTO_API_KEY=db-your-keyEvery dataset directory is self-contained: a download script, a loader
(or re-export from a parent loader), a README with the full instructions
that DataNotFoundError points readers to, and optionally a config
and exploration notebook.
Data is organized by asset class × data type, matching the Ch2 /
Ch4 taxonomy. Each asset class has market/ for OHLCV-style data, and
optionally fundamentals/, positioning/, or other type-specific
subdirectories. Cross-asset datasets (factors/, macro/,
prediction_markets/, alternative/) sit at the top level.
data/
├── __init__.py # Single import point for all loaders
├── exceptions.py # DataNotFoundError, DownloadError, MissingDependencyError
├── download_all.py # Download orchestrator
├── README.md # (this file)
│
├── equities/ # US equities
│ ├── market/ # us_equities, sp500 (daily + options), nasdaq100, microstructure
│ ├── fundamentals/ # 10-K / 10-Q / 8-K filings, XBRL financials
│ ├── positioning/ # 13F institutional holdings, Form 4 insider
│ ├── firm_characteristics/ # Chen-Pelger-Zhu panel (standalone packaged dataset)
│ └── loader.py # All equities loaders in one module
│
├── futures/ # CME futures
│ ├── market/ # Databento continuous + individual contracts
│ ├── positioning/ # CFTC Commitment of Traders (CoT)
│ └── loader.py
│
├── crypto/ # Crypto
│ ├── market/ # Binance perps OHLCV + premium index
│ ├── onchain/ # DefiLlama TVL + CoinGecko OHLCV
│ └── loader.py
│
├── fx/market/ # FX pairs (OANDA)
├── etfs/market/ # ETF universe (Yahoo)
│
├── factors/ # Fama-French, AQR (cross-asset, academic)
├── macro/ # FRED macro indicators (cross-asset)
├── prediction_markets/ # Kalshi + Polymarket events
│
└── alternative/ # Cross-asset third-party alt data
├── news/ # Bloomberg, FNSPID
└── text/ # Financial Phrasebook sentiment benchmark
Every subdirectory owns its data's lifecycle — a reader can open any leaf README and find the download command and file layout without consulting the top-level doc.
Market (OHLCV, microstructure, options):
| Loader | Dataset | Source |
|---|---|---|
load_sp500_index() |
S&P 500 index OHLCV | Bundled |
load_us_equities() |
3,199 US stocks (1962-2018) | NASDAQ DL |
load_sp500_daily_bars() |
S&P 500 daily OHLCV | AlgoSeek |
load_sp500_options() |
Raw options chains (legacy) | AlgoSeek |
load_sp500_options_eda() |
Options EDA slice (8 symbols, 2019-2020) | AlgoSeek (slim) |
load_sp500_options_straddles_raw() |
ATM-band raw chains, lifecycle-preserving (2017-2021) | AlgoSeek (slim) |
load_sp500_options_surface() |
Daily IV surface summary | Materialized |
load_sp500_options_straddles() |
Daily ATM straddles | Materialized |
load_nasdaq100_bars() |
NASDAQ-100 bars (minute default; resampling, quotes, full microstructure) | AlgoSeek |
load_nasdaq100_taq() |
TAQ tick data (AAPL, 2020-03-13 / 2020-03-16) | AlgoSeek (slim) |
load_mbo_data() |
MBO order book data | Databento |
load_nasdaq_itch() |
NASDAQ ITCH messages | NASDAQ FTP |
load_iex_hist() |
IEX DEEP/TOPS data | IEX (free) |
Fundamentals (SEC filings + XBRL):
| Loader | Dataset | Source |
|---|---|---|
load_sp500_10q_mda() |
S&P 500 10-Q MD&A text (2017-2021) | SEC EDGAR |
load_sec_filings(form_type) |
10-K / 10-Q / 8-K aggregate text | SEC EDGAR |
resolve_sec_filings_dir() |
Per-ticker filings directory (Ch22 RAG) | SEC EDGAR |
load_sec_xbrl_fundamentals() |
XBRL financial facts (CIK × quarter × concept) | SEC XBRL Frames |
Positioning (13F):
| Loader | Dataset | Source |
|---|---|---|
load_institutional_holdings_13f() |
13F holdings (per-cik, 10 curated managers) | SEC EDGAR |
load_13f_bulk_holdings(quarter) |
13F full universe (~3M rows per quarter) | SEC bulk |
load_13f_stock_features() |
Stock-level features (breadth, concentration) | Derived |
load_13f_edges() |
Institution → stock edge list (graph) | Derived |
Firm characteristics (packaged dataset):
| Loader | Dataset | Source |
|---|---|---|
load_firm_characteristics() |
Chen-Pelger-Zhu panel (~180 features, returns + accounting) | GitHub |
| Tier | Datasets | Size |
|---|---|---|
| Minimum | ETFs, Crypto, Factors | ~70 MB |
| Standard | + Macro, FX | ~75 MB |
| With Equities | + US Equities | ~740 MB |
| With Futures | + CME Futures | ~825 MB |
| Full | + AlgoSeek slim package, ITCH, MBO | ~7 GB |
All loaders return data with consistent column names:
- Entity column:
symbol(exception: CME futures useproduct) - Time column:
timestamp(for all frequencies — daily, hourly, minute, tick)
Notebooks should always use these canonical names. If older data files use legacy names like asset, date, ticker, or pair, the loaders normalize them automatically.