Home

Welcome to Turbo-TOSEC

Turbo-TOSEC is a high-performance, open-source CLI tool designed to ingest massive collections of TOSEC DAT files into a queryable DuckDB database.

It is engineered to solve the "DOM Parsing Bottleneck" inherent in processing gigabytes of XML data. By leveraging Apache Arrow for zero-copy memory transfers and Parquet for intermediate staging, Turbo-TOSEC transforms a typically slow, memory-intensive ETL process into a rapid, stream-based operation.

Key Features

• 🚀 DuckDB Powered: Utilizes DuckDB as the analytical engine, allowing complex SQL queries on millions of ROM entries with sub-second latency.
• ⚡ Zero-Copy Ingestion: The --direct mode streams XML nodes directly into PyArrow tables, inserting them into DuckDB without intermediate Python object overhead.
• 🛡️ Robust Staging Strategy: The --staged mode implements a "Map-Reduce" style ETL pipeline. Workers parse files into temporary Parquet checkpoints, enabling crash recovery and low-memory handling for massive datasets.
• 📉 Minimal Dependencies: Built with lxml (or stdlib xml) and pyarrow. No Pandas dependency, resulting in faster startup times and lower executable size.
• 💾 Legacy Support: Native support for both standard XML DATs and legacy ClrMamePro (CMP) formats via auto-detection logic.

Performance Philosophy

Turbo-TOSEC abandons the traditional "Load Everything into RAM" approach.

Strategy	Flag	Methodology	Best For
Direct Mode	--direct	XML Stream → Arrow Buffer (RAM) → DB	Maximum Speed (SSD recommended)
Staged Mode	--staged	XML Stream → Parquet (Disk) → Bulk Load	Safety & Scale (Huge datasets / HDD)
Legacy Mode	(default)	XML → DOM Tree → Python Objects	Validation / Small Collections

Quick Start

1. Basic Scan (Safe Mode)

Scans a directory for .dat files and imports them using the default staged strategy (if configured) or standard parsing.

turbo-tosec scan -i "/path/to/tosec_dats/"

2. High-Performance Scan (Direct Mode)

Use this for the fastest possible import on systems with sufficient RAM and fast storage.

turbo-tosec scan -i "/path/to/tosec_dats/" --direct

3. Mass Processing (Staged Mode)

Offloads processing to disk (Parquet), ideal for systems with limited RAM or when processing the entire TOSEC main branch (10GB+ XML).

turbo-tosec scan -i "/path/to/tosec_dats/" --staged --workers 8

Supported Formats

The parser automatically detects the file signature (Magic Bytes / Header):

• XML: Standard TOSEC/MAME schemas (<datafile>, <mame>, <?xml).
• CMP: Legacy ClrMamePro text-based format (clrmamepro ( ... )).

License

This project is licensed under the GPL v3. Developed by Depones Labs.

---

📚 Documentation Index

• Usage Guide: Command line arguments, strategies (--staged vs --direct), and performance tuning.
• Architecture & Internals: Deep dive into the ETL pipeline, Parquet handling, and DuckDB schema.
• SQL Cookbook: Ready-to-use SQL recipes to analyze your collection (Duplicates, Storage Stats, etc.).
• Troubleshooting & FAQ: Solutions for OOM errors, HDD performance, and common questions.