Substrait Compliance Framework

A decentralized compliance testing framework for Substrait implementations

Enable database engines to self-certify their Substrait compliance through standardized interfaces, pre-packaged test suites, and automated reporting.

New to the project? Read the technical introduction for the motivation, architecture, and how to get involved.

🎯 Overview

The Substrait Compliance Framework transforms how query engines validate their Substrait support. Instead of centralized testing, engines self-certify by:

Implementing standard interfaces — ComplianceEngine trait/interface
Running available test suites — TPC-H (22 queries), TPC-DS (99 queries), and function test suites
Generating compliance reports — JSON format with pass/fail results
Publishing results — Optional leaderboard/report publication

Key Features

🔄 Decentralized Testing — Engines test themselves; no central bottleneck
🌐 Multi-Language SDKs — Java, Python, Rust, C++, Go, TypeScript/JavaScript, C#/.NET, and Scala at varying maturity levels
📦 Pre-packaged Test Suites — TPC-H, TPC-DS, and function test suites included in-repo
⚡ Performance Benchmarking — Benchmarking components included across several SDKs
🤖 Automated CI/CD — GitHub Actions workflows for validation, release, and automation
🌐 REST API — Spring Boot API implementation for programmatic access (pre-release; not yet deployed to a public endpoint)
📊 Interactive Demo — Live dashboard with mock engines, query drill-down, and complexity filtering
🏆 Leaderboard Support — Report aggregation and leaderboard generation

🔧 Prerequisites

Before you begin, ensure you have:

✅ Java 17+ — Check with java -version
✅ Python 3.8+ — Check with python3 --version (optional, for Python SDK/demo)
✅ Rust 1.70+ — Check with rustc --version (optional, for Rust SDK)
✅ Git — For cloning the repository
✅ Web Browser — For viewing the interactive dashboard

Quick Verification:

java -version     # Should show 17 or higher
python3 --version # Should show 3.8 or higher
git --version     # Any recent version

🚀 5-Minute Quick Start

Experience the framework before integrating your own engine:

# 1. Clone and build
git clone https://github.com/IBM/substrait-compliance.git
cd substrait-compliance
sdk/java/gradlew shadowJar -p sdk/java

# 2. Run TPC-H demo (quick validation)
cd demo
./runner/run-simple-demo.sh                    # 22 TPC-H queries (~1-2 min)

# 3. Run comprehensive function tests (RECOMMENDED)
cd runner
./run-function-tests-python.sh                 # 5,000+ tests, 15 categories (~5-10 min)

# 4. View the interactive dashboard
cd ..
python3 -m http.server 8080

# 5. Open in your browser: http://localhost:8080

For additional testing:

# Enhanced TPC-H demo with detailed phases
cd demo && ./runner/run-enhanced-demo.sh

# Note: Java TPC-DS demo (99 queries) available but requires significant memory (8GB+)

Cold-start validation

Use this sequence to verify the public repo from a fresh clone:

# From repository root
sdk/java/gradlew shadowJar -p sdk/java

# Run tests
cd demo
./runner/run-simple-demo.sh              # TPC-H validation (~2 min)

cd runner
./run-function-tests-python.sh           # Comprehensive function tests (~10 min)

cd ..

For full SDK build verification, use the ordered checklist in docs/SDK_VERIFICATION.md.

Expected output (abridged):

📦 Loading TPC-H test suite...
✅ Loaded test suite: tpch
   Total test cases: 22

🔧 Testing: MockDB v1.0.0
   ✅ Passed: 22  ❌ Failed: 0   📊 Pass Rate: 100.0%

🔧 Testing: FastDB v2.5.0
   ✅ Passed: 21  ❌ Failed: 1   📊 Pass Rate: 95.5%

🔧 Testing: CloudDB v3.1.0
   ✅ Passed: 17  ❌ Failed: 5   📊 Pass Rate: 77.3%

🔧 Testing: DuckDB v0.10.0
   ✅ Passed: 14  ⏭️  Skipped: 8  📊 Pass Rate: 63.6%

🔧 Testing: PostgreSQL v16.0
   ✅ Passed: 14  ⏭️  Skipped: 8  📊 Pass Rate: 63.6%

🥇 MockDB       1.0.0   100.0%  🟢 VERIFIED
🥈 FastDB       2.5.0    95.5%  🟢 VERIFIED
🥉 CloudDB      3.1.0    77.3%  🔵 EDGE
🥉 DuckDB       0.10.0   63.6%  🟡 BASIC
🥉 PostgreSQL   16.0     63.6%  🟡 BASIC

✅ Demo completed successfully!

What You'll See in the Dashboard:

🥇 Leaderboard — Rankings with medals and tier badges (VERIFIED / EDGE / BASIC)
📊 Visual Charts — Bar chart and doughnut chart showing pass rates
📈 Detailed Statistics — Per-engine breakdowns and test case results
🔍 Query Drill-Down — Click any engine to see detailed query-level results
🏷️ Complexity Filtering — Filter tests by SIMPLE, MEDIUM, COMPLEX, VERY_COMPLEX

Troubleshooting:

If port 8080 is in use: python3 -m http.server 8081
If permission denied: chmod +x demo/runner/run-simple-demo.sh
Dashboard shows "Failed to load": ensure the demo ran successfully first

🔧 Test Your Own Engine

Step 1: Choose Your SDK

Java SDK (Recommended — Most Complete)

The SDK is not yet published to Maven Central or GitHub Packages. Build from source:

cd sdk/java
./gradlew build        # compile + run tests
./gradlew shadowJar    # produces build/libs/substrait-compliance-0.1.1-all.jar

# Expected output:
# BUILD SUCCESSFUL — 88 tests passed ✅

Verify build:

ls -la sdk/java/build/libs/
# substrait-compliance-0.1.1-all.jar   (fat jar with all dependencies)
# substrait-compliance-0.1.1.jar       (library jar, no deps bundled)

Python SDK (Pythonic Interface)

# Install the SDK and dev dependencies
cd sdk/python
python3 -m venv venv && source venv/bin/activate
./install-dev.sh   # sets index to PyPI, seeds build tools, installs editable

# Run tests to verify
pytest

# Expected output:
# 13 passed ✅

Verify Installation:

python3 -c "import substrait_compliance; print('SDK installed successfully!')"

C++ SDK (Native Performance)

# Build the SDK
cd sdk/cpp
cmake -S . -B build -DBUILD_TESTS=ON
cmake --build build --parallel

# Run tests to verify
ctest --test-dir build --output-on-failure

# Expected output:
# 100% tests passed ✅

Verify Installation:

ls -la libsubstrait_compliance.*
# Should see: libsubstrait_compliance.so (or .dylib on macOS, .dll on Windows)

Go SDK (Cloud-Native & Concurrent)

# Get the SDK
cd sdk/go
go mod download

# Run tests to verify
go test ./...

# Expected output:
# PASS
# ok      github.com/IBM/substrait-compliance/sdk/go

TypeScript/JavaScript SDK (Modern Web & Node.js)

# Install the SDK
cd sdk/typescript
npm install

# Build the SDK
npm run build

# Run tests to verify
npm test

# Expected output:
# PASS  tests/table-data.test.ts
# PASS  tests/runner.test.ts
# Test Suites: 2 passed, 2 total
# Tests:       17 passed, 17 total

Verify Installation:

ls -la dist/
# Should see: index.js, index.d.ts, and other compiled files

C#/.NET SDK (Enterprise & Cross-Platform)

# Build the SDK
cd sdk/csharp
dotnet restore
dotnet build

# Run tests to verify
dotnet test

# Expected output:
# Passed!  - Failed:     0, Passed:    20, Skipped:     0, Total:    20

Verify Installation:

ls -la bin/Debug/net10.0/
# Should see: Substrait.Compliance.dll and related files

Scala SDK (Functional & Type-Safe)

# Build the SDK
cd sdk/scala
sbt compile

# Run tests to verify
sbt test

# Expected output:
# [info] ComplianceEngineSpec:
# [info] - should return engine information
# [info] - should return engine capabilities
# [info] All tests passed ✅

Verify Installation:

ls -la target/scala-2.13/
# Should see: substrait-compliance_2.13-0.1.1.jar

Rust SDK (High Performance)

# Build the SDK
cd sdk/rust
cargo build --release

# Run tests to verify
cargo test

# Expected output:
# test result: ok. 22 passed ✅

Verify Installation:

ls -la target/release/
# Should see: libsubstrait_compliance.rlib

Step 2: Implement ComplianceEngine

Interface quick-reference

Every SDK exposes the same four required operations plus optional lifecycle hooks. The table below shows the exact names and signatures by language — use it as your checklist before writing any code.

	Java	Python	TypeScript	Rust	Go	C#	C++	Scala
Plan arg type	`Plan` (parsed protobuf)	`bytes`	`Uint8Array`	`&[u8]`	`[]byte`	`byte[]`	`vector<uint8_t>`	`Array[Byte]`
Execution model	Synchronous	Synchronous	`async`/`Promise`	Synchronous	`context.Context`	`async Task`	Synchronous	`Future`
Get metadata	`getEngineInfo()`	`get_info()`	`getInfo()`	`get_info()`	`GetInfo()`	`GetInfo()`	`get_info()`	`getInfo`
Get capabilities	`getCapabilities()`	`get_capabilities()`	`getCapabilities()`	`get_capabilities()`	`GetCapabilities()`	`GetCapabilities()`	`get_capabilities()`	`getCapabilities`
Execute plan	`executePlan(Plan, Map<String,TableData>)`	`execute_plan(bytes, Dict[str,TableData])`	`executePlan(Uint8Array, Map<string,TableData>)`	`execute_plan(&[u8], &HashMap<String,TableData>)`	`ExecutePlan(ctx, []byte, map[string]*TableData)`	`ExecutePlanAsync(byte[], IReadOnlyDictionary<string,TableData>)`	`execute_plan(vector<uint8_t>, TableCollection)`	`executePlan(Array[Byte], Map[String,TableData])`
Validate plan	`validatePlan(Plan)`	`validate_plan(bytes)`	`validatePlan(Uint8Array)`	`validate_plan(&[u8])`	`ValidatePlan(ctx, []byte)`	`ValidatePlanAsync(byte[])`	`validate_plan(vector<uint8_t>)`	`validatePlan(Array[Byte])`
Lifecycle init	`initialize()` (default no-op)	—	`initialize?()` (optional)	—	`Initialize(ctx)`	`InitializeAsync()` (default no-op)	`initialize()` (default no-op)	`initialize()` (default no-op)
Lifecycle cleanup	`cleanup()` (default no-op)	—	`shutdown?()` (optional)	—	`Shutdown(ctx)`	`ShutdownAsync()` (default no-op)	`shutdown()` (default no-op)	`shutdown()` (default no-op)

Key differences to be aware of:

Java receives a fully-parsed io.substrait.proto.Plan object — the SDK deserializes the protobuf before calling your engine. All other SDKs receive raw bytes and you deserialize inside execute_plan / executePlan.
TypeScript and Scala use Future-based async execution; the TypeScript runner also supports optional parallel execution (RunnerOptions.parallel). All other SDK runners execute tests sequentially.
Go passes a context.Context through every call — use it for cancellation and deadlines.
C++ uses vector<uint8_t> for plan bytes and a TableCollection typedef (unordered_map<string, TableData>) for input data.
Lifecycle: initialize is called once before the first test in a suite; cleanup/shutdown once after the last. They are not called per-test. You may keep open connections across test cases within a suite, but the engine must be stateless between test case executions (input data is passed fresh each time).

Java Example

public class MyEngine implements ComplianceEngine {
    @Override
    public EngineInfo getEngineInfo() {
        return new EngineInfo("MyEngine", "1.0.0", "MyCompany");
    }

    @Override
    public ComplianceResult executePlan(Plan plan, Map<String, TableData> inputData)
            throws ComplianceException {
        // Load data into your engine
        loadInputData(inputData);

        // Execute the Substrait plan (Plan is a parsed protobuf object)
        TableData output = executeSubstraitPlan(plan);

        return ComplianceResult.success(output, 0);
    }
}

Python Example

class MyEngine(ComplianceEngine):
    def get_info(self) -> EngineInfo:
        return EngineInfo("MyEngine", "1.0.0", "MyCompany")

    def execute_plan(self, plan_bytes: bytes, input_data: Dict[str, TableData]) -> ComplianceResult:
        # Load data into your engine
        self._load_input_data(input_data)

        # Execute Substrait plan
        output = self._execute_substrait_plan(plan_bytes)

        return ComplianceResult("test-id", TestStatus.PASSED, output_data=output)

Rust Example

impl ComplianceEngine for MyEngine {
    fn get_info(&self) -> EngineInfo {
        EngineInfo::new("MyEngine", "1.0.0", "MyCompany")
    }

    fn execute_plan(&self, plan_bytes: &[u8], input_data: &HashMap<String, TableData>)
        -> Result<ComplianceResult> {
        // Load data into your engine
        self.load_input_data(input_data)?;

        // Execute Substrait plan
        let output = self.execute_substrait_plan(plan_bytes)?;

        Ok(ComplianceResult::new("test-id", TestStatus::Passed).with_output(output))
    }
}

TypeScript Example

import { ComplianceEngine, EngineInfo, EngineCapabilities, ComplianceResult, TestStatus, TableData } from '@substrait/compliance';

class MyEngine implements ComplianceEngine {
    getInfo(): EngineInfo {
        return {
            name: 'MyEngine',
            version: '1.0.0',
            vendor: 'MyCompany',
            description: 'A high-performance query engine'
        };
    }

    getCapabilities(): EngineCapabilities {
        return {
            supportedRelations: ['read', 'filter', 'project', 'aggregate'],
            supportedFunctions: ['add', 'subtract', 'equal'],
            supportedTypes: ['i32', 'i64', 'string', 'boolean']
        };
    }

    async executePlan(planBytes: Uint8Array, inputData: Map<string, TableData>): Promise<ComplianceResult> {
        await this.loadInputData(inputData);
        const output = await this.executeSubstraitPlan(planBytes);
        return new ComplianceResult('test-id', TestStatus.PASSED, output, undefined, undefined, executionTimeMs);
    }
}

C# Example

using Substrait.Compliance;

public class MyEngine : IComplianceEngine
{
    public EngineInfo GetInfo()
    {
        return new EngineInfo(
            Name: "MyEngine",
            Version: "1.0.0",
            Vendor: "MyCompany",
            Description: "A high-performance query engine"
        );
    }

    public async Task<ComplianceResult> ExecutePlanAsync(
        byte[] planBytes,
        IReadOnlyDictionary<string, TableData> inputData)
    {
        await LoadInputDataAsync(inputData);
        var output = await ExecuteSubstraitPlanAsync(planBytes);
        return new ComplianceResult("test-id", TestStatus.Passed, output, null, null, executionTimeMs);
    }
}

Step 3: Run Against Test Suites

Java example (DuckDB):

cd examples/duckdb-java
./compile.sh
java -cp "build:../../sdk/java/build/libs/substrait-compliance-0.1.1-all.jar" \
    io.substrait.example.DuckDBComplianceExample

# Expected output (requires DuckDB JDBC with substrait extension):
# Loaded test suite: tpch
# Test cases: 22
# Pass Rate: N%   ← actual result depends on which plans your DuckDB version supports

Python example (DataFusion):

pip install datafusion datafusion-substrait pyarrow
cd examples/datafusion-python
python datafusion_compliance.py

# Expected output (when datafusion-substrait is installed):
# Loaded test suite: tpch
# Test cases: 22
# Pass Rate: N%   ← actual result; without the packages installed, tests report FAILED

Programmatic usage (Java):

// Load and run a test suite
YamlTestSuiteLoader loader = new YamlTestSuiteLoader();
TestSuite suite = loader.load("test-suites/tpch/metadata.yaml");

ComplianceRunner runner = new ComplianceRunner(new MyEngine());
ComplianceReport report = runner.runTestSuite(suite);

System.out.println("Pass Rate: " + report.getPassRate() + "%");
System.out.println("Passed: " + report.getPassedCount() + "/" + report.getTotalCount());

See examples/ for complete implementations.

Step 4: Verify Results

# Check generated reports
ls -la output/

# View report contents (requires jq)
cat output/your-engine-report.json | jq .

# Key fields in report:
# - engineInfo: { name, version, vendor }
# - testResults: [ { testId, status, executionTime } ]
# - summary: { total, passed, failed, skipped, passRate }
# - timestamp: ISO 8601 format

Step 5: Integrate with CI/CD (Optional)

Copy the template workflow to your repository:

cp .github/workflows/engine-compliance-template.yml \
   your-engine/.github/workflows/substrait-compliance.yml

Customize the workflow for your engine's build process (engine name, build commands, compliance threshold — default 80%).

See .github/workflows/README.md for details.

🧪 Test Suites

📚 Detailed Documentation:

test-suites/functions/README.md — Function tests (136 files, 5,041 test assertions, 14 categories)

test-suites/tpch/README.md — TPC-H benchmark (22 queries)

test-suites/tpcds/README.md — TPC-DS benchmark (99 queries)

TPC-H Benchmark (22 Queries)

Complete TPC-H benchmark at scale factor 0.01. Result correctness is fully verifiable — all 22 expected output files are committed and compared on every test run.

Component	Details
Queries	22 (Q1–Q22)
Data Files	8 CSV files
Total Rows	86,805
Data Size	~10.6 MB
Plan Formats	Binary (.bin) + JSON (.json)
Complexity Levels	SIMPLE, MEDIUM, COMPLEX, VERY_COMPLEX
Expected Outputs	✅ All 22 present (`expected/q01.csv` – `q22.csv`) — typed-header format

Query Complexity Breakdown:

SIMPLE (3 queries): Q1, Q6, Q14 — Single table, basic aggregations
MEDIUM (7 queries): Q3, Q4, Q10, Q12, Q13, Q16, Q19 — Joins, grouping
COMPLEX (8 queries): Q5, Q7, Q9, Q11, Q15, Q17, Q18, Q22 — Multiple joins, subqueries
VERY_COMPLEX (4 queries): Q2, Q8, Q20, Q21 — Nested subqueries, complex joins

# Navigate to TPC-H test suite
cd test-suites/tpch

# View test suite metadata
cat metadata.yaml

# Inspect test data (8 CSV files: customer, lineitem, nation, orders,
# part, partsupp, region, supplier)
ls -la data/

# View Substrait plans (44 files: q01.bin, q01.json … q22.bin, q22.json)
ls -la plans/

# View reference expected outputs (all 22 queries)
ls -la expected/

Running TPC-H tests (Java):

YamlTestSuiteLoader loader = new YamlTestSuiteLoader();
TestSuite tpchSuite = loader.load("test-suites/tpch/metadata.yaml");

ComplianceRunner runner = new ComplianceRunner(myEngine);
ComplianceReport report = runner.runTestSuite(tpchSuite);

System.out.println("TPC-H Results:");
System.out.println("  Total Queries: " + report.getTotalCount());
System.out.println("  Passed: " + report.getPassedCount());
System.out.println("  Pass Rate: " + report.getPassRate() + "%");

TPC-DS Benchmark (99 Queries)

TPC-DS (Decision Support) benchmark plans and data for complex analytical workloads:

Component	Details
Queries	99 (query01.sql – query99.sql)
Substrait Plans	198 (99 JSON + 99 binary)
Data Tables	24 CSV files (multi-channel retail schema)
Plan Formats	Binary (.bin) + JSON (.json)
Expected Outputs	✅ All 99 present — typed-header pipe-delimited CSV

Key Query Categories:

Customer behavior and profitability analysis
Multi-channel retail analytics (store, catalog, web)
Sales and returns analysis
Inventory management and marketing effectiveness

cd test-suites/tpcds
cat metadata.yaml
ls -la data/      # 24 CSV files
ls -la plans/     # 198 Substrait plan files (99 JSON + 99 binary)
ls -la expected/  # 99 reference output files (q01.csv – q99.csv)

📚 See test-suites/tpcds/README.md for complete TPC-DS documentation

Function Tests (136 Files, 5,041 Assertions)

The function test suite covers 14 semantic categories with 5,041 individual test assertions across 136 committed files:

aggregate/     (6 files)  — count, avg, stddev, variance, etc.
arithmetic/    (44 files) — add, multiply, sqrt, trigonometry, etc.
array/         (4 files)  — array operations
cast/          (1 file)   — cast, try_cast
comparison/    (19 files) — equal, gt, lt, between, coalesce, etc.
conditional/   (2 files)  — case, if
datetime/      (12 files) — date_trunc, date_diff, extract, etc.
geospatial/    (4 files)  — st_area, st_distance, st_contains, st_intersects
json/          (2 files)  — json_extract, json_parse
map/           (3 files)  — map operations
set/           (3 files)  — union, intersect, except
string/        (27 files) — concat, substring, regexp, etc.
struct/        (2 files)  — struct operations
window/        (7 files)  — row_number, rank, lag, lead, etc.

Running Function Tests (Python):

from substrait_compliance import YamlTestSuiteLoader, ComplianceRunner

loader = YamlTestSuiteLoader()
suite = loader.load("test-suites/functions/arithmetic/metadata.yaml")
runner = ComplianceRunner(my_engine)
report = runner.run_test_suite(suite)
print(f"Arithmetic Functions: {report.passed_count}/{report.total_count} passed")

Running Specific Tests

Single TPC-H query (Java):

TestSuite suite = loader.load("test-suites/tpch/metadata.yaml");

TestCase q1 = suite.getTestCases().stream()
    .filter(tc -> tc.getId().equals("q01"))
    .findFirst()
    .orElseThrow();

ComplianceResult result = myEngine.executePlan(q1.getPlan(), q1.getInputData());
System.out.println("Q1 Status: " + result.getStatus());

Filter by complexity (Java):

List<TestCase> simpleTests = suite.getTestCases().stream()
    .filter(tc -> tc.getComplexity() == Complexity.SIMPLE)
    .collect(Collectors.toList());

📦 Repository Structure

substrait-compliance/
├── 🎯 demo/                       # ⭐ START HERE — Interactive demo
│   ├── runner/
│   │   ├── run-simple-demo.sh     # ⭐ Run this first!
│   │   ├── run-demo.sh            # Full demo with dashboard
│   │   ├── run-enhanced-demo.sh   # Enhanced demo runner
│   │   └── run-function-tests.sh  # Function test runner
│   ├── dashboard/                 # Visual results dashboard (HTML/JS)
│   ├── engines/                   # Example mock engines (Java)
│   │   ├── MockDBEngine.java
│   │   ├── FastDBEngine.java
│   │   ├── CloudDBEngine.java
│   │   ├── DuckDBEngine.java
│   │   └── PostgreSQLEngine.java
│   └── output/                    # Generated reports
├── 📚 sdk/                        # Multi-language SDKs (8 languages)
│   ├── java/                      # ⭐ Most complete — JDK 17+, Gradle
│   ├── python/                    # Python 3.8+, pip/setuptools
│   ├── rust/                      # Rust 2021 edition, Cargo
│   ├── go/                        # Go 1.21+, modules
│   ├── cpp/                       # C++17, CMake 3.15+
│   ├── typescript/                # TypeScript/Node.js, npm
│   ├── csharp/                    # C#/.NET 10+, dotnet
│   └── scala/                     # Scala 2.13, sbt
├── 🧪 test-suites/                # Test suites
│   ├── functions/                 # 136 function test files, 5,041 assertions (14 categories)
│   ├── tpch/                      # TPC-H (22 queries, 8 data files, 44 plans)
│   └── tpcds/                     # TPC-DS (99 queries, 24 data files, 198 plans)
├── 💡 examples/                   # Integration examples — duckdb-java and duckdb-cpp execute plans; see examples/README.md
│   ├── datafusion-python/         # DataFusion integration (Python)
│   ├── datafusion-rust/           # DataFusion integration (Rust)
│   ├── duckdb-cpp/                # DuckDB integration (C++)
│   ├── duckdb-java/               # DuckDB integration (Java)
│   └── velox-cpp/                 # Velox integration (C++)
├── 🌐 api/                        # REST API (Spring Boot)
│   ├── src/                       # API source code
│   ├── docker-compose.yml         # Container orchestration
│   └── build.gradle               # Gradle build
├── 🤖 .github/workflows/          # CI/CD automation
│   ├── engine-compliance-template.yml  # ⭐ Template for your engine
│   ├── sdk-build-test.yml         # Tests all SDKs on every commit
│   ├── test-suite-validation.yml  # Validates test suite integrity
│   └── ...                        # API build, deploy, release workflows
├── 🔧 scripts/                    # Automation and quality tools
│   ├── generate_leaderboard.py    # Leaderboard generator
│   ├── quality_checker.py         # Test quality checker
│   ├── test_enhancer.py           # Test enhancement tool
│   └── verify_sdk_builds.sh       # SDK build verification
└── 📖 docs/                       # Documentation
    ├── REST_API_GUIDE.md
    ├── REST_API_ARCHITECTURE.md
    ├── DEPLOYMENT_GUIDE.md
    ├── PERFORMANCE_BENCHMARKING.md
    ├── SDK_VERIFICATION.md
    └── SUBSTRAIT_COMPLIANCE_FRAMEWORK_GUIDE.md

Key Directories for Developers:

🎯 demo/ — Run the interactive demo first to see the framework in action
📚 sdk/ — 8 SDKs: C++, C#, Go, Java, Python, Rust, Scala, TypeScript
🧪 test-suites/ — 136 function test files (5,041 assertions) + TPC-H + TPC-DS benchmarks
💡 examples/ — Real integration examples with DuckDB, DataFusion, and Velox

🏗️ Architecture

Decentralized Compliance Model

┌─────────────────────────────────────────────────────────────┐
│         Compliance Framework (Standards Authority)           │
│  • SDKs (8 languages)                                        │
│  • Test Suites (TPC-H, TPC-DS, Functions)                    │
│  • Documentation & CI/CD Templates                           │
└─────────────────────────────────────────────────────────────┘
                          ↓ downloads & implements
┌─────────────────────────────────────────────────────────────┐
│              Query Engines (Self-Certify)                    │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐             │
│  │  DuckDB    │  │ DataFusion │  │   Spark    │  ...        │
│  │  Java SDK  │  │ Python SDK │  │  Java SDK  │             │
│  └────────────┘  └────────────┘  └────────────┘             │
└─────────────────────────────────────────────────────────────┘
                          ↓ reports results (optional)
┌─────────────────────────────────────────────────────────────┐
│         Public Compliance Leaderboard (GitHub Pages)         │
│  • Rankings by pass rate                                     │
│  • Detailed results per engine                               │
│  • Historical trends                                         │
└─────────────────────────────────────────────────────────────┘

🔧 SDK Features

Java SDK

JDK: 17+ | Build: Gradle | Tests: 88 (unit + TPC-H pass-through integration)
Key Classes: ComplianceEngine, ComplianceRunner, YamlTestSuiteLoader, EngineCapabilities
Docs: sdk/java/README.md

Python SDK

Python: 3.8+ | Build: pyproject.toml / install-dev.sh | Tests: 13 (unit + pass-through integration)
Key Modules: engine.py, runner.py, loader.py, result.py
Docs: sdk/python/README.md

C++ SDK

Standard: C++17 | Build: CMake 3.15+ | Tests: 30 (Google Test)
Key Headers: engine.h, runner.h, loader.h, result.h, comparator.h
Features: Smart pointers, zero-copy operations, header-only option, cross-platform
Docs: sdk/cpp/README.md

Go SDK

Go: 1.21+ | Build: Go modules | Tests: 24 (pass-through + CSV loader integration)
Key Files: engine.go, runner.go, loader.go, result.go, table_data.go

Rust SDK

Edition: 2021 | Build: Cargo | Tests: 22 (unit + benchmark + pass-through integration)
Key Modules: engine.rs, runner.rs, loader.rs, result.rs

TypeScript SDK

Runtime: Node.js / Browser | Build: npm | Tests: 17 (table-data unit + runner pass-through integration)
Docs: sdk/typescript/README.md

C# SDK

Runtime: .NET 10+ | Build: dotnet | Tests: 20 (TableData unit + runner pass-through integration)
Docs: sdk/csharp/README.md

Scala SDK

Scala: 2.13 | Build: sbt | JVM: 17+ | Tests: 26
Key Files: ComplianceEngine.scala, ComplianceRunner.scala, TestSuiteLoader.scala, EngineResult.scala
Key Type: EngineResult separates engine execution from test results
Docs: sdk/scala/README.md

SDK compliance-alignment status

The table below reflects the state after all gap-closure work. Every SDK with ✅ in all four columns can produce honest compliance scores when given a test suite with expected output.

SDK	Runner compares output	Type-aware comparator	Loader carries expected output	Pass-through integration test
Java	✅ full, epsilon 1e-9	✅ `normalizeType` + `valuesMatch`	✅ typed CSV headers	✅ TPC-H 22/22 proven
Python	✅ full, `isclose` 1e-9	✅ `_normalize_type` + `_values_match`	✅ YAML `expected_output` field	✅ 5 pass-through tests
TypeScript	✅ `ResultComparator` in runner	✅ float 1e-10, NaN, Date	✅ `expectedOutput` from YAML	✅ 5 pass-through tests
C#	✅ `ResultComparator` in runner	✅ double 1e-10, float 1e-6, DateTime	✅ `ExpectedOutput` from YAML	✅ 5 pass-through tests
Go	✅ `compareOutputs()` in runner	✅ `valuesMatch` epsilon 1e-9	✅ `Load()` + `LoadCSV()` implemented	✅ 5 pass-through tests
Rust	✅ full value comparison	✅ `normalize_type` + `values_match` epsilon 1e-9	✅ reads `expected/<id>.csv`	✅ 4 pass-through tests
C++	✅ full, epsilon 1e-9	✅ `normalizeType` + `valuesMatch`	✅ typed CSV headers + YAML loader	✅ 5 pass-through tests
Scala	✅ full, epsilon 1e-9	✅ `normalizeType` + `valuesMatch`	✅ `expectedOutput` from YAML	✅ 5 pass-through tests

⚡ Performance Benchmarking

All SDKs include benchmarking components to measure engine performance:

Statistical Analysis: Min, Max, Avg, Median, P95, P99 latencies
Throughput Metrics: Operations per second
Warmup Support: Configurable warmup runs before measurement
CSV Export: Export results for external analysis

Available in all SDKs:

Java — sdk/java/src/main/java/io/substrait/compliance/benchmark/
Python — sdk/python/substrait_compliance/benchmark/benchmark_runner.py
Rust — sdk/rust/src/benchmark/mod.rs
Go — sdk/go/benchmark/benchmark_runner.go
C++ — sdk/cpp/include/substrait_compliance/benchmark_runner.hpp
TypeScript — sdk/typescript/src/benchmark/BenchmarkRunner.ts
C# — sdk/csharp/Substrait.Compliance/Benchmark/BenchmarkRunner.cs
Scala — sdk/scala/src/main/scala/io/substrait/compliance/benchmark/

See docs/PERFORMANCE_BENCHMARKING.md for detailed usage and output format specifications.

🤖 CI/CD Integration

Automated Workflows

sdk-build-test.yml — Tests all SDKs on every commit
test-suite-validation.yml — Validates test suite integrity
api-build-test.yml — API build and test
api-container-build.yml — Multi-platform container images
api-deploy-staging.yml / api-deploy-production.yml — Staged deployments
release-publish.yml — Automated versioning and publishing

For Engine Developers

Copy the template workflow to enable automated compliance testing in your own repository:

cp .github/workflows/engine-compliance-template.yml \
   .github/workflows/substrait-compliance.yml

Customize: engine name/version, build commands, test execution, compliance threshold (default 80%).

See .github/workflows/README.md for a description of every workflow file.

🔍 Troubleshooting

Common Issues and Solutions

"Permission denied" on shell scripts

chmod +x demo/runner/run-simple-demo.sh
chmod +x demo/verify-setup.sh
./demo/runner/run-simple-demo.sh

"Java version not supported"

java -version   # Should show 17 or higher

# macOS (Homebrew)
brew install openjdk@17

# Ubuntu/Debian
sudo apt update && sudo apt install openjdk-17-jdk

# Windows — download from https://adoptium.net/

"Dashboard shows 'Failed to load data'"

# Run the demo first to generate data
cd demo
./runner/run-simple-demo.sh

# Verify files exist
ls -la demo/output/leaderboard.json
ls -la demo/dashboard/data/leaderboard.json

# Serve over HTTP (not file://)
cd demo/dashboard
python3 -m http.server 8080
# Open http://localhost:8080

"Port 8080 already in use"

python3 -m http.server 8081   # or 9000, 3000, etc.

# macOS/Linux: kill the blocking process
lsof -ti:8080 | xargs kill -9

"SDK build fails"

# Java: clean rebuild
cd sdk/java
./gradlew clean build --refresh-dependencies

# Python: reinstall
cd sdk/python
pip uninstall substrait-compliance -y
./install-dev.sh   # re-seeds build tools and re-installs as editable local install

# Rust: clean rebuild
cd sdk/rust
cargo clean && cargo build --release

"Test data not found"

ls -la test-suites/tpch/data/   # Should show 8 CSV files
ls -la test-suites/tpch/plans/  # Should show 44 plan files

# If files are missing
git pull origin main

"Module not found" errors (Python)

cd sdk/python
source venv/bin/activate   # activate the venv created during install
./install-dev.sh           # re-seeds build tools and re-installs locally (editable install)
python3 -c "import substrait_compliance; print('OK')"

"Gradle build fails with 'Could not resolve dependencies'"

rm -rf ~/.gradle/caches/
cd sdk/java
./gradlew clean build --refresh-dependencies

"Dashboard shows but no data appears"

# Check browser console (F12) for errors
# Most common causes:
# 1. Opened as file:// — use a web server instead
cd demo/dashboard && python3 -m http.server 8080

# 2. JSON file malformed
cat demo/dashboard/data/leaderboard.json | python3 -m json.tool

Getting Help

Check Documentation — demo/README.md, examples/README.md
Review Examples — Working implementations in examples/ and demo/engines/
Enable Debug Logging (Java): System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "debug")
Report Issues — GitHub Issues · GitHub Discussions

✅ Verification Checklist

# Automated verification
cd demo && ./verify-setup.sh

Manual Steps:

# 1. Prerequisites
java -version    # 17+
python3 --version # 3.8+

# 2. Repository structure
ls -la demo/ sdk/ test-suites/ examples/
ls -la test-suites/tpch/data/*.csv    # 8 files
ls -la test-suites/tpch/plans/        # 44 files

# 3. Demo execution
cd demo && ./runner/run-simple-demo.sh
# Expected: 5 engines tested; summary shows 100%/95.5%/77.3%/63.6%/63.6%; "Demo completed successfully!"

# 4. Report generation
ls -la demo/output/   # mockdb/fastdb/clouddb/duckdb/postgresql-report.json + leaderboard.json
cat demo/output/leaderboard.json | python3 -m json.tool

# 5. Dashboard
cd demo/dashboard && python3 -m http.server 8080 &
curl -s http://localhost:8080 | grep -q "Substrait Compliance" && echo "OK"

# 6. SDK build (pick at least one)
cd sdk/java && ./gradlew build test    # BUILD SUCCESSFUL, 88 tests
cd sdk/python && python3 -m venv venv && source venv/bin/activate && ./install-dev.sh && pytest  # 13 passed
cd sdk/rust && cargo build --release && cargo test  # 22 passed

Success Criteria:

✅ All prerequisites installed (Java 17+, Python 3.8+)
✅ Demo runs without errors; reports generated in demo/output/
✅ Dashboard accessible at http://localhost:8080; shows 5 engines
✅ At least one SDK builds and its tests pass

📈 Compliance Leaderboard

Run the demo to generate leaderboard data from the five demo engines, then view it in the interactive dashboard. Results include fidelity tier badges (VERIFIED ≥ 90% / EDGE ≥ 70% / BASIC ≥ 50%) assigned from TPC-H pass rates.

Generate and view the leaderboard:

cd demo && ./runner/run-simple-demo.sh
cd dashboard && python3 -m http.server 8080
# Open http://localhost:8080

To generate a standalone leaderboard JSON from existing reports:

python3 scripts/generate_leaderboard.py

🌐 REST API

A Spring Boot REST API provides programmatic access to compliance results.

Features: JWT authentication, report submission, data querying with filtering/pagination, webhooks, rate limiting, caching.

# Start with Docker Compose
cd api
docker-compose up -d postgres

# In a separate terminal, start the API application
./gradlew bootRun

# Access the API
curl http://localhost:8080/api/v1/leaderboard

Pre-built container image (published on every push to main):

docker pull ghcr.io/ibm/substrait-compliance/substrait-compliance-api:latest

Documentation:

docs/REST_API_GUIDE.md — Complete API reference
docs/REST_API_ARCHITECTURE.md — Architecture and design
docs/API_IMPLEMENTATION.md — API implementation guide
docs/DEPLOYMENT_GUIDE.md — Deployment instructions
api/README.md — API module readme

Note: The REST API is pre-release — the code is in-repo and builds cleanly, but it is not deployed to a public endpoint and the auth/storage configuration is not production-hardened. It is provided for evaluation and self-hosted use only. See api/README.md for current status.

📖 Key Documentation

🎯 Getting Started

README.md — This file
demo/README.md — Interactive demo guide
demo/DASHBOARD_GUIDE.md — Dashboard features and navigation
demo/TROUBLESHOOTING.md — Demo troubleshooting
examples/README.md — Integration examples

🧪 Test Suites

test-suites/README.md — Test suites overview
test-suites/functions/README.md — Function tests
test-suites/tpch/README.md — TPC-H benchmark
test-suites/tpcds/README.md — TPC-DS benchmark

📚 SDK Documentation

sdk/java/README.md — Java SDK
sdk/python/README.md — Python SDK
sdk/rust/README.md — Rust SDK
sdk/go/README.md — Go SDK
sdk/cpp/README.md — C++ SDK
sdk/typescript/README.md — TypeScript SDK
sdk/csharp/README.md — C# SDK
sdk/scala/README.md — Scala SDK

🔧 Development & Quality

docs/PERFORMANCE_BENCHMARKING.md — Benchmarking guide
docs/SDK_VERIFICATION.md — SDK verification guide
docs/SUBSTRAIT_COMPLIANCE_FRAMEWORK_GUIDE.md — Comprehensive technical guide
scripts/README.md — Scripts documentation
scripts/TEST_ENHANCEMENT_GUIDE.md — Test enhancement workflow
.github/workflows/README.md — CI/CD workflow documentation
CHANGELOG.md — Version history

🤝 Contributing

CONTRIBUTING.md — Contribution guidelines
CODE_OF_CONDUCT.md — Community code of conduct
GOVERNANCE.md — Project governance

🛠️ Development

Building from Source

Java SDK:

cd sdk/java && ./gradlew build test

Python SDK:

cd sdk/python && python3 -m venv venv && source venv/bin/activate && ./install-dev.sh && pytest

Rust SDK:

cd sdk/rust && cargo build --release && cargo test

Running Examples

Note: duckdb-java and duckdb-cpp execute plans for real (JDBC substrait() and native from_substrait() respectively). datafusion-python executes when datafusion-substrait is installed. datafusion-rust and velox-cpp have the wiring in place but execution is not yet called. See examples/README.md for the full status table.

DuckDB (Java):

cd examples/duckdb-java && ./compile.sh
java -cp "build:../../sdk/java/build/libs/substrait-compliance-0.1.1-all.jar" \
    io.substrait.example.DuckDBComplianceExample

DataFusion (Python):

pip install datafusion datafusion-substrait pyarrow
cd examples/datafusion-python && python datafusion_compliance.py

DuckDB (C++):

cd examples/duckdb-cpp
cmake -S . -B build
cmake --build build --parallel
./build/example

DataFusion (Rust):

cd examples/datafusion-rust && cargo run

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Areas for Contribution

Additional test suites and test cases
New engine integration examples
SDK improvements and new language SDKs
Documentation enhancements
Bug fixes and optimizations

📊 Project Statistics

Metric	Value
SDKs	8 (Java, Python, Rust, Go, C++, TypeScript, C#, Scala)
Test Suites	3 (TPC-H, TPC-DS, Functions)
Function Test Files	136 (14 categories)
Function Test Assertions	5,041 individual test cases
TPC-H Queries	22
TPC-DS Queries	99
TPC-H Data Rows	86,805 (scale factor 0.01)
TPC-DS Data Tables	24
CI/CD Workflows	12
Example Implementations	5 (DuckDB Java/C++, DataFusion Python/Rust, Velox C++)
REST API Endpoints	10+

📜 License

This project is licensed under the Apache License 2.0 — see the LICENSE file for details.

🙏 Acknowledgments

Substrait Community — For the query plan specification
TPC Organization — For the TPC-H and TPC-DS benchmark queries
Engine Developers — For implementing Substrait support

📞 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: docs/

🗺️ Roadmap

See ROADMAP.md for the full phased roadmap with timelines.

Recently completed:

REST API infrastructure with Spring Boot
Comprehensive CI/CD workflows
Interactive demo system with dashboard
Multi-platform container builds
TPC-DS benchmark (99 queries)

Up next:

Performance benchmarking CI integration
Compliance badges
Spring Boot 3.x migration for REST API

Made with ❤️ for the Substrait Community

Name		Name	Last commit message	Last commit date
Latest commit History 217 Commits
.github		.github
api		api
demo		demo
docs		docs
examples		examples
scripts		scripts
sdk		sdk
test-implementations		test-implementations
test-suites		test-suites
.gitignore		.gitignore
.oss-gitignore		.oss-gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
GOVERNANCE.md		GOVERNANCE.md
LICENSE		LICENSE
MAINTAINERS.md		MAINTAINERS.md
NOTICE		NOTICE
README.md		README.md
SECURITY.md		SECURITY.md
settings.gradle		settings.gradle

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Substrait Compliance Framework

📑 Table of Contents

🎯 Overview

Key Features

🔧 Prerequisites

🚀 5-Minute Quick Start

Cold-start validation

🔧 Test Your Own Engine

Step 1: Choose Your SDK

Step 2: Implement ComplianceEngine

Interface quick-reference

Step 3: Run Against Test Suites

Step 4: Verify Results

Step 5: Integrate with CI/CD (Optional)

🧪 Test Suites

TPC-H Benchmark (22 Queries)

TPC-DS Benchmark (99 Queries)

Function Tests (136 Files, 5,041 Assertions)

Running Specific Tests

📦 Repository Structure

🏗️ Architecture

Decentralized Compliance Model

🔧 SDK Features

Java SDK

Python SDK

C++ SDK

Go SDK

Rust SDK

TypeScript SDK

C# SDK

Scala SDK

SDK compliance-alignment status

⚡ Performance Benchmarking

🤖 CI/CD Integration

Automated Workflows

For Engine Developers

🔍 Troubleshooting

Common Issues and Solutions

"Permission denied" on shell scripts

"Java version not supported"

"Dashboard shows 'Failed to load data'"

"Port 8080 already in use"

"SDK build fails"

"Test data not found"

"Module not found" errors (Python)

"Gradle build fails with 'Could not resolve dependencies'"

"Dashboard shows but no data appears"

Getting Help

✅ Verification Checklist

📈 Compliance Leaderboard

🌐 REST API

📖 Key Documentation

🎯 Getting Started

🧪 Test Suites

📚 SDK Documentation

🔧 Development & Quality

🤝 Contributing

🛠️ Development

Building from Source

Running Examples

🤝 Contributing

Areas for Contribution

📊 Project Statistics

📜 License

🙏 Acknowledgments

📞 Support

🗺️ Roadmap

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers