Home

Durak: The High-Performance Standard for Turkish NLP

Durak is a high-throughput, type-safe NLP engine designed for production-grade Turkish text processing. Built on a "Rust Core, Python Interface" architecture, it combines the developer ergonomics of Python with the raw speed and memory safety of compiled Rust.

Unlike legacy wrappers or pure Python scripts, Durak is engineered to bypass the Global Interpreter Lock (GIL), enabling true multi-core parallelism for heavy workloads such as LLM pre-training, massive corpus cleaning, and real-time inference pipelines.

Why Durak?

The "Iron Core" Engine: Critical linguistic operations (normalization, suffix stripping, dictionary lookups) are offloaded to _durak_core, a compiled Rust backend. This results in 10-100x speedups over pure Python implementations.

Research-Grade Precision: We prioritize Offset Mapping. Every token preserves its start and end character indices relative to the raw text, making Durak safe for Named Entity Recognition (NER) alignment and token classification tasks.

Native Turkish Support: Proper handling of the Turkish dotted/undotted I/ı and İ/i distinction is baked into the normalization layer, not patched with regex.

PyTorch-Style API: Pipelines are composable. Build your preprocessing flow using a modular, class-based design familiar to deep learning researchers.

No JVM Required: Get Zemberek-like morphological capabilities without the overhead of the Java Virtual Machine.

Installation

Durak is available on PyPI and supports Python 3.9+ on Linux, macOS, and Windows.

pip install durak-nlp

Quick Start

Durak uses a composable pipeline architecture. Define your steps, then process data in batches.

import durak

# 1. Define the Pipeline
# Components are backed by Rust for performance
pipeline = durak.Pipeline([
    durak.Normalizer(lowercase=True),    # Handles Turkish I/ı correctly
    durak.Tokenizer(strategy="regex"),   # Rust-backed regex engine
    durak.Lemmatizer(strategy="hybrid")  # Hybrid Lookup + Heuristic stripping
])

# 2. Process a single document
text = "İSTANBUL'da hava çok güzel."
result = pipeline(text)
# Output: "istanbul hava güzel" (depending on lemmatization depth)

# 3. High-Throughput Batch Processing
# Durak releases the GIL, allowing effective parallelism
corpus = ["Metin 1...", "Metin 2...", "Metin 3..."] * 1000
results = list(pipeline.pipe(corpus, batch_size=5000))

Project Status

Current Version: v0.3.0 (Alpha)

The project is currently finalizing the migration of all legacy Python logic to the Rust core. Expect significant performance improvements in every minor release.