Release v0.1.11: Add support for stripping LLM trailing dot truncation pattern

Introduces new content cleaning logic in Layer 1 to handle JSON responses truncated when an LLM hits `max_output_tokens` and pads the remainder with dots (e.g., Gemini API).

**Key Features:**

- **Trailing Dots Stripping**: New public function `ContentCleaning.strip_trailing_dots/1` detects 10+ consecutive trailing dots, ignoring whitespace interspersed between them.
- **Pipeline Integration**: Applied in Layer 1 pipeline before encoding normalization.
- **Real-World Fix**: Addresses output from Gemini API when structured JSON output is abruptly terminated by token limits.

**Details:**

- New module functions implemented for robust Grapheme-based detection and counting.
- Preserves legitimate ellipsis (`...`) within string values.
- Updated CHANGELOG, README, and related documentation files.
- Test coverage increased with dedicated unit and integration tests for this pattern.

Author: nshkrdotcom

CHANGELOG.md

@@ -7,6 +7,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.11] - 2025-12-03
+
+### Added
+- **Trailing dots truncation handling**: Layer 1 now detects and strips trailing dots from truncated JSON content
+  - Pattern: `{"key": "value.............` → `{"key": "value`
+  - Handles Gemini API `max_output_tokens` truncation pattern where remaining tokens are filled with dots
+  - Threshold: 10+ consecutive trailing dots trigger cleanup (preserves legitimate ellipsis)
+  - Works with mixed whitespace/newlines between dots
+  - **Test status**: ✅ 14/14 new tests passing
+- **New public API function**: `ContentCleaning.strip_trailing_dots/1` for direct access to trailing dots removal
+
+### Technical Details
+- **Module**: Enhanced `JsonRemedy.Layer1.ContentCleaning`
+- **New functions**:
+  - `strip_trailing_dots/1` - Public API for trailing dots removal
+  - `strip_trailing_dots_internal/1` - Pipeline integration function
+  - `find_trailing_dots_start/1` - Position detection
+  - `find_dots_at_end/1` - End-of-string dot detection
+  - `count_trailing_dots/4` - Dot counting with whitespace tolerance
+- **Integration**: Runs in Layer 1 pipeline after content extraction, before encoding normalization
+- **Performance**: Grapheme-based processing for proper UTF-8 handling
+
+### Documentation
+- **Test case reference**: Based on real Gemini API response in `docs/20251203/test_case/gemini_max_tokens_trailing_dots.md`
+- **Layer 1 enhancements**: Added trailing dots stripping to content cleaning pipeline
+
 ## [0.1.10] - 2025-10-28
 
 ### Fixed
@@ -353,7 +379,8 @@ This is a **100% rewrite** - all previous code has been replaced with the new la
 - Minimal memory overhead (< 8KB for repairs)
 - All operations pass performance thresholds
 
-[Unreleased]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.10...HEAD
+[Unreleased]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.11...HEAD
+[0.1.11]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.10...v0.1.11
 [0.1.10]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.9...v0.1.10
 [0.1.9]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.8...v0.1.9
 [0.1.8]: https://github.com/nshkrdotcom/json_remedy/compare/v0.1.7...v0.1.8

CLAUDE.md

@@ -22,16 +22,18 @@ JsonRemedy - A practical, multi-layered JSON repair library for Elixir that inte
 - ✅ Comment stripping (// and /* */)
 - ✅ Wrapper text extraction (HTML, prose)
 - ✅ Encoding normalization
+- ✅ Trailing dots truncation (`test/unit/layer1_trailing_dots_test.exs`) *(v0.1.11+)*
 
-**Implementation Status**: **TDD COMPLETE** 
-- ✅ Core functionality implemented (21/21 unit tests passing)
+**Implementation Status**: **TDD COMPLETE**
+- ✅ Core functionality implemented (35/35 unit tests passing)
 - ✅ LayerBehaviour contract fully implemented
 - ✅ All required callbacks: `process/2`, `supports?/1`, `priority/0`, `name/0`, `validate_options/1`
-- ✅ Public API functions: `strip_comments/1`, `extract_json_content/1`, `normalize_encoding/1`
+- ✅ Public API functions: `strip_comments/1`, `extract_json_content/1`, `normalize_encoding/1`, `strip_trailing_dots/1`
 - ✅ Context-aware processing that preserves string content
 - ✅ Performance tests passing (4/4 tests, all functions under performance thresholds)
 - ✅ Code quality checks passing (Credo, mix format)
 - ✅ Type specifications and documentation complete
+- ✅ **Trailing dots handling** (v0.1.11): Detects and strips Gemini max_output_tokens truncation pattern
 
 ### Phase 3: Layer 2 - Structural Repair ✅ COMPLETED  
 **Goal**: Fix missing/extra delimiters using state machine for context tracking
@@ -242,7 +244,7 @@ State machine approach preserves JSON semantics:
 - **✅ Production Ready**: Core 3-layer pipeline battle-tested and robust
 
 ### 📊 **Current Statistics**
-- **Total Test Suite**: 449 tests passing, 0 failures (36 excluded)
+- **Total Test Suite**: 660 tests passing, 0 failures (63 excluded)
 - **Critical Tests**: 82 tests passing, 0 failures (19 excluded)
 - **Layer 4 Tests**: 201 tests passing, 0 failures
 - **Unit Tests**: 100+ tests across all layers
@@ -266,10 +268,10 @@ The core 4-layer repair pipeline is **production-ready** and **battle-tested**:
 - **Type Safety Excellence**: Zero Dialyzer warnings with comprehensive type checking
 - **Defensive Programming**: Robust nil input handling throughout the codebase
 - **Code Quality Standards**: Zero warnings, comprehensive documentation, production-ready code
-- **Test Coverage Excellence**: 449 total tests with 100% success rate
+- **Test Coverage Excellence**: 660 total tests with 100% success rate
 
 ## Key Commands
-- `mix test` - Run all tests (449 tests, 0 failures)
+- `mix test` - Run all tests (660 tests, 0 failures)
 - `mix test test/critical` - Run critical test suite (82 tests)
 - `mix test test/layer4` - Run Layer 4 validation tests (201 tests)
 - `mix test test/unit/layer1_content_cleaning_test.exs` - Run Layer 1 tests

README.md

@@ -45,6 +45,11 @@ Malformed JSON is everywhere in real-world systems:
 {"status": "processing", "results": [{"id": 1, "name": "Alice"
 ```
 
+```text
+// Gemini API max_output_tokens truncation (fills rest with dots)
+{"title": "Report", "content": "Analysis of.............................
+```
+
 ```text
 // Human input with common mistakes
 {name: Alice, "age": 30, "scores": [95 87 92], active: true,}
@@ -67,9 +72,12 @@ Runs **before** the main layer pipeline to handle complex patterns that would ot
 ### 🧹 **Content Cleaning (Layer 1)**
 - **Code fences**: ````json ... ```` → clean JSON
 - **Comments**: `// line comments` and `/* block comments */` → removed
-- **Hash comments**: `# python-style comments` → removed  
+- **Hash comments**: `# python-style comments` → removed
 - **Wrapper text**: Extracts JSON from prose, HTML tags, API responses
 - **Trailing text removal**: `[{"id": 1}]\n1 Volume(s) created` → `[{"id": 1}]` *(v0.1.3+)*
+- **Trailing dots truncation**: `{"key": "val............` → `{"key": "val` *(v0.1.11+)*
+  - Handles Gemini API `max_output_tokens` truncation pattern
+  - Strips 10+ consecutive dots while preserving legitimate ellipsis
 - **Encoding normalization**: UTF-8 handling and cleanup
 
 ### 🏗️ **Structural Repairs (Layer 2)**
@@ -160,7 +168,7 @@ Add JsonRemedy to your `mix.exs`:
 ```elixir
 def deps do
   [
-    {:json_remedy, "~> 0.1.10"}
+    {:json_remedy, "~> 0.1.11"}
   ]
 end
 ```

docs/20251203/test_case/fix-trailing-dots.md

@@ -0,0 +1,18 @@
+Read the test case at docs/20251203/test_case/gemini_max_tokens_trailing_dots.md - this is real broken JSON from Gemini API when it hits max_output_tokens and fills the rest with dots.
+
+Tasks:
+1. Create a test case in the test suite for this trailing dots truncation pattern
+2. Devise a plan to enhance JsonRemedy to handle this (strip trailing dots, repair truncated JSON)
+3. Implement the fix with a working example
+4. Bump mix.exs version x.y.z -> x.y.(z+1)
+5. Add CHANGELOG.md entry for the new version dated 2025-12-03
+6. Update README.md to document this capability
+7. Update other docs as needed
+
+Success criteria - ALL must pass:
+- mix compile --warnings-as-errors
+- mix test
+- mix dialyzer
+- mix credo --strict
+
+Do not stop until all criteria pass.

docs/20251203/test_case/gemini_max_tokens_trailing_dots.md

@@ -0,0 +1,48 @@
+# Task: Handle Gemini Max Tokens Trailing Dots Truncation
+
+## Context
+
+See `gemini_max_tokens_trailing_dots.md` in this directory for a real-world test case where Gemini API hits `max_output_tokens` and fills the remaining output with dots (`...`) instead of completing the JSON.
+
+The JSON is cut off mid-object like:
+```json
+{
+  "excerpt": "-  Weight loss.......................................................................................
+```
+
+And continues with 14,192 dots until the response ends.
+
+## Requirements
+
+1. **Create a test case** in the test suite based on the truncated JSON in `gemini_max_tokens_trailing_dots.md`
+
+2. **Enhance JsonRemedy** to detect and handle this pattern:
+   - Detect trailing dots (`.{3,}` at end of string)
+   - Strip trailing dots
+   - Attempt to repair the truncated JSON (close open strings, arrays, objects)
+   - Return the repaired JSON with as much valid data as possible
+
+3. **Create a working example** demonstrating the fix
+
+4. **Bump version** in `mix.exs` (increment patch version x.y.z -> x.y.(z+1))
+
+5. **Update CHANGELOG.md** with entry for the new version dated 2025-12-03:
+   - Describe the new trailing dots truncation handling feature
+
+6. **Update README.md** if needed to document this capability
+
+7. **Update any other docs** as needed
+
+## Success Criteria
+
+- [ ] No compilation warnings
+- [ ] All tests pass (`mix test`)
+- [ ] Dialyzer passes (`mix dialyzer`)
+- [ ] Credo clean (`mix credo --strict`)
+
+## Implementation Notes
+
+- The dots appear when Gemini hits token limits during structured output
+- The truncation can happen mid-string, mid-array, or mid-object
+- Priority is recovering as much valid data as possible
+- The `citations` array in the test case is incomplete - we want to preserve the citations that ARE complete