Add part recover-broken command for recovering parts from s3

[otselnik]

Summary by Sourcery

Introduce a CLI command and internal tooling to recover broken ClickHouse MergeTree parts stored on S3 and export their data to TSV, including health checks, reconstruction logic, and integration tests.

New Features:

• Add chadmin part recover-broken command to repair detached MergeTree parts using S3 metadata and output recovered data as TSV.
• Implement a part recovery pipeline that classifies S3-backed part files, checks blob health, reconstructs missing metadata, and uses a running ClickHouse server to read recovered data.
• Provide JSON recovery reports summarizing missing blobs, broken columns, reconstructed files, and recovered row counts.

Enhancements:

• Add helpers to resolve part ownership, build temporary scratch tables on local disks, and safely attach recovered parts for data extraction via ClickHouse.
• Extend test ClickHouse helpers and Behave steps to manipulate parts and S3 blobs for recovery scenarios, including Wide and Compact part formats.

Tests:

• Add end-to-end BDD scenarios covering wide and compact parts, dry-run mode, critical metadata loss handling, and full/partial recovery outcomes.
• Introduce test utilities for corrupting column and data.bin blobs, copying parts to detached directories, and validating recovered output and reports.

[sourcery-ai]

Reviewer's Guide

Adds a new chadmin part recover-broken command and a full part-recovery subsystem that reconstructs MergeTree parts with missing S3 blobs by checking S3 objects, reassembling files locally, and using a scratch ClickHouse table to export recovered data to TSV, along with comprehensive BDD tests and helpers.

Sequence diagram for the new part recover-broken command

sequenceDiagram
    actor Admin
    participant part_group_recover_broken_command as recover_broken_part_command
    participant Recover as recover_broken_part
    participant S3 as S3Service
    participant CH as ClickhouseServer

    Admin->>part_group_recover_broken_command: chadmin part recover-broken
    part_group_recover_broken_command->>Recover: recover_broken_part(client,disk_conf,part_path,output_tsv,...)

    note over Recover: Scan part and collect blob keys
    Recover->>Recover: scan_blob_keys(part_path,disk_conf)
    Recover->>S3: check_blobs_parallel(disk_conf,all_blob_keys)
    S3-->>Recover: blob_status

    Recover->>Recover: classify(part_path,blob_status,disk_conf)
    alt dry_run
        Recover-->>part_group_recover_broken_command: RecoveryReport (dry-run)
    else perform recovery
        Recover->>Recover: download_part_files(disk_conf,part_files,assembled_dir)
        Recover->>Recover: _copy_local_files(assembled_dir,part_files)
        Recover->>Recover: _reconstruct_files(assembled_dir,part_files,...)
        Recover->>CH: run_server_recovery(client,part_path,assembled_dir,...)
        CH-->>Recover: rows,tsv_labels
        Recover->>Recover: build_report(part_path,part_files,rows,tsv_columns)
        Recover-->>part_group_recover_broken_command: RecoveryReport
    end

    part_group_recover_broken_command-->>Admin: TSV output file & optional report

Loading

File-Level Changes

Change	Details	Files
Introduce chadmin part recover-broken CLI command wired to a new part recovery pipeline that reads ClickHouse config, validates paths, and orchestrates recovery via recover_broken_part.	Register recover-broken subcommand under part_group with options for part path, S3 disk, output TSV, tmp dir, threads, dry-run, report, and deprecated force flag. Load S3 disk configuration from ClickHouse config and bucket prefix, validate the detached part path, and construct paths for output, tmp, and report. Create a ClickHouse client from context, invoke recover_broken_part with all parameters, log a summary from the returned report, and map CriticalLossError to process exit code 2.	ch_tools/chadmin/cli/part_group.py
Implement orchestrator and helpers for recovering MergeTree parts by classifying part files, checking S3 blobs, reconstructing metadata, and running server-side extraction into TSV.	Add recover_broken_part orchestrator that scans a part directory for S3 metadata, checks blob health in parallel, classifies files, computes broken columns, performs dry-run or real recovery, reconstructs metadata, runs server-side recovery, and writes a JSON report. Create classify module to categorize part files (critical/meta/data/mark/index) using naming regexes, parse S3 metadata, decide actions (download/zero-fill/drop/reconstruct/mark-broken), and enforce critical loss rules, including Compact vs Wide parts and columns.txt validation. Add s3_blobs module to build boto3 S3 clients, HEAD-check blobs in parallel using process pool, and download/concatenate blobs for each file into assembled part directories. Add reconstruct module to derive row counts from .mrk2 files, zero-fill stub files, generate default meta content, reconstruct count.txt, and synthesize or stub missing meta files like default_compression_codec.txt and metadata_version.txt. Add report module to aggregate recovery statistics (file health, broken columns, reconstructed/dropped files, rows recovered, TSV columns) and serialize them to JSON. Define CriticalLossError for unrecoverable conditions such as missing critical meta or Compact data.bin loss, and expose recover_broken_part from the package init.	ch_tools/chadmin/internal/part_recovery/recover.py ch_tools/chadmin/internal/part_recovery/classify.py ch_tools/chadmin/internal/part_recovery/s3_blobs.py ch_tools/chadmin/internal/part_recovery/reconstruct.py ch_tools/chadmin/internal/part_recovery/report.py ch_tools/chadmin/internal/part_recovery/exceptions.py ch_tools/chadmin/internal/part_recovery/__init__.py
Add server-side runner that uses a scratch MergeTree table on a local disk to ATTACH the reconstructed part and stream TSV output via ClickHouse HTTP.	Resolve source (database, table) from part path via UUID and system.tables, merge schema from system.columns and columns.txt, and parse ORDER BY / PARTITION BY from DDL. Select a local disk from system.disks, create a scratch database/table _chadmin_recover._recover_<rand> excluding broken columns, and validate broken columns are not used in key expressions. Copy the assembled part into the scratch table's detached/ directory, chown to ClickHouse user, ATTACH PART with validated part name, and build a SELECT that substitutes NULLs for broken columns while preserving types. Stream SELECT ... FORMAT TSV to the output file via HTTP, count rows, and compute TSV column labels with (NULL) markers; always drop the scratch table in a finally block. Expose disk-related helpers such as listing disk names by type, and building table data paths from system.tables.	ch_tools/chadmin/internal/part_recovery/server_runner.py
Extend test steps and ClickHouse helpers to corrupt S3-backed parts, manage detached/broken copies, and parse S3 metadata inside containers.	Add Behave step definitions to corrupt per-column blobs, copy parts to detached/broken_* without corruption, and corrupt Compact data.bin blobs, storing the broken part path in context and a helper file inside the container. Implement helper functions to locate active part paths via system.parts, copy parts into detached/broken_<part>, list column-related files, and read S3 metadata keys from container files using S3ObjectLocalMetaData and disk prefix handling. Add helper to write the broken part path into /tmp/broken_part_path inside the ClickHouse container for shell-based tests.	tests/steps/chadmin.py tests/modules/clickhouse.py
Add BDD feature scenarios validating chadmin part recover-broken behaviour for Wide and Compact parts, dry-run mode, and critical loss handling.	Add feature to recover Wide parts with missing column blobs, asserting full row count, broken column reporting, and successful exit codes. Add dry-run scenario that ensures no output TSV is created while command succeeds. Add scenario asserting missing columns.txt triggers exit code 2 (critical loss). Add scenarios for full recovery when all blobs are healthy for Wide and Compact parts, validating row counts and absence of broken columns in report. Add scenario for Compact parts with missing data.bin blobs, asserting exit code 2. Wire scenarios to existing test infrastructure with S3, ZooKeeper, and ClickHouse fixtures.	tests/features/chadmin_part_recovery.feature

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 3 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="ch_tools/chadmin/internal/part_recovery/server_runner.py" line_range="503-507" />
<code_context>
+    # Use streaming HTTP response to avoid loading all data into memory
+    response = client.query(sql, stream=True)
+
+    rows = 0
+    with output_path.open("wb") as fh:
+        for chunk in response.iter_content(chunk_size=65536):
+            if chunk:
+                fh.write(chunk)
+                rows += chunk.count(b"\n")
+
</code_context>
<issue_to_address>
**suggestion:** Row counting via newline bytes can undercount the last line and is slightly misleading.

Because `rows` is incremented with `chunk.count(b"\n")`, it will miss the final line when the file doesn't end with a newline and will count empty trailing lines as rows. Since this is only for reporting, consider either making it clear this is an approximate count (e.g. renaming to `approx_rows`) or obtaining the exact row count from ClickHouse (e.g. via a `COUNT()` query) so the reported value matches the logical row count.

Suggested implementation:

```python
    Returns the approximate number of rows written (based on newline-delimited output).

```

```python
    # Use streaming HTTP response to avoid loading all data into memory
    response = client.query(sql, stream=True)

    # Approximate row count based on newline-delimited output; may undercount the final line
    # if the file does not end with a newline and will include trailing empty lines.
    approx_rows = 0
    with output_path.open("wb") as fh:
        for chunk in response.iter_content(chunk_size=65536):
            if chunk:
                fh.write(chunk)
                approx_rows += chunk.count(b"\n")

```

```python
    return approx_rows

```
</issue_to_address>

### Comment 2
<location path="ch_tools/chadmin/internal/part_recovery/server_runner.py" line_range="346-103" />
<code_context>
+
+    client = clickhouse_client(ctx)
+
+    try:
+        report = recover_broken_part(
+            client=client,
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Chown logic may miss AttributeError on platforms without os.lchown.

`place_part_in_detached` currently catches only `PermissionError` and `OSError` around `os.lchown`, but on platforms/Python builds without `os.lchown`, an `AttributeError` may be raised instead. Please also handle `AttributeError` (or guard with `hasattr(os, "lchown")`) to prevent crashes in those environments.
</issue_to_address>

### Comment 3
<location path="ch_tools/chadmin/internal/part_recovery/classify.py" line_range="207-212" />
<code_context>
+
+    client = clickhouse_client(ctx)
+
+    try:
+        report = recover_broken_part(
+            client=client,
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Swallowing all metadata parsing errors in scan_blob_keys can hide genuine issues.

`scan_blob_keys` currently catches and ignores `ValueError`, `IndexError`, and `OSError` from `S3ObjectLocalMetaData.from_file`, treating any parse or I/O failure as "no blobs". This can mask real metadata or filesystem issues. Please consider at least emitting a debug log (including the filename and exception) on parse failure to aid diagnosis while preserving the current control flow.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[aalexfvk]

Integration with data-store detect-broken-partitions would be useful. Or describe scenario of using it appropriately.