columnar: gc ia meta & schema files in cloud_helper

[yongman]

What problem does this PR solve?

Issue Number: close #10995

Problem Summary:
The disk or memory can be keep increasing in the long run with no background gc.

What is changed and how it works?

1. Add gc for ia_meta files in disk. (In TiKV there has a background gc task to scan the local meta files and do the gc)
2. Add gc for schema_files map in memory.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• New Features

  • Added configuration-driven background cleanup for IA metadata and schema-file caching.
  • Introduced new controls for cleanup frequency, IA metadata capacity, and the maximum number of cached keyspaces.
  • Cache eviction now prefers recently accessed keyspaces and removes stale schema versions automatically.

• Bug Fixes

  • Improved shutdown behavior to ensure background cleanup and heartbeat-related tasks stop reliably.
  • Refreshed cache recency on snapshot access to keep cleanup decisions accurate.

• Tests

  • Extended coverage for stale pruning, recency updates, and GC behavior with and without keyspace limits.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign zanmato1984 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 2194e121-999f-4ba2-9507-64b5e994a188

📥 Commits

Reviewing files that changed from the base of the PR and between f7f5c20 and 97f3f53.

📒 Files selected for processing (1)

• contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs

---

📝 Walkthrough

Walkthrough

Schema-file cache recency tracking and global eviction are added to CloudHelper. Runtime configuration now controls IA metadata and schema cache GC, with a background thread, coordinated shutdown, and updated defaults.

Changes

Runtime garbage collection

Layer / File(s)	Summary
Schema cache tracking and eviction contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs	Adds per-keyspace cache state, recency updates, stale-version pruning, configurable keyspace eviction, and unit tests.
Snapshot access recency integration contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs	Propagates shared cache state through snapshot loading and touches the active keyspace after snapshot creation.
Configurable background GC lifecycle contrib/tiflash-columnar-hub/hub-runtime/src/run.rs	Adds GC configuration, starts periodic IA metadata and schema cache cleanup, coordinates shutdown, and joins the background thread.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EngineStoreLifecycle
  participant BackgroundGCLoop
  participant CloudHelper
  participant IAMetaFiles
  EngineStoreLifecycle->>BackgroundGCLoop: start when gc_interval is non-zero
  BackgroundGCLoop->>IAMetaFiles: run IA metadata GC
  BackgroundGCLoop->>CloudHelper: gc_schema_file_cache()
  CloudHelper->>CloudHelper: prune and evict schema-file entries
  EngineStoreLifecycle->>BackgroundGCLoop: set service_shutdown
  BackgroundGCLoop-->>EngineStoreLifecycle: exit and join

Loading

Poem

I’m a rabbit with cache in my burrow tonight,
Touching warm keyspaces, pruning old bytes from sight.
IA files hop cleanly, the GC loops with care,
Shutdown bells ring softly through the runtime air.
~( ,_,_,/)
( •ㅅ• )

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 24.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately reflects the main change: background GC for ia meta and schema files.
Description check	✅ Passed	The description covers the issue, problem, changes, tests, side effects, docs, and release note, but the commit-message block is empty.
Linked Issues check	✅ Passed	The changes implement background GC for on-disk ia_meta files and in-memory schema_files, matching issue #10995.
Out of Scope Changes check	✅ Passed	The added config and shutdown/GC loop refactors support the GC feature and do not appear unrelated to the issue.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs`:
- Around line 1390-1399: Update the cache-state insertion in the
latest_by_keyspace loop to capture the current access_seq and assign it to
last_access_seq for newly discovered keyspaces instead of 0. Keep existing
entries’ latest_version updates unchanged, so freshly loaded keyspaces are
treated as recently accessed during gc_schema_file_cache_global eviction.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 9faf5786-67f3-4e03-b576-cc762b795186

📥 Commits

Reviewing files that changed from the base of the PR and between 3d986e3 and f7f5c20.

📒 Files selected for processing (2)

• contrib/tiflash-columnar-hub/hub-runtime/src/cloud_helper.rs
• contrib/tiflash-columnar-hub/hub-runtime/src/run.rs

[yongman]

/test pull-integration-test