cmd/gc: support external sort

[zxh326]

Signed-off-by: Xuhui zhang xuhui@juicedata.io

[codecov]

Codecov Report

❌ Patch coverage is 44.52555% with 76 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.89%. Comparing base (801e8df) to head (1198a76).
⚠️ Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/meta/tkv.go	26.66%	17 Missing and 5 partials ⚠️
pkg/meta/redis.go	31.81%	12 Missing and 3 partials ⚠️
pkg/meta/sql.go	30.00%	11 Missing and 3 partials ⚠️
pkg/utils/extsort/sharded.go	72.91%	9 Missing and 4 partials ⚠️
pkg/chunk/cached_store.go	29.41%	11 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #7230       +/-   ##
===========================================
+ Coverage   33.58%   54.89%   +21.30%     
===========================================
  Files         137      139        +2     
  Lines       48007    48835      +828     
===========================================
+ Hits        16123    26807    +10684     
+ Misses      30267    19118    -11149     
- Partials     1617     2910     +1293     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds an external-sort-based GC mode to juicefs gc to reduce memory usage on large volumes by streaming slice/object records through disk-backed sorters instead of loading all slices into memory. It also extends the metadata engine interface to support streaming slice scans.

Changes:

• Add --work-dir flag to juicefs gc to enable an external sort pipeline for GC (plus supporting merge/scan helpers and tests).
• Introduce Meta.ScanSlices (Redis/SQL/TKV implementations) to stream slice records via callback, while keeping ListSlices as a wrapper.
• Add a small generic external sort wrapper (pkg/utils/extsort) and tests; plus an opt-in env flag to probabilistically skip object deletes.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/utils/extsort/sharded.go	New generic wrapper around external sorter with temp-dir lifecycle management.
pkg/utils/extsort/sharded_test.go	Unit tests for the new external sorter wrapper.
pkg/meta/interface.go	Add ScanSlices streaming API to Meta interface.
pkg/meta/redis.go	Implement ScanSlices and make ListSlices a wrapper.
pkg/meta/sql.go	Implement ScanSlices and make ListSlices a wrapper.
pkg/meta/tkv.go	Implement ScanSlices and make ListSlices a wrapper (with callback error handling).
cmd/gc.go	Add --work-dir external-sort GC path and helper functions for scanning/merging records.
cmd/gc_test.go	Update GC test fixture layout to match hashed-prefix chunk paths; add external sort coverage.
cmd/gc_external_test.go	Add focused tests for external sort merge and chunk object scanning.
pkg/chunk/cached_store.go	Add JFS_OBJECT_DELETE_SKIP behavior to optionally skip object deletes.
go.mod / go.sum	Add lanrat/extsort dependency with replace to juicedata/extsort.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jiefenghuang]

mark:

1. 相同代码抽象
2. gc主函数抽象: 初始化, 预处理(删除/回调注册), 扫描对比(外排或者全内存), 后处理, 输出结果

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cmd/gc.go:79

• This adds a user-facing GC mode, but the existing English and Chinese GC guides (docs/en/administration/status_check_and_maintenance.md and docs/zh_cn/administration/status_check_and_maintenance.md) do not mention the flag, its disk-space requirements, or how to select a work directory. Please document the new mode in both guides.

				Name:  "external-sort-dir",
				Usage: "working directory for external sort temporary files (enables external sort mode)",

pkg/utils/extsort/sharded.go:107

• The cleanup error is discarded, so repeated GC runs can report success while leaving potentially very large external-sort shards on disk. Return the RemoveAll failure from Done (while preserving any sort error) and propagate both Done results from gcExternalSort.

	_ = os.RemoveAll(s.workDir)

[jiefenghuang]

TODO:

1. ScanSlice中不要耦合cleanup, 单独抽出来一个接口.
2. 新实现接口获取异步删除清理任务是否结束. 开了delflag的话, gc要清理完待删除slice, 并且不影响外排.