restore secondary: refuse early when the backup lacks its full meta - #1185
restore secondary: refuse early when the backup lacks its full meta#1185czs007 wants to merge 1 commit into
Conversation
A secondary restore broadcasts DDL on the source's control channel and replays the source's flush-all messages per pchannel. Those fields live only in meta/full_meta.json. When that file is absent -- a backup copied between buckets with the file left behind is the case that surfaced this -- meta.Read silently falls back to the per-level files, the collection list loads fine, and the restore fails at its first broadcast with "stream: no pch in message", which points nowhere near the cause. - meta.Read now logs a warning when it falls back to the per-level files, naming the file it expected. - restore secondary checks up front that the backup carries a control channel, a pchannel list and flush-all messages, and refuses with a message that names meta/full_meta.json and the two ways it ends up missing. - The database-creation error said "broadcast create collection"; it now says which database failed. - checkTargetIsUnused reported every database-not-found as an INFO line with the full error object attached, once per collection, which reads as a failure. It now reports once per database, without the error object, and reserves a warning for errors that are not the expected not-found. Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: czs007 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@czs007 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”) |
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (43.51%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1185 +/- ##
==========================================
+ Coverage 43.36% 43.51% +0.15%
==========================================
Files 135 135
Lines 12631 12670 +39
==========================================
+ Hits 5477 5513 +36
- Misses 6743 6746 +3
Partials 411 411 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
issue: #1186
A secondary restore broadcasts DDL on the source's control channel and replays
the source's flush-all messages per pchannel. Those fields live only in
meta/full_meta.json. When that file is absent,meta.Readsilently falls backto the per-level meta files: the collection list still loads, the preflight
checks still run, and the restore fails at its first broadcast with
stream: no pch in message, which points nowhere near the cause.The same shape occurs when the backup itself never recorded them -- a backup
taken with
--strategy=skip_flushormeta_only(or the deprecated--force)records no flush point by design. A plain restore accepts such a backup; a
secondary restore cannot, because it has no position in the source's stream to
start replication from.
Reported on a 2.6 deployment: an 88-line restore log where the only error was
the final
no pch in message, preceded by twenty-odd INFO lines that read likefailures. The backup's
full_meta.jsonwas present and held all 66 collectionsand 3,435 segments, but none of the three cluster-level fields.
Changes
meta.Readlogs a warning when it falls back to the per-level files, namingthe file it expected.
restore secondarychecks up front that the backup carries a control channel,a pchannel list and flush-all messages, and refuses with a message that names
meta/full_meta.jsonand both ways it ends up missing. This runs before anyRPC, so it costs nothing and fires before the target is touched.
broadcast create collection; it now saysbroadcast create database <name>.checkTargetIsUnusedreported every database-not-found as an INFO line withthe full error object attached, once per collection. On the reported restore
that was twenty-odd lines that looked like failures while being the expected
state. It now reports once per database without the error object, and reserves
a warning for errors that are not the expected not-found.
Verification
Unit tests cover the new check for both the API and metastore spellings plus the
negative cases, and the not-found classification.
End to end on a 2.6 cluster, with the released binary as the control:
gofmt,golangci-lintv2.12.2 (the CI version) over the whole tree,go vet,go build ./...andgo testare clean.