restore: carry struct-level nullable through schema conversion - #1188
restore: carry struct-level nullable through schema conversion#1188huanghaoyuanhhy wants to merge 2 commits into
Conversation
Milvus validates nullable structs at the struct level: a nullable struct propagates the flag to every sub-field, and sub-fields must not be nullable unless their struct is. The backup proto's StructArrayFieldSchema had no nullable field, so the flag was dropped during backup and restoring a nullable-struct collection sent sub-fields marked nullable under a non-nullable struct, which the target cluster rejects. Store the struct-level flag in the backup meta and rebuild it on restore with the server's rule, deriving it from nullable sub-fields for backups written before this field existed. Signed-off-by: huanghaoyuanhhy <haoyuan.huang@zilliz.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: huanghaoyuanhhy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@huanghaoyuanhhy Please associate the related issue to the body of your Pull Request. (eg. “issue: #”) |
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (44.10%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1188 +/- ##
==========================================
+ Coverage 43.14% 44.10% +0.95%
==========================================
Files 134 136 +2
Lines 12552 12662 +110
==========================================
+ Hits 5416 5585 +169
+ Misses 6731 6664 -67
- Partials 405 413 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
364e6c7 to
2456121
Compare
Signed-off-by: huanghaoyuanhhy <haoyuan.huang@zilliz.com>
2456121 to
53b75f8
Compare
3 participants
Background
Milvus keeps nullable authoritative at the struct level of an array-of-struct field: it is rejected when set per sub-field and instead propagated onto every sub-field by proxy before persisting. Because
DescribeCollectionreturns those propagated child flags verbatim while struct-level handling depends on every consumer keepingStructArrayFieldSchemaintact, any conversion that models structs as flat field lists loses the parent bit and re-created collections fail validation (sub-field in non-nullable struct cannot be nullable individually, ...).The asymmetry itself is tracked upstream at milvus-io/milvus#52924 — this PR is deliberately a downstream compatibility pass so backup behaves like the server does, not a statement about where the UX fix belongs.
Changes
core/proto: addbool nullable = 5toStructArrayFieldSchema, regenerate generated codecore/backup: carry the struct-level flag verbatim from DescribeCollectioncore/restore(primary DDL path + secondary conv path): rebuild it per server rule via sharedconv.ApplyStructNullable; derive it from nullable children for backup metas written before this field existedTests
Unit tests cover all three meta shapes (struct flag set → propagated; legacy child-only flags → repaired; both unset → untouched) plus the describe wiring. Full build and golangci-lint clean.
Related to milvus-io/milvus#52924