Skip to content

Adopt zarr-metadata + add strict Zarr v3 validation (Core/Extra)#173

Draft
d-v-b wants to merge 89 commits into
zarr-developers:mainfrom
d-v-b:element-spec-records
Draft

Adopt zarr-metadata + add strict Zarr v3 validation (Core/Extra)#173
d-v-b wants to merge 89 commits into
zarr-developers:mainfrom
d-v-b:element-spec-records

Conversation

@d-v-b

@d-v-b d-v-b commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Integrates the zarr-metadata package (>=0.3.0) as the source of truth for Zarr metadata types, and adds strict Zarr v3 validation in two families (Core / Extra). Hand-rolled type definitions that duplicated zarr-metadata are removed.

This is a breaking release. Every breaking change is captured in a changelog fragment under changes/ and in the migration guide.

What's new (additive)

  • Strict v3 specs from pydantic_zarr.v3:
    • Core — CoreArraySpec, per-dtype classes (CoreInt32ArraySpec, CoreFloat64ArraySpec, … 15 dtypes), AnyCoreArraySpec discriminated union, CoreGroupSpec. Restricts chunk grids to regular and codecs to the seven core-spec types.
    • Extra — ExtraArraySpec + mirrors; additionally accepts rectilinear chunk grids and scale_offset / cast_value codecs.
    • Both enforce fill_value ↔ data_type coupling (e.g. float64 accepts "NaN", int64 does not) and reject unknown dtypes / codec names.
  • Validation the loose specs don't do: per-codec/grid internal checks (transpose is a permutation, blosc/gzip level in range, chunk shapes positive), array-dimensionality consistency (chunk_grid / dimension_names / sharding match the array's rank), and codec-pipeline type-flow (cast_value scalars must be dtype-correct for their pipeline position).
  • Typed builders importable directly from pydantic_zarr.v3: bytes_codec, blosc_codec, transpose_codec, sharding_indexed_codec, … plus regular_grid / rectilinear_grid and default_chunk_key_encoding / v2_chunk_key_encoding. Each returns a validated metadata dict.
  • Serialization: to_json() on v2/v3 ArraySpec/GroupSpec; to_store_json() on v2.
  • Ergonomic construction: .create() (sensible defaults for every field), strict .from_array(), and create_with_sharding(outer_chunk_shape=…, inner_chunk_shape=…).

Breaking changes

Removed exports (replaced by zarr-metadata)

NamedConfig/AnyNamedConfig, RegularChunking(Config), Default/V2ChunkKeyEncoding(Config), the FillValue union + *FillValue aliases, MemoryOrder, DimensionSeparator, CodecDict. Full replacement table in the migration guide.

Loose-spec behavioral changes

Field names on the loose ArraySpec/GroupSpec are unchanged, but adopting zarr-metadata changed some field types and validation:

Change Before After
Sequence fields on the model list tuple (v2 filters, v3 chunk_shape) — to_json() output is unchanged (still JSON arrays)
Bare-string named-configs (v3) rejected accepted (chunk_grid="regular", etc.)
fill_value / dtype type narrow unions zarr_metadata.JSONValue / zarr-metadata dtype types (loose specs still validate as syntax only)

Stricter fill_value validation

The Core/Extra strict specs reject invalid fill_values that were previously accepted (non-hex float strings, out-of-range / non-whole integers, booleans for integer dtypes, malformed complex components, out-of-range raw bytes).

Migration

docs/migration.md covers every removed symbol + replacement, the loose-spec behavioral changes, the new serialization methods, and the Core/Extra strict families. New dependency: zarr-metadata>=0.3.0.

Testing

  • Full suite green (~19.9k tests, 6 pre-existing zarr-python skips); mypy --strict, ruff, and mkdocs build --strict all clean.
  • Strict validation is covered by an independent spec-derived oracle + Hypothesis differential property tests (the oracle imports neither pydantic_zarr nor zarr_metadata, so it's a genuine cross-check). The bare-string-form rule is derived mechanically from the metadata TypedDicts and guarded by a drift test.
  • Doc examples in usage_zarr_v3.md are executed as tests.

🤖 Generated with Claude Code

d-v-b and others added 30 commits June 19, 2026 15:39
@d-v-b @claude
docs: add zarr-metadata integration design spec
40b6be6 
Design for adopting zarr-metadata>=0.3.0 in stable v2/v3: typed
to_json/to_store_json serialization, replacing hand-rolled internal
types, and an opt-in strict ArraySpec (per-dtype discriminated union
coupling data_type with fill_value).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: adopt breaking-changes posture in integration spec
f13b90c 
Significant release: drop back-compat shims (remove NamedConfig export
in favor of NamedConfigV3 and other replaced exports), add migration
guide to the docs plan. ArraySpec remains loose-by-default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: record raw-dtype hybrid-union caveat in spec
9e1dd31 
raw r<N> is a NewType(str), not a Literal, so it can't be a
Field(discriminator) member. Strict spec is a hybrid: discriminated
union over literal-named dtypes, smart-unioned with the raw member.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: add zarr-metadata integration implementation plan
e47bf10 
10-task TDD plan: dependency, v3/v2 type replacement, typed
to_json/to_store_json, _BaseArraySpec extraction, StrictArraySpec
(hybrid discriminated+raw union), drift guard, docs/migration, full gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: resolve v2 store-type contradiction in plan (Mapping, not Typed…
8d92381 
…Dict)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
build: add zarr-metadata>=0.3.0 dependency
557b4c7 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v3): replace hand-rolled metadata types with zarr-metadata (…
eee5488 
…BREAKING)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs(v3): update ArraySpec docstring NamedConfig -> MetadataV3
a175e75 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: add StrictGroupSpec (recursive strict members) to spec + plan
5b37967 
User-requested scope addition. StrictGroupSpec.members must recursively
be StrictArraySpec/StrictGroupSpec; group's own fields unchanged. Inserted
as Task 8; drift-guard/docs/gate renumbered to 9/10/11. Mechanism verified.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(v3): add typed to_json returning ArrayMetadataV3/GroupMetadataV3
36e03b7 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v2): replace hand-rolled metadata types with zarr-metadata (…
4c33471 
…BREAKING)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(v2): add typed to_json and to_store_json
ecd8766 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(v2): cover GroupSpec.to_json and to_store_json
dd63c06 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v3): extract behavior-only _BaseArraySpec for strict/loose s…
8243313 
…iblings

Move all shared fields (node_type, attributes, shape, storage_transformers,
dimension_names) and all methods (validate_dimension_names, model_dump,
to_json, from_array, from_zarr, to_zarr, like) to a new _BaseArraySpec base
class. ArraySpec becomes a thin concrete subclass that only declares the five
variant fields (data_type, chunk_grid, chunk_key_encoding, fill_value, codecs).

The cls(...) calls in from_array/from_zarr pass variant fields as keyword
arguments with # type: ignore[call-arg] because mypy cannot see the subclass
fields when cls is typed as type[Self]=type[_BaseArraySpec]; the ignore is
minimal and load-bearing. ruff PIE804 required the dict-unpacking workaround
to be abandoned in favour of direct kwargs + suppression.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v3): move field-dependent factories to ArraySpec, drop type:…
01dadd7 
… ignore

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v3): use public AnyArraySpec in ArraySpec.like signature
5f0012d 
Avoid leaking the private _BaseArraySpec into the public like() signature;
matches the GroupSpec.like/AnyGroupSpec convention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(v3): add StrictArraySpec coupling data_type with per-dtype fill_…
8786afc 
…value

Implements _strict_v3.py with 15 private per-dtype array spec classes
(_BoolArraySpec through _RawArraySpec), each pairing the zarr-metadata
<X>DataTypeName Literal with <X>FillValue, plus strict chunk_grid,
chunk_key_encoding, and codec validation.  StrictArraySpec is a hybrid
union: a discriminated union of the 14 Literal-dtype specs plus
_RawArraySpec (NewType, cannot be a discriminator member).  Re-exported
from v3.py at the bottom to avoid circular import.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(v3): add StrictGroupSpec with recursive strict members
5f8f3e6 
StrictGroupSpec is a concrete NodeSpec subclass whose members field
accepts only StrictArraySpec or StrictGroupSpec values, propagating
dtype+fill_value strictness through the entire hierarchy. Also adds a
default for attributes in _StrictBase so array specs can be used without
explicitly supplying an empty attributes mapping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
fix(v3): strict specs inherit required generic attributes (drop narro…
c29cc04 
…wing override)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: strict specs are non-generic (attributes/members not parameteri…
31a8c32 
…zed)

Decided 2026-06-19: strict per-dtype array members and StrictGroupSpec
drop TAttr/TItem; attributes: Mapping[str, object] = {}; strict group
members fixed to StrictArraySpec | StrictGroupSpec. Loose specs keep generics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(v3): make strict specs non-generic with default attributes
605ad55 
Strict per-dtype array members and StrictGroupSpec drop TAttr/TItem;
attributes: Mapping[str, object] = {}. Loose specs keep generics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: record strict-vs-loose relationship analysis (keep discriminate…
5c67d9c 
…d union)

Examined 'every strict array is also a loose array': the per-dtype
discriminated union already expresses the data_type<->fill_value
correlation with static precision, and strict fill-value types are
verified subtypes of JSONValue (value-level substitutability). Rejected
inheritance (narrowing fails mypy), runtime-only enforcement (loses
static precision), and Protocol+drop-base (cost > benefit). Keep current
sibling design. No code change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: confirm non-generic strict after re-examining generic-attributes
d4c81b8 
Re-examined per maintainer request: generic strict forces StrictArraySpec[TAttr]
as a generic union alias (mypy rejects unparametrized union of generic
members). Kept non-generic; typed attributes available via loose ArraySpec[MyAttrs].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(v3): guard loose/strict shared-field drift
bd87925 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: strict/loose usage, serialization, and migration guide
d792923 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: 'both' strict design - constructible StrictArraySpec + AnyStric…
e88f6ca 
…tArraySpec union

StrictArraySpec becomes a single constructible class (weak fill_value
annotation + model_validator runtime coupling). Public per-dtype classes
(Float64ArraySpec etc.) give static precision. AnyStrictArraySpec is the
discriminated union / validation target. StrictGroupSpec.members ->
AnyStrictArraySpec | StrictGroupSpec. Mechanism verified.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
plan: add Task 12 — rework strict to 'both' design
4da9991 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(v3): 'both' strict design — constructible StrictArraySpec + publ…
8753eec 
…ic per-dtype + AnyStrictArraySpec

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: update strict docs for 'both' design; fix stale generic-types link
d64be2c 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: Core/Extra strict families design (supersedes 'both' naming)
9bcef7a 
Two strict vocabulary levels: Core (core spec only) and Extra (core +
zarr extensions), differing in chunk_grid (regular vs +rectilinear) and
codecs (7 vs 9 + known-name-literal strings only, no arbitrary str).
Naming: ArraySpec stays loose default; Core*/Extra* for strict (no Strict
prefix). Approach A: plain non-generic per-dtype classes (~30). Replaces
interim Strict* names.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
d-v-b and others added 27 commits June 21, 2026 12:39
@d-v-b @claude
feat(strict): blosc codec module (clevel range check)
50675e3 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): transpose codec module (permutation + ndim_of)
2922796 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): sharding_indexed codec module (builder + positivity + n…
39bd9a1 
…dim_of)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): scale_offset + cast_value codec modules
c92daa1 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): regular + rectilinear chunk_grid modules
a4e4888 
Add chunk_grid implementations for regular and rectilinear chunk grid types,
including builder functions, validation, and ndim extraction helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
fix(strict): use precise metadata types in chunk_grid builders
7aa5256 
Replace bare `dict` types with `RegularChunkGridMetadata` and
`RectilinearChunkGridMetadata` in chunk grid modules' builder functions
and validators. Imports use TYPE_CHECKING pattern matching codec modules.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): codec/grid __init__ unions + dispatch maps
e7ee165 
Implement codec/__init__.py and chunk_grid/__init__.py:
- Re-export builders from all 9 codec modules and 2 grid modules
- Build CODEC_NDIM_OF, CODEC_KIND, CODEC_DTYPE_OUT, CODEC_VALIDATE dispatch maps keyed by codec name
- Build GRID_NDIM_OF, GRID_VALIDATE dispatch maps keyed by grid name
- Move _CoreCodec/_ExtraCodec unions here (identical membership to _strict_v3.py)
- Re-export RegularChunkGridMetadata/_RectilinearChunkGrid type aliases
- Add test_dispatch_maps.py verifying map coverage and spot-checks

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): cross-field dimensionality + sharding-divisibility vali…
7fe0b95 
…dator

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): codec-pipeline type-flow validator (cast_value scalar d…
bb47bf5 
…type-correctness)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): migrate codec/grid vocab to per-element modules; wi…
0212f05 
…re internal validators

- Delete inline _CoreCodec/_ExtraCodec union definitions from _strict_v3.py;
  import them from pydantic_zarr.strict.v3.codec (identical membership).
- Remove now-unused zarr_metadata codec type imports (all *CodecMetadata and
  *CodecName names previously used only to build the inline unions).
- Add _validate_codec_internal / _validate_grid_internal dispatch functions
  that look up CODEC_VALIDATE / GRID_VALIDATE and run the per-element
  validate_<x> function when the parsed value is a dict (object form).
- Wire AfterValidator(_validate_codec_internal) onto the codecs tuple element
  type and AfterValidator(_validate_grid_internal) onto chunk_grid in both
  _CoreBase and _ExtraBase, so parsed-from-JSON specs get internal checks.
- Add three targeted rejection tests: non-permutation transpose order,
  blosc clevel 99, non-positive regular chunk_shape.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): run dimensionality + divisibility + pipeline passes in …
0e99a8c 
…array validator

Add `_validate_array_consistency` model_validator (mode="after") to both
`_CoreBase` and `_ExtraBase` that calls `check_array_consistency` and
`validate_pipeline`, concatenates violations, and raises a single
`ValueError` listing all errors. Also fix `_resolve_strict_init` so that
the default chunk_grid shape matches the array shape exactly (was using
`shape_v or (1,)` which created an inconsistent 1D chunk for 0-dim arrays).

Updated 6 existing tests that were relying on the validation gap
(incomplete pipelines now correctly rejected):
- test_core_accepts_known_codec_name_string: add bytes codec before blosc
- test_extra_accepts_known_codec_name_string: same
- test_extra_accepts_scale_offset_codec: add bytes codec after scale_offset
- test_extra_accepts_scale_offset_codec_name_string: same
- test_core_array_spec_bare_construct: fixed by _resolve_strict_init fix
- test_core_codec_matches_oracle / test_extra_codec_matches_oracle: add
  _codec_pipeline_for() helper to avoid adding a second array->bytes codec
  when the candidate itself is array->bytes (e.g. "bytes")

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(strict): oracle + property coverage for internal/dimensionality/…
7757538 
…pipeline checks

Added to strict_oracle.py (stdlib-only, no pydantic-zarr imports):
- is_valid_codec_internal(name, meta): encodes transpose permutation rule,
  blosc clevel in [0,9], gzip level in [0,9], sharding/regular chunk_shape
  all-positive, other codecs -> True.
- is_valid_ndim_match(shape, chunk_grid): checks array ndim == regular
  chunk_grid ndim.

Added to test_strict_property.py (hypothesis @given tests):
- test_transpose_internal_matches_oracle: generates transpose orders (valid
  permutations + arbitrary int lists); asserts accept iff oracle says permutation.
- test_blosc_clevel_matches_oracle: generates clevel in [-3, 12]; asserts accept
  iff clevel in [0, 9].
- test_regular_grid_ndim_matches_oracle: generates (array_ndim, chunk_ndim)
  pairs; asserts accept iff equal.

Weaken/restore gap proof (Step 3):
Temporarily commented out the permutation check in validate_transpose. Hypothesis
immediately found: order=[1] accepted=True but oracle says False (order [1] is
not a permutation of range(1)=[0]). Test FAILED with:
  AssertionError: order=[1]: accepted=True but oracle says False
After restoring the check all 33 tests PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(strict): narrow transpose property test to ValidationError
71368fb 
Catch only ValidationError (not any Exception) so an unexpected error
surfaces as a test failure instead of being treated as a rejection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs(strict): document per-element builders + strengthened validation
4d842ec 
Also fix a pre-existing failing doc example: the rectilinear chunk grid
example used chunk_shapes=(5,5,5,5) (4-D) against a 1-D array in COMMON,
which the new dimensionality validation correctly rejected. Updated to
chunk_shapes=(5,) so ndim matches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
fix(strict): guard cross-field passes against bare-string codecs (no …
42abde1 
…TypeError)

A bare-string transpose/sharding_indexed codec (valid vocabulary names)
crashed the dimensionality + divisibility passes with an uncaught TypeError
(ndim_of/config indexing assumed a dict). Guard on isinstance(c, dict);
bare-string codecs contribute no dimensionality constraint. Add the two
names to the property strategy + a direct regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): add CodecSpec record + CODECS registry
b52144c 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
style(strict): hoist SPEC imports to module top (drop E402 noqa)
bbb9ec8 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): add GridSpec record + GRIDS registry
d215aa4 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
feat(strict): add chunk_key_encoding per-element package (builders + …
c40939c 
…KeyEncodingSpec + registry)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): migrate _cross_field to spec lookups
983439a 
Replace deleted CODEC_NDIM_OF/GRID_NDIM_OF parallel maps with
grid_spec_for/codec_spec_for registry lookups. Guard ndim_of calls
with isinstance(dict) so bare-string codecs never reach config
indexing — making the TypeError crash structurally impossible.
Replace local _name helper with element_name from _registry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): migrate _pipeline to spec lookups + has_dtype_depen…
a16582a 
…dent_config

Replace CODEC_KIND/CODEC_DTYPE_OUT parallel maps and _name() helper with
codec_spec_for() registry lookups; has_dtype_dependent_config replaces the
hardcoded "cast_value" name check. Dtype-threading semantics preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
refactor(strict): dispatchers use specs; move CKE union to package; w…
c22f696 
…ire CKE validator

- Replace GRID_VALIDATE/CODEC_VALIDATE dispatch-map imports with
  grid_spec_for/codec_spec_for/key_encoding_spec_for from the new
  per-element spec packages.
- Import _ChunkKeyEncoding from strict.v3.chunk_key_encoding (removes
  the local duplicate definition and drops the now-unused
  DefaultChunkKeyEncodingMetadata/V2ChunkKeyEncodingMetadata imports
  from zarr_metadata).
- Rewrite _validate_codec_internal and _validate_grid_internal to look
  up specs via the registry functions; add _validate_key_encoding_internal
  using key_encoding_spec_for.
- Wire AfterValidator(_validate_key_encoding_internal) onto
  chunk_key_encoding on both _CoreBase and _ExtraBase.
- Update test_dispatch_maps.py to use the new CODECS/GRIDS registry
  dicts (old CODEC_VALIDATE/GRID_VALIDATE symbols were removed in a
  prior task, breaking collection).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
fix(strict): bare codec short-form accepted only when config optional
5f31e0a 
Rewrote _CoreCodec/_ExtraCodec unions so bare *CodecName members appear
only for config-optional codecs (bytes, crc32c, scale_offset). Config-required
codecs (gzip, zstd, blosc, transpose, sharding_indexed, cast_value) now
contribute only their *CodecObject form.

Added test_bare_string_accepted_iff_config_optional and
test_object_form_always_accepted_for_member_codecs to test_registry.py
to lock this invariant.

Updated existing tests that asserted the buggy bare-acceptance:
- test_strict_v3.py: test_core/extra_accepts_known_codec_name_string changed
  from bare "blosc" (config-required, now rejected) to bare "crc32c" (config-optional)
- test_strict_property.py: _CODECS strategy changed from bare strings for
  config-required codecs to minimal valid object dicts so oracle property
  tests remain a vocabulary check without depending on bare-string acceptance

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(strict): differential coverage for bare-string + object forms ac…
31ad165 
…ross elements

Add _CONFIG_REQUIRED_NAMES frozenset and bare_string_allowed() to the independent
oracle (stdlib only, no pydantic_zarr/zarr_metadata import). Hand-encoded set covers:
gzip, zstd, blosc, transpose, sharding_indexed, cast_value, regular, rectilinear.
Config-optional (bare allowed, not in set): bytes, crc32c, scale_offset, default, v2.

Add three property tests to test_strict_property.py:
- test_bare_string_codec_accept_iff_oracle: feeds all 11 codec name strings as bare
  strings and asserts AnyExtraArraySpec accepts iff is_valid_codec("extra", name) and
  bare_string_allowed(name). All 36 tests pass.
- test_default_cke_object_and_bare_accepted: both object and bare "default" accepted.
- test_v2_cke_object_and_bare_accepted: both object and bare "v2" accepted.

Prove-catches-a-gap: temporarily added TransposeCodecName back into _ExtraCodec union
(re-introducing the bare-transpose acceptance bug). Result:
  AssertionError: bare codec 'transpose': accepted=True but oracle says False
  Falsifying example: test_bare_string_codec_accept_iff_oracle(codec='transpose')
Union restored; git status src/ clean before this commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
test(docs): exclude gitignored docs/superpowers from doc-example scan
ce0d35b 
find_examples('docs') recursively picked up the gitignored
docs/superpowers working area (design specs / plans), whose code
blocks are illustrative pseudocode, not runnable examples. Scope the
doc-example test to published docs only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: demonstrate strict validation errors; export builders from pyda…
84cea23 
…ntic_zarr.v3

Writing the validation-error examples surfaced that the strict builders
were only reachable via deep private-looking paths
(pydantic_zarr.strict.v3.codec.transpose) — every codec example needed a
separate deep import or a wall of nested {name, configuration} dicts. Made
all 13 builders (codecs, grids, chunk-key-encodings) importable directly
from pydantic_zarr.v3, and updated the existing builder docs to use them.

Added a 'Validation errors strict mode catches' section to usage_zarr_v3.md
demonstrating: build-time ValueError from builders; spec-level dimensionality,
sharding-divisibility, pipeline-shape, and cast_value scalar-dtype errors;
and parsed-dict rejection of bare-string codecs that require configuration.
Examples use the single-class route + exc.errors()[0]['msg'] for clean,
quotable messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: document loose-spec behavioral changes for the breaking release
9add50b 
The public-API study found that adopting zarr-metadata changed the loose
ArraySpec/GroupSpec field TYPES (names unchanged) and some validation
behavior, which the removed-exports changelog/migration did not cover:
- v2 ArraySpec.filters and v3 chunk_grid chunk_shape are now tuples on the
  model / model_dump() (to_json() output unchanged — still JSON arrays);
- v3 chunk_grid/chunk_key_encoding/codecs now accept the bare-string short
  form (previously rejected);
- fill_value/dtype are now JSONValue / zarr-metadata dtype types.

Add a changelog removal fragment + a 'Loose-spec behavioral changes' table
to the migration guide so no breaking change ships undocumented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b d-v-b marked this pull request as draft June 22, 2026 11:02
d-v-b and others added 2 commits June 22, 2026 13:12
@d-v-b @claude
test: tolerate numpy 2.5 generic-NaT deprecation from zarr-python; fi…
6b598d9 
…x changelog tuple wording

Under numpy 2.5, DateTime64/TimeDelta64.default_scalar() returns a generic-unit
np.datetime64('NaT')/np.timedelta64('NaT'), which numpy 2.5 deprecates. The dtype-example
fixtures in conftest exercise these types, so with filterwarnings=['error'] the warning failed
conftest collection across the whole CI matrix. Allow-list this specific zarr-python-originating
DeprecationWarning (narrow match: exact message + category), matching the existing pattern for
known zarr-python interactions. Reproduced under numpy 2.5.0 + zarr 3.1.0: test_pydantic_zarr
suite passes (9844 passed).

Also corrects the loose-spec changelog/migration wording: the list->tuple change is about
making the parsed spec immutable, not about JSON correctness (both serialize to JSON arrays).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@d-v-b @claude
docs: fix changelog fragment naming + correct loose-spec wording
85589df 
- Rename all changelog fragments from the +name.type.md style to the
  repo's required <issue>.<type>.md form (PR zarr-developers#173). The CI 'Check changelog
  entries' job (ci/check_changelog_entries.py, pre-existing on main) requires
  an integer issue number before the type; the + fragments all failed it, and
  '+numpy-2.5-...misc.md' additionally broke the type split on the '2.5' dot.
- Recategorize the numpy-2.5 timedelta entry misc -> bugfix (it's a real test
  fix, and misc fragments are content-less by convention).
- Correct the loose-spec migration wording: the loose specs validate
  named-config fields as SYNTAX ONLY (they don't know that e.g. a 'regular'
  chunk grid requires configuration -- that's the strict specs' job), so
  'chunk_grid="regular"' was a misleading example implying it's meaningful.
  Reframe around the actual change (generic str|NamedConfigV3 typing) and drop
  the confusing example. Also drop a stale deep-path builder reference now that
  builders are public from pydantic_zarr.v3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b