zarr_metadata.definition

zarr_metadata.definition

Reading declarations against their definitions, in either Zarr format.

Every extension point of a Zarr document (a v3 codec, data type, chunk grid, chunk key encoding or storage transformer; a v2 dtype or numcodecs codec) is a declaration: a name, and a configuration whose JSON the extension defines. A definition declares what that JSON is and what it may contain. A definition is a value, not a class to subclass. It has:

• name, the name the metadata uses;
• configuration, the TypedDict for the configuration's JSON. This is the single source of the JSON: the type checker is compiled from it, and a checked configuration has it as its static type. It is interpreted as the typing spec defines a TypedDict: total, Required, NotRequired, closed and extra_items mean what they mean to a type checker, whether or not its module postpones annotations. A member's type carries its bounds, in the same way pydantic reads them: a gzip level is Annotated[int, Interval(ge=0, le=9)];
• rules, a function over that TypedDict that yields the spec violations a type cannot express (such as constraints between members), located in the configuration.

The kind of metadata a definition defines is its type. The v3 kinds are in zarr_metadata.v3.definition, and the v2 kinds are in zarr_metadata.v2.definition. This module holds what both formats share: Definition and WithFillValue, which the kinds derive from; resolve, which reads a declaration in a scope; the three results of a reading; and Context, the protocol of a scope.

Reading JSON. There are three steps. Each feeds the next, and each can be used on its own by a caller that has only JSON:

1. check(value, SomeTypedDict), from zarr_metadata.typed_json, type-checks JSON against a TypedDict and needs nothing else. It returns a value of the TypedDict or None, and every problem, each with a location. The value contains only what the TypedDict admits. A member typed with a declaration alias is checked as declaration JSON.
2. definition.read_configuration(configuration) returns the configuration as that definition reads it. It type-checks the configuration, checks the envelope of each nested declaration (an extra member, a must_understand of false), and then runs the rules, for one configuration: GZIP_CODEC.read_configuration({"level": 12}).

3. resolve(declaration, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) reads a whole declaration in a scope. It checks the envelope, maps the name to a definition, checks the configuration, and reads each nested declaration the same way. It returns the result of the reading and every problem. The result is one of three types. AcceptedDeclaration is a declaration read by the definition that claims its name, with the configuration it checked and allowed, and its nested declarations, each read the same way. UnclaimedDeclaration is a name that nothing in scope claims. It is not checked, which is what keeps the format open. RefusedDeclaration is a declaration with problems that explain why it was rejected. Each type has the declaration's JSON, the name it is written with, the kind it was read as (read_as), the definition that claims its name (None for UnclaimedDeclaration), and its nested declarations as the scope read them (nested). The two types a model can hold, AcceptedDeclaration and UnclaimedDeclaration, have a configuration and to_json(), which returns the declaration as a document writes it. Two of them are equal when they read the same, however each was spelled. ResolvedDeclaration is the union of the three, for use with match. configuration_of(declaration, GZIP_CODEC) returns the configuration of a declaration that GZIP_CODEC accepted, typed as its TypedDict (GzipCodecConfiguration), and None for any other declaration, including one that GZIP_CODEC claimed and refused. A declaration is read as one of the kinds of its format, with or without type arguments. resolve(declaration, Definition, context) raises TypeError, because nothing is filed under Definition. A whole v3 array document is read by read_array_metadata_v3, in zarr_metadata.model. Its reading returns each declaration with its location in the document.

   from zarr_metadata.definition import resolve from zarr_metadata.v3.codec.gzip import GZIP_CODEC from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) resolved # RefusedDeclaration(..., definition=CodecDefinition(name='gzip'), ...) problems[0].loc # ('configuration', 'level') problems[0].input # 12 dict(problems[0].ctx) # {'ge': 0, 'le': 9}

   resolved, problems = resolve({"name": "gzip", "configuration": {"level": 5}}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT) resolved.definition is GZIP_CODEC # True resolved.configuration # {'level': 5}

Problems are values, not exceptions: ValidationProblem(loc, message, kind), with kind one of invalid_type, invalid_value, missing_key, unknown_key and invalid_json. A declaration with an unknown key is still read: the key is reported, and the configuration is checked without it. So a consumer that tolerates unknown keys can filter problems by kind and use the result. The declaration is valid only when there are no problems at all. Each problem stores the content of its message as data, as pydantic's errors and zod's issues do: input is the value found at loc (the 12 above), and ctx is what was expected, when that is more than a type: the bounds {"ge": 0, "le": 9}, or the values of a closed set.

Writing an extension. Write a TypedDict that declares the configuration's JSON, including bounds; a function for the rules a type cannot express; and a definition. Then build a scope that contains the definition. The TypedDict must be a typing_extensions.TypedDict, because closed and extra_items come from PEP 728, and typing.TypedDict does not accept them on the Python versions this package supports. The rules receive the configuration and its nested declarations as the scope read them. A declaration that is read stores its nested declarations as AcceptedDeclaration.nested, a Nested mapping keyed by location, so a struct's rules can access its field types. read_configuration reads in no scope, so it passes no nested declarations. A rule's message should show a value as JSON, as the package's own messages do, with zarr_metadata.problems.shown: null, [1, 2], "C". A rule reports the location of a problem. The value at that location becomes the problem's input automatically. Define each function at module top level. A model holds the definitions that read its declarations, so the model can be pickled, and compares equal after loading, only if its definitions can. A lambda or a closure cannot be pickled, and a functools.partial can be pickled but does not compare equal to its loaded copy.

from collections.abc import Iterator
from typing import Annotated, NotRequired

from annotated_types import Ge
from typing_extensions import TypedDict

from zarr_metadata import ValidationProblem
from zarr_metadata.definition import Nested
from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition


class AcmeLz4Configuration(TypedDict, closed=True):
    acceleration: Annotated[int, Ge(1)]
    dictionary: NotRequired[str]
    dictionary_size: NotRequired[Annotated[int, Ge(1)]]


def acme_lz4_rules(
    configuration: AcmeLz4Configuration, nested: Nested
) -> Iterator[ValidationProblem]:
    if "dictionary" in configuration and "dictionary_size" not in configuration:
        yield ValidationProblem(
            ("dictionary_size",), "a dictionary needs its size", "missing_key"
        )


ACME_LZ4 = CodecDefinition(
    name="acme.lz4",
    configuration=AcmeLz4Configuration,
    kind="bytes_bytes",
    size="dynamic",
    rules=acme_lz4_rules,
)
CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), ACME_LZ4])

The TypedDict declares how keys it does not declare are handled. With closed=True, such a key is a problem, as above. With extra_items=, such a key must hold that type. With closed=False, anything is allowed. A TypedDict that declares none of these is open by default and would accept a misspelled key silently, so a definition rejects it. Its members must be JSON shapes: int, float, bool, str, None, JSONValue, a Literal, tuple[T, ...] and tuple[T1, T2], a union, a TypedDict, Mapping[str, V], a NewType and a type alias. A number's type may carry bounds, written with annotated-types and read the way pydantic reads them: Gt, Ge, Lt, Le and Interval, at most one from each side, at any depth. For example, tuple[Annotated[int, Ge(1)], ...] bounds each element. A value out of bounds is an invalid_value problem. Its message says what the type admits ("expected an integer >= 1, got 0"), and its ctx holds the bounds. The rules run only on a configuration within its bounds, so a rule can rely on them, as pydantic's after-validators and zod's refinements do. Until a value out of bounds is fixed, it is the only problem reported for the configuration. Annotated may also carry a note, as a string or a Doc. Any other metadata (a MinLen, a Predicate, pydantic's Field) is rejected when the definition is built, because a type whose constraint the checker does not enforce would give false information.

A data type also declares its fill value. fill_value is the JSON shape of a fill value, as an annotation the checker reads (for example Int8FillValue, whose type carries the range). fill_value_rules is a function that yields the spec violations in a fill value of that shape that the type cannot express, such as a hex string of the wrong width, or a number of byte values that does not match the size. The rules receive the configuration, the nested declarations as the scope read them, and the typed fill value, so a struct checks each field's fill value against that field's own type. fill_value_problems(data_type, value) checks a fill value against a data type declaration the scope read. If nothing in scope claims the data type, the fill value is not checked. A data type that does not declare its fill value accepts any JSON. A fill value can have more than one spelling ("NaN" and "0x7fc00000" are the same float32), so a data type declares the canonical spelling of each value with fill_value_canonical. It receives the same arguments as the rules, plus a fill value the rules allow. Two fill values are the same value of the type exactly when their canonical spellings are written the same way: the same JSON, as json.dumps writes it. == is not the same test, because it treats -0.0 (a distinct float32) as equal to 0.0. canonical_fill_value(data_type, value) returns the canonical spelling, or UNSET for a fill value with a problem. A data type that does not declare fill_value_canonical keeps each value as written.

The simplest spelling. A declaration without problems has a simplest equivalent spelling, and declarations are compared by it. Each nested declaration is written in its own simplest spelling, then the definition's canonical is applied (blosc drops a typesize that noshuffle ignores, and a rectilinear grid run-length encodes its chunk shapes). The envelope uses the shortest form every reader accepts: a data type with nothing to configure is its bare name, and any other declaration is an object, {"name": ...}, because a Zarr v3.0 reader does not accept a short-hand name in codecs. Raw bits write their size back into the name, in decimal, so r008 becomes r8. A declaration with any problem, including an unknown key, has no simplest spelling, because a simpler spelling would erase what its author wrote. The result of canonical is checked again, and an invalid result raises ValueError, which indicates a bug in the definition. canonical_fill_value writes a fill value in the same way, using the data type that read it.

Scopes as values. A scope reads documents of one Zarr format. ZarrV3Context, in zarr_metadata.v3.definition, and ZarrV2Context, in zarr_metadata.v2.definition, are the scopes of each format. They are separate classes, and Context is the protocol both implement: the Zarr format a scope reads, and claimant, which says which definition of a kind reads a name. That is all resolve needs, so any object that provides these two can be a scope. A scope has one slot for each kind of declaration its format has, named like the document member that holds such declarations: ZarrV3Context(codecs=[...], data_types=[...]) builds one, and context.codecs["gzip"] reads one slot. A type checker rejects a definition in the wrong slot, and a scope of one format given to a reader of the other. The constructor and the reader also reject them at run time, with TypeError. Two scopes are equal when they are of one format and file the same definitions, and equal scopes have equal hashes. context.union(*others) returns the smallest scope that contains each of them, or raises a ScopeConflictError naming each name filed with two different definitions. update replaces the definitions in the slots it is given and keeps the others, and takes a slot as definitions or as a mapping by name, so context.update(codecs={**context.codecs, "gzip": MY_GZIP}) replaces one definition on purpose. A model's refined_in moves it to a scope that claims more names and contradicts nothing. refines orders two models by how much they know. A name that was unclaimed and is now read by a definition is a gain. The reverse is a loss. One name read by two different definitions is a conflict.

A definition checks itself when it is built. Each of these raises a TypeError that says what is wrong: a configuration that is not a TypedDict, that does not declare how undeclared keys are handled, or that has a member no checker reads (the error names the TypedDict that contains it); a name that is not a string; an attribute declared as a function (rules, canonical, fill_value_rules, storage, shape_rules, chunk_lengths, chunk_rules, transition, pipelines) that is not a function; a data type's fill_value that no checker reads, or that contains a declaration, which a value of the data type never does; a codec kind that is not one of the three, or a size that is not "static" or "dynamic"; a function that is never called for a codec of its kind (chunk rules or pipelines of a bytes -> bytes codec, which receives bytes, or a transition of a codec that outputs bytes); a data type named the way raw bits of one size are written; an extra field named after a member the spec defines, whose finite is not a boolean, or whose configuration does not declare must_understand. Building a definition files it nowhere: only a scope built from it does.

ClaimKey module-attribute

ClaimKey: TypeAlias = tuple[type[Definition[Any]], str]

A kind and the name a definition is filed under. A scope's claimant looks up definitions by this key.

Nested module-attribute

Nested: TypeAlias = Mapping[Loc, ResolvedDeclaration[Any]]

The declarations in a configuration, each as the scope read it, keyed by location in the configuration.

ResolvedDeclaration module-attribute

ResolvedDeclaration = TypeAliasType(
    "ResolvedDeclaration",
    "AcceptedDeclaration[D] | UnclaimedDeclaration | RefusedDeclaration[D]",
    type_params=(D,),
)

One declaration as a scope read it: accepted by the definition that claims its name, unclaimed, or refused.

AcceptedDeclaration dataclass

Bases: Generic[D]

A declaration that a definition in scope read: the name it is written with, the definition, and the configuration the definition allowed.

A problem with the envelope (an extra envelope member, a must_understand of false) is reported with the declaration, and the declaration is still read. The same applies to a problem in a nested declaration, which belongs to that nested declaration, as nested shows. Two declarations are equal when they read the same, however each was spelled, as compared by declaration_key. For example, "bytes" and {"name": "bytes"} are one declaration. So are a blosc with and without the typesize that noshuffle ignores, because the definition's canonical removes it. Equal declarations have equal hashes.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class AcceptedDeclaration(Generic[D]):
    """A declaration that a definition in scope read: the name it is written with, the definition, and the configuration the definition allowed.

    A problem with the envelope (an extra envelope member, a `must_understand` of
    `false`) is reported with the declaration, and the declaration is still read. The
    same applies to a problem in a nested declaration, which belongs to that
    nested declaration, as `nested` shows. Two declarations are equal when they read
    the same, however each was spelled, as compared by `declaration_key`. For
    example, `"bytes"` and `{"name": "bytes"}` are one declaration. So are a
    blosc with and without the `typesize` that `noshuffle` ignores, because
    the definition's `canonical` removes it. Equal declarations have equal
    hashes.
    """

    json: JSONValue
    """The declaration as written, refined (arrays as tuples). It is not used for equality."""
    name: str
    """The name the declaration is written with, for example `"r16"`, even though its definition is filed under `r*`."""
    definition: D
    """The definition that read the declaration."""
    configuration: Mapping[str, JSONValue]
    """The configuration, type-checked and allowed by the rules. For raw bits, this is the configuration encoded in the name.

    Each nested declaration is written as that declaration's `to_json` writes it, so
    the configuration shows what was read, however it was spelled. For
    example, a shard's `"crc32c"` and `{"name": "crc32c"}` are the same
    index codec.
    """
    nested: Nested = dataclasses.field(default_factory=_nothing_nested)
    """The declarations in the configuration, each as the scope read it, keyed by location in the configuration.

    For example, a struct's field types are at `("fields", 0, "data_type")`
    and a shard's codecs are at `("codecs", 0)`. A definition's functions
    use this to inspect the declarations inside its own configuration.
    """
    read_as: type[Definition[Any]] = dataclasses.field(init=False, repr=False)
    """The kind of metadata the declaration was read as: the kind of its definition."""

    def __eq__(self, other: object) -> bool:
        if not is_declaration(other):
            return NotImplemented
        return declaration_key(self) == declaration_key(other)

    def __hash__(self) -> int:
        return hash(declaration_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. A declaration built by hand (an
        # extension may do this) fails here, not later in a function that
        # trusts it.
        definition = cast("object", self.definition)
        kind = kind_of(definition)
        refusal = (
            f"a declaration read is read by a definition of a kind, got {definition!r}"
            if kind is None
            else _misread(definition, kind, self.name)
        )
        if refusal is not None:
            raise TypeError(refusal)
        object.__setattr__(self, "read_as", kind)
        # Everything the declaration returns is read-only at every level, so a
        # declaration cannot reach a state where its key, `==` and `refines`
        # disagree.
        object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(self, "configuration", frozen(self.configuration))
        object.__setattr__(self, "nested", MappingProxyType(dict(self.nested)))

    def __reduce__(self) -> tuple[Callable[..., AcceptedDeclaration[Any]], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the declaration from its constructor arguments.
        return _accepted_field, (
            copied(self.json),
            self.name,
            self.definition,
            copied(self.configuration),
            dict(self.nested),
        )

    def to_json(self) -> JSONValue:
        """The declaration as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the declaration.

        The envelope uses the shortest form every reader accepts. A data
        type with nothing to configure is written as its bare name, as core
        data types have been written since Zarr v3.0. Any other declaration is an
        object, `{"name": ...}`, because a Zarr v3.0 reader does not accept a
        bare name in `codecs`
        (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592).
        A name that encodes its configuration, as the raw bits name does, is
        written alone.
        """
        return copied(written_json(self))

configuration instance-attribute

configuration: Mapping[str, JSONValue]

The configuration, type-checked and allowed by the rules. For raw bits, this is the configuration encoded in the name.

Each nested declaration is written as that declaration's to_json writes it, so the configuration shows what was read, however it was spelled. For example, a shard's "crc32c" and {"name": "crc32c"} are the same index codec.

definition instance-attribute

definition: D

The definition that read the declaration.

json instance-attribute

json: JSONValue

The declaration as written, refined (arrays as tuples). It is not used for equality.

name instance-attribute

name: str

The name the declaration is written with, for example "r16", even though its definition is filed under r*.

nested class-attribute instance-attribute

nested: Nested = dataclasses.field(
    default_factory=_nothing_nested
)

The declarations in the configuration, each as the scope read it, keyed by location in the configuration.

For example, a struct's field types are at ("fields", 0, "data_type") and a shard's codecs are at ("codecs", 0). A definition's functions use this to inspect the declarations inside its own configuration.

read_as class-attribute instance-attribute

read_as: type[Definition[Any]] = dataclasses.field(
    init=False, repr=False
)

The kind of metadata the declaration was read as: the kind of its definition.

to_json

to_json() -> JSONValue

The declaration as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the declaration.

The envelope uses the shortest form every reader accepts. A data type with nothing to configure is written as its bare name, as core data types have been written since Zarr v3.0. Any other declaration is an object, {"name": ...}, because a Zarr v3.0 reader does not accept a bare name in codecs (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592). A name that encodes its configuration, as the raw bits name does, is written alone.

Source code in src/zarr_metadata/definition/_definition.py

def to_json(self) -> JSONValue:
    """The declaration as a document writes it, in a form every reader accepts. The configuration is as read, and the result shares no objects with the declaration.

    The envelope uses the shortest form every reader accepts. A data
    type with nothing to configure is written as its bare name, as core
    data types have been written since Zarr v3.0. Any other declaration is an
    object, `{"name": ...}`, because a Zarr v3.0 reader does not accept a
    bare name in `codecs`
    (https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L585-L592).
    A name that encodes its configuration, as the raw bits name does, is
    written alone.
    """
    return copied(written_json(self))

Conflict dataclass

One place where two readings of a name disagree: the key, the definition one reading claimed, the definition the other found, and the location in the document when known.

Source code in src/zarr_metadata/definition/_scope.py

@dataclass(frozen=True, slots=True)
class Conflict:
    """One place where two readings of a name disagree: the key, the definition one reading claimed, the definition the other found, and the location in the document when known."""

    key: ClaimKey
    """The kind, and the name the two definitions are filed under."""
    claimed: Definition[Any] | None
    """The definition one reading or scope gives the name; None when it leaves the name unclaimed."""
    found: Definition[Any] | None
    """The definition the other gives the name; None when it gives none, so the name would lose its meaning."""
    loc: Loc | None = None
    """Where the name occurs in the document; None when no document is available."""
    written: str | None = None
    """The name the document uses at the conflict location, for example `r16` for a key of `r*`; None when no document is available."""

    def __str__(self) -> str:
        kind, filed = self.key
        name = filed if self.written is None else self.written
        where = "" if self.loc is None else f" at {self.loc!r}"
        claimed, found = definition_said(self.claimed), definition_said(self.found)
        if claimed == found:
            # Nothing a description shows tells them apart, such as their
            # rules, so say that they differ.
            return f"{kind_name(kind)} {name!r}{where}: two different definitions, each {claimed}"
        return f"{kind_name(kind)} {name!r}{where}: claimed {claimed}, found {found}"

claimed instance-attribute

claimed: Definition[Any] | None

The definition one reading or scope gives the name; None when it leaves the name unclaimed.

found instance-attribute

found: Definition[Any] | None

The definition the other gives the name; None when it gives none, so the name would lose its meaning.

key instance-attribute

key: ClaimKey

The kind, and the name the two definitions are filed under.

loc class-attribute instance-attribute

loc: Loc | None = None

Where the name occurs in the document; None when no document is available.

written class-attribute instance-attribute

written: str | None = None

The name the document uses at the conflict location, for example r16 for a key of r*; None when no document is available.

Context

Bases: Protocol

A scope: the definitions in scope while metadata of one Zarr format is read.

This is what reading a declaration needs from a scope: the Zarr format of the documents it reads, and which definition of a kind claims a name. ZarrV3Context and ZarrV2Context implement it, each with one slot per kind of declaration its format has, and so does any other object that provides these two. resolve reads a declaration in any Context.

Source code in src/zarr_metadata/definition/_definition.py

class Context(Protocol):
    """A scope: the definitions in scope while metadata of one Zarr format is read.

    This is what reading a declaration needs from a scope: the Zarr format of the
    documents it reads, and which definition of a kind claims a name.
    `ZarrV3Context` and `ZarrV2Context` implement it, each with one slot
    per kind of declaration its format has, and so does any other object that
    provides these two. `resolve` reads a declaration in any `Context`.
    """

    @property
    def zarr_format(self) -> Literal[2, 3]:
        """The Zarr format of the documents this scope reads."""
        ...

    def claimant(self, kind: type[D], name: str) -> D | None:
        """The definition of `kind` in scope that reads `name` (a name as a document uses it), or None if there is none."""
        ...

zarr_format property

zarr_format: Literal[2, 3]

The Zarr format of the documents this scope reads.

claimant

claimant(kind: type[D], name: str) -> D | None

The definition of kind in scope that reads name (a name as a document uses it), or None if there is none.

Source code in src/zarr_metadata/definition/_definition.py

def claimant(self, kind: type[D], name: str) -> D | None:
    """The definition of `kind` in scope that reads `name` (a name as a document uses it), or None if there is none."""
    ...

Definition dataclass

Bases: Generic[C]

One extension's metadata as JSON: its name, its configuration TypedDict, and its rules.

configuration is the TypedDict, and so it is the single source of the JSON. The checker is compiled from it, it is the static type of a checked configuration, and a document's author writes to it. It is interpreted as the typing spec defines a TypedDict: total, Required, NotRequired, closed and extra_items mean what they mean to a type checker. It also declares how keys it does not declare are handled. With closed=True, such a key is a problem, reported and left out. With extra_items=, such a key must have that type. With closed=False, anything is allowed. rules yields each violation of the spec it finds in a configuration of that type. It receives only a configuration that has passed the check: it contains only what the TypedDict admits, each member is within the bounds its type declares, and the nested declarations are passed as the scope read them (for example, a struct's field types). The nested declarations are empty when no scope read the configuration. read_configuration runs both steps, for a caller that has JSON.

Each function receives the configuration as a read-only view, not a dict. copy.deepcopy and json.dumps reject this view. A function that simplifies a spelling must build a new mapping, such as {**configuration} without the member, instead of editing its input.

canonical is where two spellings of the configuration that mean the same thing are reduced to one.

A definition built by hand rejects inputs it could not read with: a configuration that is not a TypedDict, that does not declare how undeclared keys are handled, or that has a member no checker reads (the error names the member); and a name or rules of the wrong type.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True)
class Definition(Generic[C]):
    """One extension's metadata as JSON: its name, its configuration TypedDict, and its rules.

    `configuration` is the TypedDict, and so it is the single source
    of the JSON. The checker is compiled from it, it is the static type of
    a checked configuration, and a document's author writes to it. It is
    interpreted as the typing spec defines a TypedDict: `total`,
    `Required`, `NotRequired`, `closed` and `extra_items` mean what they
    mean to a type checker. It also declares how keys it does not declare
    are handled. With `closed=True`, such a key is a problem, reported and
    left out. With `extra_items=`, such a key must have that type. With
    `closed=False`, anything is allowed. `rules` yields each violation of
    the spec it finds in a configuration of that type. It receives only a
    configuration that has passed the check: it contains only what the
    TypedDict admits, each member is within the bounds its type declares,
    and the nested declarations are passed as the scope read them (for example, a
    struct's field types). The nested declarations are empty when no scope read
    the configuration. `read_configuration` runs both steps, for a caller
    that has JSON.

    Each function receives the configuration as a read-only view, not a
    `dict`. `copy.deepcopy` and `json.dumps` reject this view. A function
    that simplifies a spelling must build a new mapping, such as
    `{**configuration}` without the member, instead of editing its input.

    `canonical` is where two spellings of the configuration that mean the
    same thing are reduced to one.

    A definition built by hand rejects inputs it could not read with: a
    `configuration` that is not a TypedDict, that does not declare how
    undeclared keys are handled, or that has a member no checker reads
    (the error names the member); and a `name` or rules of the wrong type.
    """

    _is_kind: ClassVar[bool] = False
    """Whether this class is a kind, which is what a scope files definitions by. Set by `class Kind(Definition, kind=True, format=3)`.

    Read from the class's own namespace and never inherited: a subclass of
    a kind is a definition of that kind, and does not declare `kind`.
    """
    _format: ClassVar[Literal[2, 3] | None] = None
    """The Zarr format of the documents that contain declarations of this kind. It is declared with the kind and inherited by its definitions, and it sets the format a scope reads."""
    label: ClassVar[str] = "definition"
    """The name of the kind in error messages, for example "codec"."""
    declaration_aliases: ClassVar[tuple[TypeAliasType, ...]] = ()
    """The declaration aliases used to annotate a configuration member that holds a declaration of this kind. For a codec: `ZarrV3CodecDeclaration` and `ZarrV3StaticCodecDeclaration`."""

    name: str
    """The name the metadata uses. A scope files the definition under this name."""
    configuration: type[C]
    """The configuration TypedDict."""
    rules: Callable[[C, Nested], Iterable[ValidationProblem]] = no_rules
    """The spec violations in a well-typed configuration and its nested declarations, located in the configuration."""
    canonical: Callable[[C], C] = unchanged
    """A well-typed, allowed configuration in its simplest equivalent spelling.

    This handles only the definition's own members. `canonicalize` puts
    each nested declaration in its own canonical form, because it knows the
    location of each one.
    """

    @classmethod
    def _name_problem(cls, name: str, at: Loc) -> ValidationProblem | None:
        """The problem to report at `at` when no document of the format may use `name` for a declaration of this kind; None when it may. For Zarr v3, a name may be used when it is a valid extension name under the spec."""
        return name_problem(name, at)

    @classmethod
    def _well_named(cls, name: str) -> bool:
        """Whether a document of the format may use `name` for a declaration of this kind, as decided by `_name_problem`."""
        return cls._name_problem(name, ()) is None

    @classmethod
    def _filed_under(cls, name: str) -> tuple[str | None, dict[str, JSONValue] | None]:
        """Interpret a name used in a document: return the name its definition is filed under, and the configuration encoded in the name.

        By default a name is filed as itself and encodes nothing:
        `(name, None)`. A kind whose names encode configuration overrides
        this. For example, the v3 data type `r16` is filed under `r*` with
        `{"bits": 16}`. A name that no document uses, and that exists only
        to file a definition, returns `(None, None)`.
        """
        return name, None

    def _carrying_name(self, configuration: Mapping[str, JSONValue]) -> str | None:
        """The name that encodes `configuration` for this definition. This is the inverse of `filed_under`: `r16` for `r*` with `{"bits": 16}`. None when this definition's names encode nothing."""
        return None

    @classmethod
    def _named_configuration(
        cls, value: object
    ) -> tuple[str | None, Mapping[str, object] | None, Problems]:
        """Split `value` (a declaration as a document of the format writes it) into `(name, configuration, problems)`, as the module-level `named_configuration` does for a v3 declaration."""
        return named_configuration(value)

    @classmethod
    def _envelope_problems(cls, value: object) -> Problems:
        """Every reason `value` is not a valid envelope for a declaration of this kind in this format. The contents of the configuration are not checked."""
        return envelope_problems(value, allow_must_understand_false=False)

    @classmethod
    def _envelope_json(cls, name: str, configuration: Mapping[str, JSONValue]) -> JSONValue:
        """A declaration with `name` and `configuration`, as a document of the format writes it, in the shortest form every reader accepts. For v3 this is an object."""
        if len(configuration) != 0:
            return {"name": name, "configuration": configuration}
        return {"name": name}

    @classmethod
    def _unclaimed_json(
        cls, json: JSONValue, name: str, configuration: Mapping[str, JSONValue]
    ) -> JSONValue:
        """A declaration of this kind that no definition claims, as a document writes it: `json` is the declaration as written. By default the name and configuration go in the envelope every reader accepts, as for a declaration a definition read."""
        return cls._envelope_json(name, configuration)

    @classmethod
    def _refined(
        cls, data: object, loc: Loc, definition: Definition[Any] | None
    ) -> tuple[JSONValue | None, Problems]:
        """`data`, a declaration of this kind at `loc`, refined to JSON before it is read, with every reason it is not JSON. `definition` is the definition that claims the declaration, if any. By default every number must be finite, as RFC 8259 requires."""
        return refine_json(data, loc)

    @classmethod
    def _undeclared_parts(cls, family: str, name: str, keys: Sequence[str]) -> str:
        """The message for `name`, a name filed under `family` that encodes the configuration members `keys`, which the definition does not declare. For a kind whose names encode configuration; by default the message names the keys."""
        return (
            f"expected a name that {family!r} reads, got {shown(name)}, which encodes "
            f"{said(keys)} that it does not declare"
        )

    @classmethod
    def _in_configuration(cls, loc: Loc, place: Loc) -> Loc:
        """The location in the document of `place`, a location in the configuration of a declaration at `loc`: under `configuration` for v3, or under the declaration itself for a format that writes the parameters next to the name. `place` is `()` for the configuration itself."""
        return (*loc, "configuration", *place)

    @classmethod
    def _name_loc(cls, loc: Loc) -> Loc:
        """The location of the name of a declaration at `loc` that is written as an object: under `name` for v3."""
        return (*loc, "name")

    @classmethod
    def _alias_problems(
        cls, alias: TypeAliasType, definition: Definition[Any], loc: Loc
    ) -> Problems:
        """The problems with using `definition` in a member annotated with `alias`, one of this kind's `declaration_aliases`. Empty by default.

        A kind with an alias that admits only some of its definitions
        overrides this. For example, `ZarrV3StaticCodecDeclaration` admits only
        codecs of static size.
        """
        return ()

    def __init_subclass__(
        cls, *, kind: bool = False, format: Literal[2, 3] | None = None, **kwargs: object
    ) -> None:
        """Register a subclass. `kind=True` declares a kind, with the Zarr `format` of the documents that contain its declarations. This marks the class but not its subclasses, as `typing.Protocol` and SQLAlchemy's `__abstract__` do."""
        # Use the explicit form, not bare `super()`. A dataclass with slots
        # is rebuilt, and the cell that bare `super()` reads refers to the
        # discarded class.
        super(Definition, cls).__init_subclass__(**kwargs)
        # A dataclass with slots is built twice. The second build has no
        # class keywords but reuses the first class's namespace, and the
        # last class built is the one used to read documents. So the mark is
        # kept in the namespace, and the aliases are registered last.
        if kind:
            if format not in (2, 3):
                msg = (
                    f"{cls.__name__}: a kind declares the Zarr format its declarations belong to, "
                    f"format=2 or format=3, got {format!r}"
                )
                raise TypeError(msg)
            cls._is_kind = True
            cls._format = format
        elif format is not None:
            msg = f"{cls.__name__}: only a kind declares a format; a definition of a kind has its kind's"
            raise TypeError(msg)
        for alias in cls.__dict__.get("declaration_aliases", ()):
            _DECLARATION_KINDS[alias] = cls

    def __post_init__(self) -> None:
        refusal = _malformed(self) or self._refusal()
        if refusal is not None:
            raise TypeError(refusal)
        kind = kind_of(self)
        try:
            _vet(self.configuration, None if kind is None else format_of(kind))
        except TypeError as error:
            msg = f"{self.name!r}: {error}"
            raise TypeError(msg) from error

    def __repr__(self) -> str:
        # Keep this short, because readings that contain definitions show
        # them. A full repr would show every function the definition holds,
        # with its address.
        return f"{type(self).__name__}(name={self.name!r})"

    def _refusal(self) -> str | None:
        """A description of what is wrong with the attributes a kind adds; None if nothing is wrong, or if the kind adds no attributes."""
        return None

    def _finite_numbers(self) -> bool:
        """Whether every number in a configuration this definition reads must be finite, as RFC 8259 requires. A kind whose configuration may hold user data overrides this."""
        return True

    @property
    def requires_configuration(self) -> bool:
        """Whether a document must include a configuration, meaning the TypedDict has a required key."""
        return len(typeddict_keys(self.configuration).required) != 0

    def _check_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """Type-check `value` as this definition's configuration, and check the envelope of each nested declaration.

        `zarr_metadata.typed_json.check` does only the type check. This
        method also checks the envelope of each declaration that a
        member holds.
        """
        configuration, problems = _configuration_checked(
            value, self.configuration, loc, finite=self._finite_numbers()
        )
        return configuration, with_input(problems, value, loc)

    def read_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
        """Type-check `value`, then check it with the rules. Return the configuration if it is valid, and every problem.

        The rules run only on a configuration that passed the type check,
        including its bounds, and whose nested declarations are well formed. Such
        a configuration contains only what its TypedDict admits. So a rule
        never receives a member of the wrong type, a member out of bounds,
        or a member the type does not allow. No scope reads the nested
        declarations here, so the rules see none of them as read. A rule about a
        nested declaration (for example, a struct field whose type has values of
        varying size) finds nothing to check. `resolve` reads the declaration in a
        scope and runs every rule.
        """
        configuration, problems = self._check_configuration(value, loc)
        if configuration is None:
            return None, problems
        refused = ruled(self, lambda: self.rules(read_only(configuration), _nothing_nested()), loc)
        return (configuration if len(refused) == 0 else None), (
            *problems,
            *with_input(refused, value, loc),
        )

canonical class-attribute instance-attribute

canonical: Callable[[C], C] = unchanged

A well-typed, allowed configuration in its simplest equivalent spelling.

This handles only the definition's own members. canonicalize puts each nested declaration in its own canonical form, because it knows the location of each one.

configuration instance-attribute

configuration: type[C]

The configuration TypedDict.

declaration_aliases class-attribute

declaration_aliases: tuple[TypeAliasType, ...] = ()

The declaration aliases used to annotate a configuration member that holds a declaration of this kind. For a codec: ZarrV3CodecDeclaration and ZarrV3StaticCodecDeclaration.

label class-attribute

label: str = 'definition'

The name of the kind in error messages, for example "codec".

name instance-attribute

name: str

The name the metadata uses. A scope files the definition under this name.

requires_configuration property

requires_configuration: bool

Whether a document must include a configuration, meaning the TypedDict has a required key.

rules class-attribute instance-attribute

rules: Callable[
    [C, Nested], Iterable[ValidationProblem]
] = no_rules

The spec violations in a well-typed configuration and its nested declarations, located in the configuration.

__init_subclass__

__init_subclass__(
    *,
    kind: bool = False,
    format: Literal[2, 3] | None = None,
    **kwargs: object,
) -> None

Register a subclass. kind=True declares a kind, with the Zarr format of the documents that contain its declarations. This marks the class but not its subclasses, as typing.Protocol and SQLAlchemy's __abstract__ do.

Source code in src/zarr_metadata/definition/_definition.py

def __init_subclass__(
    cls, *, kind: bool = False, format: Literal[2, 3] | None = None, **kwargs: object
) -> None:
    """Register a subclass. `kind=True` declares a kind, with the Zarr `format` of the documents that contain its declarations. This marks the class but not its subclasses, as `typing.Protocol` and SQLAlchemy's `__abstract__` do."""
    # Use the explicit form, not bare `super()`. A dataclass with slots
    # is rebuilt, and the cell that bare `super()` reads refers to the
    # discarded class.
    super(Definition, cls).__init_subclass__(**kwargs)
    # A dataclass with slots is built twice. The second build has no
    # class keywords but reuses the first class's namespace, and the
    # last class built is the one used to read documents. So the mark is
    # kept in the namespace, and the aliases are registered last.
    if kind:
        if format not in (2, 3):
            msg = (
                f"{cls.__name__}: a kind declares the Zarr format its declarations belong to, "
                f"format=2 or format=3, got {format!r}"
            )
            raise TypeError(msg)
        cls._is_kind = True
        cls._format = format
    elif format is not None:
        msg = f"{cls.__name__}: only a kind declares a format; a definition of a kind has its kind's"
        raise TypeError(msg)
    for alias in cls.__dict__.get("declaration_aliases", ()):
        _DECLARATION_KINDS[alias] = cls

read_configuration

read_configuration(
    value: object, loc: Loc = ()
) -> tuple[C | None, Problems]

Type-check value, then check it with the rules. Return the configuration if it is valid, and every problem.

The rules run only on a configuration that passed the type check, including its bounds, and whose nested declarations are well formed. Such a configuration contains only what its TypedDict admits. So a rule never receives a member of the wrong type, a member out of bounds, or a member the type does not allow. No scope reads the nested declarations here, so the rules see none of them as read. A rule about a nested declaration (for example, a struct field whose type has values of varying size) finds nothing to check. resolve reads the declaration in a scope and runs every rule.

Source code in src/zarr_metadata/definition/_definition.py

def read_configuration(self, value: object, loc: Loc = ()) -> tuple[C | None, Problems]:
    """Type-check `value`, then check it with the rules. Return the configuration if it is valid, and every problem.

    The rules run only on a configuration that passed the type check,
    including its bounds, and whose nested declarations are well formed. Such
    a configuration contains only what its TypedDict admits. So a rule
    never receives a member of the wrong type, a member out of bounds,
    or a member the type does not allow. No scope reads the nested
    declarations here, so the rules see none of them as read. A rule about a
    nested declaration (for example, a struct field whose type has values of
    varying size) finds nothing to check. `resolve` reads the declaration in a
    scope and runs every rule.
    """
    configuration, problems = self._check_configuration(value, loc)
    if configuration is None:
        return None, problems
    refused = ruled(self, lambda: self.rules(read_only(configuration), _nothing_nested()), loc)
    return (configuration if len(refused) == 0 else None), (
        *problems,
        *with_input(refused, value, loc),
    )

EmptyConfiguration

Bases: TypedDict

The configuration of a definition with nothing to configure. Its declaration is written with its name only.

Source code in src/zarr_metadata/definition/_definition.py

class EmptyConfiguration(TypedDict, closed=True):
    """The configuration of a definition with nothing to configure. Its declaration is written with its name only."""

RefusedDeclaration dataclass

Bases: Generic[D]

A declaration that could not be read: it is not a declaration at all, not JSON, or rejected by the definition that claims its name. Its problems explain why.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class RefusedDeclaration(Generic[D]):
    """A declaration that could not be read: it is not a declaration at all, not JSON, or rejected by the definition that claims its name. Its problems explain why."""

    json: JSONValue | UNSET
    """The declaration as written, refined (arrays as tuples). `UNSET` when it is not JSON, which no document can contain."""
    name: str | None
    """The name the declaration is written with; None when it has no name."""
    read_as: type[Definition[Any]]
    """The kind of metadata the declaration was read as."""
    definition: D | None = None
    """The definition that claims the name and rejected the declaration; None when nothing in scope claims it, or when the declaration has no name."""
    nested: Nested = dataclasses.field(default_factory=_nothing_nested)
    """The nested declarations, each as the scope read it; empty when the configuration was not checked against its TypedDict."""

    def __eq__(self, other: object) -> bool:
        if not is_declaration(other):
            return NotImplemented
        return declaration_key(self) == declaration_key(other)

    def __hash__(self) -> int:
        return hash(declaration_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. Remove type arguments from
        # `read_as`, as `resolve` does.
        kind = as_kind(self.read_as)
        object.__setattr__(self, "read_as", kind)
        definition = cast("object", self.definition)
        refusal = None if definition is None else _misread(definition, kind, self.name)
        if refusal is not None:
            raise TypeError(refusal)
        if self.json is not UNSET:
            object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(self, "nested", MappingProxyType(dict(self.nested)))

    def __reduce__(self) -> tuple[Callable[..., RefusedDeclaration[Any]], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the declaration from its constructor arguments.
        return _refused_field, (
            UNSET if self.json is UNSET else copied(self.json),
            self.name,
            self.read_as,
            self.definition,
            dict(self.nested),
        )

definition class-attribute instance-attribute

definition: D | None = None

The definition that claims the name and rejected the declaration; None when nothing in scope claims it, or when the declaration has no name.

json instance-attribute

json: JSONValue | UNSET

The declaration as written, refined (arrays as tuples). UNSET when it is not JSON, which no document can contain.

name instance-attribute

name: str | None

The name the declaration is written with; None when it has no name.

nested class-attribute instance-attribute

nested: Nested = dataclasses.field(
    default_factory=_nothing_nested
)

The nested declarations, each as the scope read it; empty when the configuration was not checked against its TypedDict.

read_as instance-attribute

read_as: type[Definition[Any]]

The kind of metadata the declaration was read as.

ScopeConflictError

Bases: ValueError

Raised when two scopes, or a scope and a reading, give one name two different meanings, or when one would lose a meaning the other has.

Every conflict is stored in .conflicts, in the same way that MetadataValidationError stores every problem.

Source code in src/zarr_metadata/definition/_scope.py

class ScopeConflictError(ValueError):
    """Raised when two scopes, or a scope and a reading, give one name two different meanings, or when one would lose a meaning the other has.

    Every conflict is stored in `.conflicts`, in the same way that
    `MetadataValidationError` stores every problem.
    """

    def __init__(self, conflicts: Sequence[Conflict]) -> None:
        self.conflicts: tuple[Conflict, ...] = tuple(conflicts)
        """Every conflict, in the order found; the message joins them."""
        super().__init__("; ".join(str(conflict) for conflict in self.conflicts))

    def __reduce__(
        self,
    ) -> tuple[type[ScopeConflictError], tuple[tuple[Conflict, ...]], dict[str, object]]:
        # Pickle and copy the exception as it was raised, keeping its notes
        # and attributes. The default exception reduce calls the constructor
        # with the message, not the conflicts.
        return type(self), (self.conflicts,), dict(self.__dict__)

conflicts instance-attribute

conflicts: tuple[Conflict, ...] = tuple(conflicts)

Every conflict, in the order found; the message joins them.

UnclaimedDeclaration dataclass

A declaration that nothing in scope claims: an extension that the scope does not check. Accepting such declarations is what keeps the format open.

It is equal to another unclaimed declaration when both are of the same kind, have the same name, and a document writes the same JSON for both. Nothing in scope interprets the declaration, so it is compared as JSON text, as declaration_key defines.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, slots=True, kw_only=True)
class UnclaimedDeclaration:
    """A declaration that nothing in scope claims: an extension that the scope does not check. Accepting such declarations is what keeps the format open.

    It is equal to another unclaimed declaration when both are of the same kind,
    have the same name, and a document writes the same JSON for both.
    Nothing in scope interprets the declaration, so it is compared as JSON text,
    as `declaration_key` defines.
    """

    json: JSONValue
    """The declaration as written, refined (arrays as tuples). It is not used for equality."""
    name: str
    """The name that nothing in scope claims."""
    read_as: type[Definition[Any]]
    """The kind of metadata the declaration was read as: the kind a claiming definition would have."""
    configuration: Mapping[str, JSONValue] = dataclasses.field(init=False)
    """The configuration as written, not checked; empty when no configuration is written."""

    def __eq__(self, other: object) -> bool:
        if not is_declaration(other):
            return NotImplemented
        return declaration_key(self) == declaration_key(other)

    def __hash__(self) -> int:
        return hash(declaration_key(self))

    def __post_init__(self) -> None:
        # Check the annotations at runtime. Remove type arguments from
        # `read_as`, as `resolve` does.
        object.__setattr__(self, "read_as", as_kind(self.read_as))
        name = cast("object", self.name)
        if not isinstance(name, str) or not self.read_as._well_named(name):
            msg = (
                "a declaration nothing in scope claims is named as a document names a "
                f"{self.read_as.label}, got {name!r}"
            )
            raise TypeError(msg)
        _, written, _ = self.read_as._named_configuration(self.json)
        configuration: Mapping[str, object] = {} if written is None else written
        object.__setattr__(self, "json", frozen(self.json))
        object.__setattr__(
            self, "configuration", frozen(cast("Mapping[str, JSONValue]", configuration))
        )

    def __reduce__(self) -> tuple[Callable[..., UnclaimedDeclaration], tuple[object, ...]]:
        # Read-only views cannot be pickled, so pickle the declaration from its constructor arguments.
        return _unclaimed_field, (copied(self.json), self.name, self.read_as)

    @property
    def definition(self) -> None:
        """The definition that read the declaration. Always None, because no definition read it."""
        return None

    @property
    def nested(self) -> Nested:
        """The nested declarations as the scope read them. Always empty, because nothing read the configuration."""
        return _nothing_nested()

    def to_json(self) -> JSONValue:
        """The declaration as a document writes it, sharing no objects with the declaration. The configuration is as written, in the envelope form every reader accepts, as `AcceptedDeclaration.to_json` writes it."""
        return copied(written_json(self))

configuration class-attribute instance-attribute

configuration: Mapping[str, JSONValue] = dataclasses.field(
    init=False
)

The configuration as written, not checked; empty when no configuration is written.

definition property

definition: None

The definition that read the declaration. Always None, because no definition read it.

json instance-attribute

json: JSONValue

The declaration as written, refined (arrays as tuples). It is not used for equality.

name instance-attribute

name: str

The name that nothing in scope claims.

nested property

nested: Nested

The nested declarations as the scope read them. Always empty, because nothing read the configuration.

read_as instance-attribute

read_as: type[Definition[Any]]

The kind of metadata the declaration was read as: the kind a claiming definition would have.

to_json

to_json() -> JSONValue

The declaration as a document writes it, sharing no objects with the declaration. The configuration is as written, in the envelope form every reader accepts, as AcceptedDeclaration.to_json writes it.

Source code in src/zarr_metadata/definition/_definition.py

def to_json(self) -> JSONValue:
    """The declaration as a document writes it, sharing no objects with the declaration. The configuration is as written, in the envelope form every reader accepts, as `AcceptedDeclaration.to_json` writes it."""
    return copied(written_json(self))

WithFillValue dataclass

Bases: Definition[C]

A definition of a type whose arrays have a fill value: the fill value's JSON shape, the rules for it, and its canonical spelling.

This is not a kind. The data type kind of each format derives from it, so fill_value_problems can check a fill value for either format.

fill_value is the JSON shape of a fill value, as an annotation. The checker reads it the same way it reads a configuration's members, including any range. fill_value_rules yields the spec violations in a fill value of that shape that the type cannot express. It receives the configuration, the nested declarations as the scope read them, and the typed fill value. fill_value_canonical writes a fill value without problems in the single canonical spelling of its value. So two fill values are the same value of the type exactly when their canonical spellings are written the same way.

Source code in src/zarr_metadata/definition/_definition.py

@dataclass(frozen=True, kw_only=True, slots=True, repr=False)
class WithFillValue(Definition[C]):
    """A definition of a type whose arrays have a fill value: the fill value's JSON shape, the rules for it, and its canonical spelling.

    This is not a kind. The data type kind of each format derives from it,
    so `fill_value_problems` can check a fill value for either format.

    `fill_value` is the JSON shape of a fill value, as an annotation. The
    checker reads it the same way it reads a configuration's members,
    including any range. `fill_value_rules` yields the spec violations in
    a fill value of that shape that the type cannot express. It receives
    the configuration, the nested declarations as the scope read them, and the
    typed fill value. `fill_value_canonical` writes a fill value without
    problems in the single canonical spelling of its value. So two fill
    values are the same value of the type exactly when their canonical
    spellings are written the same way.
    """

    fill_value: object = JSONValue
    """The JSON shape of a fill value, as an annotation, such as `Int8FillValue`."""
    fill_value_rules: Callable[[C, Nested, Any], Iterable[ValidationProblem]] = no_rules
    """The spec violations in a fill value of that shape, located in the fill value."""
    fill_value_canonical: Callable[[C, Nested, Any], JSONValue] = fill_value_as_written
    """A fill value without problems, in the single canonical spelling of its value."""

    def _refusal(self) -> str | None:
        try:
            _fill_value_parser(self.fill_value)
        except TypeError as error:
            return f"{self.name!r}: fill_value: {error}"
        return None

fill_value class-attribute instance-attribute

fill_value: object = JSONValue

The JSON shape of a fill value, as an annotation, such as Int8FillValue.

fill_value_canonical class-attribute instance-attribute

fill_value_canonical: Callable[
    [C, Nested, Any], JSONValue
] = fill_value_as_written

A fill value without problems, in the single canonical spelling of its value.

fill_value_rules class-attribute instance-attribute

fill_value_rules: Callable[
    [C, Nested, Any], Iterable[ValidationProblem]
] = no_rules

The spec violations in a fill value of that shape, located in the fill value.

canonical_fill_value

canonical_fill_value(
    data_type: ResolvedDeclaration[F], value: object
) -> JSONValue | UNSET

value (a fill value of data_type, a data type declaration a scope read) in the single canonical spelling of its value; UNSET when it has a problem.

The spelling comes from the data type's fill_value_canonical. Two fill values of a data type are the same value exactly when their canonical spellings are written the same way: the same JSON as json.dumps writes it. == is not the same test, because it treats -0.0 as equal to 0.0. A fill value with a problem found by fill_value_problems has no canonical spelling, just as canonical_of returns none for a declaration with a problem. The result is UNSET and not None, because None is the JSON null, which is a valid fill value for a data type the scope did not read (such a data type keeps a fill value as written).

Source code in src/zarr_metadata/definition/_definition.py

def canonical_fill_value(data_type: ResolvedDeclaration[F], value: object) -> JSONValue | UNSET:
    """`value` (a fill value of `data_type`, a data type declaration a scope read) in the single canonical spelling of its value; `UNSET` when it has a problem.

    The spelling comes from the data type's `fill_value_canonical`. Two
    fill values of a data type are the same value exactly when their
    canonical spellings are written the same way: the same JSON as
    `json.dumps` writes it. `==` is not the same test, because it treats
    `-0.0` as equal to `0.0`. A fill value with a problem found by
    `fill_value_problems` has no canonical spelling, just as `canonical_of`
    returns none for a declaration with a problem. The result is `UNSET` and not
    `None`, because `None` is the JSON `null`, which is a valid fill value
    for a data type the scope did not read (such a data type keeps a fill
    value as written).
    """
    if len(fill_value_problems(data_type, value)) != 0:
        return UNSET
    refined, _ = refine_json(value, ())
    return spelled_canonically(data_type, refined)

canonical_of

canonical_of(
    resolved: ResolvedDeclaration[Any],
    problems: Sequence[ValidationProblem],
) -> JSONValue | None

resolved (a declaration a scope read) in its simplest equivalent spelling, as canonicalize writes it; None when it has a problem.

problems are the declaration's problems, as returned by resolve, or as with_problems returns them for each declaration of a reading. Only a declaration with no problems has a simplest spelling. A simpler spelling of a declaration with a problem would erase what its author wrote (such as a key its TypedDict does not declare), or would write something invalid. The result is built from what the scope read, without reading the declaration again, and is plain JSON that shares nothing with the declaration.

Source code in src/zarr_metadata/definition/_definition.py

def canonical_of(
    resolved: ResolvedDeclaration[Any], problems: Sequence[ValidationProblem]
) -> JSONValue | None:
    """`resolved` (a declaration a scope read) in its simplest equivalent spelling, as `canonicalize` writes it; None when it has a problem.

    `problems` are the declaration's problems, as returned by `resolve`, or as
    `with_problems` returns them for each declaration of a reading. Only a declaration
    with no problems has a simplest spelling. A simpler spelling of a
    declaration with a problem would erase what its author wrote (such as a key
    its TypedDict does not declare), or would write something invalid.
    The result is built from what the scope read, without reading the
    declaration again, and is plain JSON that shares nothing with the declaration.
    """
    if len(problems) != 0:
        return None
    simplest = _simplest(resolved)
    return None if simplest is None else copied(simplest)

configuration_of

configuration_of(
    resolved: ResolvedDeclaration[Any],
    definition: Definition[C],
) -> C | None

The configuration of resolved, typed as definition declares it, if definition accepted it.

AcceptedDeclaration stores its configuration as a read-only mapping. When called with the definition that read the declaration, or one equal to it (as with a pickled declaration), this returns a copy of it, a plain dict typed as the TypedDict, which shares nothing with the declaration. Returns None for any other declaration: one that another definition read, one that no definition claims, and one that definition claimed and refused.

Source code in src/zarr_metadata/definition/_definition.py

def configuration_of(resolved: ResolvedDeclaration[Any], definition: Definition[C]) -> C | None:
    """The configuration of `resolved`, typed as `definition` declares it, if `definition` accepted it.

    `AcceptedDeclaration` stores its configuration as a read-only mapping. When
    called with the definition that read the declaration, or one equal to it
    (as with a pickled declaration), this returns a copy of it, a plain `dict`
    typed as the TypedDict, which shares nothing with the declaration. Returns
    None for any other declaration: one that another definition read, one that
    no definition claims, and one that `definition` claimed and refused.
    """
    if not isinstance(resolved, AcceptedDeclaration) or resolved.definition != definition:
        return None
    return cast("C", copied(resolved.configuration))

fill_value_problems

fill_value_problems(
    data_type: ResolvedDeclaration[F],
    value: object,
    loc: Loc = (),
) -> Problems

The problems with value as a fill value of data_type (a data type declaration a scope read).

value is refined to JSON first. If it is not JSON, that is reported first, whatever the data type. It is then checked against the JSON shape that the data type's definition declares, and by the fill value rules, in the same way read_configuration reads a configuration: a key the shape does not declare is reported and left out, and the rules still check the rest. The rules see the nested declarations of the configuration as the scope read them, so a struct checks each field's fill value against that field's own type. If the scope did not read the data type (it is out of scope or invalid), a JSON fill value is not checked. loc prefixes every problem.

Source code in src/zarr_metadata/definition/_definition.py

def fill_value_problems(
    data_type: ResolvedDeclaration[F], value: object, loc: Loc = ()
) -> Problems:
    """The problems with `value` as a fill value of `data_type` (a data type declaration a scope read).

    `value` is refined to JSON first. If it is not JSON, that is reported
    first, whatever the data type. It is then checked against the JSON
    shape that the data type's definition declares, and by the fill value
    rules, in the same way `read_configuration` reads a configuration: a
    key the shape does not declare is reported and left out, and the rules
    still check the rest. The rules see the nested declarations of the
    configuration as the scope read them, so a struct checks each field's
    fill value against that field's own type. If the scope did not read
    the data type (it is out of scope or invalid), a JSON fill value is not
    checked. `loc` prefixes every problem.
    """
    refined, problems = refine_json(value, loc)
    if len(problems) != 0 or not isinstance(data_type, AcceptedDeclaration):
        return with_input(problems, value, loc)
    definition, configuration = data_type.definition, data_type.configuration
    typed, problems = _fill_value_parser(definition.fill_value)(refined, loc)
    if not _usable(problems):
        return with_input(problems, value, loc)
    refused = ruled(
        definition,
        lambda: definition.fill_value_rules(read_only(configuration), data_type.nested, typed),
        loc,
    )
    return with_input((*problems, *refused), value, loc)

resolve

resolve(
    data: object,
    kind: type[D],
    context: Context,
    loc: Loc = (),
) -> tuple[ResolvedDeclaration[D], Problems]

Read data (one declaration) as a kind in context. Return the result of the reading, and every problem.

This runs all three steps for one declaration. data is refined to JSON and its envelope is checked. An extra envelope member, a configuration that is not an object, and a must_understand that is not a boolean or is false are each a problem. The name is mapped to a definition in context. The configuration is checked against its TypedDict and by its rules. Each nested declaration found by the check is read the same way, in the same scope. A problem in a nested declaration belongs to that declaration and is reported at its location, as with a document's declarations. The result is an AcceptedDeclaration read by the definition that claims the name; an UnclaimedDeclaration when nothing in scope claims the name (an extension with no model, not checked, which is what keeps the format open); or a RefusedDeclaration, with the problems that explain why. loc prefixes every problem. kind must be one of the kinds of a format (for v3, those ZarrV3Kind names; for v2, those ZarrV2Kind names), with or without type arguments, and context a scope of that format. Anything else raises TypeError.

Source code in src/zarr_metadata/definition/_definition.py

def resolve(
    data: object, kind: type[D], context: Context, loc: Loc = ()
) -> tuple[ResolvedDeclaration[D], Problems]:
    """Read `data` (one declaration) as a `kind` in `context`. Return the result of the reading, and every problem.

    This runs all three steps for one declaration. `data` is refined to JSON and
    its envelope is checked. An extra envelope member, a `configuration` that is not
    an object, and a `must_understand` that is not a boolean or is `false`
    are each a problem. The name is mapped to a definition in `context`.
    The configuration is checked against its TypedDict and by its rules.
    Each nested declaration found by the check is read the same way, in the same
    scope. A problem in a nested declaration belongs to that declaration and is
    reported at its location, as with a document's declarations. The result is
    an `AcceptedDeclaration` read by the definition that claims the name; an
    `UnclaimedDeclaration` when nothing in scope claims the name (an extension
    with no model, not checked, which is what keeps the format open); or a
    `RefusedDeclaration`, with the problems that explain why. `loc` prefixes
    every problem. `kind` must be one of the kinds of a format (for v3,
    those `ZarrV3Kind` names; for v2, those `ZarrV2Kind` names), with or
    without type arguments, and `context` a scope of that format. Anything
    else raises `TypeError`.
    """
    if not _is_scope(context):
        msg = f"a declaration is read in a scope, such as CORE_AND_EXTENSIONS_CONTEXT, got {described(context)}"
        raise TypeError(msg)
    asked = as_kind(kind)
    _check_format(asked, context)
    name = asked._named_configuration(data)[0]
    definition = (
        None if name is None or not asked._well_named(name) else context.claimant(asked, name)
    )
    refined, problems = asked._refined(data, loc, definition)
    return resolve_refined(data, refined, problems, kind, context, loc)

with_problems

with_problems(
    declarations: Iterable[
        tuple[Loc, ResolvedDeclaration[Any]]
    ],
    problems: Sequence[ValidationProblem],
) -> Iterator[
    tuple[Loc, ResolvedDeclaration[Any], Problems]
]

Yield each of declarations with its location and the problems in problems located inside it, including inside its nested declarations.

declarations and problems must come from the same read: a reading's declarations() and problems, or declarations_of a declaration and the problems that resolve returned with it. A declaration's problems include those found when it was read, and those the document found with it in its position (its place in the pipeline, the chunk it receives, the array's shape). So a declaration with no problems is valid in that position, and canonical_of can write it. This groups the problems at every depth, as zod's treeifyError groups issues: a problem with a shard's inner codec belongs to the inner codec and to the shard. Each declaration comes before its nested declarations, in the order of declarations_of, so the last declaration that lists a problem is the innermost declaration containing it. A problem outside every declaration (with the fill value or the shape) is not listed under any declaration.

Source code in src/zarr_metadata/definition/_definition.py

def with_problems(
    declarations: Iterable[tuple[Loc, ResolvedDeclaration[Any]]],
    problems: Sequence[ValidationProblem],
) -> Iterator[tuple[Loc, ResolvedDeclaration[Any], Problems]]:
    """Yield each of `declarations` with its location and the problems in `problems` located inside it, including inside its nested declarations.

    `declarations` and `problems` must come from the same read: a reading's
    `declarations()` and `problems`, or `declarations_of` a declaration and the problems that
    `resolve` returned with it. A declaration's problems include those found when
    it was read, and those the document found with it in its position (its
    place in the pipeline, the chunk it receives, the array's shape). So a
    declaration with no problems is valid in that position, and `canonical_of`
    can write it. This groups the problems at every depth, as zod's
    `treeifyError` groups issues: a problem with a shard's inner codec
    belongs to the inner codec and to the shard. Each declaration comes before
    its nested declarations, in the order of `declarations_of`, so the last declaration that
    lists a problem is the innermost declaration containing it. A problem
    outside every declaration (with the fill value or the shape) is not listed
    under any declaration.
    """
    located = list(declarations)
    held: dict[Loc, list[ValidationProblem]] = {loc: [] for loc, _ in located}
    for found in problems:
        for depth in range(len(found.loc) + 1):
            holder = held.get(found.loc[:depth])
            if holder is not None:
                holder.append(found)
    for loc, declaration in located:
        yield loc, declaration, tuple(held[loc])