document

The mechanics every deeperfly document shares (ADR-0059).

A document (a run config, a manifest, a calibration file) is TOML parsed strictly into a Table: its keys are the fields, nothing is coerced beyond TOML's own types (an array is a tuple, an int widens to a float), and an unknown key, a wrong type or a missing key fails at load as [table].key. Loading also refuses a commented-out header that live keys follow and a format_version this deeperfly does not read. A run config is copied verbatim, never re-emitted.

ConfigError

Bases: ValueError

A document that cannot be loaded, reported as [table].key: what is wrong.

Source code in src/deeperfly/document.py

class ConfigError(ValueError):
    """A document that cannot be loaded, reported as `[table].key: what is wrong`."""

Named dataclass

Marks a path key that also takes names: names() are not paths.

Metadata of the key's annotation, Annotated[Path, ..., Named(names)]. A value among the names is left as typed where a path would be made absolute (overrides).

Source code in src/deeperfly/document.py

@dataclass(frozen=True)
class Named:
    """Marks a path key that also takes names: `names()` are not paths.

    Metadata of the key's annotation, `Annotated[Path, ..., Named(names)]`. A value
    among the names is left as typed where a path would be made absolute
    ([overrides][deeperfly.document.overrides]).
    """

    names: Callable[[], Container[str]]

Table

Bases: BaseModel

A document table: frozen, closed to unknown keys, and strict about types.

Subclasses declare their keys as fields. A rule that spans fields is a model_validator(mode="after") raising ValueError, which parse reports as [table]: the message; a rule on one key is a validator of that key, reported as [table].key: the message.

Fields listed in _reserved are not keys at all -- where the document is, not something it declares -- and are refused when a file names them.

Source code in src/deeperfly/document.py

class Table(BaseModel):
    """A document table: frozen, closed to unknown keys, and strict about types.

    Subclasses declare their keys as fields. A rule that spans fields is a
    `model_validator(mode="after")` raising `ValueError`, which
    [parse][deeperfly.document.parse] reports as `[table]: the message`; a rule
    on one key is a validator of that key, reported as `[table].key: the message`.

    Fields listed in `_reserved` are not keys at all -- where the document *is*,
    not something it declares -- and are refused when a file names them.
    """

    model_config = ConfigDict(frozen=True, extra="forbid", strict=True)

    _reserved: ClassVar[tuple[str, ...]] = ()

    @model_validator(mode="before")
    @classmethod
    def _arrays_are_tuples(cls, data):
        """TOML arrays as tuples, so a strict tuple field accepts them."""
        if isinstance(data, Mapping):
            named = [k for k in cls._reserved if k in data]
            if named:
                raise ValueError(
                    f"{named[0]}: not a key; it is where the document is, not "
                    "something it declares"
                )
            return _tupled(data)
        return data

at

at(path: str) -> str

A key's dotted path as the file writes it: [data].split, bare at the root.

Source code in src/deeperfly/document.py

def at(path: str) -> str:
    """A key's dotted path as the file writes it: `[data].split`, bare at the root."""
    head, _, tail = path.rpartition(".")
    return f"[{head}].{tail}" if head else tail

check_commented_headers

check_commented_headers(text: str) -> None

Refuse a commented-out table header that uncommented keys follow.

# [smoothing] looks like the way to switch a stage off, but TOML binds the keys below it to the previous table (ADR-0028).

Source code in src/deeperfly/document.py

def check_commented_headers(text: str) -> None:
    """Refuse a commented-out table header that uncommented keys follow.

    `# [smoothing]` looks like the way to switch a stage off, but TOML binds the keys
    below it to the previous table (ADR-0028).
    """
    pending: tuple[int, str] | None = None
    for i, line in enumerate(text.splitlines(), start=1):
        header = _COMMENTED_HEADER.match(line)
        if header:
            pending = (i, header.group(1))
            continue
        if _HEADER.match(line):
            pending = None
            continue
        if pending and _KEY.match(line):
            lineno, name = pending
            raise ConfigError(
                f"line {lineno}: {name} is commented out but line {i} still "
                f"declares a key, which TOML would attach to the table above. "
                f"Comment out its keys too, or delete the section -- removing a "
                f"table is how a stage is turned off."
            )

check_format_version

check_format_version(
    data: Mapping,
    kind: str,
    expected: int = DOCUMENT_FORMAT_VERSION,
    hint: str = "",
) -> None

Refuse a document written for another version of the schema.

Read before anything else, so another version's file is reported as such rather than as an unknown key; a document that declares none is version 1. kind names the document in the error and hint is appended to it.

Source code in src/deeperfly/document.py

def check_format_version(
    data: Mapping, kind: str, expected: int = DOCUMENT_FORMAT_VERSION, hint: str = ""
) -> None:
    """Refuse a document written for another version of the schema.

    Read before anything else, so another version's file is reported as such rather
    than as an unknown key; a document that declares none is version 1. `kind` names
    the document in the error and `hint` is appended to it.
    """
    version = data.get("format_version", 1)
    if version == expected:
        return
    raise ConfigError(
        f"format_version: this {kind} is version {version!r}; this deeperfly "
        f"reads version {expected}" + (f". {hint}" if hint else "")
    )

format_errors

format_errors(
    exc: ValidationError,
    cls: type[BaseModel],
    path: str = "",
) -> str

The first problem a ValidationError reports, as [table].key: ....

An unknown key lists the table's own keys; missing keys of one table are named together; a rule's own message is passed through under its table, or at its key; a type error says what was expected and what was given.

Source code in src/deeperfly/document.py

def format_errors(exc: ValidationError, cls: type[BaseModel], path: str = "") -> str:
    """The first problem a `ValidationError` reports, as `[table].key: ...`.

    An unknown key lists the table's own keys; missing keys of one table are
    named together; a rule's own message is passed through under its table, or at
    its key; a type error says what was expected and what was given.
    """
    errors = exc.errors()
    # A union of tuples fails in every member, the shorter ones only on length.
    first = next((e for e in errors if e["type"] != "too_long"), errors[0])
    loc, lands_on = _walk(cls, first["loc"])
    where = _where(path, loc)
    kind = first["type"]
    if kind == "extra_forbidden":
        owner = _walk(cls, loc[:-1])[1]
        takes = (
            [n for n in owner.model_fields if n not in owner._reserved]
            if isinstance(owner, type) and issubclass(owner, Table)
            else []
        )
        hint_text = f"; {table(_where(path, loc[:-1]))} takes " + ", ".join(takes)
        return f"{at(where)}: unknown key" + (hint_text if takes else "")
    if kind == "missing":
        parent = first["loc"][:-1]
        names = [
            str(e["loc"][-1])
            for e in errors
            if e["type"] == "missing" and e["loc"][:-1] == parent
        ]
        return f"{table(_where(path, loc[:-1]))}: missing " + ", ".join(names)
    if kind in ("value_error", "assertion_error"):
        message = first["msg"].removeprefix("Value error, ")
        if not where:
            return message
        return f"{table(where) if _is_table(lands_on) else at(where)}: {message}"
    if kind == "union_tag_not_found":
        key = first["ctx"]["discriminator"].strip("'")
        return f"{table(where)}: missing {key}"
    if kind == "union_tag_invalid":
        ctx = first["ctx"]
        key = ctx["discriminator"].strip("'")
        where = _where(path, [*loc, key])
        return f"{at(where)}: {ctx['tag']!r} is not one of {ctx['expected_tags']}"
    if kind == "literal_error":
        return (
            f"{at(where)}: {first['input']!r} is not one of {first['ctx']['expected']}"
        )
    if kind in _EXPECTED:
        return f"{at(where)}: expected {_EXPECTED[kind]}, got {_kind(first['input'])}"
    return f"{at(where)}: {first['msg']}"

load_document

load_document(path: str | Path) -> tuple[dict, Path]

Parse a config file, linted, with the directory it lives in.

Every relative path in it resolves against that directory (ADR-0085), so it travels with the document.

Source code in src/deeperfly/document.py

def load_document(path: str | Path) -> tuple[dict, Path]:
    """Parse a config file, linted, with the directory it lives in.

    Every relative path in it resolves against that directory (ADR-0085), so it
    travels with the document.
    """
    file = Path(path).expanduser()
    text = file.read_text()
    check_commented_headers(text)
    try:
        data = tomllib.loads(text)
    except tomllib.TOMLDecodeError as exc:
        raise ConfigError(f"{file}: not valid TOML: {exc}") from exc
    return data, file.resolve().parent

match_names

match_names(
    patterns: Iterable[str],
    names: Sequence[str],
    where: str,
) -> list[str]

The names a selector claims, in names order; empty patterns mean all.

A list unions its matches. An entry with a glob character must match at least one name and one without must be a name, since either miss is a typo; where names the selector in the error.

Source code in src/deeperfly/document.py

def match_names(patterns: Iterable[str], names: Sequence[str], where: str) -> list[str]:
    """The names a selector claims, in `names` order; empty patterns mean all.

    A list unions its matches. An entry with a glob character must match at least one
    name and one without must be a name, since either miss is a typo; `where` names
    the selector in the error.
    """
    patterns = list(patterns)
    if not patterns:
        return list(names)
    claimed: set[str] = set()
    for pattern in patterns:
        if any(c in pattern for c in GLOB_CHARS):
            hits = [n for n in names if fnmatchcase(n, pattern)]
            if not hits:
                raise ConfigError(
                    f"{where}: pattern {pattern!r} matches none of " + ", ".join(names)
                )
        elif pattern in names:
            hits = [pattern]
        else:
            raise ConfigError(f"{where}: {pattern!r} is not one of " + ", ".join(names))
        claimed.update(hits)
    return [n for n in names if n in claimed]

overrides

overrides(
    cls: type[Table], items: Iterable[str]
) -> dict[str, Any]

The --set KEY=VALUE items of a command, as values by dotted key.

A value is read as TOML (80, true, [0, 180, 0]), and a list may drop its brackets (0,180,0); none restores the key's default (DEFAULT); a key that takes only text takes the value verbatim; and a relative path is relative to the current directory, not to the document. cls is the document the keys belong to, which types them; an unknown key is left for validation to name.

Raises:

Type	Description
ConfigError	If an item is not KEY=VALUE.

Source code in src/deeperfly/document.py

def overrides(cls: type[Table], items: Iterable[str]) -> dict[str, Any]:
    """The `--set KEY=VALUE` items of a command, as values by dotted key.

    A value is read as TOML (`80`, `true`, `[0, 180, 0]`), and a list may drop its
    brackets (`0,180,0`); `none` restores the key's default (`DEFAULT`); a key
    that takes only text takes the value verbatim; and a relative path is relative to
    the current directory, not to the document. `cls` is the document the keys belong
    to, which types them; an unknown key is left for validation to name.

    Raises:
        ConfigError: If an item is not `KEY=VALUE`.
    """
    changes: dict[str, Any] = {}
    for item in items:
        match = _OVERRIDE.fullmatch(item)
        if match is None:
            raise ConfigError(f"{item!r} is not KEY=VALUE")
        key, raw = match.groups()
        changes[key] = _override_value(_key_type(cls, key), raw)
    return changes

parse

parse(cls: type[T], raw: Any, path: str = '') -> T

Build a table from a parsed TOML mapping, strictly.

path is the table's dotted path in the document ("render.outputs.pose2d"), which errors are named by.

Raises:

Type	Description
ConfigError	If raw is not a table or breaks any of the table's rules.

Source code in src/deeperfly/document.py

def parse[T: Table](cls: type[T], raw: Any, path: str = "") -> T:
    """Build a table from a parsed TOML mapping, strictly.

    `path` is the table's dotted path in the document (`"render.outputs.pose2d"`),
    which errors are named by.

    Raises:
        ConfigError: If `raw` is not a table or breaks any of the table's rules.
    """
    if not isinstance(raw, Mapping):
        raise ConfigError(f"{table(path)} must be a table, got {_kind(raw)}")
    try:
        return cls.model_validate(raw)
    except ValidationError as exc:
        raise ConfigError(format_errors(exc, cls, path)) from exc

table

table(path: str) -> str

A dotted path to a TABLE, as the file writes it: data is [data].

Source code in src/deeperfly/document.py

def table(path: str) -> str:
    """A dotted path to a TABLE, as the file writes it: `data` is `[data]`."""
    return f"[{path}]" if path else "the document"

under_document

under_document(value: Path | str, root: Path) -> Path

A document path resolved against root when it is relative.

root is the document's own directory (ADR-0085). The join is normalized without following symlinks, so a .. is taken as typed and the path a run prints is the one the user would type; a manifest's paths take .. as the filesystem does instead (under_folder).

Source code in src/deeperfly/document.py

def under_document(value: Path | str, root: Path) -> Path:
    """A document path resolved against `root` when it is relative.

    `root` is the document's own directory (ADR-0085). The join is normalized without
    following symlinks, so a `..` is taken as typed and the path a run prints is the
    one the user would type; a manifest's paths take `..` as the filesystem does
    instead ([under_folder][deeperfly.recording.under_folder]).
    """
    p = Path(value).expanduser()
    return p if p.is_absolute() else Path(os.path.normpath(root / p))

with_keys

with_keys(
    data: Mapping, changes: Mapping[str, Any]
) -> dict

A copy of a parsed document with values set by dotted key.

A DEFAULT value drops the key. A table the document lacks is created, and a key that is not a table where the path needs one is replaced by one, which validation then reports.

Source code in src/deeperfly/document.py

def with_keys(data: Mapping, changes: Mapping[str, Any]) -> dict:
    """A copy of a parsed document with values set by dotted key.

    A `DEFAULT` value drops the key. A table the document lacks is created, and a
    key that is not a table where the path needs one is replaced by one, which
    validation then reports.
    """
    out: dict = {k: _copied(v) for k, v in data.items()}
    for key, value in changes.items():
        *tables, name = key.split(".")
        owner = out
        for part in tables:
            inner = owner.get(part)
            if not isinstance(inner, dict):
                inner = owner[part] = {}
            owner = inner
        if value is DEFAULT:
            owner.pop(name, None)
        else:
            owner[name] = value
    return out

write_keys

write_keys(text: str, changes: Mapping[str, Any]) -> str

A document's text with values set by dotted key, as with_keys sets them.

Only the keys' lines change, so comments and layout survive: a key is rewritten where it stands, or added under its table's header, and a table the document lacks is appended. A document the edit cannot place (a table written inline or with dotted keys) is re-emitted instead, without its comments.

Source code in src/deeperfly/document.py

def write_keys(text: str, changes: Mapping[str, Any]) -> str:
    """A document's text with values set by dotted key, as [with_keys][] sets them.

    Only the keys' lines change, so comments and layout survive: a key is rewritten
    where it stands, or added under its table's header, and a table the document
    lacks is appended. A document the edit cannot place (a table written inline or
    with dotted keys) is re-emitted instead, without its comments.
    """
    want = with_keys(tomllib.loads(text), changes)
    out = text
    for key, value in changes.items():
        written = _write_key(out, key, value)
        if written is None:
            return toml.dumps(want)
        out = written
    return out if _parses(out) and tomllib.loads(out) == want else toml.dumps(want)