Spec Getter Protocols for Read and Write

Spec Getter Protocols for Read and Write#

This page explains the protocol interfaces used by xarray-binfile and shows:

the protocol contracts for read and write operations
the implementations shipped with the package
how to implement your own getters for custom binary layouts

Why spec getters exist#

Raw binary files are not self-describing. They do not store enough metadata for xarray to know:

dimension names and order
coordinate values
variable name
dtype and byte order

xarray-binfile solves this by asking you for callables that provide metadata for reads and file splitting rules for writes.

File-layout examples#

The protocol approach is useful because real projects organize files differently.

Time-series variables only#

caseA/
	ux001.bin
	ux002.bin
	ux003.bin
	uy001.bin
	uy002.bin
	uy003.bin

In this layout, the read specs getter usually parses:

variable name (ux, uy)
step index (001, 002, 003)

and returns coords including a single-value time coordinate for each file.

Mixed time-series plus static fields#

caseB/
	ux001.bin
	ux002.bin
	ux003.bin
	uy001.bin
	uy002.bin
	uy003.bin
	epsi.bin

Here, ux* and uy* are time-dependent while epsi.bin is static.

A custom read specs getter can support this by using two filename rules:

time-series rule: parse variable + step, include time=[step]
static rule: parse static variable (for example epsi), return coords without time

In xarray workflows, static variables without a time dimension can be naturally broadcast against time-dependent variables during computations.

Read protocol#

The read protocol is ReadSpecsGetterProtocol.

Contract:

input: pathlib.Path
output: ReadSpecs

ReadSpecs fields:

filepath: path to the binary file
dtype: data type (for example np.float32, "<f4", ">f8")
coords: mapping of dimension name to coordinate array
name: variable name
attrs: optional attributes

See the API reference for authoritative signatures and behavior:

ReadSpecs: API reference
ReadSpecsGetterProtocol: API reference

Write protocol#

The write protocol is WriteSpecsGetterProtocol.

Contract:

input: xarray.DataArray
output: Iterator[WriteSpecs]

WriteSpecs fields:

filename: output file name
sub_array: array slice to write into that file
dtype: optional on-disk data type; when set, the sub-array is cast right before serialization (including byte order, for example "<f4"). When omitted, the in-memory dtype and native byte order are written as-is, so make sure they match what your read specs getter declares.

Warning

Writes are eager and whole-file only. Each sub_array is fully loaded into memory (a Dask compute for lazy data) and its file is written in a single pass — the backend never appends to, patches, or resumes a file. Files are staged in a unique temporary directory inside the destination and atomically moved into place with os.replace once complete, so an interrupted write never leaves a truncated file behind. That protects against partially-updated, corrupted files, but it means every sub_array you yield must fit in memory. Split large arrays into more, smaller files, and reach for xarray-supported formats such as NetCDF or Zarr when you need streaming or partial writes.

See the API reference for authoritative signatures and behavior:

WriteSpecs: API reference
WriteSpecsGetterProtocol: API reference

Implementations shipped with xarray-binfile#

The package includes a built-in reference implementation in xarray_binfile.tutorial.FileSpecsGetter.

It provides both protocol implementations:

FileSpecsGetter.reader: implements ReadSpecsGetterProtocol
FileSpecsGetter.writer: implements WriteSpecsGetterProtocol

It uses a common pattern:

filename convention like {name}-{digits:04}.bin
regex parsing for variable name and time index
base coordinates plus a time coordinate extracted from filename

This shipped helper targets a single time-series naming convention. Mixed layouts (for example ux001.bin together with static epsi.bin) are typically handled by creating your own getter based on the same protocol contracts.

A second shipped helper, xarray_binfile.tutorial.DatasetGenerator, consumes a read specs getter and generates synthetic datasets for tests/tutorials.

These implementations are documented in the API section and should be considered the source of truth:

FileSpecsGetter: API reference
DatasetGenerator: API reference

How to create your own read specs getter#

Create a callable that follows ReadSpecsGetterProtocol. In practice:

Parse your filename convention.
Resolve variable identity (for example variable name and step index).
Build explicit coords and dtype.
Return a ReadSpecs object.

Use FileSpecsGetter.reader as the reference behavior and adapt only what differs for your project.

How to create your own write specs getter#

Create a callable that follows WriteSpecsGetterProtocol. In practice:

Decide how one in-memory array maps to one or more output files.
Yield one WriteSpecs object per output file.
Ensure each sub_array is transposed to the expected on-disk dimension order.
Keep each sub_array small enough to fit in memory, since it is fully materialized before its file is written.
Use deterministic filename rules that your read getter can parse back.

Use FileSpecsGetter.writer as the reference behavior and adapt only what differs for your project.

Validation checklist for custom implementations#

Validate filename parsing with explicit errors.
Keep dimension order explicit and consistent.
Make dtype and endianness explicit.
Test read/write round-trips (xarray.testing.assert_allclose).
Start from FileSpecsGetter and replace only what differs.