API reference

API reference#

Read backend#

class xarray_binfile.read.RawBinaryEntrypoint#

Bases: BackendEntrypoint

Backend entry point for reading binary files in Xarray.

Variables:

  • open_dataset_parameters (ClassVar[tuple | None]) – Parameters accepted by the open_dataset method.

  • description (ClassVar[str]) – Description of the backend.

  • url (ClassVar[str]) – URL to the backend documentation.

open_dataset(filename_or_obj: str | PathLike[Any], *, read_specs_getter: ReadSpecsGetterProtocol, drop_variables: str | Iterable[str] | None = None) → Dataset#

Open a dataset from a binary file.

Parameters:

  • filename_or_obj – Path to the binary file or a file-like object.

  • read_specs_getter – A callable that generates read specifications for the binary file.

  • drop_variables – Variables to drop from the dataset. Defaults to None.

Returns:

The opened Xarray dataset.

Raises:

  • ValueError – If filename_or_obj is not a valid file path.

  • ValueError – If there is an error reading the metadata from the file path.

class xarray_binfile.read.ReadSpecs(filepath: Path, dtype: DTypeLike, coords: Mapping[str, ArrayLike], name: str, attrs: Mapping[Any, Any] | None = None)#

Bases: object

Immutable metadata contract for interpreting one binary file.

A raw binary file does not carry enough semantic information to be decoded safely by itself. This object provides that missing context so the backend can expose the data as a labeled xarray variable.

Notes

  • Dimension order is inferred from coords key order.

  • shape is inferred from coordinate lengths.

  • dtype should be explicit about byte order when portability matters (for example "<f4" for little-endian float32).

Variables:

  • filepath (pathlib._local.Path) – Path to the binary file.

  • dtype (DTypeLike) – Data type of the binary file.

  • coords (Mapping[str, ArrayLike]) – Coordinates of the data in the binary file.

  • name (str) – Name of the dataset or variable.

  • attrs (Mapping[Any, Any] | None) – Additional attributes for the dataset or variable.

property dims: tuple[str, ...]#

Gets the dimension names of the data.

Returns:

Dimension names.

property shape: tuple[int, ...]#

Gets the shape of the data based on the coordinates.

Returns:

Shape of the data.

class xarray_binfile.read.ReadSpecsGetterProtocol(*args, **kwargs)#

Bases: Protocol

Structural protocol for read spec getter implementations.

Any callable matching (path: Path) -> ReadSpecs can be used as a read specs getter. No inheritance is required.

Typical responsibilities:

  • Parse filename conventions.

  • Resolve variable identity (for example variable name, step index).

  • Provide explicit dtype, coordinates, and optional attrs.

__call__(path: Path) → ReadSpecs#

Generate read specifications for a binary file.

Parameters:

path – Path to the binary file.

Returns:

The metadata required by the binary backend to decode path.

Write accessors#

class xarray_binfile.write.BinaryEngineDataArray(data_array: DataArray)#

Bases: object

An accessor with extra utilities for xarray.DataArray.

to_file(write_specs_getter: WriteSpecsGetterProtocol, directory: Path | None = None) → None#

Writes the data array to binary files.

Writes are eager and whole-file only. For each write specification, the entire sub_array is loaded into memory (triggering a Dask compute for lazy data) and the target file is written in full, in a single pass. There is no partial, appending, or resuming write mode: re-writing a file always replaces its whole content instead of trying to guess or patch existing bytes, which avoids leaving files in a partially-updated, corrupted state.

Writes are also atomic per file: the bytes are first serialized into a unique temporary directory created inside the destination directory (so the final move stays on the same filesystem), and each file is moved to its final path with os.replace() only once it is complete. An interrupted write never leaves a truncated file at the destination, and the temporary directory is removed automatically.

Plan the write specifications so that every individual output file fits comfortably in memory, for example by splitting the array into one file per time step. If you need streaming, incremental, or partial writes, prefer one of the other file formats supported by xarray, such as NetCDF or Zarr.

Each file is written with the in-memory dtype and native byte order, unless the write specification sets dtype, in which case the values are cast right before serialization.

Parameters:

  • write_specs_getter – A callable that generates write specifications for the data array.

  • directory – The directory where the binary files will be written. Defaults to the current working directory.

class xarray_binfile.write.BinaryEngineDataset(data_set: Dataset)#

Bases: object

An accessor with extra utilities for xarray.Dataset.

to_file(write_specs_getter: WriteSpecsGetterProtocol, directory: Path | None = None) → None#

Writes the dataset to binary files.

Every data variable is delegated to BinaryEngineDataArray.to_file(), so the same eager, atomic, whole-file write semantics apply: each output file is fully materialized in memory (triggering a Dask compute for lazy variables), written in a single pass, and moved into place only once complete. See that method for guidance on sizing files and on alternatives when streaming or partial writes are needed.

Parameters:

  • write_specs_getter – A callable that generates write specifications for the data arrays.

  • directory – The directory where the binary files will be written. Defaults to the current working directory.

class xarray_binfile.write.WriteSpecs(filename: str, sub_array: DataArray, dtype: DTypeLike | None = None)#

Bases: NamedTuple

Immutable write instruction for one output binary file.

Each item defines both the output filename and the exact DataArray slice to serialize into that file.

Variables:

  • filename (str) – The name of the binary file.

  • sub_array (xarray.core.dataarray.DataArray) – The portion of the DataArray to be written.

  • dtype (DTypeLike | None) – Optional on-disk data type. When set, the sub-array is cast to this dtype (including byte order, for example "<f4") right before serialization. When None, the in-memory dtype and native byte order are written as-is, so make sure they match what your read specs getter declares.

dtype: DTypeLike | None#

Alias for field number 2

filename: str#

Alias for field number 0

sub_array: DataArray#

Alias for field number 1

class xarray_binfile.write.WriteSpecsGetterProtocol(*args, **kwargs)#

Bases: Protocol

Structural protocol for write spec getter implementations.

Any callable matching (data_array: xr.DataArray) -> Iterator[WriteSpecs] can be used as a write specs getter. No inheritance is required.

Typical responsibilities:

  • Define filename conventions.

  • Split a DataArray into one or more per-file slices.

  • Ensure each slice is transposed to the expected on-disk dimension order.

  • Keep each per-file slice small enough to fit in memory, since every sub_array is fully materialized (Dask compute) before its file is written in a single pass.

__call__(data_array: DataArray) → Iterator[WriteSpecs]#

Generate write instructions for a DataArray.

Parameters:

data_array – The data array for which to generate write specifications.

Returns:

An iterator over per-file write instructions.

Tutorial helpers#

class xarray_binfile.tutorial.DatasetGenerator(read_specs_getter: ReadSpecsGetterProtocol)#

Bases: object

Utility that generates synthetic datasets from read specs metadata.

This helper is intended for tutorials and tests where you want deterministic mock data that follows the same metadata conventions expected by the backend.

Variables:
__call__(iter_filepath: Iterator[Path]) → Dataset#

Generate and merge datasets for the provided file paths.

Each path is converted to ReadSpecs using read_specs_getter. A synthetic variable is generated for each metadata object, and all variables are merged into one Dataset.

Parameters:

iter_filepath – An iterator over file paths.

Returns:

A merged xarray Dataset.

class xarray_binfile.tutorial.FileSpecsGetter(base_coords: dict[str, ArrayLike], dtype: DTypeLike = <class 'numpy.float64'>, filename_template: str = '{name}-{digits:04}.bin', filename_regex: ~re.Pattern = re.compile('(?P<name>\\w+)-(?P<digits>\\d{4})\\.bin'))#

Bases: object

Reference implementation of both read and write spec getter protocols.

This helper is intended for tutorials, tests, and simple projects with a filename convention that encodes variable name and step index.

It is intentionally narrow: the default regex expects time-indexed files (for example ux-0001.bin). For mixed layouts that include static files (for example epsi.bin), implement a custom getter that still follows the same read/write protocol contracts.

It implements:

  • reader(path) -> ReadSpecs

  • writer(data_array) -> Iterator[WriteSpecs]

Variables:

  • base_coords (dict[str, ArrayLike]) – Base coordinates for the data.

  • dtype (DTypeLike) – Data type of the binary file. Defaults to np.float64.

  • filename_template (str) – Template for generating filenames.

  • filename_regex (re.Pattern) – Regular expression for parsing filenames.

dtype#

alias of float64

reader(path: Path) → ReadSpecs#

Build ReadSpecs from a file path.

The implementation parses path.name using filename_regex and appends a single-value time coordinate based on the extracted step number.

Parameters:

path – Path to the binary file.

Returns:

Metadata for reading the binary file.

Return type:

ReadSpecs

Raises:

ValueError – If the filename does not match the expected pattern.

writer(data_array: DataArray) → Iterator[WriteSpecs]#

Yield WriteSpecs for one DataArray.

The default behavior writes one file per time coordinate value using filename_template and transposes each slice to base_coords dimension order before serialization. Each slice is cast to dtype on write, so files always round-trip with reader().

Parameters:

data_array – The data array to generate write specifications for.

Returns:

An iterator over write specifications.