Quick Start

Quick Start#

This notebook goes from an installed package to a lazy dataset in a few steps:

describe the file convention once, with coordinates, dtype, and filename pattern
open a folder of binary files as one lazy, Dask-backed xarray.Dataset
compute with the usual xarray API
write a derived result back to the same raw layout

The sample files are generated on the fly, so the notebook runs as is. To try it on your own data, skip the generation step and point data_directory at your folder, adjusting the coordinates, dtype, and pattern to match. The shipped conventions are described in Spec getter protocols.

import pathlib
import tempfile

import numpy as np
import xarray as xr

from xarray_binfile.conventions import Layout, StepIndexedFiles
from xarray_binfile.tutorial import DatasetGenerator

temporary_directory = tempfile.TemporaryDirectory()
data_directory = pathlib.Path(temporary_directory.name)

Describe the convention#

Layout holds what every file has in common: the coordinates in on-disk order and the dtype. StepIndexedFiles adds the filename pattern that tells the files apart, here a variable name and an integer step that becomes the time coordinate.

convention = StepIndexedFiles(
    layout=Layout(
        {"x": np.linspace(0, 1, 8), "y": np.linspace(0, 1, 4)},
        dtype=np.float32,
    ),
    pattern="{name}-{step:04d}.bin",  # ux-0000.bin, ux-0001.bin, ...
)
convention

StepIndexedFiles(layout=Layout(coords={'x': array([0.        , 0.14285714, 0.28571429, 0.42857143, 0.57142857,
       0.71428571, 0.85714286, 1.        ]), 'y': array([0.        , 0.33333333, 0.66666667, 1.        ])}, dtype=<class 'numpy.float32'>, order='C', coord_attrs=None), pattern=FilenamePattern(template='{name}-{step:04d}.bin', exact_width=False), check_coords=True, stacks=(), name_of=None, names=None, time_dim='time', time_dtype=None, time_step=None)

Generate sample files#

DatasetGenerator builds random data with the shape and dtype the convention declares for each filename, and .binary_engine.to_file() writes it to disk following the same convention. The files are raw float32 values with no header; the convention is what gives them a shape, coordinates, and a name.

filenames = [f"{name}-{step:04d}.bin" for name in ("ux", "uy") for step in range(3)]
sample = DatasetGenerator(convention.reader)(map(pathlib.Path, filenames))
sample.binary_engine.to_file(convention.writer, data_directory)
sorted(path.name for path in data_directory.iterdir())

['ux-0000.bin',
 'ux-0001.bin',
 'ux-0002.bin',
 'uy-0000.bin',
 'uy-0001.bin',
 'uy-0002.bin']

Open the folder lazily#

convention.open() lists the files that match the pattern and hands them to xarray.open_mfdataset() with engine="binfile". The chunks argument keeps the arrays lazy and Dask-backed, so nothing is read from disk yet. The step from each filename becomes the time coordinate, and the files are combined along it.

dataset = convention.open(data_directory, chunks={"time": 1})
dataset

<xarray.Dataset> Size: 888B
Dimensions:  (x: 8, y: 4, time: 3)
Coordinates:
  * x        (x) float64 64B 0.0 0.1429 0.2857 0.4286 0.5714 0.7143 0.8571 1.0
  * y        (y) float64 32B 0.0 0.3333 0.6667 1.0
  * time     (time) int64 24B 0 1 2
Data variables:
    ux       (x, y, time) float32 384B dask.array<chunksize=(8, 4, 1), meta=np.ndarray>
    uy       (x, y, time) float32 384B dask.array<chunksize=(8, 4, 1), meta=np.ndarray>

Compute with xarray#

The object returned above is a regular xarray.Dataset, and each variable a regular xarray.DataArray. Everything xarray offers applies: indexing by label with sel(), reshaping with expand_dims(), to_dataarray() and to_dataset(), combining with xarray.merge(), plotting, and so on. Only the chunks a calculation needs are read, and compute() materializes the result in memory.

mean = dataset["ux"].mean(dim="time").compute()
mean

<xarray.DataArray 'ux' (x: 8, y: 4)> Size: 128B
array([[0.4008663 , 0.8160922 , 0.8085046 , 0.5083308 ],
       [0.6048036 , 0.41626215, 0.56569785, 0.7081222 ],
       [0.3264173 , 0.18718915, 0.5167758 , 0.580631  ],
       [0.6041058 , 0.33329576, 0.601585  , 0.46592084],
       [0.59133196, 0.6177352 , 0.6806824 , 0.50749046],
       [0.5730499 , 0.5830164 , 0.5244477 , 0.7062941 ],
       [0.47323307, 0.56948906, 0.57231945, 0.5212994 ],
       [0.64451104, 0.6346267 , 0.7130688 , 0.522614  ]], dtype=float32)
Coordinates:
  * x        (x) float64 64B 0.0 0.1429 0.2857 0.4286 0.5714 0.7143 0.8571 1.0
  * y        (y) float64 32B 0.0 0.3333 0.6667 1.0

Write a derived result back#

The same convention writes as well as it reads. The writer takes the variable name from the array and the step from each value along time, so the result must keep the dimensions declared in the layout. Here the fluctuation around the time mean is still lazy; writing it computes one file at a time.

fluctuation = (dataset["ux"] - mean).rename("ux_fluctuation")
fluctuation.binary_engine.to_file(convention.writer, data_directory)
sorted(path.name for path in data_directory.iterdir())

['ux-0000.bin',
 'ux-0001.bin',
 'ux-0002.bin',
 'ux_fluctuation-0000.bin',
 'ux_fluctuation-0001.bin',
 'ux_fluctuation-0002.bin',
 'uy-0000.bin',
 'uy-0001.bin',
 'uy-0002.bin']

The standard xarray entry points#

convention.open() is a convenience. The same files open through xarray.open_dataset() and xarray.open_mfdataset() with engine="binfile", which fits code that already uses them.

temporary_directory.cleanup()

The read and write tutorials expand on each step, and Parallel and larger-than-memory covers Dask schedulers and datasets that do not fit in memory.