Write Derived Results Back To Raw Binary Files

This notebook focuses on the write side of xarray-binfile. The workflow is:

• open a binary dataset lazily with xarray and Dask

• build a derived result without immediately loading everything

• compute a small summary when you need an actual value

• write a derived array back to .bin files with .binary_engine.to_file(...)

• read the result again to verify the round trip

Note

Raw binary output is useful when you must match an existing external convention. If you control the output format, prefer NetCDF or Zarr for portability, metadata preservation, and interoperability across tools.

import pathlib
import tempfile

import numpy as np
import xarray as xr

from xarray_binfile.conventions import Layout, StepIndexedFiles
from xarray_binfile.tutorial import DatasetGenerator

input_directory_holder = tempfile.TemporaryDirectory()
output_directory_holder = tempfile.TemporaryDirectory()
input_directory = pathlib.Path(input_directory_holder.name)
output_directory = pathlib.Path(output_directory_holder.name)
input_directory, output_directory

(PosixPath('/tmp/tmpist14_au'), PosixPath('/tmp/tmpe2w0pgwa'))

base_coords = {
    "x": np.linspace(0.0, 2.0, num=24, dtype=np.float32),
    "y": np.linspace(-1.0, 1.0, num=18, dtype=np.float32),
    "z": np.linspace(0.0, 1.5, num=12, dtype=np.float32),
}

input_specs_getter = StepIndexedFiles(
    layout=Layout(base_coords, dtype=np.float32),
    pattern="{name}-{step:04d}.bin",
)

dataset_generator = DatasetGenerator(input_specs_getter.reader)
filenames = (
    input_specs_getter.pattern.format(name=name, step=step)
    for name in ("ux", "uy", "uz")
    for step in range(8)
)
source_dataset = dataset_generator(map(pathlib.Path, filenames))
source_dataset.binary_engine.to_file(input_specs_getter.writer, input_directory)
sorted(path.name for path in input_directory.glob("*.bin"))[:6]

['ux-0000.bin',
 'ux-0001.bin',
 'ux-0002.bin',
 'ux-0003.bin',
 'ux-0004.bin',
 'ux-0005.bin']

lazy_dataset = xr.open_mfdataset(
    sorted(input_directory.glob("*.bin")),
    engine="binfile",
    read_specs_getter=input_specs_getter.reader,
    chunks={"x": 6, "y": 6, "z": 4, "time": 2},
    parallel=True,
)
lazy_dataset

<xarray.Dataset> Size: 498kB
Dimensions:  (x: 24, y: 18, z: 12, time: 8)
Coordinates:
  * x        (x) float32 96B 0.0 0.08696 0.1739 0.2609 ... 1.739 1.826 1.913 2.0
  * y        (y) float32 72B -1.0 -0.8824 -0.7647 -0.6471 ... 0.7647 0.8824 1.0
  * z        (z) float32 48B 0.0 0.1364 0.2727 0.4091 ... 1.091 1.227 1.364 1.5
  * time     (time) int64 64B 0 1 2 3 4 5 6 7
Data variables:
    ux       (x, y, z, time) float32 166kB dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>
    uy       (x, y, z, time) float32 166kB dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>
    uz       (x, y, z, time) float32 166kB dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • x: 24
  • y: 18
  • z: 12
  • time: 8
• Coordinates: (4)
  • x
    (x)
    float32
    0.0 0.08696 0.1739 ... 1.913 2.0

    array([0.      , 0.086957, 0.173913, 0.26087 , 0.347826, 0.434783, 0.521739,
           0.608696, 0.695652, 0.782609, 0.869565, 0.956522, 1.043478, 1.130435,
           1.217391, 1.304348, 1.391304, 1.478261, 1.565217, 1.652174, 1.73913 ,
           1.826087, 1.913043, 2.      ], dtype=float32)

  • y
    (y)
    float32
    -1.0 -0.8824 -0.7647 ... 0.8824 1.0

    array([-1.      , -0.882353, -0.764706, -0.647059, -0.529412, -0.411765,
           -0.294118, -0.176471, -0.058824,  0.058824,  0.176471,  0.294118,
            0.411765,  0.529412,  0.647059,  0.764706,  0.882353,  1.      ],
          dtype=float32)

  • z
    (z)
    float32
    0.0 0.1364 0.2727 ... 1.364 1.5

    array([0.      , 0.136364, 0.272727, 0.409091, 0.545455, 0.681818, 0.818182,
           0.954545, 1.090909, 1.227273, 1.363636, 1.5     ], dtype=float32)

  • time
    (time)
    int64
    0 1 2 3 4 5 6 7

    array([0, 1, 2, 3, 4, 5, 6, 7])

• Data variables: (3)
  • ux
    (x, y, z, time)
    float32
    dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>

    Array Chunk Bytes 162.00 kiB 576 B Shape (24, 18, 12, 8) (6, 6, 4, 1) Dask graph 288 chunks in 17 graph layers Data type float32 numpy.ndarray

  • uy
    (x, y, z, time)
    float32
    dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>

    Array Chunk Bytes 162.00 kiB 576 B Shape (24, 18, 12, 8) (6, 6, 4, 1) Dask graph 288 chunks in 17 graph layers Data type float32 numpy.ndarray

  • uz
    (x, y, z, time)
    float32
    dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>

    Array Chunk Bytes 162.00 kiB 576 B Shape (24, 18, 12, 8) (6, 6, 4, 1) Dask graph 288 chunks in 17 graph layers Data type float32 numpy.ndarray

speed = np.sqrt(
    lazy_dataset["ux"] ** 2 + lazy_dataset["uy"] ** 2 + lazy_dataset["uz"] ** 2
).rename("speed")
speed

<xarray.DataArray 'speed' (x: 24, y: 18, z: 12, time: 8)> Size: 166kB
dask.array<sqrt, shape=(24, 18, 12, 8), dtype=float32, chunksize=(6, 6, 4, 1), chunktype=numpy.ndarray>
Coordinates:
  * x        (x) float32 96B 0.0 0.08696 0.1739 0.2609 ... 1.739 1.826 1.913 2.0
  * y        (y) float32 72B -1.0 -0.8824 -0.7647 -0.6471 ... 0.7647 0.8824 1.0
  * z        (z) float32 48B 0.0 0.1364 0.2727 0.4091 ... 1.091 1.227 1.364 1.5
  * time     (time) int64 64B 0 1 2 3 4 5 6 7

xarray.DataArray

'speed'

• x: 24
• y: 18
• z: 12
• time: 8

• dask.array<chunksize=(6, 6, 4, 1), meta=np.ndarray>

  Array Chunk Bytes 162.00 kiB 576 B Shape (24, 18, 12, 8) (6, 6, 4, 1) Dask graph 288 chunks in 57 graph layers Data type float32 numpy.ndarray

• Coordinates: (4)
  • x
    (x)
    float32
    0.0 0.08696 0.1739 ... 1.913 2.0

    array([0.      , 0.086957, 0.173913, 0.26087 , 0.347826, 0.434783, 0.521739,
           0.608696, 0.695652, 0.782609, 0.869565, 0.956522, 1.043478, 1.130435,
           1.217391, 1.304348, 1.391304, 1.478261, 1.565217, 1.652174, 1.73913 ,
           1.826087, 1.913043, 2.      ], dtype=float32)

  • y
    (y)
    float32
    -1.0 -0.8824 -0.7647 ... 0.8824 1.0

    array([-1.      , -0.882353, -0.764706, -0.647059, -0.529412, -0.411765,
           -0.294118, -0.176471, -0.058824,  0.058824,  0.176471,  0.294118,
            0.411765,  0.529412,  0.647059,  0.764706,  0.882353,  1.      ],
          dtype=float32)

  • z
    (z)
    float32
    0.0 0.1364 0.2727 ... 1.364 1.5

    array([0.      , 0.136364, 0.272727, 0.409091, 0.545455, 0.681818, 0.818182,
           0.954545, 1.090909, 1.227273, 1.363636, 1.5     ], dtype=float32)

  • time
    (time)
    int64
    0 1 2 3 4 5 6 7

    array([0, 1, 2, 3, 4, 5, 6, 7])

The derived speed array is still lazy. You can compute a lightweight summary first, then decide whether to persist a full result.

speed_time_series = speed.mean(dim=("x", "y", "z")).compute()
speed_time_series

<xarray.DataArray 'speed' (time: 8)> Size: 32B
array([0.9637632 , 0.9667021 , 0.9645986 , 0.95585877, 0.9610805 ,
       0.95524615, 0.9609951 , 0.95937735], dtype=float32)
Coordinates:
  * time     (time) int64 64B 0 1 2 3 4 5 6 7

xarray.DataArray

'speed'

• time: 8

• 0.9638 0.9667 0.9646 0.9559 0.9611 0.9552 0.961 0.9594

  array([0.9637632 , 0.9667021 , 0.9645986 , 0.95585877, 0.9610805 ,
         0.95524615, 0.9609951 , 0.95937735], dtype=float32)

• Coordinates: (1)
  • time
    (time)
    int64
    0 1 2 3 4 5 6 7

    array([0, 1, 2, 3, 4, 5, 6, 7])

Custom write specifications

The derived array has the same layout as the inputs; only the filename differs. Declaring a second StepIndexedFiles with another pattern is enough, and because the regex used for reading is derived from that same pattern, the files can be read back without repeating the convention. When the shipped conventions do not fit, implement a callable compatible with WriteSpecsGetterProtocol as described in Spec getter protocols.

The dtype of the Layout pins the on-disk data type (including byte order): the values are cast right before serialization, so the files always match what the corresponding reader declares.

output_specs_getter = StepIndexedFiles(
    layout=Layout(base_coords, dtype=np.float32),
    pattern="derived-{name}-{step:04d}.bin",
)
output_specs_getter

StepIndexedFiles(layout=Layout(coords={'x': array([0.        , 0.08695652, 0.17391305, 0.26086956, 0.3478261 ,
       0.4347826 , 0.5217391 , 0.6086956 , 0.6956522 , 0.7826087 ,
       0.8695652 , 0.95652175, 1.0434783 , 1.1304348 , 1.2173913 ,
       1.3043479 , 1.3913044 , 1.4782609 , 1.5652174 , 1.6521739 ,
       1.7391304 , 1.826087  , 1.9130435 , 2.        ], dtype=float32), 'y': array([-1.        , -0.88235295, -0.7647059 , -0.64705884, -0.5294118 ,
       -0.4117647 , -0.29411766, -0.1764706 , -0.05882353,  0.05882353,
        0.1764706 ,  0.29411766,  0.4117647 ,  0.5294118 ,  0.64705884,
        0.7647059 ,  0.88235295,  1.        ], dtype=float32), 'z': array([0.        , 0.13636364, 0.27272728, 0.4090909 , 0.54545456,
       0.6818182 , 0.8181818 , 0.95454544, 1.0909091 , 1.2272727 ,
       1.3636364 , 1.5       ], dtype=float32)}, dtype=<class 'numpy.float32'>, order='C', coord_attrs=None), pattern=FilenamePattern(template='derived-{name}-{step:04d}.bin', exact_width=False), check_coords=True, stacks=(), name_of=None, names=None, time_dim='time', time_dtype=None, time_step=None)

speed.binary_engine.to_file(output_specs_getter.writer, output_directory)
sorted(path.name for path in output_directory.glob("*.bin"))[:6]

['derived-speed-0000.bin',
 'derived-speed-0001.bin',
 'derived-speed-0002.bin',
 'derived-speed-0003.bin',
 'derived-speed-0004.bin',
 'derived-speed-0005.bin']

Writing triggers the actual per-file materialization of each selected slice. That is often exactly what you want when handing data back to an external solver or post-processing pipeline.

Warning

.binary_engine.to_file(...) never writes a file partially. Each sub_array is fully loaded into memory (a Dask compute for lazy data) and the whole file is written in a single pass — there is no appending, patching, or resuming. Each file is also written atomically: the bytes go to a temporary file next to the destination and are moved into place with os.replace only when complete, so an interrupted write never leaves a truncated file behind. This keeps the on-disk files consistent instead of risking partially-updated, corrupted content, but it means every individual output file must fit in memory. Plan your chunks and the number of output files accordingly, for example one file per time step. If you need streaming or partial writes, use one of the other xarray-supported formats such as NetCDF or Zarr.

roundtrip_dataset = xr.open_mfdataset(
    sorted(output_directory.glob("derived-*.bin")),
    engine="binfile",
    read_specs_getter=output_specs_getter.reader,
    chunks={"time": 2},
    parallel=True,
)
roundtrip_dataset

<xarray.Dataset> Size: 166kB
Dimensions:  (x: 24, y: 18, z: 12, time: 8)
Coordinates:
  * x        (x) float32 96B 0.0 0.08696 0.1739 0.2609 ... 1.739 1.826 1.913 2.0
  * y        (y) float32 72B -1.0 -0.8824 -0.7647 -0.6471 ... 0.7647 0.8824 1.0
  * z        (z) float32 48B 0.0 0.1364 0.2727 0.4091 ... 1.091 1.227 1.364 1.5
  * time     (time) int64 64B 0 1 2 3 4 5 6 7
Data variables:
    speed    (x, y, z, time) float32 166kB dask.array<chunksize=(24, 18, 12, 1), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • x: 24
  • y: 18
  • z: 12
  • time: 8
• Coordinates: (4)
  • x
    (x)
    float32
    0.0 0.08696 0.1739 ... 1.913 2.0

    array([0.      , 0.086957, 0.173913, 0.26087 , 0.347826, 0.434783, 0.521739,
           0.608696, 0.695652, 0.782609, 0.869565, 0.956522, 1.043478, 1.130435,
           1.217391, 1.304348, 1.391304, 1.478261, 1.565217, 1.652174, 1.73913 ,
           1.826087, 1.913043, 2.      ], dtype=float32)

  • y
    (y)
    float32
    -1.0 -0.8824 -0.7647 ... 0.8824 1.0

    array([-1.      , -0.882353, -0.764706, -0.647059, -0.529412, -0.411765,
           -0.294118, -0.176471, -0.058824,  0.058824,  0.176471,  0.294118,
            0.411765,  0.529412,  0.647059,  0.764706,  0.882353,  1.      ],
          dtype=float32)

  • z
    (z)
    float32
    0.0 0.1364 0.2727 ... 1.364 1.5

    array([0.      , 0.136364, 0.272727, 0.409091, 0.545455, 0.681818, 0.818182,
           0.954545, 1.090909, 1.227273, 1.363636, 1.5     ], dtype=float32)

  • time
    (time)
    int64
    0 1 2 3 4 5 6 7

    array([0, 1, 2, 3, 4, 5, 6, 7])

• Data variables: (1)
  • speed
    (x, y, z, time)
    float32
    dask.array<chunksize=(24, 18, 12, 1), meta=np.ndarray>

    Array Chunk Bytes 162.00 kiB 20.25 kiB Shape (24, 18, 12, 8) (24, 18, 12, 1) Dask graph 8 chunks in 17 graph layers Data type float32 numpy.ndarray

computed_speed = speed.compute().to_dataset()
xr.testing.assert_allclose(roundtrip_dataset.compute(), computed_speed)
computed_speed["speed"].isel(time=0, z=0).plot(cmap="magma", robust=True)

<matplotlib.collections.QuadMesh at 0x7f7fadff0d70>

For general exchange and long-term storage, use xarray’s standard formats when possible: Dataset.to_netcdf(...) and Dataset.to_zarr(...) preserve metadata and make collaboration easier. Use raw binary output when you need strict compatibility with an external file convention.