Benchmark On Your Own Data

Benchmark On Your Own Data#

There is no single answer to “how fast is it” for a raw-binary backend. Throughput depends on your data and your machine: the grid or dimension size, the data type (float32 moves half the bytes of float64), the number of files and timesteps, the chunk shape, disk speed, CPU core count, and the operating system file cache.

The reliable approach is to measure the operation you actually run, on data shaped like yours, on your own hardware. This notebook is a small, dependency-free harness you can copy, point at your files, and adapt. Treat the absolute numbers as a template, not a reference, and focus on the method and the relative trends.

Caution

These timings run inside the documentation build, on shared hardware, with tiny synthetic data and a warm file cache. At this scale fixed overhead dominates, so the numbers mainly illustrate the measurement itself. Re-run the harness on your real files before drawing any conclusion.

import os
import pathlib
import platform
import statistics
import tempfile
import time

import numpy as np
import pandas as pd
import xarray as xr

import xarray_binfile  # noqa: F401  (registers the .binary_engine accessors)
from xarray_binfile.tutorial import DatasetGenerator, FileSpecsGetter

workdir_holder = tempfile.TemporaryDirectory()
workdir = pathlib.Path(workdir_holder.name)
workdir

PosixPath('/tmp/tmpsea3en4g')

Record your environment#

A benchmark is only meaningful next to the machine that produced it. Capture the basics so results stay comparable when you change hardware or revisit them later.

print("platform:", platform.platform())
print("python:", platform.python_version())
print("logical CPU cores:", os.cpu_count())

platform: Linux-6.17.0-1018-azure-x86_64-with-glibc2.39
python: 3.13.14
logical CPU cores: 4

A tiny benchmarking harness#

Two small helpers are enough. make_dataset synthesizes a cube of num_points**3 values per file so you can vary the grid size and data type, and benchmark runs an operation a few times after a warm-up, returning the best and median wall-clock time. The best time filters out noise; the median hints at typical behavior.

To benchmark your data instead, skip make_dataset and point the getter at your real directory, as shown in Read Binary Files Lazily With Xarray and Dask.

def make_dataset(
    directory, *, num_points, dtype, num_steps=8, variables=("ux", "uy", "uz")
):
    """Write ``len(variables) * num_steps`` files of ``num_points**3`` points each."""
    directory.mkdir(parents=True, exist_ok=True)
    coords = {
        axis: np.linspace(0.0, 1.0, num=num_points, dtype=np.float32)
        for axis in ("x", "y", "z")
    }
    getter = FileSpecsGetter(base_coords=coords, dtype=dtype)
    generator = DatasetGenerator(getter.reader)
    filenames = (
        getter.filename_template.format(name=name, digits=step)
        for name in variables
        for step in range(num_steps)
    )
    dataset = generator(map(pathlib.Path, filenames))
    dataset.binary_engine.to_file(getter.writer, directory)
    return getter


def benchmark(operation, *, repeats=3, warmup=1):
    """Time ``operation`` and return the best and median seconds over ``repeats`` runs."""
    for _ in range(warmup):
        operation()
    samples = []
    for _ in range(repeats):
        start = time.perf_counter()
        operation()
        samples.append(time.perf_counter() - start)
    return min(samples), statistics.median(samples)

Sweep data type and dimension size#

Here the operation (open the whole collection lazily, then reduce it) and the chunking stay fixed while the data type and grid size vary. Each scenario writes to its own directory, so the reads do not overlap. The result is a small table you can extend with more sizes, dtypes, or operations.

def open_reduce_operation(getter, directory, chunks):
    def run():
        dataset = xr.open_mfdataset(
            sorted(directory.glob("*.bin")),
            engine="binfile",
            read_specs_getter=getter.reader,
            chunks=chunks,
            parallel=True,
        )
        dataset.mean().compute()

    return run


chunks = {"time": 4}
sizes = {"small": 24, "large": 40}
dtypes = {"float32": np.float32, "float64": np.float64}

rows = []
for dtype_name, dtype in dtypes.items():
    for size_label, num_points in sizes.items():
        directory = workdir / f"{dtype_name}-{size_label}"
        getter = make_dataset(directory, num_points=num_points, dtype=dtype)
        operation = open_reduce_operation(getter, directory, chunks)
        best, median = benchmark(operation, repeats=3, warmup=1)
        megabytes = sum(p.stat().st_size for p in directory.glob("*.bin")) / 1024**2
        rows.append(
            {
                "dtype": dtype_name,
                "points_per_axis": num_points,
                "dataset_MiB": round(megabytes, 2),
                "best_s": round(best, 4),
                "median_s": round(median, 4),
                "MiB_per_s": round(megabytes / best, 1),
            }
        )

results = pd.DataFrame(rows)
results
dtype points_per_axis dataset_MiB best_s median_s MiB_per_s
0 float32 24 1.27 0.1194 0.1194 10.6
1 float32 40 5.86 0.1188 0.1193 49.3
2 float64 24 2.53 0.1181 0.1200 21.4
3 float64 40 11.72 0.1195 0.1195 98.1

Reading the results#

A few patterns usually emerge, and your hardware will shift them:

  • For the same grid, float64 moves about twice the bytes of float32. When a run is bandwidth-bound, that shows up directly as more time; when it is overhead-bound (as in this tiny build), it may not.

  • Larger grids process more data but amortize fixed per-call and scheduling overhead, so MiB_per_s typically climbs with size until it plateaus near your disk or memory bandwidth.

  • Absolute times are environment-specific. A faster NVMe drive, more CPU cores, or a cold cache can each reorder the rows, which is exactly why the number that matters is the one you measure on your setup.

Tuning knobs to try#

Extend the harness by changing one variable at a time:

  • Chunk shape: smaller chunks add scheduling overhead; larger chunks raise peak memory. See Parallel And Larger-Than-Memory Workflows for how chunks become parallel tasks.

  • Scheduler and workers: wrap the operation in dask.config.set(scheduler=..., num_workers=...) to compare threads against processes.

  • The operation itself: a full .compute() that loads everything stresses I/O and memory differently from a reduction or a single-point slice. Swap the body of open_reduce_operation to match your real workload.

  • Cold versus warm cache: the first read from disk is slower than repeats served from the operating system cache. Decide which case reflects your usage and measure that one.

With the method in hand, the conclusions are yours to draw from your data and your hardware.