Migration guide: from prm.dataset to the lazy API

Migration guide: from `prm.dataset` to the lazy API#

Version 1.5.0 introduces a lazy, Dask-backed way to read and write the raw binary files of XCompact3d, built on xarray-binfile. The on-demand loader prm.dataset keeps working in the 1.x series but emits a FutureWarning, and it will be removed in version 2.0. This page maps every call to its replacement.

Why#

Lazy and parallel. prm.open_dataset() describes the whole simulation at once; values are read only when needed, one file per Dask task, so larger-than-memory workflows and parallel evaluation come for free.
One description of the files. Naming, byte order, dtype, mesh, time and the stacked u/phi are declared once, in Xcompact3dConvention, and reused for reading, writing and discovery.
Standard xarray. The same convention is registered as the xarray engine xcompact3d, so xarray.open_mfdataset works out of the box.
Less toolbox-specific code. The file mechanics now live in xarray-binfile, which is shared with other solvers.

What is deprecated#

Access to prm.dataset itself, and therefore all of its methods and settings. The two APIs are independent: the lazy API never reads prm.dataset.filename_properties, data_path, drop_coords, snapshot_step, stack_velocity or stack_scalar. Configure one or the other, never both.

Call by call#

On-demand loader (deprecated)	Lazy API
`prm.dataset.set(data_path="./data/")`	pass `data_path` to `prm.open_dataset(...)` / `prm.write_dataset(...)`; the default is `data/` next to the parameters file (`prm.default_data_path`)
`prm.dataset.filename_properties.set(separator="-", file_extension=".bin", number_of_digits=3)`	`prm.open_dataset(filename_properties={"separator": "-", "file_extension": ".bin", "number_of_digits": 3})`; the same defaults apply
`prm.dataset.set(drop_coords="z")`	`prm.open_dataset(drop_coords="z")`
`prm.dataset.set(snapshot_step="iprocessing")`	`prm.open_dataset(snapshot_step="iprocessing")`
`prm.dataset.set(stack_velocity=True, stack_scalar=True)`	always stacked; `prm.open_dataset(stack=False)` to see `ux`, `uy`, `uz`, `phi1`, … as on disk. `stack_names={"i": {"u", "vort"}}` also reassembles your own fields (`vortx`, `vorty`, `vortz` into `vort`); writing splits any array carrying `i` or `n`, whatever its name
`prm.dataset.set(set_of_variables={"ux", "uy"})`	`prm.open_dataset(variables=["ux", "uy"])`, or `variables=["u"]` for the stacked name
`x3d.param["mytype"] = np.float32`	still the default dtype; override per call with `prm.open_dataset(dtype=np.float32)`
`prm.dataset["ux"]` (time series of one variable)	`prm.open_dataset(variables=["ux"], stack=False)["ux"]`
`prm.dataset[10]` (one snapshot)	`ds.sel(t=10 * prm.dt * prm.ioutput, method=\"nearest\")` with `ds = prm.open_dataset()`: the loader indexed by file counter, the lazy dataset by time label. `ds.isel(t=10)` is positional and only equivalent when every snapshot from `000` is on disk
`prm.dataset[0:101:10]`, `prm.dataset[:]`	`ds.sel(t=slice(0, 100 * prm.dt * prm.ioutput)).isel(t=slice(None, None, 10))` (label range, then every tenth), `prm.open_dataset()`
`for ds in prm.dataset:` / `prm.dataset(0, 101, 5)`	`ds = prm.open_dataset()` once, then `for time in ds.t.values: snapshot = ds.sel(t=time)` (iterating `ds.t` yields time values, not datasets); or better, express the computation on the whole series and let Dask split the work
`prm.dataset.load_array("ux-000.bin")`	`xr.open_dataset("data/ux-000.bin", engine="xcompact3d", prm=prm)`
`prm.dataset.load_array("epsilon.bin", add_time=False)`	`prm.open_dataset(variables=["epsilon"])["epsilon"]` (static files are opened with the snapshots)
`prm.dataset.load_snapshot(10, list_of_variables=["ux"])`	`prm.open_dataset(variables=["ux"]).isel(t=10)`
`prm.dataset.load_time_series("ux")`	`prm.open_dataset(variables=["ux"], stack=False)["ux"]`
`prm.dataset.write(vort, file_prefix="w3")`	`prm.write_dataset(vort, file_prefix="w3")`
`prm.dataset.write(ds)` (variables with `file_name`)	`prm.write_dataset(ds)` (same rule, same warning for the others)
`prm.dataset.write_xdmf()`	not migrated yet; keep calling `prm.dataset.write_xdmf()` (the warning is the only difference)
`prm.dataset.load_wind_turbine_data()`	not migrated yet; keep calling it on `prm.dataset`

len(prm.dataset), which guessed the number of snapshots from ilast and ioutput, has no equivalent: the lazy dataset lists what is on disk, prm.open_dataset().sizes["t"].

It is a plain xarray.Dataset#

prm.open_dataset() returns an ordinary, lazy xarray.Dataset. The toolbox cannot anticipate every case, so when the defaults do not fit, open with stack=False or with xr.open_mfdataset(..., engine="xcompact3d", prm=prm) and shape the result with xarray itself: xr.concat to stack components along a new dimension, xr.merge and xr.combine_by_coords to join folders or runs, Dataset.stack, rename, assign_coords, chunk, isel/sel, and so on. Everything stays lazy until you compute.

Files in sub-folders#

XCompact3d versions and user cases lay files out differently, so no folder tree is assumed. Declare the ones you have:

ds = prm.open_dataset(
    folders={
        "xy_planes": {"drop_coords": "z"},
        "3d": {},
        "geometry": {"static": True},
    }
)

Each entry is built from the same parameters with the given overrides, or it can be a ready xarray-binfile convention. On write, an array named geometry/epsilon (through its file_name attribute) goes to that folder.

Static planes from the sandbox#

init_dataset produces inflow planes such as bxx1 on (y, z). prm.write_dataset writes them as static files, like the old writer did, but prm.open_dataset never lists 2D static files: it is a reader for the simulation output. Read a plane with xr.open_dataset(path, engine="xcompact3d", prm=prm, drop_coords="x").

Sandbox and genepsi#

init_epsi, init_dataset and gene_epsi_3d now write through the lazy API and accept an explicit data_path. Without it they default to prm.default_data_path, or to a changed prm.dataset.data_path while that setting exists.

Known differences#

Output names. The old writer accepted any file_prefix and appended the extension when missing, so prm.dataset.write(vort, "w3-mean") or "ux-000.bin" worked. The lazy API derives the separator, step counter and extension from the filename pattern, so a name is a word made of letters, digits and underscores, optionally under sub-folders ("geometry/epsilon"). Other names are rejected with a message saying so: use w3_mean instead of w3-mean, and never include the counter or extension yourself.

Assigning prm.dataset. Still possible (prm.dataset = Dataset(...)), with the same deprecation warning as reading it.

The old writer computed the step of a snapshot as int(t / dt), which truncates (0.3 / 0.1 is 2.999...) and could make two snapshots overwrite each other. The lazy API rounds and refuses values that are not within tolerance of an integer step.