OxiH5 0.2.2 Released — 44 Agents, 33 Conformance Defects, and Every File Now Verified Against h5py and netCDF4-python

Forty-four agents. Thirty-three distinct ways libhdf5 or netCDF-C rejected, misread, or crashed on a file OxiH5’s own reader had happily accepted as fine. Every one of them is fixed.

Today we released OxiH5 0.2.2 — the release where every file FileWriter and NcFileWriter produce is verified byte-openable by h5py 3.16 (libhdf5 2.0.0) and netCDF4-python 1.7.4, not merely by OxiH5’s own reader. That distinction matters: a reader that only ever checks its own writer’s output can mask an entire class of defects, and 0.2.2 exists because that’s exactly what had been happening.

No libhdf5. No FFI. No -sys crates. OxiH5 parses and writes the real HDF5 binary format — superblock, object headers, B-trees, heaps, the global heap, the filter pipeline, all eleven datatype classes — using nothing but std. #![forbid(unsafe_code)] guards oxih5-core; the facade crate carries exactly one documented unsafe block, the mmap call behind open_mmap. It compiles to a single static binary, needs no system libraries and no C toolchain at build time, and stays WASM-friendly by construction.

Why OxiH5 0.2.2 is a game changer

0.2.1 fixed the write path’s own internal correctness — chunk B-trees, group symbol tables, layout-v4 decoding. 0.2.2 asked a harder question: does the reference implementation agree these files are valid at all? A 44-agent differential interop audit answered it — each agent wrote a permutation of datatype, layout, filter and attribute, opened the result with both h5py and netCDF4-python, and bisected every divergence down to a minimal reproducer. The audit surfaced 33 confirmed conformance defects and 19 capability gaps no amount of self-consistency testing had caught:

Every small variable-length dataset — every vlen string under ~4 KB — was unreadable by libhdf5. GlobalHeapWriter sized each collection exactly to its contents, but libhdf5 rejects any global-heap collection declared under 4096 bytes before it even tries to parse it. Every create_vlen_string_dataset output, every netCDF station-name array, gone at h5py.File().
The global heap’s free-space padding was a size-0 terminator object — which hangs libhdf5’s deserializer outright, because it advances its parse cursor by the object’s own declared size.
The on-disk global-heap object index was truncated with as u16, silently corrupting any vlen dataset that pushed a collection past 16-bit slot space.
An empty scalar fixed-string attribute emitted a size-0 datatype — which made libhdf5 unable to read any attribute on that object, not just the empty one.
Vlen strings declared ASCII charset while carrying UTF-8 data, and fixed strings used NULLTERM with no byte of room for the terminator where libhdf5 expects NULLPAD.
A zero-length dataset wrote a defined data address with size 0, which libhdf5 reports as file corruption rather than “empty.”
netCDF’s DIMENSION_LIST was a plain object-reference array instead of H5T_VLEN{H5T_REFERENCE} — its actual datatype — which didn’t just misread under netCDF4-python, it segfaulted netCDF-C on open.

OxiH5 0.2.2 ends all of that — and closes 9 of the 19 capability gaps the same audit surfaced:

Shuffle and Fletcher32 filters on write, composable with DEFLATE in one real multi-filter pipeline (set_shuffle → set_deflate → set_fletcher32, applied in that order on write and inverted in reverse on read).
Fixed-maxshape tiling via set_chunking — a chunked dataset that reports its real bounded maxshape instead of always claiming maxshape[0] = ∞.
Custom fill values (set_fill_value_*, all ten fixed-width types) instead of the always-zero-length default.
Boolean datasets, fixed-length string datasets, compact-layout datasets, and the full fixed-width + array + vlen-objref + compound attribute set.
True netCDF coordinate variables — a dimension and a variable sharing a name, emitted as one dimension-scale dataset with its own data, instead of a fabricated [0, 1, …, n−1] phantom.

Technical Deep Dive: how a differential audit finds what a reader can’t

The harness never trusts OxiH5’s own reader as the oracle. Each of the 44 agents built a file through FileWriter/NcFileWriter, then handed it to h5py and netCDF4-python — real, independently-implemented parsers — and treated any exception, wrong value, or crash as a confirmed defect, bisected to the smallest input that reproduces it. A defect that both OxiH5’s reader and its writer agree on is invisible to self-consistency testing by construction; this is the only harness shape that catches it.
Global heap conformance, four bugs deep. H5HG_MINSIZE = 4096 is a hidden libhdf5 precondition, not a documented format rule — collections are now floored at 4096, rounded to a power of two, capped at 65536; the free-space padding is a real index-0 object spanning the remainder; the object index is 32-bit throughout; vlen-string lengths are stored raw at strlen instead of strlen + 1.
The global heap reader had its own alignment bug, found by the same audit. Multi-object collections at a file offset not 8-byte-aligned to the file dropped every object past the first — libhdf5 aligns each object relative to the collection start, not the file; netCDF-C routinely places a GCOL at an address like 4763, which the old code silently mishandled.
DIMENSION_LIST’s datatype was wrong, not just its values. netCDF’s dimension-linkage attribute is a variable-length array of object references (H5T_VLEN{H5T_REFERENCE}) — a nested vlen-of-reference type distinct from a flat reference array. Writing the flat form parses as something structurally different, which is exactly the shape of bug that only crashes on the code path that assumes the correct encoding.

Getting Started

cargo add oxih5

use oxih5::FileWriter;

let path = std::env::temp_dir().join("out.h5");
let mut w = FileWriter::new();

// New in 0.2.2: shuffle -> deflate -> fletcher32 compose in one real pipeline.
w.write_dataset_f64("/sensors/temperature", &[21.5, 21.6, 21.4, 21.7], &[4])?;
w.set_shuffle("/sensors/temperature")?;
w.set_deflate("/sensors/temperature", 6)?;
w.set_fletcher32("/sensors/temperature")?;

// New in 0.2.2: a real fill value instead of the always-zero default.
w.write_dataset_i32("/sensors/status", &[0, 1, 0, 1], &[4])?;
w.set_fill_value_i32("/sensors/status", -1)?;

// New in 0.2.2: numpy-compatible boolean datasets.
w.write_dataset_bool("/sensors/active", &[true, true, false, true], &[4])?;

w.build(&path)?;
# Ok::<(), oxih5::OxiH5Error>(())

What’s New in 0.2.2

33 confirmed conformance defects fixed, spanning the global heap (4 bugs), attribute encoding (6), object headers (5), the chunk index (4), the reader (5), and netCDF conventions (4) — see the full list in CHANGELOG.md.
9 writer capability gaps closed: shuffle + Fletcher32 pipelines, fixed-maxshape tiling (set_chunking), variable-length-reference and compound attributes, the full fixed-width attribute type set plus 1-D array siblings, custom fill values, fixed-length string datasets, boolean datasets, compact-layout datasets, and true netCDF coordinate variables.
netCDF reader completeness: old-style groups spanning multiple symbol-table nodes now enumerate fully, vlen strings/sequences resolve across multiple global-heap collections, and array sizing comes from the dataspace instead of raw byte length — no more phantom trailing elements on padded scalar attributes.
862 tests pass (--all-features; 841 with default features) + 15 doc tests, across all four crates (~26.7k SLOC) — zero clippy/rustdoc warnings.
Known limitations, unchanged from 0.2.1: extensible-array chunk indexes are located but not decoded; hyperslab selections with block > 1 still drop elements on chunked datasets.

Tips

If you wrote a vlen-string dataset under ~4 KB with OxiH5 before 0.2.2, regenerate it. libhdf5 rejected the whole file outright with OSError: global heap size is too small — this wasn’t a degraded-but-readable case, it was a hard open failure.
Filters now compose in write order. set_shuffle → set_deflate → set_fletcher32 on the same path builds one pipeline message applied in that order on write and unwound in reverse on read — call them in the order you want the transforms to run, not the order you want them undone.
set_fill_value_* is typed, not coercive. Calling set_fill_value_f32 on an i32 dataset returns a typed error instead of silently reinterpreting bytes — pick the accessor that matches the dataset’s actual element type.
Prefer set_chunking over create_dataset_unlimited when you know the bound. create_dataset_unlimited always declares maxshape[0] = ∞; set_chunking reports the dataset’s real bounded maxshape to h5py and rejects a chunk extent that overruns a fixed dimension.
NcFileWriter no longer fabricates coordinate axes. A dimension and variable sharing a name now become one real coordinate variable with REFERENCE_LIST/_Netcdf4Coordinates wired correctly — if you were post-processing away a phantom [0, 1, …, n−1] axis, that workaround is no longer needed.

This is the foundation

OxiH5 belongs to NoFFI — the COOLJAPAN initiative retiring every C/C++/Fortran/-sys FFI dependency in the Rust ecosystem with a clean, memory-safe, Pure-Rust implementation. It already has real consumers pinned at 0.2.2 across the workspace:

oxinetcdf — ships in the same workspace; its coordinate-variable and DIMENSION_LIST fixes land directly in every file it writes.
OxiGeo — depends on both oxih5 and oxinetcdf at 0.2.2 for its HDF5/NetCDF raster paths.
OxiProj — depends on oxih5 0.2.2 for coordinate-reference-system data.
SciRS2 — scirs2-io and scirs2-datasets depend on oxih5-core/oxih5-format 0.2.2, staged behind an optional hdf5_io feature alongside the C hdf5 crate until write-path coverage clears the last gap.

Repository: https://github.com/cool-japan/oxih5

Star the repo ⭐ if “passes our own tests” shouldn’t be the bar for a file format library — the reference implementation’s opinion should be.

The era of a Pure-Rust writer whose correctness claim rested on its own reader is over. Pure Rust HDF5 that libhdf5 and netCDF-C actually agree on — that’s 0.2.2.

— KitaSan at COOLJAPAN OÜ July 22, 2026