OxiH5 0.1.4 Released — Virtual Dataset Support, Chunked Variable-Length Reads, and a Global-Heap Decode Bug Fixed

A parser that refuses to open a file is an inconvenience. A parser that opens it and quietly hands back the wrong values is a hazard — OxiH5 0.1.4 closes one of each.

Today we released OxiH5 0.1.4 — a release that closes five real gaps in the COOLJAPAN Pure-Rust HDF5 reader: Virtual Dataset (VDS) resolution, chunked variable-length dataset reads (with a new dataset_vlen_sequences API), szip RAW-mode chunk decoding, soft-link→external-link chain resolution, and a fix for a global-heap reference bug that was silently mis-decoding real-world variable-length data.

No libhdf5. No FFI. No -sys crates. OxiH5 parses the published HDF5 binary format — superblock, object headers, B-trees, heaps, filter pipelines, all eleven datatype classes — directly from bytes, using nothing but std. #![forbid(unsafe_code)] guards oxih5-core; the facade crate carries exactly one documented unsafe block, the mmap call behind open_mmap. It compiles to a single static binary, needs no system libraries and no C toolchain at build time, and stays WASM-friendly by construction.

Why OxiH5 0.1.4 is a game changer

Even with a working Pure-Rust HDF5 reader in hand, a handful of real-world file patterns still forced a fallback to libhdf5 — or, worse, produced a wrong answer with no error at all:

A Virtual Dataset (HDF5 layout class 3) — the layout large archives and multi-file sensor records use to stitch several physical files into one logical array — always failed outright with NotImplemented.
A chunked dataset of variable-length strings or sequences — the combination, not either alone — always failed with “variable-length element size not supported.”
An szip-compressed chunk stored in RAW mode (a bitstream with no HDF5 framing header) always failed with UnsupportedFilter.
A soft link pointing at an external link — a two-hop indirection some tools emit — always failed with NotImplemented.
Worst of all: a vlen string or sequence dataset backed by a global-heap collection at a nonzero address decoded to the wrong values, silently — every existing unit test happened to exercise heap address 0, so the bug shipped unnoticed until now.

OxiH5 0.1.4 ends all of that.

Virtual Datasets resolve end-to-end. A new oxih5_format::vds module decodes both mapping-block encodings (version 0, and the filename-deduplicated version 1) and both hyperslab-selection versions (1 and 3) that appear inside a VDS mapping; oxih5::File then opens each source dataset a mapping references — same file or external, resolved relative to the containing file’s directory — and scatters the selected regions into the virtual dataset’s buffer, transparently, through the same dataset()/read_dataset() calls you already use.
Chunked variable-length reads work, hyperslabs included. A new on_disk_elem_footprint helper accounts for the fixed 16-byte global-heap-reference footprint of vlen elements on disk, and filters that don’t apply to vlen data (shuffle, fletcher32, nbit, scaleoffset) are now explicitly rejected instead of silently misapplied.
File::dataset_vlen_sequences(path) is a new public method decoding a variable-length sequence dataset (datatype class 9) into Vec<Value>, for both contiguous and chunked layouts.
szip RAW-mode chunks decode. A new apply_pipeline_sized accepts the caller’s known decoded chunk size — the one piece of information a header-less RAW stream can’t carry itself — behind a new szip feature on the oxih5 crate.
A silent data-corruption bug is fixed. parse_vlen_ref now decodes the on-disk vlen reference in HDF5’s real byte order — length(4) + heap_address(8) + object_index(4) — instead of the previous, wrong layout; FileWriter’s writer path is corrected to match, so files OxiH5 writes are now byte-layout-compatible with libhdf5/h5py here too.
Two crafted-file panics are now typed errors. A zero chunk dimension no longer divides by zero when computing overlapping chunk-grid cells, and a Fixed Array header’s implausible element count (e.g. u64::MAX) is bounded to 16Mi (1 << 24) and cross-checked against the file’s remaining bytes before it ever reaches a Vec::with_capacity.

Technical Deep Dive: how VDS, chunked vlen, and the hardening fit together

Virtual Dataset resolution — oxih5_format::vds + File. VdsMapping / VdsEntry / VdsSelection (None / All / Hyperslab) model the HDF5 spec’s global-heap VDS mapping block; parse_vds_mapping / parse_vds_block decode it, and selection_element_offsets turns a selection into concrete row-major element positions. Point-list (H5S_SEL_POINTS) selections inside a mapping are recognized and rejected with a named error rather than silently mishandled — the same “honest gap” discipline as the rest of OxiH5.
Chunked vlen assembly — the chunked-read path in oxih5-format. The new on_disk_elem_footprint helper tells the chunk assembler how many bytes a vlen element actually occupies on disk (always 16, the global-heap reference width) versus its logical size, so chunk boundaries and hyperslab intersections compute correctly for vlen data for the first time.
szip RAW mode — filters::apply_pipeline_sized. apply_pipeline is now a thin wrapper around apply_pipeline_sized(..., expected_out_len: None); only szip’s RAW mode ever needs Some(len), since it’s the one filter whose bitstream carries no self-describing length.
Link resolution and hardening. Soft-link and external-link navigation moved out of oxih5/src/lib.rs into a new internal links.rs module — both to keep source files under the workspace’s 2000-line limit and to host the new soft→external chain logic. Independently, fractal_heap.rs‘s header parser now reads the “I/O Filters’ Encoded Length” field at its correct byte offset and width (it was reading the wrong single byte before), so a filtered fractal-heap root direct block is finally detected instead of silently rejected.

Getting Started

cargo add oxih5

use oxih5::open;

// Virtual Datasets (HDF5 layout class 3) now resolve transparently through
// the same read APIs — no code changes required, they simply stop
// returning `OxiH5Error::NotImplemented`.
let f = open("ensemble.h5")?;
let ds = f.dataset("/stitched_temperature")?;
let values: Vec<f32> = ds.as_f32()?;

// New in 0.1.4: decode a variable-length *sequence* dataset (datatype
// class 9), contiguous or chunked, in one call.
let sequences = f.dataset_vlen_sequences("/variable_length_records")?;
println!("{} sequences decoded", sequences.len());

// Chunked variable-length strings and sequences now support hyperslab
// slicing too.
let region = f.dataset_slice("/chunked_vlen_strings", &[0..10])?;

Reading szip RAW-mode chunks needs the new feature flag:

[dependencies]
oxih5 = { version = "0.1.4", features = ["szip"] }

What’s New in 0.1.4

Virtual Dataset (VDS) reading resolves end-to-end (layout class 3), including external-file source datasets and hyperslab selections (versions 1 and 3), via the new oxih5_format::vds module.
Chunked variable-length dataset reads — datasets of vlen strings or sequences can now be read in full and via dataset_slice/hyperslab selections, including a selection that straddles a chunk boundary.
New File::dataset_vlen_sequences(path) decodes vlen sequence datasets (datatype class 9) into Vec<Value>.
szip RAW-mode chunk decoding via a new apply_pipeline_sized and a new szip feature on the oxih5 crate.
Filtered fractal-heap root direct blocks (e.g. deflate/shuffle-compressed) now decode correctly, thanks to a header-field offset/width fix.
Soft link → external link chains now resolve through File::dataset instead of failing.
Breaking change: oxih5_format::message::LayoutInfo::VirtualDataset’s entry_count: u32 field is renamed to heap_index: u32 — it was always the global-heap object index, never an entry count, which is why VDS layouts couldn’t be resolved before this release. Update any code matching this enum variant.
Fixed: the on-disk vlen (global-heap) reference byte layout — the correct encoding is length(4) + heap_address(8) + object_index(4); the previous code silently mis-decoded vlen data whenever the referenced heap collection had a nonzero address. FileWriter’s writer path is corrected to match.
Fixed: FileWriter::write_dataset_f32/f64/i32/i64/u8 now validates that supplied data length matches shape.iter().product() × element_size, returning OxiH5Error::Format instead of risking a corrupt file or a later panic.
Security: a crafted zero chunk dimension no longer triggers a divide-by-zero panic; a crafted Fixed Array header’s implausible element count is now bounded to 16Mi and cross-checked against the file’s remaining bytes.
oxiarc-deflate and oxiarc-szip bumped from 0.3.3 to 0.3.6.
20 new tests this release; all 486 tests pass (cargo test --workspace --all-features).

Tips

Match on heap_index, not entry_count. If your code pattern-matches oxih5_format::message::LayoutInfo::VirtualDataset directly, 0.1.4 renames the field — it was always a global-heap object index, never a count, so update the field name when you upgrade.
Enable the szip feature for any file using SZIP/AEC compression (filter id 4), RAW mode included — szip = ["oxih5-format/szip"] pulls in oxiarc-szip.
Reach for dataset_vlen_sequences, not dataset, for datatype class 9. Vlen sequences decode through the new method; vlen strings still go through dataset_strings — mixing the two up returns OxiH5Error::TypeMismatch rather than wrong data.
VDS point-list selections aren’t supported yet. 0.1.4 covers None, All, and regular Hyperslab virtual/source selections; an explicit point-list selection inside a VDS mapping still returns a named error rather than a wrong read.
If you decoded vlen data with an OxiH5 version before 0.1.4, re-verify it. The global-heap byte-layout bug only bit when the heap collection lived at a nonzero address — files where it happened to sit at address 0 already decoded correctly, so the real blast radius is production files, not test fixtures.
Don’t rely on FileWriter silently ignoring a shape/data-length mismatch. It now returns OxiH5Error::Format up front instead of building a dataset that could corrupt the file or panic later.

This is the foundation

OxiH5 belongs to NoFFI — the COOLJAPAN initiative retiring every C/C++/Fortran/-sys FFI dependency in the Rust ecosystem with a clean, memory-safe, Pure-Rust implementation. OxiH5’s assignment is hdf5-sys/hdf5 (and netcdf-sys on the read path); 0.1.4’s VDS and chunked-vlen support close two of the largest remaining gaps against that goal.

It already has real consumers pinned at 0.1.4 across the workspace:

OxiArc — oxiarc-deflate and oxiarc-szip (both bumped to 0.3.6 this release) provide the Pure-Rust compression behind HDF5’s deflate and szip filters.
oxinetcdf — ships in the same workspace, reading NetCDF-4 conventions (NcFile, NcVariable, NcDimension) on top of the parser this release hardened.
OxiGDAL — the Pure-Rust GDAL replacement depends on both oxih5 and oxinetcdf at 0.1.4 for its HDF5/NetCDF raster paths.
OxiProj — the Pure-Rust PROJ replacement depends on oxih5 0.1.4 as well.
SciRS2 — the root workspace, plus scirs2-io and scirs2-datasets, depend on oxih5-core/oxih5-format 0.1.4; the latter two stage it behind an optional hdf5_io feature and honestly keep the C hdf5 crate available in parallel until OxiH5’s own write support matures far enough to drop it.

Repository: https://github.com/cool-japan/oxih5

Star the repo ⭐ if you want HDF5’s trickiest read-path features — virtual datasets, chunked variable-length data, RAW-mode szip — working in Rust without libhdf5 anywhere in the build.

The era of a scientific-data parser choosing between “fail outright” and “succeed with a silently wrong answer” is over. Pure Rust HDF5 is here — and as of 0.1.4, more of it works correctly than ever.

— KitaSan at COOLJAPAN OÜ July 17, 2026