OxiONNX 0.1.7 Released — A 40-Op CUDA Backend, 17x Batched MatMul, and a Silent Two-Stream Race That Returned Wrong Answers

A GPU kernel that silently returns the wrong number is worse than one that doesn’t run at all — the second one at least fails loudly.

Today we released OxiONNX 0.1.7 — the release where oxionnx-cuda grows from a working-but-partial 25-op backend into a 40-op one, and where we found and fixed the exact kind of bug GPU-accelerated inference should be most afraid of: a kernel that looked correct and quietly returned the wrong answer.

No C. No C++. No ONNX Runtime binaries. No cuDNN, no cuBLAS, no external protobuf — the CUDA backend talks to the driver through COOLJAPAN’s own Pure Rust oxicuda stack.
No unsafe code outside a handful of documented, audited call sites.
Just memory-safe ONNX model execution that compiles to a single static binary (or WASM) and runs everywhere — from laptops to browsers to edge devices to GPUs.

Why OxiONNX 0.1.7 is a game changer

A partial GPU backend fails in the boring way — it just falls back to CPU. A wrong GPU backend fails in the dangerous way — it keeps running and hands you a number that looks fine:

• Conv — the single most common neural-net op — declined to CUDA unconditionally; is_supported_op’s Conv arm had been hardcoded false since the op’s CPU-only stub was written
• Every CUDA dispatch re-allocated, re-uploaded, and sometimes re-JIT-compiled from scratch — a batch of 16 MatMul calls cost 48 device allocations and 32 fences to run kernels that individually take microseconds
• MatMul (and ReduceSum/ReduceMax) could silently return the wrong answer — a two-stream race, not a kernel bug: a 64×64×64 all-ones case came back with 3,456 of 4,096 elements wrong, every wrong element reading back 0.0
• OXIONNX_CUDA_STRICT=1 — the flag meant to catch exactly that kind of mismatch — could still exit 0 on a run whose own shadow verification had just caught it disagreeing with the CPU oracle
• oxionnx-gpu’s dispatch-or-decline thresholds were seven flat constants blind to adapter class or shape, and opening a second GPU-backed session in one process could panic on its first convolution

OxiONNX 0.1.7 ends all of that.

• Real Conv on CUDA — 40 accelerated ops total, up from 25 — selecting ImplicitGemmConv/Conv1x1/DepthwiseConv per shape, validated against two independently-written CPU oracles (conv_tests.rs, 1,114 lines, plus the crate’s existing reference::ref_conv)
• A session-lifetime device-buffer pool, weight-residency cache, and compiled-PTX cache replace the per-dispatch cuMemAlloc/cuMemFree/JIT pattern — steady-state frames upload zero weight bytes, asserted directly in tests/batched_matmul_gpu.rs
• True batched MatMul/Gemm dispatch — up to 17x faster, measured on an RTX A4000 ([16,64,128]x[16,128,64]: 5.89 ms → 0.34 ms)
• CUDA activation residency, shared with the wgpu backend through one generic DeviceActivation trait — a five-node all-CUDA chain now costs exactly one upload, one download, one fence, instead of five of each
• Two silent-wrong-answer bugs fixed: the MatMul/Reduce* two-stream race, and OXIONNX_CUDA_STRICT’s exit-code blind spot
• oxionnx-gpu dispatch tuning is now per-adapter, per-shape — arithmetic intensity gates Gemm instead of raw FLOP count, and the multi-context panic is fixed

Technical Deep Dive: four layers that make the CUDA path honest

1. The buffer/weight/module residency stack (oxionnx-cuda/src/residency.rs, new, 1,537 lines) — a size-classed device-buffer free list (DevicePool, 512 MiB budget), an initializer-keyed weight cache (ResidentWeights, identity checked against the cached entry’s recorded host address and length on every lookup, not trusted), and a per-context compiled-PTX cache for the elementwise/softmax kernels the crate JIT-generates itself.
2. Real Conv dispatch (oxionnx-cuda/src/conv.rs, 110 → 1,168 lines) — conv::cuda_conv picks ImplicitGemmConv, Conv1x1, or DepthwiseConv per shape, deliberately bypassing oxicuda_dnn’s own auto-selector (which can route into a separately-gated Winograd path), and declines to CPU rather than guess on asymmetric pads or group/channel mismatches.
3. A shared, backend-generic activation-residency trait (src/session/gpu_activations.rs) — RunActivations is now generic over DeviceActivation, with GpuActivations/CudaActivations as the two instantiations. CUDA uses a more permissive keep policy than wgpu’s: a value stays resident if any capable consumer can bind it in place, because a value that must still be read back for one CPU-only consumer costs one read-back either way.
4. Device- and shape-aware dispatch tuning (oxionnx-gpu/src/context/tuning.rs, new, 868 lines) — GpuTuning replaces seven flat threshold constants; for Gemm it gates on arithmetic intensity (I = 2mkn/(mk+kn+mn)) instead of raw FLOP count, and a software adapter (lavapipe, WARP) now declines every size outright since it is the CPU it would be racing.

Getting Started

cargo add oxionnx --features cuda

The core inference API is unchanged in 0.1.7:

use oxionnx::{Session, Tensor};
use std::collections::HashMap;

let session = Session::from_file("model.onnx".as_ref())?;

let mut inputs = HashMap::new();
inputs.insert("input", Tensor::new(vec![1.0, 2.0, 3.0], vec![1, 3]));

let outputs = session.run(&inputs)?;
println!("{:?}", outputs);

Checking for a CUDA device directly (needs OXIONNX_CUDA=1 and the cuda feature):

use oxionnx_cuda::CudaContext;

// Returns None unless OXIONNX_CUDA=1 is set, and also None with no CUDA
// device present. No panic, no unwrap required either way.
if let Some(ctx) = CudaContext::try_new() {
    println!("CUDA device ready: {:?}", ctx.driver_context());
}

In practice you don’t need to touch oxionnx-cuda directly — Session dispatches to CUDA automatically once the cuda feature is on, OXIONNX_CUDA=1 is set, and a compatible GPU is present.

What’s New in 0.1.7

• Added: real CUDA Conv (40 accelerated ops, up from 25); session-lifetime device-buffer pool + weight-residency cache + compiled-PTX-module cache; true batched MatMul/Gemm dispatch; CUDA activation residency shared with wgpu; six new data-movement op families (MaxPool/AveragePool, Resize, Pad, Slice, Concat, the zero-kernel Reshape family); channel-broadcast Add/Sub/Mul/Div, PRelu, BatchNormalization/OxiInstanceNorm; opt-in CUDA Graph capture/replay for MatMul/Gemm; a real on-device integration-test suite gated behind a gpu-tests feature so a plain cargo test never touches a GPU; oxionnx-gpu’s try_new_diagnosed/try_new_diagnosed_async distinguishing “no GPU present” from “a GPU is present but unreachable”; 9 new AVX2 dispatch-path regression tests in oxionnx-ops.
• Changed: oxionnx-gpu dispatch-or-decline thresholds now derived per-adapter and per-shape instead of seven flat constants; released GPU activation buffers now recycle into the context’s pool instead of being destroyed — measured at 96.1%/98.9% hit rates across two workloads, up from 4.9%; Auto placement now also routes the new data-movement and normalization ops to CUDA; the oxicuda-* stack updated 0.5.3 → 0.5.5.
• Fixed: MatMul/ReduceSum/ReduceMax could silently return wrong numbers from a two-stream race, not a kernel bug; OXIONNX_CUDA_STRICT=1 could exit 0 on a run it had just caught disagreeing with the CPU oracle; opening a second GpuContext in one process could panic on its first convolution; a flaky w2_async_run test assertion.

Tips

• OXIONNX_CUDA_STRICT=1 now actually tells you when it caught something. Check CudaDispatchError::is_verify_mismatch() (or oxionnx_cuda::is_verify_mismatch(), re-exported from the crate root) instead of trusting a bare exit code — a shadow-verification mismatch and an ordinary dispatch failure used to collapse into the same CPU-fallback path.
• Session::cuda_cache_counters()/cached_device_bytes()/is_weight_resident() are how you confirm residency is actually working, not just enabled — on a steady-state frame, weight_bytes_uploaded should read zero.
• CUDA Graph capture is opt-in for a reason — measure before you flip it. OXIONNX_CUDA_GRAPH wins up to 10% on small, launch-overhead-bound shapes and loses by as much on batched dispatch; the sign is a property of the shape, not of the release. It’s deliberately not enabled for Conv, where the cost is the weight transfer itself and a graph can’t remove that.
• OXIONNX_CUDA_GRAPH and OXIONNX_CUDA_STRICT don’t compose — the shadow-verification path leaves graph capture off with a warning rather than trying to verify a replayed graph.
• The new gpu-tests feature is required to build any on-device CUDA/wgpu integration test — a bare cargo test -p oxionnx-cuda builds and runs none of them, so CI without a GPU stays green without special-casing.
• If you’re routing ops with execution_providers::Auto, more ops now land on CUDA automatically — Reshape/Squeeze/Unsqueeze/Flatten/Slice/Concat/Pad and PRelu/BatchNormalization/OxiInstanceNorm/ReduceMean all route once CUDA’s own predicate claims them, not just Conv.

This is the foundation

OxiONNX is the ONNX inference backend across the COOLJAPAN stack — all bumped to oxionnx = "0.1.7" as part of this release:

• VoiRS — multimodal speech+vision inference
• OxiMedia — vision model deployment
• OxiWhisper — speech-model inference
• OxiGeo — geospatial ML inference
• ToRSh — model-hub inference integration
• OxiFY — LLM workflow orchestration
• OxiFace — face-model inference pipelines
• Oxi3D — 3D/graphics model inference
• OxiBonsai (oxionnx-proto) — protobuf model parsing

Repository: https://github.com/cool-japan/oxionnx

Star the repo if you want a CUDA backend that catches its own wrong answers instead of shipping them.

The era of “GPU accelerated” meaning “trust the kernel” is over.

Pure Rust ONNX inference — verified against its own CPU oracle, not just fast — is here.

— KitaSan at COOLJAPAN OÜ August 14, 2026