◉ COOLJAPAN
← All posts
August 11, 2026 • 8 min read

OxiONNX 0.1.6 Released — GPU Weight Residency, a 692 GFLOP/s Conv2D Kernel, and WebAssembly Inference That Finally Runs

OxiONNX 0.1.6 ships GPU weight/activation residency and a direct implicit-GEMM Conv2D kernel (~692 GFLOP/s on M3) — plus the fix making wasm32 CPU inference actually run, and a CoreML compile cache ending 857 GB of orphaned recompiles. 3,212 tests passing. The sovereign Pure Rust ONNX layer for COOLJAPAN.

release oxionnx onnx inference ai machine-learning gpu wasm pure-rust scirs2

A GPU feature flag that makes things slower isn’t a GPU feature — it’s a benchmark you haven’t run yet.

On August 11 we released OxiONNX 0.1.6 — a release built around one question the previous GPU work never actually answered: for this op, on this hardware, does the GPU path win or lose against the tuned CPU kernel it’s supposed to replace? Answering it honestly meant building weight and activation residency so the bus stops being the bottleneck, a direct Conv2D kernel fast enough to be worth dispatching in the first place, and a measured size gate that declines the ops which still lose even after that — plus the unrelated but overdue fix that makes wasm32 CPU inference actually execute in a browser instead of merely compiling.

No C. No C++. No ONNX Runtime binaries. No external protobuf or CUDA dependencies.
No unsafe code outside a handful of documented, audited call sites.
Just memory-safe ONNX model execution that compiles to a single static binary (or WASM) and runs everywhere — from laptops to browsers to edge devices to GPUs.

Why OxiONNX 0.1.6 is a game changer

A GPU feature flag is easy to ship and easy to get wrong in a way nobody notices, because “GPU accelerated” sounds like a strict improvement right up until someone measures it:

OxiONNX 0.1.6 ends all of that.

Technical Deep Dive: four layers that make the GPU path honest

  1. Async GPU execution (src/session/run/sequential_async.rs, gpu_owner.rs) — Session::run_gpu_async is a second, smaller node-execution loop that runs the same nodes in the same order as Session::run, differing in exactly one place: a GPU dispatch is .awaited instead of blocked on, so the calling task yields to an event loop instead of blocking a thread a browser page may never have. try_gpu_dispatch_async is now the real dispatcher everywhere; the synchronous path is a thin pollster::block_on wrapper around it.
  2. Session-lifetime weight residency (oxionnx-gpu/src/context/resident.rs) — ResidentBuffers caches a TrackedBuffer per graph-initializer identity for the life of the GpuContext, keyed and checked (not trusted) against the kernel slot and byte length it was uploaded for, so a key collision uploads fresh bytes instead of silently serving the wrong ones. Keyed per numeric format too, so flipping Session::set_f16_compute() mid-session is safe.
  3. Run-scoped activation residency (src/session/gpu_activations.rs) — the other half of the transfer problem: a GPU node’s output can now stay in its device buffer for the next GPU consumer to bind in place, precomputed per-run so a name qualifies only when every consumer can accept it resident and it isn’t a graph output. Toggle with Session::activation_residency_enabled().
  4. The direct Conv2D kernel (oxionnx-gpu/src/shaders/conv2d.rs) — the im2col gather happens in-register inside the shader as the input tile stages into workgroup memory, instead of materializing a column matrix kH*kW times the input size and re-uploading it every call. Bias and activation fuse into the epilogue. group > 1 convolutions still decline to the old hybrid path.

Getting Started

cargo add oxionnx

The core inference API is unchanged in 0.1.6:

use oxionnx::{Session, Tensor};
use std::collections::HashMap;

let session = Session::from_file("model.onnx".as_ref())?;

let mut inputs = HashMap::new();
inputs.insert("input", Tensor::new(vec![1.0, 2.0, 3.0], vec![1, 3]));

let outputs = session.run(&inputs)?;
println!("{:?}", outputs);

New in 0.1.6 — genuinely asynchronous GPU execution, awaited instead of blocked on (needs the gpu feature):

use oxionnx::{Session, Tensor};
use std::collections::HashMap;

let mut session = Session::from_file("model.onnx".as_ref())?;

let mut inputs = HashMap::new();
inputs.insert("input", Tensor::new(vec![1.0, 2.0, 3.0], vec![1, 3]));

if session.enable_gpu_async().await {
    let outputs = session.run_gpu_async(&inputs).await?;
    println!("{:?}", outputs);
}

What’s New in 0.1.6

Tips

This is the foundation

OxiONNX is the ONNX inference backend across the COOLJAPAN stack — all bumped to oxionnx = "0.1.6" as part of this release:

Repository: https://github.com/cool-japan/oxionnx

Star the repo if you want a GPU feature flag that has to prove it’s actually faster before it gets to run.

The era of “GPU accelerated” meaning “trust us” is over.

Pure Rust ONNX inference — measured, not assumed — is here.

— KitaSan at COOLJAPAN OÜ August 11, 2026

↑ Back to all posts