◉ COOLJAPAN
← All posts
August 7, 2026 • 8 min read

OxiONNX 0.1.5 Released — Async Execution, Streaming Generation, and a 232-Finding Production Hardening Pass

OxiONNX 0.1.5 adds async execution, cancellation, streaming generation, and session serialization — backed by a 12-lens, 232-finding hardening pass across parsing, spec-conformance, and GPU/CUDA/DirectML dispatch. 23 new operators, 2,946 tests passing. The sovereign Pure Rust ONNX layer for COOLJAPAN.

release oxionnx onnx inference ai machine-learning gpu wasm pure-rust scirs2

A 232-finding audit is either a scandal or a release note, depending entirely on whether you went looking for the findings yourself.

Today we released OxiONNX 0.1.5 — a production-hardening release for the COOLJAPAN Pure Rust ONNX inference engine. A 12-lens audit (spec conformance ×3, engine, proto robustness, panics, GPU, CUDA/CoreML, stubs, API/release, performance, test gaps) covering the whole workspace produced 232 findings, fixed across three waves of parallel implementation under strict file-ownership partitioning — alongside three new runtime capabilities: async execution, cancellation tokens, and streaming token generation for autoregressive models.

No C. No C++. No ONNX Runtime binaries. No external protobuf or CUDA dependencies.
No unsafe code outside a handful of documented, audited call sites.
Just memory-safe ONNX model execution that compiles to a single static binary (or WASM) and runs everywhere — from laptops to browsers to edge devices to GPUs.

Why OxiONNX 0.1.5 is a game changer

A production hardening pass is where an inference engine earns trust or loses it — a wrong answer that never panics is worse than a crash, because nothing tells you it happened. This release’s 12-lens audit went looking for exactly that class of bug across the whole workspace:

OxiONNX 0.1.5 ends all of that.

Technical Deep Dive: three new runtime primitives, one routing rewrite

  1. Async execution (src/session/async_run.rs) — Arc<Session>::run_async(inputs) -> RunFuture starts the model on its own thread immediately and returns a future you can .await under tokio/async-std/smol, or drive with the crate’s own dependency-free block_on. spawn_run() returns a blocking RunHandle for callers with no async executor at all.
  2. Cancellation tokens (src/session/cancellation.rs) — SessionBuilder::with_session_cancellation(token) makes every operator check a CancellationToken before it runs, unwinding with OnnxError::Cancelled at the first node boundary — on the sequential path, the rayon parallel path, and inside If/Loop/Scan bodies.
  3. Streaming generation + session serialization (src/streaming.rs, src/session/serialize.rs) — session.generate(prompt, config) runs one forward pass per next(), feeding present.* key/value outputs back in as the next step’s past.* inputs. session.save_optimized(path) persists the post-optimization graph so constant folding/CSE/fusion never re-run on load — proven by counting operator executions during load: zero.
  4. Placement unification (src/execution_providers.rs) — select_accelerator(op) now consults each backend’s own op-support predicate in priority order Cuda > DirectMl > Gpu, with a hard floor (MIN_GPU_DISPATCH_BYTES, 4096 bytes) below which even an explicit accelerator pin falls back to CPU, since a discrete-GPU round trip costs more than the op it would run.

Getting Started

cargo add oxionnx

The core inference API is unchanged in 0.1.5:

use oxionnx::{Session, Tensor};
use std::collections::HashMap;

let session = Session::from_file("model.onnx".as_ref())?;

let mut inputs = HashMap::new();
inputs.insert("input", Tensor::new(vec![1.0, 2.0, 3.0], vec![1, 3]));

let outputs = session.run(&inputs)?;
println!("{:?}", outputs);

New in 0.1.5 — streaming, token-by-token generation. There’s no tokenizer built in: token ids in, token ids out.

use oxionnx::{Session, GenerationConfig};

let session = Session::from_file("model.onnx".as_ref())?;
let prompt: Vec<i64> = vec![1, 464, 3290]; // from your own tokenizer

let config = GenerationConfig::default()
    .with_max_new_tokens(64)
    .with_eos_token_id(2);

for step in session.generate(&prompt, config)? {
    let step = step?;
    print!("{} ", step.token);
}

What’s New in 0.1.5

Tips

This is the foundation

OxiONNX is the ONNX inference backend across the COOLJAPAN stack — all bumped to oxionnx = "0.1.5" as part of this release:

Repository: https://github.com/cool-japan/oxionnx

Star the repo if you want an ONNX runtime where the hardening work is public — all 232 of these findings and their fixes are in this release’s history, not a private incident report.

The era of trusting a “just works” ONNX runtime you can’t audit is over.

Pure Rust ONNX inference — hardened, sovereign, and inspectable end to end — is here.

— KitaSan at COOLJAPAN OÜ August 7, 2026

↑ Back to all posts