◉ COOLJAPAN
← All posts
July 22, 2026 • 8 min read

OxiBonsai 0.2.3 Released — 109 Agents, 91 Confirmed Bugs, and a Silent CUDA Correctness Fix

OxiBonsai 0.2.3 ships a two-pass, 109-agent production-hardening audit: a P0 fix for a silent CUDA prefill/decode KV-cache desync, full Metal GPU coverage for Q4_0/Q8_0/K-quant and FP8 batch prefill, real OpenAI-compatible penalties and logprobs, and tokenizer fidelity fixes for CJK/emoji text — 91 confirmed findings closed, 5,158 tests passing. Sub-2-bit Pure Rust sovereign AI inference for the COOLJAPAN ecosystem.

release oxibonsai llm inference pure-rust quantization metal cuda security production-hardening

A CUDA prompt of 17+ tokens was silently corrupting decode output — the batch-prefill KV cache and the per-token decode KV cache had drifted apart. A 109-agent audit found it, and 90 other real bugs, before any of them reached a user.

Today we released OxiBonsai 0.2.3 — a large, two-pass production-hardening release: 91 confirmed findings across 23 implementation packages, closed by a systematic audit rather than by waiting for bug reports.

No llama.cpp. No BLAS. No C, no C++, no Fortran runtime. OxiBonsai is the first Pure Rust, zero-FFI inference engine for PrismML’s sub-2-bit Bonsai model family — the 1-bit line (Q1_0_g128) and the ternary line (TQ2_0_g128) — running on CPU SIMD (AVX2/AVX-512/NEON/WASM), Apple Silicon (Metal), and NVIDIA (CUDA NVRTC), all on top of SciRS2, OxiBLAS, OxiFFT, OxiARC, and OxiONNX.

Why OxiBonsai 0.2.3 matters

Most inference engines find correctness gaps by shipping and waiting for a GitHub issue. That works — slowly, and only for the bugs someone happens to trigger and report. 0.2.3 took a different approach: 15 domain-focused sweeps plus adversarial per-finding verification, run against the entire 0.2.2 codebase, then a same-day second pass that went back for the platform gaps the first pass had only documented as honest limitations.

The headline result is the kind of bug that doesn’t announce itself:

0.2.3 ends the “it compiles and the demo works” bar and replaces it with “an adversarial pass tried to break every claim in the README, and 91 of those attempts found something real.”

Technical Deep Dive

1. Unified KV-cache ownership (oxibonsai-kernels, CUDA). cuda_prefill/state.rs now delegates to the same cuda_full_layer::acquire_kv_cache that per-token decode uses, instead of maintaining a parallel cache of its own. The missing per-token d_pos_seqlen upload in the Q1 prefill loop is fixed in the same pass. Q4_0/Q8_0/K-quant CUDA batch prefill hit the same bug class and are disabled by default (clean Err → sequential fallback, bit-correct, just not batched) until they get the same unification.

2. Metal model-layer dispatch (oxibonsai-model). New Metal GEMV kernels for Q4_0/Q8_0 and the full K-quant family sit behind the same short-circuit-then-fallback pattern the CUDA path already used: Linear*::forward() tries Metal, falls back to CPU on any error. FP8 batch prefill is a deliberate hybrid — linear projections (fused QKV, attn-output, gate/up SwiGLU, FFN down) run as batched FP8 GEMMs on the GPU, while q/k-norm, RoPE, attention, and the K/V store stay on the CPU against one shared cache. That design sidesteps the split-KV-cache bug class by construction, rather than needing the CUDA-style fix after the fact. Both are parity-validated on Apple Silicon M3 (GEMV ≤1e-4 rel / 5e-3 abs; FP8 prefill logit cos=1.000000, identical greedy continuation).

3. A real penalty/logprobs seam (oxibonsai-runtime). Sampler::sample_with_history applies PenaltyParams (OpenAI’s frequency/presence formula, [-2.0, 2.0]) over recent-token history; the no-penalty case stays bit-identical to prior output. InferenceEngine::generate_with_logprobs captures logits during generation instead of needing a second pass. Both are wired through /v1/chat/completions, the extended chat endpoint, and /v1/completions — plus GGUF-resolved eos_token_id(), which previously fell back to a hardcoded value regardless of the loaded model.

4. Server-boundary hardening (oxibonsai-serve, oxibonsai-runtime). max_tokens/temperature/top_p/n now get honest 400s instead of silently accepting an unbounded request; RateLimitConfig::trusted_proxies closes a spoofable X-Forwarded-For/X-Real-IP bypass by only trusting those headers from an allowlisted proxy peer; bearer-token comparison is constant-time; max_concurrent_requests/per_request_timeout_ms are genuinely enforced via a tower admission stack (overload → 503, timeout → 408).

Getting Started

Install the CLI:

cargo install oxibonsai-cli

Serve with the new production admission controls — bearer auth, a concurrency cap, and per-request timeouts:

oxibonsai serve \
  --model models/Ternary-Bonsai-1.7B.gguf \
  --bearer-token "$OXI_TOKEN" \
  --max-concurrent-requests 64 \
  --request-timeout-ms 30000

Then call it exactly like OpenAI’s API — penalties and logprobs now do something:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $OXI_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bonsai",
    "messages": [{"role": "user", "content": "Explain ternary quantization in one sentence."}],
    "frequency_penalty": 0.4,
    "logprobs": true
  }'

What’s New in 0.2.3

Tips

This is the foundation

OxiBonsai is the inference end of the COOLJAPAN ecosystem — sub-2-bit Bonsai models from PrismML, served on top of SciRS2, OxiBLAS, OxiFFT, OxiARC, and OxiONNX, with no FFI and no C/C++/Fortran runtime anywhere underneath. 0.2.3 doesn’t add a feature you can see in a demo GIF; it closes the gap between what the README claimed and what an adversarial pass could actually verify — on both the CPU/GPU correctness side and the API-surface honesty side.

Repository: https://github.com/cool-japan/oxibonsai

Star the repo if you’d rather an audit find your inference engine’s bugs than your users do.

Pure Rust inference that’s been through 109 agents looking for reasons not to trust it is here — fast, safe, and sovereign.

— KitaSan at COOLJAPAN OÜ July 22, 2026

↑ Back to all posts