◉ COOLJAPAN
← All posts
August 17, 2026 • 10 min read

OxiLLaMa 0.1.4 Released — Security Hardening Against Real llama.cpp, a GPU Offload Backend, and 37.5x Faster Decode

OxiLLaMa 0.1.4 is the Pure Rust LLM inference engine and sovereign alternative to llama.cpp. This release fixes a remotely-crashable GGUF parser, an unauthenticated /admin/*, and a shipped server that ran with every middleware disabled; adds a GGUF-embedded tokenizer, K-quant encoders, and a GPU offload backend; and verifies logits token-for-token against real llama.cpp — plus a 37.5x Qwen3 decode speedup. 3,751 tests passing.

release oxillama llm-inference gguf llama.cpp pure-rust security gpu scirs2

A production LLM server that runs with authentication, rate limiting, and CORS silently disabled is worse than one that refuses to start — a crash is loud; dead middleware is silent.

Today we released OxiLLaMa 0.1.4 — the correctness-and-security-hardening release. The theme this cycle wasn’t a new architecture count; it was comparing OxiLLaMa against something other than itself for the first time, and fixing every real bug that comparison found.

No C. No C++. No Fortran. No FFI. No system libraries. Where llama.cpp drags along a C++ toolchain, fragile build dependencies, and the segfault class of bugs that comes with manual memory management, OxiLLaMa compiles to a single static native binary — or to WebAssembly, or to embedded targets — from one codebase. The tagline still holds: Pure Rust LLM Inference Engine — The Sovereign Alternative to llama.cpp. It is built on the COOLJAPAN stack: SciRS2 for tensor primitives and neural ops, OxiBLAS for Pure Rust BLAS, and OxiFFT for the FFT behind RoPE.

Why OxiLLaMa 0.1.4 is a game changer

Most inference engines only ever check their own numbers against themselves. This cycle, OxiLLaMa was compared against a real, upstream-built llama.cpp for the first time — and that comparison surfaced bugs no amount of self-consistency testing could have caught:

0.1.4 fixes all five, this time golden-tested against values produced by compiling and running upstream llama.cpp’s own C code — not the previously-wrong internal reference. On top of the fixes:

As of 0.1.4 that is roughly 164,000 lines of Pure Rust across 11 crates, with 3,751 tests passing (--all-features; 3,631 on default features), zero compiler warnings in either configuration.

Technical Deep Dive: where the hardening landed

OxiLLaMa is organized as 11 focused crates — oxillama (meta), oxillama-gguf, oxillama-quant, oxillama-arch, oxillama-runtime, oxillama-server, oxillama-bench, oxillama-gpu, oxillama-py, oxillama-wasm, and oxillama-cli. Here is where 0.1.4 landed.

oxillama-gguf — parser hardening and zero-copy loading. Beyond the integer-overflow bounds-check bypass, the parser also had unbounded metadata-array recursion (crashable from 12 input bytes per nesting level), unchecked u64 arithmetic on tensor offsets, allocation bombs from unvalidated dimension counts, and silent duplicate-tensor collapse — all fixed. Separately, a new SharedBytes/ByteOwner abstraction lets QuantTensor hold an Arc-backed view straight into the mmap instead of copying into a private Vec<u8>; ten architecture loaders were migrated off .to_vec(). On a Qwen3-4B Q4_K_M checkpoint: peak footprint 4.204 GB → 148 MB, weight-load wall time 0.604 s → 0.014 s.

oxillama-quant — real K-quant encoders and a correctness pass. New byte-for-byte ports of llama.cpp’s quantize_row_q{2,3,4,5,6}_K_ref reference encoders (including the round-half-to-even nearest_int bit trick and the make_qx_quants/make_q3_quants/make_qkx2_quants helpers) mean oxillama quantize can now actually produce K-quant files, not just read them. Alongside that: the seven-format weight-layout fix described above, plus AVX-512 IQ kernels and NEON SDOT integer dot products (asm!("sdot …"), since the stable vdotq_s32 intrinsic is still gated behind an unstable feature) for an additional +11.8% decode on Apple M3.

oxillama-arch + oxillama-runtime — new loaders, a real tokenizer, and GPU wiring. Five architectures (Falcon, GPT-NeoX, StableLM, OLMo2, MiniCPM) gained direct GGUF loaders where previously only pre-materialized Vec<f32> weights worked, and Mixtral’s attention — previously a stub computing Q = K = V = norm(hidden) — is now a real block. The LLaMA-family RoPE convention was fixed (these architectures used NeoX half-split rotation instead of llama.cpp’s actual interleaved-pairs convention; synthetic weights couldn’t tell the two apart, which is exactly why real-checkpoint parity testing is what finally caught it). On the runtime side: the GGUF-embedded tokenizer, an O(1) speculative-decoding KV resync via InferenceEngine::truncate, and the new GpuPolicy/GpuOptions wiring that routes eligible weights to oxillama-gpu behind the gpu feature — CPU behavior is byte-for-byte unchanged when it’s off.

oxillama-server — the middleware fix and a real chat template. Beyond the build_app_with_config fix and the /admin/* auth fix, the five-site fabricated-chat-template bug is gone: model-family detection and rendering (Llama-3 / ChatML / Mistral / Alpaca, from GGUF metadata and vocab special tokens) now lives in one place, oxillama_runtime::chat_template, and every route resolves it once at model load. finish_reason is no longer hardcoded to "stop" — it’s threaded through chat, completions, WebSocket, and batch records, so hitting max_tokens correctly reports "length". WebSocket inference now streams real tokens instead of a hardcoded stub response.

Getting Started

Add the library to your project:

cargo add oxillama

Or grab the CLI:

cargo install oxillama-cli

OxiLLaMa targets Rust 1.89+. This is Recipe 1 from the crate’s doctested RECIPES.md — load a GGUF and stream 100 tokens:

use oxillama::runtime::{EngineConfig, InferenceEngine, SamplerConfig};

fn main() -> anyhow::Result<()> {
    let config = EngineConfig {
        model_path: "llama-3-8b-instruct.Q4_K_M.gguf".to_string(),
        num_threads: 4,
        ..EngineConfig::default()
    };

    let mut engine = InferenceEngine::new(config);
    engine.load_model()?;

    let sampler = SamplerConfig {
        temperature: 0.8,
        top_k: 40,
        ..SamplerConfig::default()
    };

    let prompt = "The Rust programming language is great because";

    engine.generate_with_config(prompt, 100, sampler, |tok| {
        use std::io::Write as _;
        print!("{tok}");
        let _ = std::io::stdout().flush();
    })?;
    println!();

    Ok(())
}

Prefer the server? Start it and it now actually enforces what it claims to:

oxillama serve --model models/llama-3-8b-q4_k_m.gguf --port 8080

What’s New in 0.1.4

Per-crate test breakdown for 0.1.4: gguf=354, quant=502, arch=1036, runtime=701, server=357, cli=175, bench=146, gpu=266 (--features gpu), wasm=59, py=131.

Tips

This is the foundation

OxiLLaMa fits the COOLJAPAN Pure Rust stack as it stands today. Tensor primitives and neural ops come from SciRS2 (scirs2-core / scirs2-linalg / scirs2-neural 0.6.5); the dense math leans on OxiBLAS (0.2.2); RoPE rides on OxiFFT (0.4.2); serialization uses oxicode (0.2.6); the GPU path runs on wgpu (30.0.0); and remote GGUF/Hub downloads use hf-hub 1.0 over reqwest. The aggressive 1-bit Q1_0_G128 quantization that powers Bonsai-8B was absorbed directly into oxillama-quant, in-house. Every layer is Pure Rust, all the way down.

Repository: https://github.com/cool-japan/oxillama

Star the repo if you believe production LLM inference should be checked against the real thing, not just against itself. Pure Rust LLM inference is here — fast, safe, and sovereign.

— KitaSan at COOLJAPAN OÜ August 17, 2026

↑ Back to all posts