OxiLLaMa 0.1.0 Released — Pure Rust LLM Inference Engine, Sovereign Alternative to llama.cpp

The LLM inference foundation of the COOLJAPAN ecosystem just became fully sovereign.

Today we released OxiLLaMa 0.1.0 — a complete, production-grade pure Rust LLM inference engine that is the clean-room, memory-safe alternative to llama.cpp.

No C. No C++. No Fortran. No FFI. No system libraries.
No unsafe code in hot paths. No build hell.
Just clean, memory-safe, high-performance LLM inference that compiles to a single static binary (or WASM) and runs everywhere — from laptops to browsers to edge devices to cloud GPUs.

Why OxiLLaMa 0.1.0 is a game changer

For years, fast local LLM inference meant depending on the excellent but C++-based llama.cpp (with its complex build system, memory risks, and limited portability).

These tools are powerful but suffer from:

• C/C++ memory unsafety and segfault risks
• Heavy build dependencies and platform-specific binaries
• Poor WASM/embedded support
• Limited native integration with safe Rust ecosystems
• Difficulty achieving full auditability and sovereignty

OxiLLaMa 0.1.0 ends all of that.

It delivers ≥80% of llama.cpp throughput while being 100% memory-safe and fully auditable.
Notable results (target on 8-core AVX2):

• LLaMA-3-8B Q4_K_M: ≥25 tokens/s
• Bonsai-8B Q1_0_G128: ≥22 tokens/s
• Mistral-7B Q4_K_M: ≥27 tokens/s

Technical Deep Dive: How We Built a Production-Grade LLM Engine in Pure Rust

The architecture is trait-based and built directly on the COOLJAPAN stack:

1. GGUF Engine (oxillama-gguf)
   Full GGUF v3 parser and tensor loader.

2. Quantization Layer (oxillama-quant)
   25 formats with SIMD kernels: Q4_0–Q8_1, all K-Quants (Q2_K–Q6_K), I-Quants (IQ1–IQ4), Q1_0_G128 (OxiBonsai), FP16/BF16/FP32.

3. Model Architectures (oxillama-arch)
   8 families: LLaMA (3.x/4.x + Mixtral MoE), Qwen3/Bonsai, Mistral, Gemma 2/3, Phi-3/4, Command-R, StarCoder, LLaVA-1.5 (multimodal).

4. Runtime & Server
   KV cache, sampling engine, OpenAI-compatible HTTP API server (oxillama-server).

5. Hardware & Bindings
   Optional wgpu GPU backend, WASM target, PyO3 Python bindings, CLI (oxillama run/serve/info).

Key Rust advantages:

• 100% Pure Rust — zero C/C++/Fortran, zero FFI
• Built on SciRS2, OxiBLAS, OxiFFT (and MeCrab for Japanese tokenization)
• Full memory safety with arena allocators and zero-copy tensors
• Single codebase for native + WASM + embedded
• Enterprise observability and graceful error recovery

What’s inside 0.1.0 (released April 15)

• Full GGUF v3 support and 25 quantization formats
• 8 model architectures (including LLaVA multimodal)
• OpenAI-compatible API server + CLI tools
• wgpu GPU backend and WASM bindings
• Production readiness confirmed with comprehensive tests across 11 crates
• 56,200 lines of pure Rust — zero warnings, Clippy0 + fail0 enforced

This is the foundation

OxiLLaMa is now the official LLM inference backend for the entire COOLJAPAN stack (total ecosystem: 21M+ SLoC Rust, 597 crates, 40+ production-grade libraries):

• SciRS2 / NumRS2 — all tensor and neural operations
• OxiONNX — hybrid ONNX + GGUF workflows
• OxiMedia / VoiRS — multimodal and voice-enabled agents
• OxiHuman / Spintronics — physics-informed and avatar LLMs
• ToRSh / OxiRAG — high-throughput RAG pipelines
• Future integration with OxiLean for formally verified inference

Repository: https://github.com/cool-japan/oxillama

Star the repo if you want sovereign, memory-safe LLM inference without llama.cpp’s C++ dependencies.

The era of “just compile llama.cpp” is over.

Pure Rust LLM inference is here — fast, safe, auditable, and sovereign.

— KitaSan at COOLJAPAN OÜ April 15, 2026