◉ COOLJAPAN
← All posts
July 9, 2026 • 11 min read

TrustformeRS 0.2.0 Released — CUDA Catches Up to Metal's Resident Attention, libtorch Is Gone for Good

TrustformeRS 0.2.0 gives CUDA the same device-resident attention pipeline Metal already had, adds batched CUDA matmul and refcounted GPU buffers, fixes a silent CUDA data race, and drops the torch/libtorch backend workspace-wide. The sovereign transformer layer for the COOLJAPAN ecosystem.

release trustformers rust transformers cuda gpu pure-rust metal machine-learning

Metal has had a fully device-resident attention chain — QKV split, RoPE, causal softmax, KV-cache append, head merge, zero trips back to the host — for a while now. As of 0.2.0, CUDA finally has the same chain, and the workspace’s last C++ tensor backend is gone.

Today we released TrustformeRS 0.2.0 — a release that closes CUDA’s attention-pipeline gap with Metal, adds batched and NumPy-style broadcasting CUDA matmul, gives GPU buffers a real refcounted lifecycle, mounts three subsystems and ten pipeline scaffolds that were sitting unwired in the tree, fixes a silent CUDA correctness race, and removes the torch/tch (libtorch) backend from the workspace entirely.

TrustformeRS is Pure Rust Hugging Face Transformers: transformer/LLM loading and inference, tokenizers, training, and serving — no Python required, no PyTorch, and as of 0.2.0, no libtorch anywhere in the workspace either. The torch Cargo feature, the tch dependency, and the Tensor::Torch variant are gone from trustformers-core, trustformers-training, trustformers, and trustformers-c — tch links libtorch, a C++ shared library, and cutting it is a direct step forward on the Pure-Rust default feature tree. candle remains as the one optional non-CPU tensor-framework integration, off by default. TrustformeRS still compiles to a single static binary — or to WASM, or onto mobile — and runs anywhere Rust runs.

Why TrustformeRS 0.2.0 is a game changer

Running TrustformeRS on CUDA before 0.2.0 meant living with a few real gaps:

TrustformeRS 0.2.0 ends most of that:

Technical Deep Dive: what 0.2.0 actually changed under the hood

1. trustformers-core — the GPU backend layer. The new gpu_ops::cuda::oxicuda::attention module (gather_heads_gpu_to_gpu, rope_neox_gpu_to_gpu, softmax_causal_gpu_to_gpu, attention_prefill_gpu_to_gpu, attention_decode_gpu_to_gpu, concat_v_cache_gpu_to_gpu, add_gpu_to_gpu) sits alongside a new gpu_ops::cuda::oxicuda::batched module (BatchedMatmulPlan, dispatch_oxicuda_matmul/dispatch_oxicuda_matmul_resident). CudaTensorData is restructured around the new BufferHandle — a breaking change: the old public buffer_id: BufferId field is now buffer: BufferHandle plus buffer_id()/device_id() accessors. OxicudaCudaBackend::new validates a requested device_id against the real enumerated device count instead of surfacing an opaque driver failure, and default_cuda_device_id() reads a new TRUSTFORMERS_CUDA_DEVICE environment variable for host-dispatched matmuls on multi-GPU machines. Underneath it all, oxicuda-{blas,dnn,memory,driver,metal,backend} moved 0.4.0 → 0.4.1, fixing a GEMM transpose-flag bug and a batched-GEMM launch-tuple corruption bug that made gemm_strided_batched produce incorrect results in 0.4.0 — part of the motivation for shipping the resident-attention and batched-matmul work in the same release.

2. trustformers-models — two new architecture families, and an honest breaking fix. swin/deit Cargo features mount SwinModel/SwinForImageClassification and DeiTModel/DeiTForImageClassification (with distillation-token support). Separately, mamba, rwkv, s4, falcon, stablelm, and linformer had Cargo features that existed but were never attached via #[cfg(feature = ...)] to their pub mod declarations, so they compiled unconditionally regardless of the flag; that’s fixed now, which means a build that relied on the old, accidental always-on behavior needs to add the feature explicitly. The all feature aggregate was also missing llama3_2 and mistral_v3 — --features all now actually builds every supported architecture.

3. trustformers (the umbrella crate) — three subsystems and ten pipelines come out of the basement. cache, finetuning, and loading were fully implemented modules that simply were never pub mod-declared; 0.2.0 mounts all three, plus ten trustformers::pipeline task modules — audio generation, document classification, feature extraction, image segmentation, speech recognition, table QA, text-to-image, video classification, visual grounding, zero-shot audio classification — that were in the same boat. Worth being precise about what shipped: the pipeline mounts are a real API-surface milestone, not a production-inference one — all ten currently return deterministic mock output pending real model backends. cache and finetuning are the real thing today (see Tips below).

4. Training, optimizers, tokenizers. trustformers-training::hpo is now mounted: MultiObjectiveHpo/ParetoFront/compute_pareto_front/hypervolume_indicator/non_domination_sort for Pareto-front search, plus AutoLrSelector/LrRangeTest for automatic learning-rate range tests. trustformers-optim re-exports 21 fsdp/optimizer_surgery/per_layer_quant types directly at its crate root (locked in by a new tests/crate_root_reexports.rs), and trustformers-tokenizers gains NFKCNormalizer/NFKDNormalizer alongside the existing NFC/NFD pair.

Getting Started

cargo add trustformers

use trustformers::prelude::*;
use trustformers::{AutoModel, AutoTokenizer, Tensor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load model and tokenizer
    let tokenizer = AutoTokenizer::from_pretrained("bert-base-uncased")?;
    let model = AutoModel::from_pretrained("bert-base-uncased")?;

    // Tokenize input
    let tokenized = tokenizer.encode("Hello, Rust world!")?;

    // AutoModel's `Model` impl is Tensor-in/Tensor-out, so wrap the token IDs
    // as a Tensor before running inference.
    let ids: Vec<f32> = tokenized.input_ids.iter().map(|&id| id as f32).collect();
    let len = ids.len();
    let inputs = Tensor::from_vec(ids, &[len])?;

    let outputs = model.forward(inputs)?;
    println!("Output shape: {:?}", outputs.shape());
    Ok(())
}

To exercise 0.2.0’s new resident CUDA attention chain, move a GPT-2 model onto a GPU device explicitly:

// Cargo.toml: trustformers-core = { version = "0.2", features = ["cuda"] }  // or "metal"
use trustformers_core::Device;
use trustformers_models::gpt2::{Gpt2Config, Gpt2Model};

let device = Device::CUDA(0); // or Device::Metal(0)
let mut model = Gpt2Model::new_with_device(Gpt2Config::default(), device)?;
model.weights_to_gpu_cuda(&device)?;   // Metal: `weights_to_gpu`
let outputs = model.forward(inputs)?;  // prefill AND decode now run the full
                                        // resident attention chain on CUDA

On a multi-GPU host, point host-dispatched matmuls at a specific card without touching code:

TRUSTFORMERS_CUDA_DEVICE=1 cargo run --release --features cuda

What’s New in 0.2.0

Added

Changed

Fixed

Removed

Tips

This is the foundation

TrustformeRS 0.2.0 rides on OxiCUDA 0.4.1 (bumped this release, fixing the GEMM transpose-flag and batched-GEMM launch-tuple bugs that motivated the new resident-attention and batched-matmul work above) for CUDA compute, oxicuda-metal for Apple Silicon, SciRS2 0.6.0 for numerics, OxiBLAS for Pure-Rust BLAS/LAPACK, OxiCode for serialization, and OxiARC 0.3.5 (oxiarc-zstd/-deflate/-lz4/-archive) for compression. It sits beside ToRSh, SkleaRS, and TenfloweRS in the COOLJAPAN model-training and serving stack.

Repository: https://github.com/cool-japan/trustformers

Star the repo if you want a CUDA backend that doesn’t quietly cut the corners Metal already learned not to cut — and a tensor stack with no C++ library left to link against. The era of leaking GPU caches and an untrusted libtorch dependency is over. Pure Rust GPU-accelerated transformers are here — fast, safe, and sovereign.

— KitaSan at COOLJAPAN OÜ
July 9, 2026

↑ Back to all posts