◉ COOLJAPAN
← All posts
July 13, 2026 • 10 min read

TenfloweRS 0.2.0 Released — Real Gradients for Every Layer, Real Steps for Every Optimizer

TenfloweRS 0.2.0 rewires the Python-facing, PyTorch-style implicit autograd system for real: every layer type gets tape-backed .backward()/.grad(), all 9 optimizers perform genuine gradient updates, and a correctness sweep fixes wrong Softmax/BatchNorm/GroupNorm/LayerNorm gradients — 14,536+ tests passing.

release tenflowers machine-learning deep-learning tensorflow rust autograd pytorch

Call .backward() on anything but a bare Dense layer in TenfloweRS 0.1.x, and PyTorch-style training either raised or silently computed the wrong gradient — GroupNorm’s backward pass was wrong by up to 760% for non-uniform gamma, and the loss curve gave you no way to tell.

Today we released TenfloweRS 0.2.0 — a complete rewrite of the Python-facing, PyTorch-style implicit autograd system in tenflowers-ffi. Every layer type now has real, tape-backed .backward()/.grad() support, all 9 optimizers perform genuine gradient-based parameter updates through a new optimizer_bridge module, and a sweep of gradient-correctness bugs across tenflowers-autograd and tenflowers-core — wrong Softmax/LogSoftmax backward, hardcoded-zero BatchNorm eval gradients, a GroupNorm/LayerNorm gamma-before-reduction bug, stubbed Slice/Gather backward, a row-major stride bug, and two tape-registry lifecycle bugs — are fixed and pinned down with finite-difference gradient-check tests.

TenfloweRS is the pure-Rust answer to TensorFlow. The tensor engine, the autodiff tape, the neural layers, the dataset pipeline, and the GPU backend are all Rust — no C runtime, no CUDA-C kernels, and no Python interpreter required at inference time (Python is opt-in, through a PyO3 FFI crate that stays publish = false and ships via maturin develop rather than crates.io). It compiles to a single static binary and runs everywhere the SciRS2/NumRS2 stack beneath it already runs.

Why TenfloweRS 0.2.0 is a game changer

Hand-rolled autograd systems fail in a specific, insidious way:

TenfloweRS 0.2.0 ends all of that.

Technical Deep Dive: what actually changed under the hood

  1. tenflowers-ffi — the per-layer backward wiring lives here: Dense, the conv/pooling family, Embedding/EmbeddingBag, all four normalization layers, MultiheadAttention, both Transformer layer types, and the full RNN family each participate for real in the tape, alongside a rewritten neural::losses and the new optimizer_bridge module that reads .grad() and writes updates back to parameters for all 9 optimizers. attention, conv_layers, recurrent, transformer, and extended_optimizers were each split into mod.rs + a dedicated tests.rs gaining its own gradient-check suite; PyTensor::slice/transpose/reshape now call the same record_and_link_unary hook used by every other tracked op. 337 Rust tests plus 55 pytest and 13 integration_test.py Python tests back this crate, including the three new end-to-end tests/test_training_convergence.py cases.
  2. tenflowers-autograd — home of the correctness fixes: Softmax/LogSoftmax backward now recomputes the forward output on the tape and applies the correct Jacobian-vector product; BatchNorm’s eval-mode grad_gamma/grad_beta are derived from running statistics instead of hardcoded to zero, with a fixed 3-D (NCL) shape case; LayerNorm and GroupNorm both had the same gamma-before-reduction bug — gamma applied to the final reduced result instead of folded in before the per-axis reduction sums, silently wrong for non-uniform gamma and measured up to 760% relative error in GroupNorm. A new dedicated Conv1D backward (ops/convolution_ops/conv1d.rs) and six new gradient-check test suites (conv1d_gradient_test, conv3d_gradient_test, group_instance_norm_gradient_check, normalization_gradient_check, slice_concat_stack_split_gather_gradient_test, activation_gaps_gradient_test) pin all of it against finite-difference/closed-form references. 575 tests, 5 skipped.
  3. tenflowers-core — a row-major-vs-Fortran-order stride bug in slice_with_stride (ops::manipulation::indexing) was silently producing wrong elements for any non-square, non-1-D sliced array, caught by an LSTM/GRU gate-slicing finite-difference test — exactly the kind of bug a real neural workload finds and a synthetic one doesn’t. Also fixed: a fallback module global that lived in an unsafe static mut synchronized only for its first write, now a safe OnceLock<RwLock<FallbackConfig>>; a missing #[cfg(feature = "parallel")] gate in histogram.rs that broke --no-default-features --features std builds (the Miri workflow); and improved WASM SharedArrayBuffer/SIMD-capability detection.
  4. Workspace hygiene — implicit_autograd.rs and six tenflowers-neural modules (attention, conv_layers, recurrent, transformer, normalization, optimizers) were split from single files into mod.rs + tests.rs submodule directories, keeping every file under COOLJAPAN’s 2000-line refactor policy while the new gradient-check suites landed alongside the code they verify.

Getting Started

Add the meta-crate:

cargo add tenflowers

Ordinary eager tensor math plus explicit autograd, in Rust:

use tenflowers_core::{Tensor, Device, Context};
use tenflowers_autograd::GradientTape;

// Create a context for eager execution
let ctx = Context::new()?;

let a = Tensor::<f32>::ones(&[2, 3]);
let b = Tensor::<f32>::from_vec(vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0], &[2, 3])?;
let c = a.add(&b)?;
let d = c.matmul(&b.transpose()?)?;

// watch() wraps a Tensor in a TrackedTensor; pow() is tensor-tensor, and
// gradient() takes slices of TrackedTensor for both targets and sources.
let tape = GradientTape::new();
let x = tape.watch(Tensor::<f32>::from_vec(vec![1.0, 2.0, 3.0], &[3])?);
let y = x.pow(&x)?;
let grads = tape.gradient(&[y], &[x])?;

0.2.0’s headline feature, though, is on the Python side — a real, tape-backed training loop in the same shape PyTorch users already know:

pip install maturin
maturin develop --release

import numpy as np
import tenflowers as tf

X = np.random.uniform(-1.0, 1.0, size=(16, 4)).astype(np.float32)
y = np.random.uniform(-1.0, 1.0, size=(16, 1)).astype(np.float32)
x_tensor = tf.tensor_from_numpy(X)
y_tensor = tf.tensor_from_numpy(y)

layer = tf.PyDense(4, 1, activation=None)
for param in layer.parameters():
    param.set_requires_grad(True)

optimizer = tf.SGD(learning_rate=0.1)

for step in range(50):
    y_pred = layer.forward(x_tensor)
    loss = tf.mse_loss(y_pred, y_tensor)

    loss.backward()
    optimizer.step(layer)
    optimizer.zero_grad(layer)

print(f"final loss: {float(tf.tensor_to_numpy(loss)[()]):.6f}")

Every layer type listed above, and every one of the 9 optimizers, now follows this same .backward() / optimizer.step() / optimizer.zero_grad() pattern — see crates/tenflowers-ffi/tests/test_training_convergence.py for the full end-to-end tests that assert the loss curve actually goes down.

What’s New in 0.2.0

Tips

This is the foundation

TenfloweRS leans on the rest of the COOLJAPAN ecosystem for everything below the tensor engine: NumRS2 for n-dimensional arrays, the SciRS2 stack for the scientific core and LAPACK-backed linear algebra, OxiBLAS (oxiblas-ndarray, behind the blas-oxiblas feature) for pure-Rust BLAS, OxiFFT for FFT ops, and Oxicode for serialization. It stands alongside ToRSh, TrustformeRS, and sklears in the COOLJAPAN machine-learning stack.

Repository: https://github.com/cool-japan/tenflowers

Star the repo if a hand-rolled autograd engine that gradient-checks itself against finite differences, layer by layer, is something you want to see more of. The era of trusting a .backward() call just because it didn’t crash is over. Pure Rust ML is here — fast, safe, and sovereign.

— KitaSan at COOLJAPAN OÜ July 13, 2026

↑ Back to all posts