◉ COOLJAPAN
← All posts
August 24, 2026 • 12 min read

PandRS 0.4.1 Released — Every Chi-Squared p-Value Was Silently Wrong, and GroupBy Drops from O(n²) to O(n)

PandRS 0.4.1 fixes a chi2_sf sign bug that made chi-squared hypothesis tests under-report significance, corrects the Student-t/F CDFs, rewrites GroupBy from O(n²) to O(n), and replaces fabricated GPU/Python-binding results with real computations. 2,817 tests passing. The sovereign DataFrame layer for COOLJAPAN.

release pandrs dataframe pandas pure-rust correctness statistics gpu scirs2

A statistical test that always says “not significant” is worse than no test at all — it fails silently, with the full confidence of a passing test suite behind it.

Today we released PandRS 0.4.1 — a correctness-and-hardening release built on a second, independent, more exhaustive audit of the codebase. It closes a chi-squared p-value bug that survived 0.4.0’s own correctness pass, rewrites the typed GroupBy path from O(n²) to O(n), replaces fabricated GPU and Python-binding results with real computations, and fixes a data-loss bug in the columnar core — on top of a build that is, for the first time, genuinely warning-free.

No pandas. No scikit-learn or SciPy C-extensions. No Cython. No Python GIL.
No conda environments, no platform-specific wheels, no native segfaults hiding under a friendly API.
Just a memory-safe, high-performance DataFrame that compiles to a single static binary (or WASM) and computes the same answer everywhere — laptop, server, edge, browser.

Why PandRS 0.4.1 is a game changer

0.4.0 replaced a wave of fabricated ML and statistics stubs with real algorithms — including a native chi2_sf implementation to power real χ²-distributed p-values. What a second, independent, more exhaustive audit found in the months since is that fixing a stub is not the same as fixing it correctly:

PandRS 0.4.1 ends all of that.

Technical Deep Dive: Four Layers That Got Re-Audited

1. The shared statistical core (stats::special).
This is the architectural fix behind the p-value bugs above: a single, numerically-correct special-functions module — Lanczos ln_gamma; Numerical-Recipes regularized incomplete gamma (gser/gcf) and incomplete beta (betacf); exact CDF/survival/quantile for the normal, Student-t, chi-squared, and F distributions — now shared across stats and time_series, which previously carried their own, independently-buggy copies of the same math. TDistribution::ln_gamma’s one-term Stirling approximation (which put lnΓ(1) at −0.081 instead of 0, corrupting every t/χ²/F PDF built on it) is gone, replaced by the accurate Lanczos form. OLS regression p-values now use the Student-t distribution at the residual degrees of freedom instead of an anti-conservative normal approximation, and Shapiro-Wilk uses the real Royston (1992) AS R94 coefficients — W now matches published reference values (n=5 → W≈0.984, versus the former 0.943 stub).

2. The columnar data-integrity layer.
Column::from_any’s downcast now correctly recovers the real column type instead of silently returning an empty Int64Column. DataFrame::to_arrow() converts real column data with correct Arrow types instead of fabricating values from the row index or column name; CSV/JSON/Parquet (de)serialization read and write actual data instead of hardcoded ["Alice","Bob","Charlie"], "{}", or silent no-ops; joins perform a correct hash join instead of self-recursing or returning empty, and the right_join side-inversion bug is fixed. OptimizedDataFrame ↔ DataFrame conversion now preserves NA by upcasting per type — a NULL-free Int64Column still converts natively, but a column with a real NULL widens (Int64Column → Series<f64> with NaN, BooleanColumn → Series<String>) rather than defaulting to 0/false. On the performance side, apply(Axis::Row), duplicated, and drop_duplicates went from O(N²·C) to O(N·C) by extracting each column once instead of per row, and .iat random-position access is now O(1) instead of O(n) per call.

3. GPU honesty, CPU fallback and Python bindings.
The #[cfg(not(cuda_available))] matrix-multiply, elementwise, and reduce paths in src/gpu/cuda.rs now compute real ndarray CPU results instead of Array2::zeros; cuda_qr/svd/eigen/matrix_inverse and GPU DataFrame extensions return Error::NotImplemented instead of the unchanged input or an identity matrix standing in as if it were a real answer; stats::gpu/ml::gpu linear regression solves the real normal equations (was 0.1·i). Fabricated speedup numbers, a benchmark “speedup” column, hardcoded p-values, and fabricated P2P/synchronize_all device claims are gone, replaced by real cudarc-0.19 device queries. The Python GPU bindings (py_bindings/src/py_gpu.rs) — gpu_corr, gpu_pca, gpu_kmeans, gpu_linear_regression — now compute genuine Pearson correlation, PCA fit+transform, k-means clustering, and OLS instead of returning fabricated zeros.

4. Security, lints, and release engineering.
A cyclic ReBAC relationship graph previously crashed the process with SIGABRT; resolution now uses a visited-set with a bounded MAX_RESOLUTION_DEPTH and returns Ok(false). API keys are hashed at rest, JWT verification is real end-to-end, secret comparisons are constant-time, and credential encryption now carries AAD. object_store moved 0.13.2 → 0.14.1, closing RUSTSEC-2026-0194 and RUSTSEC-2026-0195, and a new deny.toml makes cargo deny check bans part of the release gate. On the hygiene side, the crate-wide #![allow(clippy::all)] is gone (clippy::correctness denied, clippy::suspicious/clippy::perf warned and clean), RUSTDOCFLAGS="-D warnings" cargo doc is clean on both the all-safe and docs.rs feature sets (~47 broken intra-doc links fixed), and package.exclude trims the published crate to what a library consumer actually needs.

Getting Started

cargo add pandrs

use pandrs::column::{Column, Float64Column};
use pandrs::OptimizedDataFrame;

fn main() -> pandrs::error::Result<()> {
    let mut df = OptimizedDataFrame::new();
    df.add_column(
        "x".to_string(),
        Column::Float64(Float64Column::new(vec![1.0, 2.0, 3.0, 4.0])),
    )?;
    df.add_column(
        "y".to_string(),
        Column::Float64(Float64Column::new(vec![3.0, 5.0, 7.0, 9.0])),
    )?;

    // New in 0.4.1: OLS directly on a columnar OptimizedDataFrame — no
    // conversion to the row-oriented DataFrame required first.
    let model = df.linear_regression("y", &["x"])?;
    println!(
        "intercept: {:.4}, slope: {:.4}, R²: {:.4}",
        model.intercept, model.coefficients[0], model.r_squared
    );
    Ok(())
}

What’s New in 0.4.1

New APIs

Statistical correctness

Performance

DataFrame / IO / Arrow correctness

GPU

Time series

Security

API hygiene

Dependencies

Tips

This is the foundation

PandRS is the DataFrame layer of the mature COOLJAPAN scientific stack, and 0.4.1 tightens the connections 0.4.0 first wired in:

Together they form a broad, mature, sovereign analytics stack: load a DataFrame, get a p-value you can trust, fit a real model on GPU or CPU, and serialize through Pure Rust compression — all in one static binary, with no Python interpreter and no native dependency chain to fight.

Repository: https://github.com/cool-japan/pandrs

Star the repo if you want a DataFrame whose hypothesis tests, GroupBy, and GPU paths all compute what they claim to compute.

The era of a p-value you can compute but can’t trust is over.

Pure Rust data analytics is here — fast, safe, correct, and sovereign.

— KitaSan at COOLJAPAN OÜ August 24, 2026

↑ Back to all posts