How Google Made AI Memory
6× Smaller

Every time an AI model generates a word, it stores a “memory” of every previous word in the conversation. This KV cache is the single biggest memory consumer during inference. Google Research found a way to compress it 6× with zero quality loss — using a single elegant math trick that the industry is already racing to adopt.

Every Bit Costs Extra Bits

When you quantize numbers — storing them in fewer bits to save memory — you need to keep track of how you quantized them. Every block of quantized data needs a zero point and a scale factor stored in full precision. Depending on block size, this overhead adds 1–2 extra bits per number, partially negating the whole point of compressing in the first place.

Random Rotation Makes Everything Predictable

The key discovery is deceptively simple. If you multiply a vector by a random rotation matrix before quantizing it, the resulting coordinates follow a known mathematical distribution — regardless of what the original data looked like. Since you know the distribution in advance, you can precompute the optimal quantization codebook once and reuse it forever. No per-block statistics. No zero points. No scales. Zero overhead.

Three Papers, One Family

TurboQuant is the culmination of a three-paper research arc from the same team at Google Research. Each paper builds on the last, and together they form a complete theoretical and practical framework.

QJL

AAAI 2025 · The Foundation

Apply a random Gaussian projection to key embeddings, then keep only the sign bit. 1-bit quantization with zero overhead. The asymmetric trick: quantize the key but leave the query full-precision, yielding an unbiased inner product estimator.

>5× memory reduction at 3 bits, no accuracy drop. Open-source CUDA kernels on GitHub.

PolarQuant

AISTATS 2026 · Multi-Bit Precision

Random rotation, then convert to polar coordinates and quantize the angles. After rotation, angle distributions are analytically known — concentrated around π/4 at higher recursion levels. Flexible per-level bit allocation.

4.2× compression. Best quality scores among all methods on long-context benchmarks.

TurboQuant

ICLR 2026 · The Unified Framework

Combines optimal Cartesian scalar quantization (not polar) after rotation with QJL residual correction for unbiased inner products. Provably within 2.7× of the information-theoretic optimum.

At 3.5 bits: identical to full precision. At 2.5 bits: 0.62 points below. Vector search 500,000× faster indexing than PQ.

The Math That Makes It Provably Optimal

These aren’t engineering heuristics — they’re backed by information-theoretic proofs showing exactly how close to optimal TurboQuant is, and exactly how much room is left to improve.

Random Rotation → Known Distribution

After multiplying by a random orthogonal matrix, each coordinate of a unit vector follows a Beta((d−1)/2, (d−1)/2) distribution. In high dimensions, this converges to N(0, 1/d). The distribution is data-independent — no calibration needed.

LEMMA 1 — TURBOQUANT

Optimal Codebook via Panter-Dite

Since the distribution is known, the optimal scalar quantizer (Lloyd-Max) can be precomputed once. For b>4 bits, the Panter-Dite formula gives distortion = (1/12)·(∫f(x)^1/3dx)³·4^−b. The 1/3 power arises from balancing quadratic distortion penalty against linear bit budget.

THEOREM 1 — TURBOQUANT

QJL Makes It Unbiased

The MSE-optimal quantizer has multiplicative bias (2/π at 1 bit). Applying a 1-bit Johnson-Lindenstrauss transform to the residual cancels this bias exactly. Quantizing the key to sign bits while leaving the query unquantized gives an asymmetric unbiased estimator.

THEOREM 2 — TURBOQUANT

Provable Lower Bound

Via Shannon’s rate-distortion theory and Yao’s minimax principle: no quantizer can achieve MSE < 1/4^b. TurboQuant achieves (√3·π/2)/4^b ≈ 2.7/4^b. The gap factor is 2.7× worst-case, and just 1.45× at 1 bit.

THEOREM 3 — TURBOQUANT

Smaller Memory, Same Quality

Evaluated on Llama-3.1-8B-Instruct across the LongBench suite. TurboQuant at 3.5 bits exactly matches full 16-bit precision. Even at 2.5 bits — over 6× compression — the quality drop is negligible.

Four Days from Paper to Working Code

The Google Research blog post went up March 24, 2026. Within four days, the two most important open-source LLM serving frameworks had working implementations. This is one of the fastest research-to-production transitions I’ve seen in ML infrastructure.

Why This Matters

This isn’t incremental. A 6× reduction in the single biggest memory consumer during LLM inference changes the economics of AI deployment. Here’s how.

6× Longer Context, Same GPU

If KV cache was your bottleneck at 32K tokens, you can now handle 192K on the same hardware. Long-context applications — document analysis, multi-turn agents, RAG over large corpora — become practical where they weren’t before.

CONTEXT LENGTH

6× More Users Per GPU

For serving providers, KV cache memory determines how many concurrent users a GPU can handle. Shrinking it 6× means serving 6× more requests simultaneously — or using fewer GPUs for the same load.

COST REDUCTION

Vector Search Revolution

TurboQuant isn’t just for LLMs. For vector databases, it achieves better recall than Product Quantization while building indices 500,000× faster (0.002s vs. 494s at d=3072). No dataset-specific tuning required.

VECTOR SEARCH

Fully Open Research

Complete papers with proofs, algorithms, and pseudocode are on arXiv. QJL has open-source CUDA kernels. The community implemented working integrations in under a week. The math is a public good.

OPEN ACCESS