PrismML Compressed a 27.8B-Parameter Model to 5.9GB and Kept 98.2% of Its Benchmarks

PrismML released Bonsai 2 27B on September 17, 2026, a 27.8-billion-parameter model that fits in 5.9GB while retaining 98.2% of the aggregate benchmark performance of its full-precision counterpart Qwen3.8 27B. The compression converts weights to a ternary format holding only -1, 0 and +1, adds FP16 group-wise scaling, and lands at 1.76 effective bits per weight, more than a 9x reduction in footprint. The aggregate score across a 20-benchmark suite is 83.9 against the original's 85.4, and on instruction following the compressed model scores 82.66 against 81.25. The license is Apache 2.0 and throughput reaches up to 143 tokens per second on an NVIDIA GeForce RTX 5090. ASAP separates where that 1.5-point loss actually landed.

Ternary weights cut the number of states, not just the bit width

The compression in Bonsai 2 27B restricts every weight to the three values -1, 0 and +1, then attaches FP16 group-wise scaling to reach 1.76 effective bits per weight. Three states require 1.585 bits in principle, and the gap to 1.76 comes from the FP16 scale stored per group. Converted to footprint, a 27.8-billion-parameter model occupies 5.9GB, more than a 9x reduction against full precision.

Ordinary quantization narrows the width of the representation. Dropping from 16 bits to 8 or 4 reduces how many values a weight can take while leaving those values on a continuous scale. Ternary is a different move. A weight now holds one of three states, on, off, or on in the opposite direction, and matrix multiplication moves closer to signed addition. That is where PrismML's speed and power claims originate.

The rest of the specification tracks the original. The context window is 262K tokens, input is multimodal across text and images, and PrismML states the model supports reasoning, coding, vision and agentic tool use. Training ran on Google v5 TPUs.

Instruction following is the one category where the compressed model wins

Across the seven categories PrismML published, Bonsai 2 27B beats Qwen3.8 27B on instruction following by 1.41 points at 82.66 against 81.25, and trails on the other six. The aggregate is 83.9 against 85.4, computed over a suite of 20 benchmarks.

The size of the gap is uneven by category. Vision lands at 78.59 against 81.64, a 3.05-point drop. Knowledge and reasoning lands at 83.95 against 86.66, a 2.71-point drop. Agentic and tool calling lands at 77.57 against 79.74, a 2.17-point drop. On the other side, coding falls only 0.59 points from 82.17 to 81.58, and math only 0.49 points from 97.06 to 96.57.

Founder and CEO Babak Hassibi told TechCrunch that compression will likely always have some impact. The same report states that the previous generation, Bonsai 1, matched 95% of aggregate benchmark scores, and that Bonsai 2 reaches 98%.

The hardware list is the model's real claim

PrismML reports up to 143 tokens per second on an NVIDIA GeForce RTX 5090 and 46.8 tokens per second on an Apple M5 Max, plus 0.714 milliwatt-hours per token on an RTX 4090, which it states is 40% more energy-efficient than a full-precision 8B model. Two execution paths are supported. NVIDIA GPUs run through CUDA, and Mac, iPhone and iPad run through MLX.

Distribution matches that list. The Bonsai 2 collection on Hugging Face carries a GGUF build, an MLX 2-bit build, and a demo space that runs the model in a browser on WebGPU. The license is Apache 2.0, and the model became a free download on the September 17, 2026 release date.

The company comes out of Caltech. PrismML was founded by Caltech researchers and is led by Babak Hassibi, a Caltech professor specializing in compression. Ion Stoica, Databricks co-founder and director of Berkeley's Sky Computing Lab, serves as an adviser. TechCrunch reports a $22.25 million seed round from Khosla Ventures, Cerberus Capital and Caltech, while PrismML's own release lists Khosla Ventures, Cerberus, Google and Samsung among past supporters.

Where the 1.5 points landed decides what the model is for

The 1.5-point drop in the aggregate is an average, and the actual loss concentrates in vision at 3.05 points, knowledge and reasoning at 2.71 points, and agentic tool calling at 2.17 points. Against that, math gives up 0.49 points and coding 0.59. Quoting 98.2% alone erases that distribution.

Read by use case, the distribution splits the decision. For on-device work with a fixed shape, such as code assistance or arithmetic, the price of compression sits near measurement noise. For reading images or answering questions that require broad recall, the gap to the original is more than three times wider. A team embedding a model in a laptop or phone should first determine which of those two groups its workload belongs to, not read the aggregate.

The instruction-following result, where the compressed model leads by 1.41 points, deserves separate handling. Benchmark scores do not reproduce deterministically, and 1.41 points in a single category can sit inside measurement variance. Reading it as evidence that compression improved the model goes too far. What it does establish is that compression loss is not spread evenly, varying in both direction and magnitude by category, which means evaluating a compressed model on a single aggregate discards information.

The axis shifts from building small models to folding large ones

The goal of Bonsai 2 27B is to put a 27.8-billion-parameter model directly onto laptops and phones, which is a different premise than training a smaller model from fewer parameters. On-device AI has mostly followed two routes. Either design and train small from the start, or distill a large teacher into a smaller student. Both produce a model that differs from the original, so evaluation becomes the problem of scoring a new model on its own terms.

Post-hoc compression changes the shape of the question. The comparison target is not another model but the full-precision version of the same model, and evaluation reduces from how good is it to how much was lost. That is why the phrase 98.2% works at all. Within this frame the room for improvement lies in pushing the loss rate toward zero rather than in new architectures, which is also what PrismML means when it describes moving from 95% to 98%.

The approach carries a structural dependency. Compression requires something to compress, and the original behind Bonsai 2 27B is Qwen3.8 27B. However good the compression gets, the ceiling on the result is set by whichever open-weight model exists at that moment. That ties a company like PrismML to the open-weight release cycle, and whether the next jump in on-device performance comes from the compression side or the base-model side depends on that structure.

What the energy figure means in practice, and what remains unverified

The 0.714 milliwatt-hours per token figure is a battery and thermal budget question rather than a performance comparison, and for device makers and app teams it lands before the scoreboard does. The real constraint when running a model on a phone or laptop is usually sustainable power and temperature rather than accuracy. The 46.8 tokens per second on an M5 Max is closer to an on-device planning number than the 143 tokens per second on a desktop GPU. Shipping an MLX build for Apple silicon and naming iPhone and iPad support points the same way.

Several items still stand unverified in this release. First, every published score is PrismML's own measurement and no third-party evaluation has been attached. Second, 98.2% is a figure over a suite of 20 benchmarks, and whether the same retention holds on tasks outside that suite is a separate question. Third, this material carries no numbers on whether the 262K context window stays stable through its full range after compression.

What a team evaluating this should do is simple. GGUF and MLX builds are public under Apache 2.0, so running the original and the compressed version side by side on your own task data and measuring the per-category gap is available today. How much ternary compression costs on a given job comes out of that measurement, not out of a single aggregate line.

Sources: Introducing Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint (PrismML, September 17, 2026) · PrismML hopes its tiny LLM could change how we all use AI (TechCrunch, September 17, 2026)