Running 753B GLM-5.2 on a single workstation GPU: FreeToken doubles llama.cpp's throughput

FreeToken serves the 753B-parameter GLM-5.2 on one RTX PRO 6000 at 14.9 tokens per second, exactly 2.0x llama.cpp's 7.3 on the same box, according to arXiv paper 2608.16157 released on August 17, 2026. "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution," by Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica and colleagues, also demonstrates a 35B model on an 8GB laptop GPU and a 284B model on a gaming desktop. The code ships under Apache 2.0 and the GitHub repository had collected 3,346 stars as of August 24. ASAP summarizes the result from the paper and the repository as primary sources.

Abandoning fixed offloading in favor of measured bandwidth

What separates FreeToken from existing edge inference engines is that it never decides in advance what belongs on the GPU. llama.cpp splits expert weights between GPU and CPU on a static, routing-blind boundary, and KTransformers fixes placement from the routing it observes during prefill. FreeToken treats a personal machine not as a small GPU but as one elastic inference platform spanning GPU, CPU, memory and interconnect, and continuously remaps computation and model state onto whatever resources remain available.

Three mechanisms carry that idea. In prefill, full-layer double buffering overlaps expert transfer with computation. Each 8,192-token prefill chunk completes in 1.19 to 1.22 seconds, which is exactly the time needed to stream the 64.4GB expert pool once at the measured 52.7GB/s of a PCIe 5.0 x16 link. Computation therefore hides entirely behind transfer, and prefill throughput climbs to 6,700 tokens per second at 16k tokens. Disabling the second buffer costs 19% of throughput at 4k tokens, 25% at 8k and 26% at 16k, the penalty growing with prompt length as the share of hidden computation rises.

In decode, the expert cache is a global LRU updated on every miss. The paper replays identical routing traces against all three engines' placement policies at equal capacity. At the RTX 5090 serving capacity, 37% of Qwen3.6's expert pool and 11% of DeepSeek-V4-Flash's, FreeToken misses 16% and 39% of decode-time expert reads. KTransformers misses 41% and 59% at the same capacity, and llama.cpp misses 62% and 89%. The third mechanism, elastic memory management, reallocates VRAM at runtime, because on a personal machine a browser or a game can claim the same GPU at any moment.

35B on an 8GB laptop, 753B on a 96GB workstation

Measurements span six systems: three rented servers with an RTX 3090, 4090 and 5090, one RTX 5090 desktop, one RTX 4060 laptop with 8GB, and the RTX PRO 6000 with 96GB that hosts the frontier-scale demonstration. The rented servers carry CPUs far beyond any edge host, so every run on them is capped at 6 CPU threads and pinned to the GPU's NUMA node. Capped that way the servers deliver 56.7 to 77.3GB/s of host bandwidth, the same scale the real edge machines reach at full threads, 53.8GB/s on the desktop's 16 cores and 47.5GB/s on the laptop's 14. Every bandwidth figure is measured on deployed tensor shapes rather than taken from platform specifications.

On the RTX 5090, FreeToken sustains 77 to 83 tokens per second on Qwen3.6-35B-A3B and 22 to 25 on DeepSeek-V4-Flash, a 284B model with 13B active. That is 1.8x to 2.3x and 1.5x to 1.9x the strongest baseline in each workload. Repeating the coding-agent workload across the five consumer systems, FreeToken leads by 1.3x on the RTX 3090 and 4090, 1.9x on the 5090 server, 2.1x on the 5090 desktop and 1.8x on the RTX 4060 laptop, where the NVFP4 build sustains 39.3 tokens per second on an 8GB, PCIe x8 machine, or 92% of the RTX 4090 rate.

The sharpest contrast appears between the two machines that share the same GPU silicon. Moving from the many-channel server to a dual-channel consumer desktop costs FreeToken 4% of its decode rate, while llama.cpp keeps only 80% of its own as its CPU-resident experts starve on two DDR5 channels. At the frontier tier, FreeToken serves GLM-5.2, 753B with 40B active and a 433GB NVFP4 checkpoint, on the single RTX PRO 6000 at 14.9 tokens per second against llama.cpp's 7.3, with bit-identical expert weights and comparable mean time to first token at 7.5 versus 7.8 seconds. KTransformers cannot serve that combination at all: its GLM-5.2 methods require 753GB to 1.5TB of host-resident experts against the box's 512GiB, and its CPU kernels do not read the NVFP4 layout.

The real result is the 44-second tail, not the average

The most operationally important number in the paper is not throughput but the tail of time to first token. FreeToken's worst turn stays below 44 seconds in every cell. Each baseline crosses 150 seconds somewhere: llama.cpp at 232 seconds, Ollama at 179, KTransformers at 946. Averages erase that distinction. On the AIME workload, with its short isolated prompts, llama.cpp actually posts the better mean.

The distinction matters because agent clients enforce per-request timeouts, not averages. The paper cites two thresholds: OpenClaw ships a 120-second idle watchdog, and Claude Code's default request timeout is roughly ten minutes. A 946-second tail is not a slow response, it is a failed request, and the session drops regardless of how many tokens per second the engine would eventually have produced. Given that most local inference benchmarks report only mean latency and steady-state throughput, redefining the tail as an availability boundary is itself a change in evaluation standard.

A second observation cuts the same way. FreeToken stays within 12% of its single-turn rate across all three agent workloads, while KTransformers on DeepSeek-V4-Flash, the most context-sensitive baseline, has already lost 31% of its first-workload rate by the second. MoE-Infinity serves only the first workload, at 8.8 tokens per second. The conclusion follows directly: single-stream benchmarks overstate baseline agentic performance, which means a good share of the local-LLM speed tables in circulation rest on that trap.

The cost of owning a frontier model gets recalculated

What FreeToken changes is not speed but the boundary of which models are candidates on which hardware. Running a 753B open-weight model has so far presumed a multi-GPU node, so the licensing freedom of open weights was in practice a right only organizations with datacenter budgets could exercise. At 14.9 tokens per second a single workstation is slow for conversational use, but it sits comfortably in the usable range for batch classification, document processing and overnight pipelines.

The organizations where this lands hardest are those that cannot move data off premises. Teams in healthcare, finance and the public sector, for whom external API calls are structurally difficult, have been trading away substantial capability for small local models or deferring adoption entirely. A single workstation plus an open-weight checkpoint changes the premise of that calculation. The condition attached is real, though: the GLM-5.2 checkpoint alone is 433GB, and the RTX PRO 6000 box in the paper carries 512GiB of host memory. Total cost of ownership has to be recomputed including storage and RAM, not read off a GPU price list.

Worth noting too is that the contribution here is neither a new model nor a new quantization scheme but a co-design of the entire serving stack. Recent discussion around NVIDIA's results argued that the harness now matters more than the model; FreeToken repeats that argument one layer down, in inference infrastructure. When identical weights at identical bit precision on identical hardware differ by a factor of two, the remaining performance was living in placement policy and memory management, not in the model. That is one of the places competition in the open-weight ecosystem is likely to move next.

What to check before taking the numbers at face value

The stated runtime constraints are narrow: FreeToken targets Linux x86_64 with an NVIDIA GPU and driver r580 or newer under CUDA 13. Mac and Windows users, and anyone on AMD or Apple silicon, are outside the current scope. Three of the six measured systems are servers capped to 6 CPU threads to emulate edge conditions rather than actual edge hardware. The paper justifies the emulation by showing it lands in the same measured bandwidth range as the real desktop and laptop, but an emulation is still an emulation.

The nature of the measurements deserves scrutiny as well. Reported figures are decode throughput under single-user interactive serving; batched multi-request serving is outside this experiment. KTransformers is absent from the GLM-5.2 comparison not because it lost but because it had no servable path, so the frontier-tier 2.0x is effectively a comparison against llama.cpp alone. The laptop's 39.3 tokens per second comes from an NVFP4 build rather than BF16, giving it a different precision from other figures in the same column.

One question the results leave open: 14.9 tokens per second is tolerable at human reading speed but still slow for a coding agent that makes many tool calls. It is worth noting that the paper evaluates agentic workloads yet reports throughput and latency rather than task completion or accuracy. Whether running frontier models at the edge is actually useful will be settled not by a speed table but by what those models finish locally.

Source: ASAP summary based on Shuo Yang et al., "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution" (arXiv:2608.16157, August 17, 2026) and the GitHub repository FlashML-org/FreeToken.