Contents
Yesterday Nvidia announced that Groq 3 LPX, the first chip to ship from its $20 billion acquisition of Groq’s LPU technology, has entered full production. Nebius signed on as the first cloud to deploy it, and SpaceX confirmed it is pairing Nvidia’s new Vera CPUs with the same racks for its own agentic workloads. The headline number is 3,400 tokens per second running Google’s Gemma 4 31B at a 100,000 token input context, a benchmark Nvidia says is roughly 4x the nearest publicly available alternative.
The reason this is worth your attention even if you never touch a rack is what it signals about where the money is moving. For two years the AI hardware conversation was almost entirely about training, bigger clusters, more FLOPs, longer runs. Inference, the part that actually serves your product’s traffic, was treated as a byproduct you’d figure out with whatever GPUs you already owned. Nvidia spending $20B to buy a company whose entire premise was that inference deserves its own chip architecture is the clearest signal yet that assumption is over.
Why a separate chip for inference at all
A GPU is a general-purpose parallel processor, good at training because training is throughput-bound: you want to chew through as many examples as possible, and latency on any single one barely matters. Inference, especially the interactive kind your users are waiting on, is latency-bound and memory-bandwidth-bound in a different way. Groq’s LPU (Language Processing Unit) architecture was built from the start around deterministic, single-pass token generation, trading the flexibility of a GPU for a data path tuned specifically to how transformer decoding actually moves data. That specialization is what produces a 4x gap on the same class of model rather than an incremental improvement.
The 100K context number is the real story
Most published inference benchmarks quote throughput at short context, where key-value cache pressure is minimal and the numbers flatter every architecture. The Groq 3 LPX figure was measured at 100,000 input tokens, the range where memory bandwidth to the KV cache typically becomes the bottleneck and throughput on conventional GPU inference collapses. Holding 3,400 tokens/sec at that context length is the part hardware engineers are actually arguing about this week, not the peak number itself. If it holds up under independent load outside Nvidia’s own benchmark, it changes the calculus for anyone running long-context agents, RAG pipelines, or coding assistants that regularly push six-figure token windows.
What this changes for you, concretely
You are not procuring racks, but you are choosing between inference providers, and those providers are about to start advertising LPX-backed endpoints the way they advertise GPU generation today. The number that should change your evaluation is not raw tokens/sec, it is tokens/sec at the context length your workload actually runs at. A provider quoting speed at 4K context tells you almost nothing about how your 80K-token agent trace will perform.
Try 800 tokens against the GPU-cluster profile versus the LPX profile. The gap you see is the actual user-facing difference between “the response streams in as you read” and “there is a pause before anything happens,” at the context lengths agentic workloads now run at routinely.
Where I land on this
Specialized inference silicon stops being a curiosity the moment a $20B acquisition ships hardware that clears independent scrutiny, and Nvidia choosing to fold Groq’s architecture into its own roadmap rather than compete against it is the tell. I would not switch providers on a vendor benchmark alone, Artificial Analysis and others will run their own numbers over the next few weeks and that is the numbers to wait for. But if your product does anything with long-context agents or RAG over large documents, it is worth asking your inference provider today what their tokens/sec looks like at the context length you actually use, not the one on their landing page.
Source: The Register’s coverage of the Groq 3 LPX benchmark and Nvidia’s announcement.