Every few months a new technique shows up in model release notes that sounds like science fiction and matters enormously for your API bill. Multi-token prediction (MTP) is one of those. Google, Meta, and DeepSeek have all shipped models that use it. Here's what it actually is — without the math degree.
The one-word-at-a-time problem
Large language models generate text the way a typewriter writes: one character at a time, in strict sequence. Technically, a standard LLM predicts one token (roughly a word fragment) per forward pass of the neural network. Generate a 500-token paragraph and the model has run 500 full passes through a neural network that might contain hundreds of billions of parameters.
This is called autoregressive generation, and it's the single biggest reason large models feel slow. Each token depends on all the tokens before it, so the work fundamentally can't be parallelized — the model literally cannot know token 42 until token 41 exists.
Multi-token prediction: writing ahead
Multi-token prediction flips this. Instead of forcing the model to commit to one token per pass, the model is trained to predict several future tokens at once — say, the next 2, 3, or 4 tokens simultaneously from a single forward pass.
Think of it like an experienced typist. A beginner looks at the screen after every keystroke. A touch typist's hands are already two or three words ahead of what's appeared on screen. MTP gives the model that same "hands ahead" ability.
In practice it works like this:
- The model's main network (the "backbone") does one forward pass.
- Attached lightweight prediction heads each guess one of the next few tokens.
- The highest-confidence token sequence gets accepted and appended to the output.
- Repeat — but now you're jumping 2–4 tokens per pass instead of 1.
The result: generation speedups of 2–4x with little to no quality loss, because the extra prediction heads are tiny compared to the backbone network.
Not the same as speculative decoding
You may have heard of speculative decoding — a different speed trick where a small "draft" model guesses several tokens and the big model checks them in one pass. The two approaches solve the same problem from opposite directions:
- Speculative decoding: two separate models (one small, one big) cooperating. No retraining of the big model needed.
- Multi-token prediction: one model, trained from scratch with extra output heads baked in. Needs training-time commitment, but needs only one model deployed.
DeepSeek's V3 and V4 families famously used MTP modules during training — and, cleverly, you can reuse the same technique at inference time for the speed boost.
Why it matters for your costs
Speed isn't just about user experience — it's about throughput and price. Faster token generation means:
- More users per GPU. If a serving stack generates 3x more tokens per second, providers fit more traffic on the same hardware.
- Cheaper per-token prices. Those efficiency gains get competed away into lower prices — which is a big reason output tokens from newer models often cost less than older, smaller models did two years ago.
- Lower latency for agents. Agent workloads generate thousands of tokens per task; halving generation time halves wall-clock time on entire pipelines.
This is part of a broader pattern worth internalizing: raw model quality is improving slowly and expensively, but inference efficiency — MTP, speculative decoding, KV cache compression, quantization — is improving fast and continuously. When you compare models on Qubax, you're often seeing these efficiency gains competed into the price, sometimes 5–20x below official retail list prices.
The trade-offs
MTP isn't free magic:
- Training complexity. The model must be trained with the extra heads from the start (or expensively retrofitted).
- Acceptance rate matters. When the model's "guess ahead" tokens are wrong, they get discarded, and you fall back to standard speed. Performance gains depend on how often the guesses are right — which is why highly unpredictable text (creative writing, code with many valid paths) sees smaller gains than predictable text (boilerplate, structured output).
- Serving stack support. You need an inference engine that supports MTP heads; not every deployment does.
The takeaway
Multi-token prediction is a reminder that the AI systems you use daily are the product of dozens of stacked engineering tricks, not just "a bigger brain." The next time a model feels noticeably faster and cheaper than its predecessor with similar benchmark scores, there's a good chance tricks like MTP, FP4 KV caches, and speculative decoding are doing quiet, heavy lifting behind the scenes.
Want to see what those efficiency gains look like on your bill? Browse live model prices on Qubax — the open market where compute providers compete on price.
FAQ
What is multi-token prediction in simple terms?
It's a training technique where an LLM learns to predict several future tokens at once instead of just the next one, letting it generate 2–4 tokens per pass through the network and dramatically speeding up text generation.
How is multi-token prediction different from speculative decoding?
Speculative decoding uses a separate small "draft" model to guess ahead, with the large model verifying. Multi-token prediction bakes the prediction ability into one model using extra output heads during training. Both reduce the number of expensive forward passes.
Does multi-token prediction reduce output quality?
When implemented well, quality is essentially unchanged — the main network is identical, and the extra heads only speed up generation. Predictable text sees bigger speedups than highly unpredictable text, since wrong guesses are discarded.
Which models use multi-token prediction?
DeepSeek's V3/V4 families are the best-known adopters, and variants of the technique appear in several other frontier and open-weight models. Check individual model cards for serving-stack support.
How do I get the fastest, cheapest generation for my app?
Compare providers on an open marketplace. On Qubax, competing inference providers mean efficiency gains are passed to you as lower prices — often well below official retail rates.
