What Is Speculative Decoding? The Trick That Makes LLMs 2-3x Faster (Simple Explanation)

If you've ever wondered why some AI APIs respond dramatically faster than others for the same model — or how providers serve flagship models at a fraction of the usual latency — one of the biggest secrets isn't a bigger GPU. It's an inference trick called speculative decoding.

Speculative decoding is one of those techniques that sounds like magic when you first hear it: a small model writes guesses, the big model checks them in one pass, and you get the big model's quality at a fraction of the time. No quality is lost. Nothing is approximated. The output is mathematically identical to what the big model would have produced on its own. Here's how it actually works, in plain language.

First, Why Is LLM Generation Slow?

Large language models generate text one token at a time — roughly one word or word-piece per step. For each token, the model has to run its entire neural network: every transformer layer, every parameter.

Here's the painful part: generating 500 tokens means running the full model 500 times, and each run is dominated not by computation but by memory bandwidth — the time it takes to shuttle the model's billions of parameters from GPU memory to the processors. The GPU does a tiny amount of arithmetic per parameter, then spends most of its time just reading the model. It's like driving a semi-truck across town to deliver a single envelope.

Engineers call this the memory-bandwidth bottleneck, and it's why a model with 10x more parameters doesn't generate text 10x slower — but also why raw compute often sits idle during generation. Speculative decoding attacks exactly this waste.

The Core Idea: Draft, Then Verify

Speculative decoding pairs the big model (the target) with a much smaller, faster model (the draft) that has been trained on similar data. Then generation works like this:

1. The draft model guesses ahead. The small model rapidly generates, say, 4–8 candidate tokens — cheaply, because it's tiny. This costs almost nothing.
2. The target model checks them all in one pass. Here's the key insight: processing 8 tokens in a single forward pass through the big model costs barely more than processing 1 token, because the bottleneck is reading the model's weights, not the sequence length. So the big model evaluates all 8 draft tokens simultaneously.
3. Accept or reject. Each draft token is compared against what the big model would have chosen. Tokens the big model agrees with are accepted — kept. At the first token the big model disagrees with, everything after it is thrown away, and the big model's own choice for that position is used instead.
4. Repeat. The accepted tokens become part of the context, and the loop runs again.

The result: if the draft model's guesses are right (say, 3–4 out of 4 on easy text), you've generated 3–4 tokens for the price of one big-model pass. On a 500-token response, that's a 2–3x speedup — sometimes more.

"But What If the Draft Is Wrong?" — The Guarantee

This is the part people find hardest to believe, so let's be precise.

A rejected draft token costs you only the small model's wasted effort — microseconds. And critically, the acceptance rule is designed so that the final text distribution is mathematically identical to sampling from the big model alone. Not "approximately the same quality." Identical. Speculative decoding is a lossless acceleration.

The draft model doesn't need to be great — it just needs to be right often enough to be worth consulting. And language is highly predictable most of the time: common phrases, code syntax, boilerplate, closing sentences. The draft nails those constantly. Hard, surprising tokens (a rare name, a key API argument) get rejected, and you pay only the normal price for them.

An analogy: it's like having a junior assistant pre-write sections of an email that a senior editor then reviews. Where the junior is right, the editor stamps it through instantly. Where the junior is wrong, the editor rewrites that one line. The final email is 100% the editor's work — it just took a fraction of the editor's time.

Variations You'll See in the Wild

The basic scheme has spawned a family of techniques, and if you read model cards or provider docs you'll see these names:

• Draft-model speculative decoding — the classic version above (e.g., a 1B-parameter draft for a 70B+ target).
• Self-speculative / Medusa heads — the model carries extra lightweight "heads" that predict several future tokens at once, skipping a separate draft model entirely. Popular for models where no good small sibling exists.
• EAGLE and lookahead decoding — cleverer schemes that reuse the target model's own internal states to draft, pushing acceptance rates higher.
• N-gram / prompt-lookup speculation — for tasks like editing or code refactoring, where much of the output copies the input, the system drafts by literally looking up repeated phrases from the prompt. Near-free speedups for those workloads.

Why Developers Should Care

Speculative decoding isn't academic trivia — it's why the same model can feel completely different across providers:

• Latency without quality trade-offs. Providers that use it serve identical model outputs in 2–3x less time. When you compare models on Qubax, the same model served by different compute providers can have visibly different speeds — inference optimization is one reason.
• Cost can drop too. If a provider prices by time or passes efficiency savings on, faster generation means cheaper tokens. This is one driver behind the aggressive per-token prices on open marketplaces, where compute providers compete on both price and speed.
• It compounds with batching. Serving many users at once means batching requests; speculative decoding shortens each request, so more requests fit through the same hardware.

A Quick Mental Checklist

When evaluating an API provider or self-hosting a model, ask:

• Does the serving stack support speculative decoding (vLLM, SGLang, and TensorRT-LLM all do)?
• Is there a matched draft model for the target model?
• Is the workload predictable? Code editing and templated generation benefit most; highly creative text benefits least (more rejections).
• Is speed a hard requirement? If you need low latency for a chat UI or voice agent, speculative decoding is often the single biggest lever after choosing a smaller model.

The Bottom Line

Speculative decoding is the rare free lunch in systems engineering: a small model's cheap guesses plus a big model's single verification pass yield the exact same text, substantially faster. It's one of the main reasons today's API prices and latencies look so much better than two years ago — the hardware didn't get magically better; the scheduling did.

If you're choosing where to run your workloads, compare speed and price across providers on Qubax's model marketplace, or check the API docs to route requests to the fastest (and cheapest) option for each task.

FAQ

Does speculative decoding change the model's output?

No. The acceptance rule is constructed so the output distribution is mathematically identical to sampling from the large model alone. Quality is unchanged; only speed improves.

Why not just use the small model for everything?

Because quality drops. The small model is only good enough to guess frequently predictable tokens — it would make noticeably more errors if left in charge. The big model remains the final arbiter of every token.

How much faster is it in practice?

Typically 2–3x for chat-style generation, sometimes higher for code editing or tasks where output resembles the input. Gains shrink for highly unpredictable text, where the draft is rejected often.

Do I need to configure anything to benefit from it?

If you're using an API, no — it's a server-side optimization. If you self-host, use an inference framework like vLLM or SGLang and supply (or enable) a draft model or Medusa/EAGLE heads for your target model.