What Is Speculative Decoding? A Simple Explanation of How AI Models Generate Text Faster

If you've ever wondered why some AI models seem to generate responses almost instantly while others make you wait, you're not alone. The answer often comes down to a technique called speculative decoding — one of the most important but least understood innovations in modern large language model inference.

In this article, we'll break down speculative decoding in plain English, explain why it works, and show you why it's becoming the default way that API providers serve frontier models like GPT-5.6, Claude Opus 4.8, and DeepSeek V4.

The Problem: LLMs Generate One Token at a Time

To understand speculative decoding, you first need to understand the bottleneck it solves.

Large language models generate text autoregressively — one token at a time. To produce the next token, the model must process the entire input context plus everything it has generated so far, then predict the next word. This is inherently sequential: you can't generate token N+1 until you've generated token N.

On modern hardware (GPUs and TPUs), the actual computation for a single token is fast — often just a few milliseconds. But the memory bandwidth bottleneck is the real killer. For each token, the model must load all its weights from GPU memory into compute units. For a 400-billion-parameter model, that means reading hundreds of gigabytes of data from memory for every single token.

This is called being memory-bound: the GPU's compute units sit idle most of the time, waiting for data to arrive from memory. It's like having a Formula 1 engine but a fuel pump that can only deliver one drop at a time.

The Insight: Most Tokens Are Predictable

Here's the key observation that makes speculative decoding possible: most tokens in a model's output are easy to predict.

Think about the sentence: "The capital of France is ___." The next token is almost certainly "Paris." A small, fast model could guess that. A large model would also say "Paris," but it would take much longer to confirm because it's loading all its weights to compute one obvious token.

Speculative decoding exploits this asymmetry. Instead of using the large model to generate every token, it uses a small, fast "draft" model to guess several tokens ahead, then asks the large model to verify them all at once.

How Speculative Decoding Works: Step by Step

Here's the process in detail:

Step 1: The Draft Model Generates

A small, fast model (the "draft model" or "speculator") generates a sequence of candidate tokens — typically 4 to 8 tokens at a time. This is fast because the draft model is much smaller (e.g., a 1B-parameter model drafting for a 400B-parameter model).

Step 2: The Large Model Verifies

The large model (the "target model") processes all the draft tokens in a single forward pass. Because the large model is already loading its weights from memory for the first token, verifying additional tokens is nearly free — the compute cost of processing 8 tokens is barely more than processing 1, because the bottleneck is memory bandwidth, not compute.

Step 3: Accept or Reject

The large model checks each draft token against what it would have predicted:

• If the draft token matches what the large model would have produced, it's accepted.
• If it doesn't match, that token and everything after it is rejected, and the large model generates the correct token from that point.

This is the magic: the acceptance check is free because the large model was going to do a forward pass anyway. The only cost is the extra compute for the additional tokens, which is negligible compared to the memory cost.

Step 4: Repeat

The process repeats: the draft model generates another batch of speculative tokens, the large model verifies, and so on.

Why It Works: The Math

The speedup depends on the acceptance rate — the fraction of draft tokens that the large model accepts. If the draft model is good (often a smaller version of the same model family), acceptance rates can be 70–90%.

• If the draft model guesses 8 tokens and 6 are accepted on average, you've generated 6 tokens in the time it normally takes to generate 1–2. That's a 3–6× speedup.
• If all 8 are accepted (which happens for highly predictable text), the speedup approaches 8×.

Even with a modest acceptance rate of 50%, you typically see 2–3× speedups — significant for high-volume API workloads.

The Different Flavors of Speculative Decoding

Speculative decoding has evolved beyond the original draft-model approach. Several variants are now in production:

Vanilla Speculative Decoding

The classic approach: a small model drafts, a large model verifies. Simple and effective. This is what most people mean when they say "speculative decoding."

Self-Speculative Decoding

Instead of using a separate draft model, the large model uses its own earlier layers as a draft. The model skips some middle layers to generate a quick prediction, then runs the full network to verify. No separate model needed. This is how some providers achieve speedups without maintaining a second model.

Medusa Heads

The model is augmented with extra "heads" that predict multiple future tokens simultaneously from a single forward pass. Each head predicts a token at a different position ahead. This avoids the need for a separate draft model entirely.

EAGLE and EAGLE-2

More advanced draft models that use the hidden states (not just the output tokens) of the target model to make better predictions. These achieve higher acceptance rates and larger speedups than vanilla speculative decoding.

N-gram Speculation

The simplest form: use an n-gram model (essentially a lookup table of common token sequences) as the drafter. No neural network needed. Works surprisingly well for repetitive or formulaic text like code generation.

Why Speculative Decoding Matters for API Users

If you're calling AI APIs, speculative decoding affects you even if you never think about it:

Lower latency. Faster token generation means your applications feel more responsive. Streaming chatbots update more quickly, and batch jobs finish sooner.

Lower costs. When providers can serve models more efficiently, they can offer lower prices. The compute savings from speculative decoding are one reason API prices have been dropping across the industry. You can see current pricing for hundreds of models on Qubax AI's model catalog.

Better throughput. Higher throughput means providers can handle more concurrent requests, reducing queue times and rate limits during peak hours.

Enables longer outputs. When generation is faster, it becomes practical to generate longer responses — detailed analyses, full code files, comprehensive explanations — without timing out.

The Trade-offs

Speculative decoding isn't free. The main costs:

Draft model overhead. Running a separate draft model requires additional memory and compute. On memory-constrained hardware, this can reduce the batch size you can serve.

Diminishing returns on hard tokens. For highly unpredictable text (creative writing, rare technical content), acceptance rates drop, and the overhead of generating rejected tokens can actually slow things down.

Implementation complexity. The verification logic, especially handling the rejection point and resuming from the correct token, is non-trivial. Most inference frameworks (vLLM, TensorRT-LLM, SGLang) now support it natively, but tuning it for optimal performance requires effort.

Quality is preserved. This is the key guarantee: speculative decoding produces exactly the same output as standard autoregressive generation. The large model still verifies every token. There's no quality degradation — only speedup or slowdown.

Which Models Use Speculative Decoding?

As of 2026, speculative decoding is widespread in production API serving:

• OpenAI uses it across the GPT-5.x family, contributing to the fast response times on GPT-5.6 Luna and GPT-5.6 Sol
• Anthropic employs speculative decoding for Claude Opus 4.8 Fast and Claude Opus 5 (Fast) variants
• DeepSeek uses it in V4 Flash and V4 Pro, helping achieve their aggressive price points
• Google applies it to Gemini 3.1 Flash and Gemini 3.5 Flash
• Meta uses it for Llama 3.3 70B serving
• Most open-source inference frameworks (vLLM, SGLang, TensorRT-LLM) support it out of the box

When you see a "Fast" or "Flash" variant of a model, speculative decoding is often part of the story.

A Simple Analogy

Think of speculative decoding like a senior developer reviewing a junior developer's code:

1. The junior developer (draft model) writes a chunk of code quickly.
2. The senior developer (target model) reviews it all at once.
3. The parts that are correct get approved instantly.
4. The first mistake is caught, and the senior developer writes the correction.
5. The process repeats.

The senior developer could have written everything from scratch, but reviewing is faster than writing — especially when the junior developer is right most of the time. The result is the same quality of code, produced much faster.

The Future of Speculative Decoding

The technique continues to evolve. Current research focuses on:

• Adaptive draft lengths — dynamically adjusting how many tokens to speculate based on the current acceptance rate
• Better draft models — using reinforcement learning to train draft models specifically for high acceptance rates
• Multi-model speculation — using multiple draft models with different strengths
• Hardware co-design — building GPUs and accelerators that natively support speculative verification

As models grow larger and inference becomes the dominant cost of AI, speculative decoding will only become more important. It's one of the few techniques that gives you a free lunch: same quality, faster speed, lower cost.

Getting Started

If you're building applications with AI APIs, you don't need to implement speculative decoding yourself — your provider handles it. What you should do is:

1. Choose models that use it. Look for "Fast" or "Flash" variants in the Qubax model catalog — these typically use speculative decoding.
2. Benchmark latency yourself. Use streaming to measure time-to-first-token and tokens-per-second for your specific workload.
3. Consider cost per token. Faster models often have lower prices, and speculative decoding is a big reason why. Check the Qubax documentation for API integration guides.

Speculative decoding is one of those behind-the-scenes innovations that makes modern AI feel fast and affordable. The next time your AI assistant responds almost instantly, there's a good chance a small model guessed most of the answer — and a big model confirmed it.

FAQ

What is speculative decoding in simple terms?

Speculative decoding is a technique where a small, fast AI model guesses several upcoming tokens, and a large model verifies them all at once. If the guesses are correct, you save time. If they're wrong, the large model corrects them. The result is identical output but faster generation.

Does speculative decoding change the quality of the output?

No. The large model still verifies every token, so the output is exactly the same as if the large model generated it token by token. There is zero quality degradation — only a speed improvement (or occasionally a slowdown for highly unpredictable text).

How much faster is speculative decoding?

Typically 2–6× faster, depending on the acceptance rate of the draft model. For highly predictable text (like code or formulaic responses), speedups can approach 8×. For unpredictable text, the benefit is smaller.

Do I need to change my API calls to use speculative decoding?

Usually not. Most API providers handle speculative decoding transparently on their end. You just see faster responses and lower prices. Some providers offer explicit "Fast" or "Flash" model variants that use it.

What's the difference between speculative decoding and prompt caching?

Speculative decoding speeds up output generation by guessing future tokens. Prompt caching speeds up input processing by storing the computed representations of repeated input prefixes. They're complementary — many providers use both together for maximum speedup.

Which models support speculative decoding?

Most frontier models served via APIs in 2026 use some form of speculative decoding, including GPT-5.x Fast variants, Claude Opus Fast variants, Gemini Flash variants, and DeepSeek V4 Flash. Check the Qubax model catalog for current options.