Speculative Decoding (Multi-Token Decoding)

Speculative decoding makes generation faster without changing the output. A lightweight proposer drafts several tokens ahead, and the model verifies them in a single batched forward pass — so a step that would normally produce one token can commit several.

For greedy decoding (temperature: 0) every committed token is still the model's own top choice, so you get the same answer, sooner. The text usually matches normal generation character for character; it can differ where two candidate tokens are nearly tied (see Limitations).

Speculative decoding is OFF by default

Every request runs plain autoregressive decoding unless you ask for a proposer in the speculative field. This is intentional:

• The speedup is workload-dependent. Grounded prompts (RAG, summarization, code refactor) get 1.5–2.5×. Free-form prompts (creative writing, open-ended chat) get ≈ 1× — same speed as plain AR, but with a tiny per-round overhead.
• On some model/device combinations the proposer's multi-token verify forward is a measured regression vs plain AR (small-m quantized GEMM on Metal — Bonsai 1-bit, certain Q4_K_M targets). Auto-enabling there would silently halve throughput.
• You almost always know your workload better than the server does. An RAG product enables it once at integration time; a creative- writing demo leaves it off; an A/B-test toggles it per request.

If you're not sure whether your workload benefits, the which proposer should I use? matrix below has a quick decision tree.

Enabling it

Add a speculative field to your chat completion request:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Summarize the text above."}],
    "speculative": { "proposer": "prompt_lookup", "k": 5 }
  }'

Field	Meaning
proposer	Which proposer to use — prompt_lookup (alias pld), lookahead, or mtp (alias nextn, see Self-speculation (MTP)). Case-insensitive.
k	Draft length: tokens drafted per verify round. Optional; default 5, or 2 for mtp.

Omit speculative (or pass null) and generation runs normally. An unknown proposer name falls back to normal decoding (you lose the speedup, not the request) — "none" is the conventional explicit opt-out.

Why turn it on?

For the right workload, the speedup is significant and free:

Workload	Proposer	Measured speedup (Metal, M1 Max)
RAG / Q&A over documents	prompt_lookup	1.5–2.8× AR
Summarization with entity preservation	prompt_lookup	1.5–2.5× AR
Code refactoring / editing	prompt_lookup	2–3× AR
Multi-turn chat with long history	prompt_lookup	1.3–2× AR
Translation with proper-noun preservation	prompt_lookup	1.2–1.5× AR
Llama-3.1-8B on a grounded code prompt (measured)	prompt_lookup	2.10× AR (92.9 % accept)

For other workloads it's a wash:

Workload	Speedup
Creative writing / story generation	~ 1.0× (no n-gram overlap to find)
Open-ended chat / brainstorming	~ 1.0×
Math reasoning that re-derives	~ 1.0×
Short prompt → long generation	~ 1.0× (tiny haystack)

Greedy output is the model's own choice either way (see Limitations for near-ties). The only penalty for enabling it on a workload where it doesn't help is the proposer's own per-round overhead — a few microseconds for prompt_lookup (it's a CPU n-gram lookup), more substantial for draft-model / EAGLE-3 / DFlash proposers (which is why we don't expose those as default options today).

Which proposer should I use?

There are two general-purpose proposers exposed through the API:

prompt_lookup (often abbreviated PLD)

The recommended choice for almost every speedup use case.

• How it works. No extra model. No GPU draft cost. For every generation step, the proposer searches the prompt + already- generated tokens for a recent repeat of the last 2–3 tokens, and drafts the K tokens that follow that match site. When it doesn't find a match, it returns nothing and the verifier falls back to AR for that round — so the cost on a workload where it doesn't help is essentially zero.
• When to use it. Any input-grounded task: RAG / Q&A, summary, code editing, translation, multi-turn chat with long history, agent-style traces that quote earlier turns.
• When NOT to use it. Free-form creative output where the model isn't echoing the prompt — the n-gram lookup almost always fails, costing a few microseconds per round for no benefit. (Still safe; just pointless.)
• Tuning. k = 5 is a good default. Higher k makes each speculative round commit more tokens on a hit, but every miss becomes more wasteful. The PLD paper sweeps n = 2–3 and reports n = 3 as the sweet spot; we use both with longest-first fallback.

lookahead

Pick this if prompt_lookup doesn't help your workload but you still want a general-purpose speedup.

• How it works. Training-free n-gram parallel decoding. Predicts multiple positions in parallel using an n-gram table built from prior decoding, rather than from the prompt.
• When to use it. Open-ended generation that doesn't echo its prompt but does have local structure / repetition (e.g. bullet-pointed lists, generated tables, structured code without a template).
• Expected speedup. ~ 1.3× AR — smaller than PLD on grounded tasks but more general.

When NOT to enable speculative decoding

Some model/device combinations have a correctness or performance veto independently of which proposer you pick. The server enforces these gates automatically — if a gate fails, the request transparently falls back to AR with a log line, you don't have to handle errors.

Case	What happens	Why
Recurrent / hybrid models (Qwen3.5, Qwen3.6, Nemotron-H)	Server falls back to AR, except quantized Qwen 3.5-architecture models (qwen35 GGUF) on CUDA, which speculate	DeltaNet / Mamba recurrent state advances through drafts even when they get rejected, so a rejected draft must be rolled back. Quantized Qwen 3.5 on CUDA records a rollback slot per drafted token during verify and rewinds exactly. The other recurrent paths have no rollback and keep plain AR: Metal, CPU, float Qwen 3.5, Qwen 3.6 (qwen35moe) and Nemotron-H.
Structured generation (response_format) + model without batched verify	Server falls back to AR	The verify path has to enforce the grammar; per-token verify can't.

You can still send speculative on those targets — it just won't take effect, and the server will log why. There is no error and no behavior change vs the plain AR baseline.

For the user-controlled "should I bother" question, the matrix at the top of Which proposer should I use is the answer. The two short rules:

1. Don't enable it for plain creative-writing or open-ended chat. You're paying microseconds per round for nothing. Not a regression, not a win.
2. Don't enable it on Bonsai 1-bit on Metal. Measured 0.43× AR regression — the small-m quantized verify forward is more expensive than running AR. (On CUDA Bonsai is fine; on Metal small-m quantized GEMM doesn't amortize.)

Everything else is either a win (grounded workloads) or a no-op (non-grounded workloads, where the proposer harmlessly fails to find matches).

Self-speculation (MTP) on quantized Qwen

Some Qwen 3.5 and Qwen 3.6 checkpoints ship an appended MTP (Multi-Token Prediction) block — a nextn head that drafts the next token from the model's own hidden state, so no separate draft model is needed. The engine now loads that head for the quantized hybrid Qwen targets:

• QQwen3_5MtpHead — quantized Qwen 3.5 (qwen35 GGUF, dense SwiGLU MTP block).
• QQwen3_6MtpHead — quantized Qwen 3.6 (qwen35moe GGUF, full-attn + MoE MTP block).

Detection is automatic: the loader reads the {arch}.nextn_predict_layers GGUF metadata and loads the last block (blk.{num_hidden_layers}, which carries the nextn.* fusion tensors) as the MTP head. If the GGUF dropped its nextn tensors at quantization time — as the Qwen3.5-{0.8B,2B,4B,9B}-Q4_K_M builds in the asset store did — the model loads trunk-only and behaves exactly as before.

Using MTP

Request it with "proposer": "mtp". The draft length defaults to 2. The head is trained for one-step prediction, so acceptance falls with depth; on Qwen3.8-27B-Q4_K_S (one RTX 3090, 300 tokens greedy) K=2 won or tied on every prompt:

K	python	prose	shell
1	71.2	62.6	66.1
2	81.2	64.9	70.3
3	81.3	56.5	65.3

Plain AR on the same model runs at about 43.5 tok/s.

MTP takes effect only when both of these hold. Otherwise the request runs plain AR; the server logs why at debug level:

• The GGUF ships its nextn tensors. The asset store's Qwen3.5-{0.8B,2B,4B,9B}-Q4_K_M builds dropped them, so they load trunk-only.
• The model can roll back its recurrent state during verify. Today that means quantized qwen35 checkpoints on CUDA. On Metal, and for Qwen 3.6 (qwen35moe), the recurrent gate in the table above applies. The Qwen 3.6 head is loaded and validated (~84.6% draft acceptance) but not driven for generation.

Structured generation

Speculative decoding composes with structured generation: set both speculative and response_format on the same request and the grammar is enforced through the speculative verify path — every emitted token is schema-legal and is the model's own greedy choice under the grammar.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Describe an NPC."}],
    "speculative": { "proposer": "prompt_lookup" },
    "response_format": { "type": "json_schema", "json_schema": { ... } }
  }'

This needs a model that supports batched verify:

• float Llama (1B and up), Gemma 4 and Qwen 3;
• Bonsai 1-bit;
• on CUDA, Llama-loader GGUFs (Llama, Qwen 3) and quantized Qwen 3.5.

Anywhere else the request still works, just without the speculative speedup: it runs plain constrained AR.

SDK examples

Rust SDK (atelico-sdk)

use atelico_sdk::{ChatCompletionRequest, SpeculativeConfig, RequestMessage};

let req = ChatCompletionRequest {
    model: "in-memory::Llama-3.1-8B-Instruct".into(),
    messages: vec![RequestMessage {
        role: "user".into(),
        content: "Summarize the document above.".into(),
    }],
    // Off by default — opt in:
    speculative: Some(SpeculativeConfig {
        proposer: "prompt_lookup".into(),
        k: Some(5),
    }),
    ..Default::default()
};

Python (OpenAI-compatible client)

client.chat.completions.create(
    model="in-memory::Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarize..."}],
    extra_body={"speculative": {"proposer": "prompt_lookup", "k": 5}},
)

(Most OpenAI-client libraries route unknown fields via extra_body.)

Godot

var response = atelico.chat_completion({
    "model": "in-memory::Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarize..."}],
    "speculative": {"proposer": "prompt_lookup", "k": 5}
})

Unity / Unreal (C / FFI)

The C ABI takes the request as a JSON string; same shape:

const char* req_json = R"({
    "model": "in-memory::Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarize..."}],
    "speculative": {"proposer": "prompt_lookup", "k": 5}
})";
atelico_chat_completion(handle, req_json, response_buf, response_buf_len);

Limitations

• The speedup depends on how predictable the output is. Highly templated or input-echoing responses gain the most; high-entropy creative text gains the least.
• Greedy (temperature: 0) text can differ from plain AR at a near-tie. Verify scores drafts through a forward that doesn't always run the exact kernels single-token decode uses (batched projections, a different logits path). The logits therefore agree only to about 1e-4. When two candidate tokens are closer than that, the pick can flip and the continuations part ways. Both texts are valid greedy outputs; neither is garbage.
  • Measured on 200-token greedy runs: Llama 3.x, SmolLM3 and Gemma 4 E2B Q4_K_M diverged from AR on some prompts, after 45 to 630 characters, on both Metal (M1 Max) and CUDA (RTX 3090). They matched exactly on others, and float Llama 3.2 3B matched on Metal.
  • If you need byte-identical output across runs, compare runs made with the same speculative setting.
• With sampling, the distribution is preserved but individual tokens may differ from a non-speculative run with the same seed. That is the same trade-off any speculative-decoding paper describes.
• The MTP / EAGLE-3 / DFlash proposers (visible in the source tree) are not exposed through this API. The only proposer names the request accepts are prompt_lookup (alias pld) and lookahead; any other name (mtp, eagle3, dflash, draft_model, …) is unrecognized and falls back to plain AR. They underperform AR on the hardware we currently ship — see the engine's internal PROPOSER_GUIDE.md for the empirical breakdown. See Self-speculation (MTP) on quantized Qwen below for the current status of the MTP work specifically.

Quick decision tree

Will your output quote / paraphrase / edit the prompt?
├─ YES → enable `prompt_lookup`. Try k=5.
└─ NO ─→ Is the output bullet-points / code / structured text without a template?
           ├─ YES → try `lookahead`. Modest speedup.
           └─ NO ─→ Leave speculative off. You won't lose anything.

Are you on Bonsai 1-bit on Metal?
└─ YES → Leave speculative off regardless. The verify forward
         regresses there (AIENG-67).