Speculative Decoding (Multi-Token Decoding)

Speculative decoding makes generation faster without changing the output. A lightweight proposer drafts several tokens ahead, and the model verifies them in a single batched forward pass — so a step that would normally produce one token can commit several.

For greedy decoding (temperature: 0) the output is bit-identical to normal generation. You get the same text, sooner.

Speculative decoding is OFF by default

Every request runs plain autoregressive decoding unless you ask for a proposer in the speculative field. This is intentional:

• The speedup is workload-dependent. Grounded prompts (RAG, summarization, code refactor) get 1.5–2.5×. Free-form prompts (creative writing, open-ended chat) get ≈ 1× — same speed as plain AR, but with a tiny per-round overhead.
• On some model/device combinations the proposer's multi-token verify forward is a measured regression vs plain AR (small-m quantized GEMM on Metal — Bonsai 1-bit, certain Q4_K_M targets). Auto-enabling there would silently halve throughput.
• You almost always know your workload better than the server does. An RAG product enables it once at integration time; a creative- writing demo leaves it off; an A/B-test toggles it per request.

If you're not sure whether your workload benefits, the which proposer should I use? matrix below has a quick decision tree.

Enabling it

Add a speculative field to your chat completion request:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Summarize the text above."}],
    "speculative": { "proposer": "prompt_lookup", "k": 5 }
  }'

Field	Meaning
proposer	Which proposer to use — prompt_lookup (alias pld) or lookahead. Case-insensitive.
k	Draft length: tokens drafted per verify round. Optional, default 5.

Omit speculative (or pass null) and generation runs normally. An unknown proposer name falls back to normal decoding (you lose the speedup, not the request) — "none" is the conventional explicit opt-out.

Why turn it on?

For the right workload, the speedup is significant and free:

Workload	Proposer	Measured speedup (Metal, M1 Max)
RAG / Q&A over documents	prompt_lookup	1.5–2.8× AR
Summarization with entity preservation	prompt_lookup	1.5–2.5× AR
Code refactoring / editing	prompt_lookup	2–3× AR
Multi-turn chat with long history	prompt_lookup	1.3–2× AR
Translation with proper-noun preservation	prompt_lookup	1.2–1.5× AR
Llama-3.1-8B on a grounded code prompt (measured)	prompt_lookup	2.10× AR (92.9 % accept)

For other workloads it's a wash:

Workload	Speedup
Creative writing / story generation	~ 1.0× (no n-gram overlap to find)
Open-ended chat / brainstorming	~ 1.0×
Math reasoning that re-derives	~ 1.0×
Short prompt → long generation	~ 1.0× (tiny haystack)

The output stays bit-identical to plain AR in all cases. The only penalty for enabling it on a workload where it doesn't help is the proposer's own per-round overhead — a few microseconds for prompt_lookup (it's a CPU n-gram lookup), more substantial for draft-model / EAGLE-3 / DFlash proposers (which is why we don't expose those as default options today).

Which proposer should I use?

There are two general-purpose proposers exposed through the API:

prompt_lookup (often abbreviated PLD)

The recommended choice for almost every speedup use case.

• How it works. No extra model. No GPU draft cost. For every generation step, the proposer searches the prompt + already- generated tokens for a recent repeat of the last 2–3 tokens, and drafts the K tokens that follow that match site. When it doesn't find a match, it returns nothing and the verifier falls back to AR for that round — so the cost on a workload where it doesn't help is essentially zero.
• When to use it. Any input-grounded task: RAG / Q&A, summary, code editing, translation, multi-turn chat with long history, agent-style traces that quote earlier turns.
• When NOT to use it. Free-form creative output where the model isn't echoing the prompt — the n-gram lookup almost always fails, costing a few microseconds per round for no benefit. (Still safe; just pointless.)
• Tuning. k = 5 is a good default. Higher k makes each speculative round commit more tokens on a hit, but every miss becomes more wasteful. The PLD paper sweeps n = 2–3 and reports n = 3 as the sweet spot; we use both with longest-first fallback.

lookahead

Pick this if prompt_lookup doesn't help your workload but you still want a general-purpose speedup.

• How it works. Training-free n-gram parallel decoding. Predicts multiple positions in parallel using an n-gram table built from prior decoding, rather than from the prompt.
• When to use it. Open-ended generation that doesn't echo its prompt but does have local structure / repetition (e.g. bullet-pointed lists, generated tables, structured code without a template).
• Expected speedup. ~ 1.3× AR — smaller than PLD on grounded tasks but more general.

When NOT to enable speculative decoding

Some model/device combinations have a correctness or performance veto independently of which proposer you pick. The server enforces these gates automatically — if a gate fails, the request transparently falls back to AR with a log line, you don't have to handle errors.

Case	What happens	Why
Recurrent / hybrid models (Qwen3.5, Qwen3.6, Nemotron-H)	Server falls back to AR	DeltaNet / Mamba recurrent state advances through drafts even when they get rejected, corrupting the output. This holds even though quantized Qwen 3.5 / 3.6 now ship MTP heads — see Self-speculation (MTP) on quantized Qwen.
Structured generation (response_format) + model without batched verify	Server falls back to AR	The verify path has to enforce the grammar; per-token verify can't.

You can still send speculative on those targets — it just won't take effect, and the server will log why. There is no error and no behavior change vs the plain AR baseline.

For the user-controlled "should I bother" question, the matrix at the top of Which proposer should I use is the answer. The two short rules:

1. Don't enable it for plain creative-writing or open-ended chat. You're paying microseconds per round for nothing. Not a regression, not a win.
2. Don't enable it on Bonsai 1-bit on Metal. Measured 0.43× AR regression — the small-m quantized verify forward is more expensive than running AR. (On CUDA Bonsai is fine; on Metal small-m quantized GEMM doesn't amortize.)

Everything else is either a win (grounded workloads) or a no-op (non-grounded workloads, where the proposer harmlessly fails to find matches).

Self-speculation (MTP) on quantized Qwen

Some Qwen 3.5 and Qwen 3.6 checkpoints ship an appended MTP (Multi-Token Prediction) block — a nextn head that drafts the next token from the model's own hidden state, so no separate draft model is needed. The engine now loads that head for the quantized hybrid Qwen targets:

• QQwen3_5MtpHead — quantized Qwen 3.5 (qwen35 GGUF, dense SwiGLU MTP block).
• QQwen3_6MtpHead — quantized Qwen 3.6 (qwen35moe GGUF, full-attn + MoE MTP block).

Detection is automatic: the loader reads the {arch}.nextn_predict_layers GGUF metadata and loads the last block (blk.{num_hidden_layers}, which carries the nextn.* fusion tensors) as the MTP head. If the GGUF dropped its nextn tensors at quantization time — as the Qwen3.5-{0.8B,2B,4B,9B}-Q4_K_M builds in the asset store did — the model loads trunk-only and behaves exactly as before.

This is infrastructure only — it is not user-facing today, and you do not enable it through a request field. Two things keep it off the API surface:

1. No speedup on this architecture. Published Qwen MTP heads are depth-1 (one drafted token per round). On the recurrent DeltaNet trunk the per-round overhead plus the MTP head's own forward outweigh the single-token draft — measured ~0.67–0.85× AR, i.e. a regression, not a speedup. (The trained Qwen 3.6 35B-A3B head does draft well — ~84.6% accept — confirming the head is numerically correct; the wall-clock loss is structural, not a quality problem.)
2. The recurrent fallback gate still applies. The server routes recurrent / hybrid models (Qwen 3.5, Qwen 3.6, Nemotron-H) to plain AR regardless of the speculative field, because the recurrent state drifts across speculative verify rounds (greedy output stops being bit-identical to AR on tight-margin targets). MTP doesn't change that.

So on a quantized Qwen 3.5 / 3.6 model, sending speculative has no effect: the request runs plain AR, the same as omitting the field. The MTP head is loaded and validated but not driven for generation. This section exists to document why — if Qwen MTP self-speculation becomes a request-time option in a future release, this is the page where it will be enabled.

Structured generation

Speculative decoding composes with structured generation: set both speculative and response_format on the same request and the grammar is enforced through the speculative verify path — every emitted token is schema-legal, and greedy output is identical to running the schema without speculation.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Describe an NPC."}],
    "speculative": { "proposer": "prompt_lookup" },
    "response_format": { "type": "json_schema", "json_schema": { ... } }
  }'

This needs a model that supports batched verify (the float models — Llama, Gemma4, Qwen3, Parcae). For a quantized model the request still works, just without the speculative speedup (verify falls back to per-token mode).

SDK examples

Rust SDK (atelico-sdk)

use atelico_sdk::{ChatCompletionRequest, SpeculativeConfig, RequestMessage};

let req = ChatCompletionRequest {
    model: "in-memory::Llama-3.1-8B-Instruct".into(),
    messages: vec![RequestMessage {
        role: "user".into(),
        content: "Summarize the document above.".into(),
    }],
    // Off by default — opt in:
    speculative: Some(SpeculativeConfig {
        proposer: "prompt_lookup".into(),
        k: Some(5),
    }),
    ..Default::default()
};

Python (OpenAI-compatible client)

client.chat.completions.create(
    model="in-memory::Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarize..."}],
    extra_body={"speculative": {"proposer": "prompt_lookup", "k": 5}},
)

(Most OpenAI-client libraries route unknown fields via extra_body.)

Godot

var response = atelico.chat_completion({
    "model": "in-memory::Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarize..."}],
    "speculative": {"proposer": "prompt_lookup", "k": 5}
})

Unity / Unreal (C / FFI)

The C ABI takes the request as a JSON string; same shape:

const char* req_json = R"({
    "model": "in-memory::Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarize..."}],
    "speculative": {"proposer": "prompt_lookup", "k": 5}
})";
atelico_chat_completion(handle, req_json, response_buf, response_buf_len);

Limitations

• The speedup depends on how predictable the output is. Highly templated or input-echoing responses gain the most; high-entropy creative text gains the least.
• Output bit-identity is guaranteed for greedy decoding only (temperature: 0). With sampling, the distribution is preserved but individual tokens may differ from a non-speculative run with the same seed — exactly the same trade-off any speculative-decoding paper describes.
• The MTP / EAGLE-3 / DFlash proposers (visible in the source tree) are not exposed through this API. The only proposer names the request accepts are prompt_lookup (alias pld) and lookahead; any other name (mtp, eagle3, dflash, draft_model, …) is unrecognized and falls back to plain AR. They underperform AR on the hardware we currently ship — see the engine's internal PROPOSER_GUIDE.md for the empirical breakdown. See Self-speculation (MTP) on quantized Qwen below for the current status of the MTP work specifically.

Quick decision tree

Will your output quote / paraphrase / edit the prompt?
├─ YES → enable `prompt_lookup`. Try k=5.
└─ NO ─→ Is the output bullet-points / code / structured text without a template?
           ├─ YES → try `lookahead`. Modest speedup.
           └─ NO ─→ Leave speculative off. You won't lose anything.

Are you on Bonsai 1-bit on Metal?
└─ YES → Leave speculative off regardless. The verify forward
         regresses there (AIENG-67).