GPU Scheduling

The engine runs every modality — chat, embeddings, image generation, text-to-speech, speech-to-text, classifiers — on one GPU, which it also shares with whatever else is on the machine: your game's renderer, a browser, a video call. Two settings decide how that GPU is divided, and they answer different questions.

	what it answers	scope
Device mode	how much GPU the engine takes from the rest of the machine	one setting for the whole engine
Request priority	who goes first among the engine's own work	one per request

Set neither and the engine behaves as it always has: everything competes evenly, and the renderer gets whatever is left over.

Request priority

Any request can declare what it is for:

{
  "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct",
  "messages": [{ "role": "user", "content": "What do you know about the north road?" }],
  "priority": "interactive"
}

• interactive — a player is waiting on this right now: the line an NPC is about to speak, the answer to something they just typed.
• normal — the default, and what every request that omits the field gets.
• background — off-screen work: indexing memories, warming a cache, pre-rendering a portrait for a menu three screens away.

The field is optional and works on chat, text and response completions, image generation, video, text-to-speech, speech-to-text, background removal, and text, image and audio embeddings. Omitting it produces exactly the request body older versions sent, so nothing changes until you opt in.

Speech defaults to interactive. Text-to-speech and speech-to-text are a player talking, or being talked to, and speech that arrives late is a conversation that stalled. Everything else defaults to normal. A bulk offline transcription job should say background and get out of the way.

A load inherits the request that triggered it. If an interactive request has to load a model first, that load runs at interactive too — otherwise you would be blocked on work that keeps losing its place in the queue. An explicit preload (a splash screen warming a model nobody has asked for yet) runs at background, so a request whose model is already loaded can go ahead of it.

Device mode

The device mode says how greedy the engine should be with the machine as a whole. Your host application sets it, and can change it at runtime — many games switch to PrioritizeCompute in a menu and back to PrioritizeGraphics in the world.

• PrioritizeGraphics — the renderer comes first. Inference hands the GPU back at every opportunity, costing throughput to keep the frame rate steady.
• Balance — the default. Inference hands the GPU over when something else in the engine is waiting for it, and otherwise runs flat out.
• PrioritizeCompute — inference comes first. Use it while the game is paused, on a loading screen, or in a menu.

On CUDA, the mode also sets the stream's hardware priority

On NVIDIA hardware the device mode does one more thing: it picks the scheduling priority of the CUDA stream the engine submits its work on. That is a hint to the GPU's own scheduler, below anything the engine does in software.

device mode	CUDA stream priority
PrioritizeGraphics	lowest — the renderer's work is scheduled ahead of inference
Balance	the driver default
PrioritizeCompute	highest

The exact numbers are per-device; the engine queries the supported range at startup and maps onto its ends. (On an RTX 5080, whose range is −5..=0, the three modes come out as 0, −2 and −5 — CUDA's convention is that a lower number means higher priority.)

:::caution The stream's priority is fixed when the device is created Set the mode before you build the engine. A CUDA stream's priority cannot be changed after creation, so switching modes at runtime still changes the yielding behaviour described above — that part works whenever you change it — but it cannot re-prioritise a stream that already exists. If your application has a mode it spends most of its time in, declare that one at startup. :::

Metal has no equivalent control: there is no per-queue scheduling priority to set, so on Apple hardware the device mode governs yielding only.

Bounding what one request can allocate

Two further knobs on the scheduling config guard the card against a single oversized request. Both are off by default.

max_kv_tokens caps the KV-cache tokens (prompt + generation) one request may require. Over-ceiling requests fail cleanly with a typed error before any cache allocation happens, which is the point: the engine drops the existing cache before allocating its replacement, so an allocation that fails partway leaves the engine cacheless, the freed memory hoarded by the CUDA pool, and the pointers baked into captured decode graphs invalid. One oversized request could wedge the card for every request after it. Refusing it up front keeps the persistent cache provably intact, and the next in-budget request proceeds.

For scale: a single 30k-token request at F32 KV is a ~7.5 GB allocation sitting next to 16.8 GB of weights on a 24 GB card.

pool_reserve_mb bounds how much freed memory CUDA's pool holds between requests. Absent, candle's default on a discrete GPU is to hoard everything — which also makes nvidia-smi read the card as full when it is not. It is applied by exporting CANDLE_CUDA_MEMPOOL_RELEASE_THRESHOLD before the first CUDA device is created; the fork reads that variable exactly once, so it cannot be changed later in the process, and an operator-set variable wins over the config knob. It is process-global: a multi-device setup shares one threshold.

let engine = EngineBuilder::new()
    .max_kv_tokens(8192)   // refuse a request needing more KV than this
    .build()?;

How the two combine

A long job — a chat generation, an image generation, a video — checks both at natural boundaries in its work: a decode token, a prefill chunk, a denoise step.

device mode ↓ / priority →	interactive	normal	background
PrioritizeCompute	never yields	never yields	yields if anything is queued
Balance	yields if something higher is queued	same	yields if anything is queued
PrioritizeGraphics	yields regularly	yields regularly	yields regularly

Two properties are worth knowing because they shape what you can expect:

Checking is free when nothing is waiting. The check is a lock and a peek at the queue — no GPU work — so it happens often without costing anything. The handover itself is only paid when there is genuinely someone to hand over to.

This is cooperation, not preemption. A job hands the GPU over at a boundary it chooses; it is never interrupted mid-kernel. So an interactive request queued behind an image generation waits roughly one denoise step, not the whole image — but it does wait for that step.

Measured on an M1 Max: an interactive embedding issued against a continuously running background generation went from 13.8 s at p99 (it was waiting for the entire generation) to 80 ms, while the background job completed slightly more work in the same window, not less.

What priorities cannot do

Two chat requests to the same model cannot be prioritised against each other. The engine serves one language model on a single worker thread, so a second chat queues behind the first before the GPU scheduler ever sees it. Priorities order work across modalities — a chat against an embedding, an embedding against an image — which is where contention actually bites. If you need one NPC's line to jump ahead of another's, issue them in the order you want them answered.

A background request can wait indefinitely. There is no ageing or automatic promotion: under sustained higher-priority traffic, background work may simply not run. That is deliberate — it degrades first, by design — but it means background work needs a plan for not completing. Two things help:

• ATELICO_GPU_ACQUIRE_TIMEOUT_SECS (default 30) bounds how long a request waits before failing with a GPU busy error that names what is holding the device — held by 'llm.decode' for 12.4s, 19 waiting. Set it below your application's own timeout, so the engine gives up first and can tell you why.
• The gate watchdog logs the holder of a device held longer than ATELICO_GPU_WATCHDOG_SECS.

A model load is exempt from that timeout. Loads legitimately take minutes on a cold cache, and failing a request that is merely waiting for a cold start would turn a slow first interaction into an error.

Using it from the SDKs

Chat, completions and embeddings take a JSON request in every binding, so priority works by adding the field:

// Unity
engine.Llm.ChatCompletion(@"{
  ""model"": ""in-memory::my-model"",
  ""messages"": [{ ""role"": ""user"", ""content"": ""Hello"" }],
  ""priority"": ""interactive""
}");

# Godot
var response = engine_node.llm_chat(JSON.stringify({
    "model": "in-memory::my-model",
    "messages": [{"role": "user", "content": "Hello"}],
    "priority": "interactive",
}))

# Python
engine.llm.chat(json.dumps({
    "model": "in-memory::my-model",
    "messages": [{"role": "user", "content": "Hello"}],
    "priority": "interactive",
}))

Unreal's typed request carries it as an enum, settable from Blueprint or C++:

FAtelicoChatRequest Request;
Request.ModelId = TEXT("in-memory::my-model");
Request.Priority = EAtelicoPriority::Interactive;

Lua's embedding helpers take it as an optional trailing argument:

local vector = engine:embed_image("in-memory::dinov2-small", path, "background")