Chat Completions API

The chat completions endpoint generates responses from a conversation. It's OpenAI-compatible, so if you've used the OpenAI API before, this works the same way.

POST /v1/chat/completions

Basic Request

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  }
}

Request Parameters

Parameter	Type	Default	Description
model	string	required	Model identifier (e.g., in-memory::meta-llama/Llama-3.2-3B-Instruct)
messages	array	required	Conversation messages (see below)
stream	boolean	false	Enable token-by-token streaming
temperature	float	model default	Sampling temperature. Lower = more deterministic, higher = more creative. 0 = greedy.
top_k	integer	model default	Keep only the k most-likely tokens before sampling. 0 disables.
top_p	float	model default	Nucleus sampling: keep the smallest set of tokens whose probabilities sum to p. 1.0 disables.
min_p	float	model default	Keep tokens whose probability is at least min_p × the top token's probability. 0 disables.
repetition_penalty	float	model default	Penalize tokens already in the context. 1.0 = no penalty; higher discourages repetition.
presence_penalty	float	0	OpenAI-style penalty applied once to any token that has appeared.
frequency_penalty	float	0	OpenAI-style penalty scaled by how often a token has appeared.
seed	integer	a fixed default	Seed for the sampler. The same seed + params + prompt yields a byte-identical stream. Omitting it does not randomise — the engine uses one fixed seed, so identical requests repeat.
max_tokens	integer	model default	Maximum tokens to generate
response_format	object	null	Constrain output format (see Structured Generation)
enable_thinking	boolean	null	Toggle the extended-thinking span on models that support it (Qwen 3.5/3.6, SmolLM3). Ignored — with a warning — on other models. See Thinking / Reasoning.
continue_final_message	boolean	null	Continue a partial assistant turn instead of opening a fresh reply. See Continuing a Partial Reply.
adapter_scale	float	the model's default, else 1.0	Strength of the LoRA adapter this request runs with — the one named by the model::adapter suffix, or the model's default. 1.0 as trained, 0.0 the base model, values between blend. See LoRA Adapters.
prefix_cache	object	null	Reuse the KV state of a shared prompt prefix instead of re-prefilling it every call. { "scope": "auto" } matches the longest cached prefix; { "scope": { "leading_messages": N } } names the first N messages; optional namespace isolates reuse groups. Output is byte-identical to a non-cached greedy decode — only prefill cost changes. Available on every binding (the field rides the request JSON: FFI, Python, Lua, Godot, Unreal, Unity).

The sampling parameters above (temperature through seed) are all optional — when you omit them, each model falls back to its own recommended defaults. See Sampling below. You rarely need to set them.

Message Roles

Each message in the messages array has a role and content:

Role	Purpose
system	Sets the AI's behavior, personality, or constraints. Placed first.
user	The human's message.
assistant	The AI's previous response. Used for multi-turn context.

System Prompts

System prompts define how the model behaves. They're essential for game NPCs, assistants, or any specialized behavior:

Python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {
            "role": "system",
            "content": "You are a ship AI aboard a deep-space freighter. You speak formally, "
                       "address the player as Captain, and provide status reports when asked. "
                       "You are concerned about a recent anomaly in sector 7.",
        },
        {"role": "user", "content": "Status report."},
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

Godot (GDScript)

func get_ship_ai_response(player_input: String) -> void:
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": [
            {
                "role": "system",
                "content": "You are a ship AI aboard a deep-space freighter. You speak formally, address the player as Captain, and provide status reports when asked. You are concerned about a recent anomaly in sector 7."
            },
            {"role": "user", "content": player_input}
        ],
        "temperature": 0.7
    }
    engine.async_chat_completions(JSON.stringify(request))

Unity (C#)

public async Task<string> GetShipAIResponse(string playerInput)
{
    var request = new
    {
        model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        messages = new object[]
        {
            new { role = "system", content = "You are a ship AI aboard a deep-space freighter. You speak formally, address the player as Captain, and provide status reports when asked." },
            new { role = "user", content = playerInput }
        },
        temperature = 0.7
    };

    var json = JsonSerializer.Serialize(request);
    var content = new StringContent(json, Encoding.UTF8, "application/json");
    var response = await client.PostAsync($"{BaseUrl}/chat/completions", content);
    var responseJson = await response.Content.ReadAsStringAsync();

    using var doc = JsonDocument.Parse(responseJson);
    return doc.RootElement.GetProperty("choices")[0]
        .GetProperty("message").GetProperty("content").GetString();
}

Unreal (C++)

void UAtelicoClient::GetShipAIResponse(const FString& PlayerInput)
{
    TSharedPtr<FJsonObject> Body = MakeShareable(new FJsonObject);
    Body->SetStringField("model", "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M");
    Body->SetNumberField("temperature", 0.7);

    TArray<TSharedPtr<FJsonValue>> Messages;

    TSharedPtr<FJsonObject> SystemMsg = MakeShareable(new FJsonObject);
    SystemMsg->SetStringField("role", "system");
    SystemMsg->SetStringField("content",
        "You are a ship AI aboard a deep-space freighter. You speak formally, "
        "address the player as Captain, and provide status reports when asked.");
    Messages.Add(MakeShareable(new FJsonValueObject(SystemMsg)));

    TSharedPtr<FJsonObject> UserMsg = MakeShareable(new FJsonObject);
    UserMsg->SetStringField("role", "user");
    UserMsg->SetStringField("content", PlayerInput);
    Messages.Add(MakeShareable(new FJsonValueObject(UserMsg)));

    Body->SetArrayField("messages", Messages);
    SendRequest(Body); // see Getting Started for full HTTP setup
}

Multi-Turn Conversations

Include previous messages to give the model context of the conversation:

Python

response = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {"role": "system", "content": "You are a helpful tavern keeper named Boris."},
        {"role": "user", "content": "What do you have on the menu?"},
        {"role": "assistant", "content": "We have roasted boar, mushroom stew, and fresh bread. The stew is my specialty!"},
        {"role": "user", "content": "I'll have the stew. Any rumors lately?"},
    ],
)

print(response.choices[0].message.content)

Godot (GDScript)

# Keep conversation history in an array
var conversation: Array = [
    {"role": "system", "content": "You are a helpful tavern keeper named Boris."}
]

func talk_to_npc(player_input: String) -> void:
    conversation.append({"role": "user", "content": player_input})
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": conversation
    }
    engine.async_chat_completions(JSON.stringify(request))

func _on_async_request_completed(_job_id: int, response: String) -> void:
    var parsed = JSON.parse_string(response)
    var reply = parsed["choices"][0]["message"]["content"]
    conversation.append({"role": "assistant", "content": reply})
    dialogue_label.text = reply

Unity (C#)

private List<object> conversation = new()
{
    new { role = "system", content = "You are a helpful tavern keeper named Boris." }
};

public async Task<string> TalkToNPC(string playerInput)
{
    conversation.Add(new { role = "user", content = playerInput });

    var request = new { model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M", messages = conversation };
    var json = JsonSerializer.Serialize(request);
    var content = new StringContent(json, Encoding.UTF8, "application/json");
    var response = await client.PostAsync($"{BaseUrl}/chat/completions", content);
    var responseJson = await response.Content.ReadAsStringAsync();

    using var doc = JsonDocument.Parse(responseJson);
    var reply = doc.RootElement.GetProperty("choices")[0]
        .GetProperty("message").GetProperty("content").GetString();

    conversation.Add(new { role = "assistant", content = reply });
    return reply;
}

Unreal (C++)

// Store conversation as TArray<TSharedPtr<FJsonValue>>
TArray<TSharedPtr<FJsonValue>> Conversation;

void UAtelicoClient::InitConversation()
{
    TSharedPtr<FJsonObject> SystemMsg = MakeShareable(new FJsonObject);
    SystemMsg->SetStringField("role", "system");
    SystemMsg->SetStringField("content", "You are a helpful tavern keeper named Boris.");
    Conversation.Add(MakeShareable(new FJsonValueObject(SystemMsg)));
}

void UAtelicoClient::TalkToNPC(const FString& PlayerInput)
{
    TSharedPtr<FJsonObject> UserMsg = MakeShareable(new FJsonObject);
    UserMsg->SetStringField("role", "user");
    UserMsg->SetStringField("content", PlayerInput);
    Conversation.Add(MakeShareable(new FJsonValueObject(UserMsg)));

    TSharedPtr<FJsonObject> Body = MakeShareable(new FJsonObject);
    Body->SetStringField("model", "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M");
    Body->SetArrayField("messages", Conversation);

    // On response callback, parse assistant reply and append to Conversation
    SendRequest(Body);
}

Streaming

Set "stream": true to receive tokens as they're generated. This is ideal for typewriter-style dialogue UI.

SSE Format: Each token arrives as a Server-Sent Event. The stream ends with data: [DONE].

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" upon"},"finish_reason":null}]}
...
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Python

stream = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {"role": "system", "content": "You are a narrator for a fantasy RPG."},
        {"role": "user", "content": "Describe the entrance to the dungeon."},
    ],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()

Godot (GDScript)

# Streaming uses the built-in signal-based API
func stream_narration(prompt: String) -> void:
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": [
            {"role": "system", "content": "You are a narrator for a fantasy RPG."},
            {"role": "user", "content": prompt}
        ]
    }
    engine.stream_chat_completions(JSON.stringify(request))

# Called once per token as it arrives
func _on_inference_token_generated(_job_id: int, token: String) -> void:
    dialogue_label.text += token  # typewriter effect

func _on_inference_completed(_job_id: int) -> void:
    print("Stream finished")

Unity (C#)

public async Task StreamToDialogue(string prompt, TMPro.TextMeshProUGUI dialogueText)
{
    var request = new
    {
        model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        messages = new[]
        {
            new { role = "system", content = "You are a narrator for a fantasy RPG." },
            new { role = "user", content = prompt }
        },
        stream = true
    };

    var json = JsonSerializer.Serialize(request);
    var httpContent = new StringContent(json, Encoding.UTF8, "application/json");
    var httpRequest = new HttpRequestMessage(HttpMethod.Post, $"{BaseUrl}/chat/completions")
    {
        Content = httpContent
    };

    var response = await client.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead);
    using var stream = await response.Content.ReadAsStreamAsync();
    using var reader = new StreamReader(stream);

    dialogueText.text = "";
    while (await reader.ReadLineAsync() is { } line)
    {
        if (line.StartsWith("data: ") && line != "data: [DONE]")
        {
            var chunk = JsonDocument.Parse(line.Substring(6));
            var delta = chunk.RootElement.GetProperty("choices")[0].GetProperty("delta");
            if (delta.TryGetProperty("content", out var c))
                dialogueText.text += c.GetString();
        }
    }
}

Unreal (C++)

void UAtelicoClient::StreamNarration(const FString& Prompt)
{
    auto Request = FHttpModule::Get().CreateRequest();
    Request->SetURL(TEXT("http://localhost:11434/v1/chat/completions"));
    Request->SetVerb(TEXT("POST"));
    Request->SetHeader(TEXT("Content-Type"), TEXT("application/json"));

    // Build JSON with "stream": true and messages array
    // ... (see Getting Started for JSON building pattern) ...

    // Handle chunked SSE responses via progress callback
    Request->OnRequestProgress().BindLambda(
        [this](FHttpRequestPtr Req, int32 BytesSent, int32 BytesReceived)
    {
        FString Content = Req->GetResponse()->GetContentAsString();
        // Parse new SSE lines since last callback
        // Extract delta.content tokens
        // Append to dialogue UTextBlock
    });

    Request->ProcessRequest();
}

Sampling

Every sampling parameter is optional. Each model comes with recommended defaults tuned for it, so you get good results without setting anything — a bare request like the basic example above already samples the way the model is meant to. Set a parameter only when you want to override the default for a particular request.

Sampling parameters

Parameter	What it does	Optional / Default
temperature	Controls randomness. Lower is more focused and repeatable; higher is more varied and creative. 0 always picks the single most-likely token.	Optional — model default
top_k	Only consider the k most-likely next tokens. Lower values keep output on-topic.	Optional — model default
top_p	Nucleus sampling: only consider the smallest set of tokens whose probabilities add up to p.	Optional — model default
min_p	Ignore tokens less likely than this fraction of the most-likely token. Filters out long-shot tokens.	Optional — model default
repetition_penalty	Discourages repeating words and phrases already in the conversation. 1.0 means no penalty; higher values push harder against repetition.	Optional — model default
presence_penalty	Discourages reusing any token that has already appeared (applied once per token).	Optional — 0
frequency_penalty	Like presence_penalty, but the penalty grows the more often a token has appeared.	Optional — 0
seed	Set it to vary output on purpose, or to reproduce a specific generation: the same seed with the same parameters and prompt produces the same result every time.	Optional — a fixed default, not random
max_tokens	Caps the length of the response.	Optional — model default

Temperature quick reference

Temperature is the knob you'll reach for most often. For game applications:

Use Case	Temperature	Why
Factual responses, game rules	0.1 - 0.3	Consistent, predictable
NPC dialogue, general conversation	0.6 - 0.8	Natural variation
Creative writing, storytelling	0.9 - 1.2	More surprising, diverse

Examples

Rely on the model's recommended defaults — pass no sampling parameters at all:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [{"role": "user", "content": "Describe a misty harbor at dawn."}]
  }'

Override a couple of parameters for tighter, reproducible output:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [{"role": "user", "content": "Describe a misty harbor at dawn."}],
    "temperature": 0.4,
    "top_p": 0.9,
    "seed": 42
  }'

Finish Reasons

The finish_reason field tells you why generation stopped:

Value	Meaning
stop	Model finished naturally (end of response)
length	Hit the max_tokens limit

length means the output was truncated — common with structured/JSON output. When you see it, raise max_tokens and regenerate; for a best-effort salvage of a cut-off JSON response, see repair_json.

Continuing a Partial Reply

Set continue_final_message: true and end messages with a role: "assistant" turn, and the model resumes that turn rather than starting a new one. Only the continuation comes back — the prefix you sent is not echoed. This mirrors vLLM's flag of the same name (and Mistral/DeepSeek's prefix).

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::PrismML/Bonsai-1.7B",
    "continue_final_message": true,
    "messages": [
      {"role": "user", "content": "Write a haiku about GPUs."},
      {"role": "assistant", "content": "Silicon fireflies"}
    ]
  }'

The reply picks up after Silicon fireflies instead of writing a fresh haiku.

With a schema: resuming mid-document

The real payoff is with response_format. The grammar is primed over the partial assistant content, so structured output resumes inside the document — letting you grow a partial JSON result instead of regenerating it:

{
  "model": "in-memory::PrismML/Bonsai-1.7B",
  "continue_final_message": true,
  "messages": [
    {"role": "user", "content": "Generate a 6-card deck."},
    {"role": "assistant", "content": "{\"cards\":[{\"name\":\"Ember Fox\"},{\"name\":\"Tide Warden\"}"}
  ],
  "response_format": {
    "type": "json_schema",
    "json_schema": { "name": "deck", "schema": { "...": "6-card schema" } }
  }
}

The engine emits only cards 3–6 plus the closing brackets. The cards you already had are preserved byte-for-byte, because they were never regenerated.

caution

The partial content must be a legal prefix of the schema. If it isn't, the request fails at priming time rather than silently generating unconstrained text. Compact your schema with "x-guidance": { "whitespace_flexible": false } so the partial document you feed back matches the whitespace the grammar expects.

Limits. If the last message isn't role: "assistant", the flag is ignored and the engine logs a warning. tools and speculative are not supported on this path and are ignored (also logged). The same field works on /v1/responses, where the trailing assistant item of input is continued and the schema comes from text.format.

Thinking / Reasoning

Some models support an extended "thinking" (chain-of-thought) phase before the final answer. How a model handles this is a per-model capability, and the enable_thinking request field only has a defined meaning for one group:

Model family	Behavior with enable_thinking
Qwen 3.5, Qwen 3.6, SmolLM3	Honor the flag: true gates a <think>…</think> reasoning span; false skips it.
Gemma 4	Reasons in its own native format, driven by the model/template — not by enable_thinking. Passing true is ignored and can degrade output; pass false.
All other models (Llama, etc.)	No extended-thinking mode. enable_thinking: true is ignored.

If you pass enable_thinking: true to a model that doesn't honor the flag (Gemma 4 or any non-thinking model), the engine logs a warning and runs the request anyway — the flag is simply ignored. The safe default is to omit enable_thinking (or set it false) unless you're targeting a model in the first row.

:::warning Gemma 4 known issue Gemma 4's native reasoning can diverge during long agentic / tool-use decoding. See the Gemma-4 agentic decoding divergence known-issue note. :::

Error Handling

Errors return standard HTTP status codes with an OpenAI-compatible error body:

{
  "error": {
    "message": "Model 'nonexistent-model' not found",
    "type": "invalid_request_error",
    "param": "model",
    "code": null
  }
}

Status	Meaning
400	Invalid request (bad model name, malformed JSON)
500	Server error (model failed to load, inference error)

Scheduling priority

A chat request can tell the engine what it is for, so a line a player is waiting to read is not queued behind background work:

{
  "model": "in-memory::my-model",
  "messages": [{ "role": "user", "content": "Hello" }],
  "priority": "interactive"
}

interactive, normal (the default) or background. Optional — omit it and the request behaves exactly as before. See GPU Scheduling for how this combines with the engine-wide device mode, and for the one case it cannot help with: two chats to the same model, which are serialised before the GPU scheduler sees them.

Sampling seed

seed controls the sampler's RNG, so the same prompt can be answered differently on purpose:

{
  "model": "in-memory::my-model",
  "messages": [{ "role": "user", "content": "Greet the traveller." }],
  "temperature": 0.9,
  "seed": 4271
}

Absent, the engine uses a fixed default — identical requests keep producing identical output, which is the behaviour every caller had before this field existed. Supply one when you want variation you control: the same line delivered differently by different NPCs, or a retry that is a genuine second opinion rather than a replay.

Two things worth knowing:

• The same seed reproduces the same completion, so a seed is also how you make a specific generation repeatable — for a bug report, or a test fixture.
• Greedy decoding ignores it. At temperature: 0 there is no draw to seed, so varying the seed changes nothing. That is not a bug; it is what greedy means.