Chat Completions API

The chat completions endpoint generates responses from a conversation. It's OpenAI-compatible, so if you've used the OpenAI API before, this works the same way.

POST /v1/chat/completions

Basic Request

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  }
}

Request Parameters

Parameter	Type	Default	Description
model	string	required	Model identifier (e.g., in-memory::meta-llama/Llama-3.2-3B-Instruct)
messages	array	required	Conversation messages (see below)
stream	boolean	false	Enable token-by-token streaming
temperature	float	model default	Sampling temperature. Lower = more deterministic, higher = more creative. 0 = greedy.
top_k	integer	model default	Keep only the k most-likely tokens before sampling. 0 disables.
top_p	float	model default	Nucleus sampling: keep the smallest set of tokens whose probabilities sum to p. 1.0 disables.
min_p	float	model default	Keep tokens whose probability is at least min_p × the top token's probability. 0 disables.
repetition_penalty	float	model default	Penalize tokens already in the context. 1.0 = no penalty; higher discourages repetition.
presence_penalty	float	0	OpenAI-style penalty applied once to any token that has appeared.
frequency_penalty	float	0	OpenAI-style penalty scaled by how often a token has appeared.
seed	integer	random	Seed for the sampler. The same seed + params + prompt yields a byte-identical stream.
max_tokens	integer	model default	Maximum tokens to generate
response_format	object	null	Constrain output format (see Structured Generation)
enable_thinking	boolean	null	Toggle the extended-thinking span on models that support it (Qwen 3.5/3.6, SmolLM3). Ignored — with a warning — on other models. See Thinking / Reasoning.

The sampling parameters above (temperature through seed) are all optional — when you omit them, each model falls back to its own recommended defaults. See Sampling below. You rarely need to set them.

Message Roles

Each message in the messages array has a role and content:

Role	Purpose
system	Sets the AI's behavior, personality, or constraints. Placed first.
user	The human's message.
assistant	The AI's previous response. Used for multi-turn context.

System Prompts

System prompts define how the model behaves. They're essential for game NPCs, assistants, or any specialized behavior:

Python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {
            "role": "system",
            "content": "You are a ship AI aboard a deep-space freighter. You speak formally, "
                       "address the player as Captain, and provide status reports when asked. "
                       "You are concerned about a recent anomaly in sector 7.",
        },
        {"role": "user", "content": "Status report."},
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

Godot (GDScript)

func get_ship_ai_response(player_input: String) -> void:
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": [
            {
                "role": "system",
                "content": "You are a ship AI aboard a deep-space freighter. You speak formally, address the player as Captain, and provide status reports when asked. You are concerned about a recent anomaly in sector 7."
            },
            {"role": "user", "content": player_input}
        ],
        "temperature": 0.7
    }
    engine.async_chat_completions(JSON.stringify(request))

Unity (C#)

public async Task<string> GetShipAIResponse(string playerInput)
{
    var request = new
    {
        model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        messages = new object[]
        {
            new { role = "system", content = "You are a ship AI aboard a deep-space freighter. You speak formally, address the player as Captain, and provide status reports when asked." },
            new { role = "user", content = playerInput }
        },
        temperature = 0.7
    };

    var json = JsonSerializer.Serialize(request);
    var content = new StringContent(json, Encoding.UTF8, "application/json");
    var response = await client.PostAsync($"{BaseUrl}/chat/completions", content);
    var responseJson = await response.Content.ReadAsStringAsync();

    using var doc = JsonDocument.Parse(responseJson);
    return doc.RootElement.GetProperty("choices")[0]
        .GetProperty("message").GetProperty("content").GetString();
}

Unreal (C++)

void UAtelicoClient::GetShipAIResponse(const FString& PlayerInput)
{
    TSharedPtr<FJsonObject> Body = MakeShareable(new FJsonObject);
    Body->SetStringField("model", "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M");
    Body->SetNumberField("temperature", 0.7);

    TArray<TSharedPtr<FJsonValue>> Messages;

    TSharedPtr<FJsonObject> SystemMsg = MakeShareable(new FJsonObject);
    SystemMsg->SetStringField("role", "system");
    SystemMsg->SetStringField("content",
        "You are a ship AI aboard a deep-space freighter. You speak formally, "
        "address the player as Captain, and provide status reports when asked.");
    Messages.Add(MakeShareable(new FJsonValueObject(SystemMsg)));

    TSharedPtr<FJsonObject> UserMsg = MakeShareable(new FJsonObject);
    UserMsg->SetStringField("role", "user");
    UserMsg->SetStringField("content", PlayerInput);
    Messages.Add(MakeShareable(new FJsonValueObject(UserMsg)));

    Body->SetArrayField("messages", Messages);
    SendRequest(Body); // see Getting Started for full HTTP setup
}

Multi-Turn Conversations

Include previous messages to give the model context of the conversation:

Python

response = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {"role": "system", "content": "You are a helpful tavern keeper named Boris."},
        {"role": "user", "content": "What do you have on the menu?"},
        {"role": "assistant", "content": "We have roasted boar, mushroom stew, and fresh bread. The stew is my specialty!"},
        {"role": "user", "content": "I'll have the stew. Any rumors lately?"},
    ],
)

print(response.choices[0].message.content)

Godot (GDScript)

# Keep conversation history in an array
var conversation: Array = [
    {"role": "system", "content": "You are a helpful tavern keeper named Boris."}
]

func talk_to_npc(player_input: String) -> void:
    conversation.append({"role": "user", "content": player_input})
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": conversation
    }
    engine.async_chat_completions(JSON.stringify(request))

func _on_async_request_completed(_job_id: int, response: String) -> void:
    var parsed = JSON.parse_string(response)
    var reply = parsed["choices"][0]["message"]["content"]
    conversation.append({"role": "assistant", "content": reply})
    dialogue_label.text = reply

Unity (C#)

private List<object> conversation = new()
{
    new { role = "system", content = "You are a helpful tavern keeper named Boris." }
};

public async Task<string> TalkToNPC(string playerInput)
{
    conversation.Add(new { role = "user", content = playerInput });

    var request = new { model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M", messages = conversation };
    var json = JsonSerializer.Serialize(request);
    var content = new StringContent(json, Encoding.UTF8, "application/json");
    var response = await client.PostAsync($"{BaseUrl}/chat/completions", content);
    var responseJson = await response.Content.ReadAsStringAsync();

    using var doc = JsonDocument.Parse(responseJson);
    var reply = doc.RootElement.GetProperty("choices")[0]
        .GetProperty("message").GetProperty("content").GetString();

    conversation.Add(new { role = "assistant", content = reply });
    return reply;
}

Unreal (C++)

// Store conversation as TArray<TSharedPtr<FJsonValue>>
TArray<TSharedPtr<FJsonValue>> Conversation;

void UAtelicoClient::InitConversation()
{
    TSharedPtr<FJsonObject> SystemMsg = MakeShareable(new FJsonObject);
    SystemMsg->SetStringField("role", "system");
    SystemMsg->SetStringField("content", "You are a helpful tavern keeper named Boris.");
    Conversation.Add(MakeShareable(new FJsonValueObject(SystemMsg)));
}

void UAtelicoClient::TalkToNPC(const FString& PlayerInput)
{
    TSharedPtr<FJsonObject> UserMsg = MakeShareable(new FJsonObject);
    UserMsg->SetStringField("role", "user");
    UserMsg->SetStringField("content", PlayerInput);
    Conversation.Add(MakeShareable(new FJsonValueObject(UserMsg)));

    TSharedPtr<FJsonObject> Body = MakeShareable(new FJsonObject);
    Body->SetStringField("model", "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M");
    Body->SetArrayField("messages", Conversation);

    // On response callback, parse assistant reply and append to Conversation
    SendRequest(Body);
}

Streaming

Set "stream": true to receive tokens as they're generated. This is ideal for typewriter-style dialogue UI.

SSE Format: Each token arrives as a Server-Sent Event. The stream ends with data: [DONE].

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Once"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" upon"},"finish_reason":null}]}
...
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Python

stream = client.chat.completions.create(
    model="in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    messages=[
        {"role": "system", "content": "You are a narrator for a fantasy RPG."},
        {"role": "user", "content": "Describe the entrance to the dungeon."},
    ],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()

Godot (GDScript)

# Streaming uses the built-in signal-based API
func stream_narration(prompt: String) -> void:
    var request = {
        "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        "messages": [
            {"role": "system", "content": "You are a narrator for a fantasy RPG."},
            {"role": "user", "content": prompt}
        ]
    }
    engine.stream_chat_completions(JSON.stringify(request))

# Called once per token as it arrives
func _on_inference_token_generated(_job_id: int, token: String) -> void:
    dialogue_label.text += token  # typewriter effect

func _on_inference_completed(_job_id: int) -> void:
    print("Stream finished")

Unity (C#)

public async Task StreamToDialogue(string prompt, TMPro.TextMeshProUGUI dialogueText)
{
    var request = new
    {
        model = "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
        messages = new[]
        {
            new { role = "system", content = "You are a narrator for a fantasy RPG." },
            new { role = "user", content = prompt }
        },
        stream = true
    };

    var json = JsonSerializer.Serialize(request);
    var httpContent = new StringContent(json, Encoding.UTF8, "application/json");
    var httpRequest = new HttpRequestMessage(HttpMethod.Post, $"{BaseUrl}/chat/completions")
    {
        Content = httpContent
    };

    var response = await client.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead);
    using var stream = await response.Content.ReadAsStreamAsync();
    using var reader = new StreamReader(stream);

    dialogueText.text = "";
    while (await reader.ReadLineAsync() is { } line)
    {
        if (line.StartsWith("data: ") && line != "data: [DONE]")
        {
            var chunk = JsonDocument.Parse(line.Substring(6));
            var delta = chunk.RootElement.GetProperty("choices")[0].GetProperty("delta");
            if (delta.TryGetProperty("content", out var c))
                dialogueText.text += c.GetString();
        }
    }
}

Unreal (C++)

void UAtelicoClient::StreamNarration(const FString& Prompt)
{
    auto Request = FHttpModule::Get().CreateRequest();
    Request->SetURL(TEXT("http://localhost:11434/v1/chat/completions"));
    Request->SetVerb(TEXT("POST"));
    Request->SetHeader(TEXT("Content-Type"), TEXT("application/json"));

    // Build JSON with "stream": true and messages array
    // ... (see Getting Started for JSON building pattern) ...

    // Handle chunked SSE responses via progress callback
    Request->OnRequestProgress().BindLambda(
        [this](FHttpRequestPtr Req, int32 BytesSent, int32 BytesReceived)
    {
        FString Content = Req->GetResponse()->GetContentAsString();
        // Parse new SSE lines since last callback
        // Extract delta.content tokens
        // Append to dialogue UTextBlock
    });

    Request->ProcessRequest();
}

Sampling

Every sampling parameter is optional. Each model comes with recommended defaults tuned for it, so you get good results without setting anything — a bare request like the basic example above already samples the way the model is meant to. Set a parameter only when you want to override the default for a particular request.

Sampling parameters

Parameter	What it does	Optional / Default
temperature	Controls randomness. Lower is more focused and repeatable; higher is more varied and creative. 0 always picks the single most-likely token.	Optional — model default
top_k	Only consider the k most-likely next tokens. Lower values keep output on-topic.	Optional — model default
top_p	Nucleus sampling: only consider the smallest set of tokens whose probabilities add up to p.	Optional — model default
min_p	Ignore tokens less likely than this fraction of the most-likely token. Filters out long-shot tokens.	Optional — model default
repetition_penalty	Discourages repeating words and phrases already in the conversation. 1.0 means no penalty; higher values push harder against repetition.	Optional — model default
presence_penalty	Discourages reusing any token that has already appeared (applied once per token).	Optional — 0
frequency_penalty	Like presence_penalty, but the penalty grows the more often a token has appeared.	Optional — 0
seed	Set it to get reproducible output: the same seed with the same parameters and prompt produces the same result every time.	Optional — random
max_tokens	Caps the length of the response.	Optional — model default

Temperature quick reference

Temperature is the knob you'll reach for most often. For game applications:

Use Case	Temperature	Why
Factual responses, game rules	0.1 - 0.3	Consistent, predictable
NPC dialogue, general conversation	0.6 - 0.8	Natural variation
Creative writing, storytelling	0.9 - 1.2	More surprising, diverse

Examples

Rely on the model's recommended defaults — pass no sampling parameters at all:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [{"role": "user", "content": "Describe a misty harbor at dawn."}]
  }'

Override a couple of parameters for tighter, reproducible output:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "in-memory::meta-llama/Llama-3.2-3B-Instruct-Q4_K_M",
    "messages": [{"role": "user", "content": "Describe a misty harbor at dawn."}],
    "temperature": 0.4,
    "top_p": 0.9,
    "seed": 42
  }'

Finish Reasons

The finish_reason field tells you why generation stopped:

Value	Meaning
stop	Model finished naturally (end of response)
length	Hit the max_tokens limit

length means the output was truncated — common with structured/JSON output. When you see it, raise max_tokens and regenerate; for a best-effort salvage of a cut-off JSON response, see repair_json.

Thinking / Reasoning

Some models support an extended "thinking" (chain-of-thought) phase before the final answer. How a model handles this is a per-model capability, and the enable_thinking request field only has a defined meaning for one group:

Model family	Behavior with enable_thinking
Qwen 3.5, Qwen 3.6, SmolLM3	Honor the flag: true gates a <think>…</think> reasoning span; false skips it.
Gemma 4	Reasons in its own native format, driven by the model/template — not by enable_thinking. Passing true is ignored and can degrade output; pass false.
All other models (Llama, etc.)	No extended-thinking mode. enable_thinking: true is ignored.

If you pass enable_thinking: true to a model that doesn't honor the flag (Gemma 4 or any non-thinking model), the engine logs a warning and runs the request anyway — the flag is simply ignored. The safe default is to omit enable_thinking (or set it false) unless you're targeting a model in the first row.

:::warning Gemma 4 known issue Gemma 4's native reasoning can diverge during long agentic / tool-use decoding. See the Gemma-4 agentic decoding divergence known-issue note. :::

Error Handling

Errors return standard HTTP status codes with an OpenAI-compatible error body:

{
  "error": {
    "message": "Model 'nonexistent-model' not found",
    "type": "invalid_request_error",
    "param": "model",
    "code": null
  }
}

Status	Meaning
400	Invalid request (bad model name, malformed JSON)
500	Server error (model failed to load, inference error)