Unreal: Embeddings & Search

Embeddings, vector search, and hybrid search in Unreal Engine. For the concepts behind each step see the guide pages on Embedding Models, Vector Search, and Hybrid & Lexical Search.

KV-store entries and queries are JSON strings; the store takes vectors you produce with EmbedText, not raw text. embed_dim must match the model — 384 for MiniLM/bge-small, 768 for bge-base.

:::info Changed in 0.12.1 Embed(...) is now EmbedText(...), alongside new EmbedImage / EmbedAudio; ClassifierPredict(...) is now ClassifierPredictText(...), alongside ClassifierPredictImage / ClassifierPredictAudio. All are BlueprintCallable. There are no compatibility aliases — rename any 0.12.0 calls. :::

Embed text

const FString Model = TEXT("in-memory::sentence-transformers/all-MiniLM-L6-v2");

const FString Resp = Atelico->EmbedText(FString::Printf(TEXT(R"({
    "model": "%s",
    "input": ["The guard saw a thief enter the keep at midnight."]
})"), *Model));
// Parse Resp -> data[0].embedding into a TArray<float>.

// Or a direct cosine similarity between two strings:
float Sim = Atelico->Similarity(Model, TEXT("a knight's blade"), TEXT("a warrior's sword"));

Create a store, insert, query

// 1. Create
Atelico->CreateKvStore(TEXT(R"({
    "store_id": "npc-memory",
    "db_path": "./data/npc-memory",
    "embed_dim": 384
})"));

// 2. Insert — key_embedding is the vector from Embed (EmbeddingCsv = comma-joined floats)
const FString Entries = FString::Printf(TEXT(R"([{
    "id": "mem-001",
    "key_text": "The guard saw a thief enter the keep at midnight.",
    "key_embedding": [%s],
    "facets": {"npc_id": "guard-01", "location": "keep"},
    "priority": 0.8
}])"), *EmbeddingCsv);
Atelico->KvStoreInsert(TEXT("npc-memory"), Entries);

// 3. Query — embed the question first, pass the vector
const FString Query = FString::Printf(TEXT(R"({
    "query_embedding": [%s],
    "facet_filters": {"npc_id": "guard-01"},
    "limit": 5
})"), *QueryEmbeddingCsv);
FString Results = Atelico->KvStoreQuery(TEXT("npc-memory"), Query);
// Results: JSON array of {id, key_text, similarity, priority, combined_score}

uint64 Count = Atelico->KvStoreCount(TEXT("npc-memory"), TEXT(""));

Hybrid (vector + keyword) search

Atelico->KvStoreCreateFtsIndex(TEXT("npc-memory"), TEXT("key_text"));

const FString Hybrid = FString::Printf(TEXT(R"({
    "embeddings": [%s],
    "text": "thief keep",
    "fts_column": "key_text",
    "limit": 5,
    "vector_weight": 0.6,
    "lexical_weight": 0.4
})"), *QueryEmbeddingCsv);
FString Response = Atelico->KvStoreHybridQuery(TEXT("npc-memory"), Hybrid);

See Hybrid & Lexical Search for the scoring details and the lexical-only (KvStoreLexicalQuery) variant.

Embed images and audio

EmbedImage and EmbedAudio take a base64-encoded file in the request and return the same response shape as EmbedText. With a cross-modal model id the image/audio vectors share the text space.

// Image — model ids: "clip", "siglip", "dinov2-small", "dinov2-large"
const FString ImgResp = Atelico->EmbedImage(FString::Printf(TEXT(R"({
    "model": "clip",
    "image": "%s"
})"), *ImageBase64));

// Audio — model id "clap". WAV / RIFF only (not MP3/FLAC).
const FString AudResp = Atelico->EmbedAudio(FString::Printf(TEXT(R"({
    "model": "clap",
    "audio": "%s"
})"), *AudioBase64));
// Parse resp -> data[0].embedding into a TArray<float>.

EmbedImage / EmbedAudio read the model from the on-device cache and never launch a silent blocking download — gate with is_model_cached(ModelId) and model_download_async(ModelId) first. See Multimodal Embedding for the model table.

Classify text, images, and audio

ClassifierLoad registers a trained classifier; the ClassifierPredict* calls return a JSON string with a top label, its probability, and a top list of the top-k {label, probability} pairs.

Atelico->ClassifierLoad(TEXT("steering"), TEXT("/abs/path/to/models/steering"));

// top_k is optional: omit for all labels ranked, 0 for just the top label
// (empty `top` list), N for the N best.
const FString TextResult = Atelico->ClassifierPredictText(TEXT(R"({
    "model_id": "sentiment", "text": "I love this!", "top_k": 3
})"));

// Image classifier over DINOv2-Small embeddings (labels e.g. "left"/"right").
const FString ImageResult = Atelico->ClassifierPredictImage(TEXT(R"({
    "model_id": "steering", "image_path": "/abs/path/frame.png", "top_k": 0
})"));

// Audio classifier — WAV / RIFF input only.
const FString AudioResult = Atelico->ClassifierPredictAudio(TEXT(R"({
    "model_id": "ambience", "audio_path": "/abs/path/clip.wav", "top_k": 3
})"));

The image/audio embedder the classifier was trained with ("dinov2-small", "clap") must be cached first. See Classifiers for training and the full result schema.