Unity: Embeddings & Search

Embeddings, vector search, and hybrid search in Unity. For the concepts behind each step see the guide pages on Embedding Models, Vector Search, and Hybrid & Lexical Search.

KV-store entries and queries are JSON strings; the engine takes vectors you produce with Embeddings, not raw text. embed_dim must match the model — 384 for MiniLM/bge-small, 768 for bge-base.

:::info Changed in 0.12.1 Embeddings.Embed(...) is now Embeddings.EmbedText(...), alongside new EmbedImage / EmbedAudio; Classifiers.Predict(...) is now Classifiers.PredictText(...), alongside PredictImage / PredictAudio. There are no compatibility aliases — rename any 0.12.0 calls. :::

Embed text

const string MODEL = "in-memory::sentence-transformers/all-MiniLM-L6-v2";

string resp = engine.Embeddings.EmbedText(JsonSerializer.Serialize(new {
    model = MODEL,
    input = new[] { "The guard saw a thief enter the keep at midnight." },
}));
// Parse resp -> data[0].embedding into a float[].

// Or a direct cosine similarity between two strings:
float sim = engine.Embeddings.Similarity(MODEL, "a knight's blade", "a warrior's sword");

Create a store, insert, query

// 1. Create
ulong storeId = engine.KvStore.Create(JsonSerializer.Serialize(new {
    store_id = "npc-memory",
    db_path  = "./data/npc-memory",
    embed_dim = 384,
}));

// 2. Insert — key_embedding is the float[] from Embeddings.Embed
engine.KvStore.Insert("npc-memory", JsonSerializer.Serialize(new[] { new {
    id = "mem-001",
    key_text = "The guard saw a thief enter the keep at midnight.",
    key_embedding = vector,                 // float[]
    facets = new { npc_id = "guard-01", location = "keep" },
    priority = 0.8f,
}}));

// 3. Query — embed the question first, pass the vector
string results = engine.KvStore.Query("npc-memory", JsonSerializer.Serialize(new {
    query_embedding = queryVector,          // float[]
    facet_filters = new { npc_id = "guard-01" },
    limit = 5,
}));
// results: JSON array of {id, key_text, similarity, priority, combined_score}

ulong n = engine.KvStore.Count("npc-memory", "");

Hybrid (vector + keyword) search

engine.KvStore.CreateFtsIndex("npc-memory", "key_text");

string response = engine.KvStore.HybridQuery("npc-memory", JsonSerializer.Serialize(new {
    embeddings = queryVector,
    text = "thief keep",
    fts_column = "key_text",
    limit = 5,
    vector_weight = 0.6f,
    lexical_weight = 0.4f,
}));

See Hybrid & Lexical Search for the scoring details and the lexical-only (KvStore.LexicalQuery) variant.

Embed images and audio

EmbedImage and EmbedAudio take a base64-encoded file in the request and return the same response shape as EmbedText. With a cross-modal model id the image/audio vectors share the text space.

// Image — model ids: "clip", "siglip", "dinov2-small", "dinov2-large"
string imgResp = engine.Embeddings.EmbedImage(JsonSerializer.Serialize(new {
    model = "clip",
    image = Convert.ToBase64String(File.ReadAllBytes("car.png")),
}));

// Audio — model id "clap". WAV / RIFF only (not MP3/FLAC).
string audResp = engine.Embeddings.EmbedAudio(JsonSerializer.Serialize(new {
    model = "clap",
    audio = Convert.ToBase64String(File.ReadAllBytes("bark.wav")),
}));
// Parse resp -> data[0].embedding into a float[].

EmbedImage / EmbedAudio read the model from the on-device cache and never launch a silent blocking download — gate with is_model_cached(modelId) and model_download_async(modelId) first. See Multimodal Embedding for the model table.

Classify text, images, and audio

Classifiers.Load registers a trained classifier; the Predict* calls return a JSON string with a top label, its probability, and a top list of the top-k {label, probability} pairs.

engine.Classifiers.Load("steering", "/abs/path/to/models/steering");

// top_k is optional: omit for all labels ranked, 0 for just the top label
// (empty `top` list), N for the N best.
string textResult = engine.Classifiers.PredictText(JsonSerializer.Serialize(new {
    model_id = "sentiment", text = "I love this!", top_k = 3,
}));

// Image classifier over DINOv2-Small embeddings (labels e.g. "left"/"right").
string imageResult = engine.Classifiers.PredictImage(JsonSerializer.Serialize(new {
    model_id = "steering", image_path = "/abs/path/frame.png", top_k = 0,
}));

// Audio classifier — WAV / RIFF input only.
string audioResult = engine.Classifiers.PredictAudio(JsonSerializer.Serialize(new {
    model_id = "ambience", audio_path = "/abs/path/clip.wav", top_k = 3,
}));

The image/audio embedder the classifier was trained with ("dinov2-small", "clap") must be cached first. See Classifiers for training and the full result schema.