Skip to main content
Version: 0.12

Multimodal Embedding

Ordinary embedding models turn text into vectors. A cross-modal (multimodal) model goes further: it embeds text, images, and audio into one shared vector space. Because every modality lands in the same space, a text query can retrieve images, an image can retrieve audio, and any modality can retrieve any other — the nearest neighbours by cosine similarity are the retrieval results.

This is what powers "search my screenshots by description", "find the sound effect that matches this caption", or "given this image, find similar audio". You embed your corpus once (images, clips, captions — whatever you have), embed the query, and rank by cosine similarity in the shared space.

:::note Same model on both sides Vectors are only comparable when produced by the same cross-modal model. A CLIP image vector and a CLIP text vector share a space; a CLIP image vector and a CLAP audio vector do not. Pick one model that covers all the modalities you need to compare, and use its id for every side of the query. :::

Models​

Each model covers a different set of modalities and produces a different shared-space dimension. Pick the smallest model that spans the modalities you need to retrieve across.

Modeltextimageaudiovideo3DShared dim
CLIP✓✓512
SigLIP✓✓768
CLAP✓✓512
ImageBind✓✓✓1024
EBind✓✓✓✓✓1024
DINOv2✓image-only

Notes:

  • CLIP / SigLIP — text ↔ image. The workhorses for caption-to-image and image-to-image retrieval.
  • CLAP — text ↔ audio. Caption-to-sound and audio-to-audio retrieval.
  • ImageBind — tri-modal (text ↔ image ↔ audio) in one 1024-d space, including the image ↔ audio direction that CLIP + CLAP can't bridge on their own.
  • EBind — the most complete self-hostable joint embedder: five modalities (text + image + audio + video + 3D point cloud) in one 1024-d space.
  • DINOv2 — an image-only vision encoder (no text tower). Use it for pure image ↔ image similarity where you don't need text queries; it produces strong visual features but cannot embed a text caption into the same space.

Weights download from HuggingFace on first use and are cached for subsequent offline runs.

Model ids​

ModalityModel ids
Image"clip", "siglip", "dinov2-small", "dinov2-large"
Audio"clap"
Textuse the same cross-modal model's id (e.g. "clip" for CLIP)

The text side of a query uses the cross-modal model's own id — text embedded with "clip" lands in CLIP's space and is comparable to images embedded with "clip".

Usage​

Text, image, and audio embedding are exposed from every SDK (Godot, Unity, Unreal, Python, Lua) as well as the Rust SDK and C FFI. The three calls are embed_text, embed_image, and embed_audio; point each at the cross-modal model's id so all sides share a space.

:::info Changed in 0.12.1 The text-embedding call was renamed from embed(...) to embed_text(...) to sit alongside embed_image(...) / embed_audio(...) (e.g. Python engine.embed → engine.embed_text, C FFI atelico_embed → atelico_embed_text). There are no compatibility aliases. Godot and the other plugin SDKs also expose a high-level AtelicoEmbedderNode / embedder object whose embed_text / embed_image / embed_audio take a string or a file path and return a plain float vector (Godot PackedFloat32Array). :::

:::caution Audio format: WAV / RIFF only embed_audio decodes WAV / RIFF audio only — MP3, FLAC, and other compressed formats are not supported. Convert to WAV first. :::

:::note Weights download on first use, never silently mid-call embed_image / embed_audio resolve the model from the on-device cache that model_download_async populates. They do not kick off a blocking download when the weights are absent — the call fails / returns empty so you gate. Check with is_model_cached(model_id) and pre-fetch with model_download_async(model_id) before embedding. :::

use atelico_sdk::{
    Engine, EmbeddingRequest, ImageEmbeddingRequest, AudioEmbeddingRequest,
};

let engine = Engine::new()?;

// Embed an image into CLIP's shared space (image bytes as base64 PNG):
let img = engine.embeddings().embed_image_sync(ImageEmbeddingRequest {
    model: "clip".into(),
    image: base64_png,            // base64-encoded PNG
})?;
let image_vec = &img.data[0].embedding;   // 512-d (CLIP)

// Embed an audio clip into CLAP's shared space (base64 WAV):
let aud = engine.embeddings().embed_audio_sync(AudioEmbeddingRequest {
    model: "clap".into(),
    audio: base64_wav,            // base64-encoded WAV
})?;
let audio_vec = &aud.data[0].embedding;   // 512-d (CLAP)

// Embed the text query side with the SAME cross-modal model id so it shares
// the image space:
let q = engine.embeddings().embed_sync(EmbeddingRequest {
    model: "clip".into(),
    input: vec!["a red sports car at sunset".into()],
    ..Default::default()
})?;
let query_vec = &q.data[0].embedding;     // comparable to image_vec above

:::note No HTTP route for image/audio embedding Today, image and audio embedding are library/SDK-level only (Rust SDK, C FFI, Python, Godot, Unity, Unreal). There is no HTTP /v1/embeddings/image or /v1/embeddings/audio route. (Text embeddings do have the OpenAI-compatible /v1/embeddings route — see Embedding Models.) :::

Worked example: text → image retrieval​

The whole point of a shared space is retrieval. Embed a corpus of images once, embed a text query, and cosine-rank. With CLIP, both sides use the "clip" id, so the vectors are directly comparable.

import base64, json
import numpy as np
from atelico import Engine

engine = Engine()

def embed_image(path):
    b64 = base64.b64encode(open(path, "rb").read()).decode()
    r = json.loads(engine.embed_image(json.dumps({"model": "clip", "image": b64})))
    return np.array(r["data"][0]["embedding"], dtype=np.float32)

def embed_text(text):
    r = json.loads(engine.embed_text(json.dumps({"model": "clip", "input": [text]})))
    return np.array(r["data"][0]["embedding"], dtype=np.float32)

# 1. Embed the image corpus once.
corpus = ["car.png", "beach.png", "cat.png", "mountain.png"]
vectors = np.stack([embed_image(p) for p in corpus])

# 2. Embed the text query.
query = embed_text("a red sports car at sunset")

# 3. Cosine-rank (CLIP/CLAP vectors are L2-normalized, so a dot product is cosine).
scores = vectors @ query
order = np.argsort(scores)[::-1]
for i in order:
    print(f"{corpus[i]:14s}  {scores[i]:.3f}")
# car.png        0.31
# beach.png      0.18
# ...

The same pattern works any → any within a model: embed audio with CLAP and query it by a text caption, embed images with ImageBind and query them by an audio clip, and so on.

Storing and searching at scale​

For more than a handful of vectors you'll want a persistent index instead of an in-memory matrix. The cross-modal vectors are ordinary float vectors, so they drop straight into the engine's vector store — set the store's embed_dim to the model's shared dimension (512 for CLIP/CLAP, 768 for SigLIP, 1024 for ImageBind/EBind):

The demos/multimodal-retrieval desktop app is a complete reference: it builds one KV store per model, indexes your own images / audio / text, and lets you query by any modality — results span every indexed modality, ranked by cosine similarity.

See also​