Audio-to-Face (A2F)

Audio-to-Face turns speech into facial animation: feed it a voice clip and it emits ARKit-52 facial blendshape weights at 60 fps that drive a rigged character's lips, jaw, brows, cheeks, and eyes — lip-sync without an animator in the loop.

It's a Candle port of NVIDIA's Audio2Face-3D pipeline: a HuBERT speech encoder feeds a diffusion network that emits the 52 ARKit blendshape trajectories. Three built-in character identities are bundled — Claire, James, and Mark — each trained on a different speaker, so picking the closest match to your TTS voice gives the cleanest sync. The model bundle (~741 MB) downloads from HuggingFace on first use and is cached for subsequent offline runs.

:::note Pairs naturally with TTS A2F takes raw speech PCM, so it slots directly behind text-to-speech: synthesize a line with Audio (TTS), decode the WAV to mono PCM, and feed that PCM to A2F to animate the character saying it. :::

Pipeline

text  ──TTS──▶  speech WAV  ──decode──▶  mono PCM f32  ──A2F──▶  blendshape frames (60 fps)
                (or any voice clip)                              [52 weights] per frame

1. Get speech audio — synthesize it with TTS, or use any recorded voice clip.
2. Decode it to mono PCM f32 samples plus the sample rate.
3. Run A2F. It returns one frame per 1/60 s; each frame is a vector of 52 blendshape weights in [0, 1], aligned channel-for-channel to the 52 canonical ARKit names (jaw, lips, brows, cheeks, eyes, tongue).
4. Send the per-frame weights to your character rig, or interpolate to the rig's native frame rate.

Output format

Each frame carries a timestamp and the 52 ARKit weights in canonical ARFaceAnchor.BlendShapeLocation order. A few representative channels so you know the shape of the output:

Index	Name	Region
0	eyeBlinkLeft	Eyes
7	eyeBlinkRight	Eyes
17	jawOpen	Jaw
18	mouthClose	Mouth
23	mouthSmileLeft	Mouth
24	mouthSmileRight	Mouth
42	browInnerUp	Brow
45	cheekPuff	Cheek
51	tongueOut	Tongue

(52 channels total; the full ordered list is the ARKit-52 set — eyeBlinkLeft … tongueOut.)

Usage

Audio-to-Face is a standalone handle — it is not a method on the Engine. You create an A2F instance, process PCM through it, and dispose of it. The three characters are addressed by id: Claire (0), James (1), Mark (2).

Rust SDK

use atelico_sdk::a2f::{A2f, A2fCharacter};

// Load the A2F model bundle (downloads ~741 MB on first use).
let a2f = A2f::load()?;

// `pcm` is mono f32 samples; pass the sample rate so A2F can resample.
let frames = a2f.process(&pcm, 24_000, A2fCharacter::Claire, None)?;

for frame in &frames {
    // frame.weights is [f32; 52] — one weight per ARKit blendshape.
    let jaw_open = frame.weights[17];
    let smile_l  = frame.weights[23];
    // drive your rig at frame.time_seconds ...
}

The fourth argument is an optional noise seed (None lets A2F pick one); passing a fixed seed makes the output deterministic.

C FFI

// Create the A2F handle (loads the model bundle).
AtelicoA2f* a2f = atelico_a2f_create();

// Process mono f32 PCM. character: 0 = Claire, 1 = James, 2 = Mark.
// noise_seed = UINT64_MAX lets A2F choose; pass a fixed value for determinism.
const char* out_json = NULL;
atelico_a2f_process(a2f,
    pcm, pcm_len, sample_rate,
    /* character  */ 0,
    /* noise_seed */ UINT64_MAX,
    &out_json);
// out_json: [{"time_seconds":0.0,"weights":[52 floats]},
//            {"time_seconds":0.0166,"weights":[...]}, ...]

atelico_a2f_destroy(a2f);

The output pointer is owned by the FFI layer's thread-local return buffer — copy it before the next API call on the same thread.

Python

import json
from atelico import A2f

# Create the A2F handle (loads the model bundle on first use).
a2f = A2f()

# pcm is a list of mono f32 samples; character is "claire" | "james" | "mark".
frames = json.loads(a2f.process(pcm_list, 24_000, "claire"))

for frame in frames:
    t = frame["time_seconds"]
    weights = frame["weights"]        # 52 floats in [0, 1]
    jaw_open = weights[17]
    # drive your rig at time t ...

Example: TTS → A2F lip-sync

A typical pipeline synthesizes a line, decodes it to PCM, and animates a character speaking it:

import base64, json, io, wave, struct
from atelico import Engine, A2f

engine = Engine()
a2f = A2f()

# 1. Synthesize a line of dialogue (see the Audio guide for TTS details).
resp = json.loads(engine.audio_synthesize(json.dumps({
    "model": "in-memory::tts",
    "input": "Welcome to the keep, traveller.",
    "voice": "af_heart",
})))
wav_bytes = base64.b64decode(resp["audio_b64"])

# 2. Decode the WAV to mono f32 PCM.
w = wave.open(io.BytesIO(wav_bytes), "rb")
sr = w.getframerate()
raw = w.readframes(w.getnframes())
ints = struct.unpack("<%dh" % (len(raw) // 2), raw)
pcm = [s / 32768.0 for s in ints]      # mono f32 in [-1, 1]

# 3. Drive A2F with the speech PCM.
frames = json.loads(a2f.process(pcm, sr, "claire"))
print(f"{len(frames)} blendshape frames @ 60 fps")
# Stream frames[i]["weights"] (52 values) to your character rig.

The demos/audio-to-face desktop app is the recommended starting point for wiring A2F into a game engine: it combines TTS + A2F with a live ARKit-52 dashboard and a 3D head reconstruction, so you can see the blendshapes drive a face before hooking them into a character rig.

See also

• Audio (TTS & STT) — synthesize the speech that drives A2F
• Multimodal Embedding — another audio-driven capability, for cross-modal retrieval