Benchmarks

This page collects measured performance figures for the Atelico AI Engine across LLM decode, prefix caching, GPU sharing, image generation, speech, and audio-to-face.

note

All figures are point-in-time and hardware-specific. They were measured on the GPUs named in each table, with the engine and model versions current as of late May 2026. Run-to-run variance on Apple Silicon is ±2–3% due to thermals, so deltas under ~5% should be read as ties. Numbers will change as the engine, drivers, and models evolve — treat them as a snapshot, not a guarantee.

Methodology

LLM throughput uses the industry-standard pure-decode protocol so every engine measures the same quantity:

• Prompt: BOS-only (empty) prompt.
• Generated tokens: 128.
• Runs: median of 5 measured runs (3 on some larger models / secondary GPUs), after a warmup run.
• Sampling: deterministic, greedy (temperature 0).
• What's timed: decode-loop time only (forward passes plus sampling) — no prefill, no HTTP. Time-to-first-token (TTFT) and prefill rate are reported separately.

This is equivalent to llama-bench -p 0 -n 128 -r 5 and to MLX-LM --max-tokens 128 with an empty prompt. Atelico numbers come from the bench_minimal binary.

Unless a different format is noted next to the model name, all LLM numbers are GGUF Q4_K_M. Exceptions: Qwen3 0.6B is Q8_K_M, and the Bonsai models are 1-bit MLX. For every model, all engines were benchmarked on the same weights for an apples-to-apples comparison.

LLM Decode Throughput

Decode throughput and time-to-first-token for the Atelico AI Engine, measured per model and GPU. TTFT here is the per-request first-token latency reported by bench_minimal; a dash means the metric was not recorded for that run.

Apple Silicon · M1 Max (Metal)

Model	Format	Decode (tok/s)	TTFT (ms)
LLaMA 3.2 1B	GGUF Q4_K_M	252.7	6.3
LLaMA 3.2 3B	GGUF Q4_K_M	116.8	13.4
LLaMA 3.1 8B	GGUF Q4_K_M	61.7	27.2
Qwen3 0.6B	Q8_K_M	249.8	6.9
Qwen3.5 0.8B	GGUF Q4_K_M	164.9	10.3
Qwen3.5 2B	GGUF Q4_K_M	119.0	12.8
Qwen3.5 4B	GGUF Q4_K_M	64.6	23.4
Qwen3.5 9B	GGUF Q4_K_M	48.0	31.3
Gemma4 E2B	GGUF Q4_K_M	90.4	16.8
Gemma4 26B-A4B	GGUF Q4_K_M	49.5	28.0
Nemotron-H 4B	GGUF Q4_K_M	79.4	17.5
Nemotron-3 Elastic 12B	GGUF Q4_K_M	59.2	—
SmolLM3 3B	GGUF Q4_K_M	97.0	15.2
Bonsai 1.7B	MLX 1-bit	245.7	7.0
Bonsai 8B	MLX 1-bit	86.4	15.7

NVIDIA · RTX 3090 (CUDA)

Model	Format	Decode (tok/s)	TTFT (ms)
LLaMA 3.2 1B	GGUF Q4_K_M	613.8	5.8
LLaMA 3.2 3B	GGUF Q4_K_M	281.3	12.6
LLaMA 3.1 8B	GGUF Q4_K_M	160.3	22.1
Qwen3 0.6B	Q8_K_M	496.6	7.9
Qwen3.5 0.8B	GGUF Q4_K_M	435.0	9.6
Qwen3.5 2B	GGUF Q4_K_M	322.6	11.6
Qwen3.5 4B	GGUF Q4_K_M	187.2	19.6
Qwen3.5 9B	GGUF Q4_K_M	123.7	27.7
Gemma4 E2B	GGUF Q4_K_M	263.2	16.2
Gemma4 26B-A4B	GGUF Q4_K_M	132.4	9.9
Nemotron-H 4B	GGUF Q4_K_M	225.0	5.1
Nemotron-3 Elastic 12B	GGUF Q4_K_M	219.7	—
SmolLM3 3B	GGUF Q4_K_M	266.2	13.6
Bonsai 1.7B	MLX 1-bit	202.2	15.3
Bonsai 8B	MLX 1-bit	60.7	49.3

NVIDIA · RTX 4090 (CUDA)

These rows were measured with a 3-run protocol and do not include TTFT.

Model	Format	Decode (tok/s)
LLaMA 3.2 1B	GGUF Q4_K_M	391.1
LLaMA 3.2 3B	GGUF Q4_K_M	192.9
LLaMA 3.1 8B	GGUF Q4_K_M	119.4
Qwen3 0.6B	Q8_K_M	358.6
Qwen3.5 0.8B	GGUF Q4_K_M	250.5
Qwen3.5 2B	GGUF Q4_K_M	223.8
Qwen3.5 4B	GGUF Q4_K_M	151.3
Qwen3.5 9B	GGUF Q4_K_M	111.4
Gemma4 E2B	GGUF Q4_K_M	259.8
Gemma4 26B-A4B	GGUF Q4_K_M	100.5
Nemotron-H 4B	GGUF Q4_K_M	224.1
SmolLM3 3B	GGUF Q4_K_M	218.6
Bonsai 1.7B	MLX 1-bit	220.1

NVIDIA · RTX 5080 (CUDA)

These rows were measured with a 3-run protocol and do not include TTFT.

Model	Format	Decode (tok/s)
LLaMA 3.2 1B	GGUF Q4_K_M	331.7
LLaMA 3.2 3B	GGUF Q4_K_M	166.7
LLaMA 3.1 8B	GGUF Q4_K_M	105.4
Qwen3 0.6B	Q8_K_M	316.7
Qwen3.5 0.8B	GGUF Q4_K_M	197.1
Qwen3.5 2B	GGUF Q4_K_M	178.5
Qwen3.5 4B	GGUF Q4_K_M	135.4
Qwen3.5 9B	GGUF Q4_K_M	104.8
Gemma4 E2B	GGUF Q4_K_M	247.8
Nemotron-H 4B	GGUF Q4_K_M	213.8
Bonsai 1.7B	MLX 1-bit	178.0

NVIDIA · RTX 2080 Ti (CUDA)

These rows were measured with a 3-run protocol and do not include TTFT.

Model	Format	Decode (tok/s)
LLaMA 3.2 1B	GGUF Q4_K_M	389.3
LLaMA 3.2 3B	GGUF Q4_K_M	177.5
LLaMA 3.1 8B	GGUF Q4_K_M	91.7
Qwen3 0.6B	Q8_K_M	325.1
Qwen3.5 0.8B	GGUF Q4_K_M	340.3
Qwen3.5 2B	GGUF Q4_K_M	236.1
Qwen3.5 4B	GGUF Q4_K_M	122.8
Qwen3.5 9B	GGUF Q4_K_M	81.3
Gemma4 E2B	GGUF Q4_K_M	166.1
Nemotron-H 4B	GGUF Q4_K_M	153.9
Bonsai 1.7B	MLX 1-bit	123.8

Vs. Other Inference Engines

Decode throughput (tok/s, higher is better) for the same model, weights, and protocol across serving engines. n/a means the engine cannot run that (model, backend) combination (for example, MLX has no pre-quantized repo for the Qwen 3.5 family, and vLLM does not run on Metal). The fastest value in each row is in bold.

A note on vLLM: it is built for high-throughput batched server deployments. The single-user, batch-1, real-time scenario these tables measure is its weakest mode by design, which is why its numbers are lower here.

Apple Silicon · M1 Max (Metal)

Model	Atelico AI Engine	llama.cpp	Ollama	MLX
LLaMA 3.2 1B	252.7	194.3	166.5	314.7
LLaMA 3.2 3B	116.8	77.2	77.6	145.5
LLaMA 3.1 8B	61.7	29.8	30.2	69.5
Qwen3 0.6B (Q8_K_M)	249.8	150.6	191.8	n/a
Qwen3.5 0.8B	164.9	154.3	70.9	n/a
Qwen3.5 2B	119.0	97.9	69.1	n/a
Qwen3.5 4B	64.6	41.9	30.7	n/a
Qwen3.5 9B	48.0	26.8	24.3	n/a
Gemma 4 E2B	90.4	82.7	67.0	n/a
Gemma 4 26B-A4B	49.5	48.4	43.2	n/a
Nemotron-H 4B	79.4	45.1	48.3	n/a
SmolLM 3 3B	97.0	75.6	91.8	n/a

Bonsai 1.7B (245.7) and Bonsai 8B (86.4) run only on the Atelico AI Engine on this backend; the other engines have no 1-bit MLX path.

NVIDIA · RTX 3090 (CUDA)

Model	Atelico AI Engine	llama.cpp	Ollama	vLLM
LLaMA 3.2 1B	613.8	539.3	602.7	181.6
LLaMA 3.2 3B	274.2	252.3	266.7	102.0
LLaMA 3.1 8B	145.4	139.2	147.1	51.0
Qwen3 0.6B (Q8_K_M)	496.6	434.5	434.3	n/a
Qwen3.5 0.8B	435.0	393.0	308.3	n/a
Qwen3.5 2B	322.6	298.3	222.8	n/a
Qwen3.5 4B	187.2	165.5	139.9	n/a
Qwen3.5 9B	123.7	117.3	99.6	n/a
Gemma 4 E2B	263.2	185.6	3.6	n/a
Gemma 4 26B-A4B	131.1	129.6	111.1	n/a
Nemotron-H 4B	225.0	199.5	209.8	n/a
SmolLM 3 3B	266.2	242.0	255.6	n/a

Bonsai 1.7B (202.2) and Bonsai 8B (60.7) run only on the Atelico AI Engine on this backend.

Prefix-Cache Speed-up

Multi-turn conversations reuse the prompt prefix from one turn to the next. The engine's automatic token-replay prefix cache replays the cached KV state for a byte-identical prefix instead of recomputing it, turning time-to-first-token from a function of the whole conversation into a function of just the new tail.

These figures are the mean TTFT across 5 turns of a character-dialogue (LotR Q&A) workload using Qwen3 0.6B Q8_0, at three prefix sizes. Cold = no cache (the persona is re-prefilled every turn); Warm = the persona is reused from the previous turn.

Apple Silicon · M1 Max (Metal)

Prefix size	~Tokens	Cold TTFT (ms)	Warm TTFT (ms)	Speed-up
Short	115	85	63	1.3×
Medium	1,060	347	83	4.2×
Long	1,981	632	175	3.6×

NVIDIA · RTX 3090 (CUDA)

Prefix size	~Tokens	Cold TTFT (ms)	Warm TTFT (ms)	Speed-up
Short	115	58	56	1.0×
Medium	1,060	125	72	1.7×
Long	1,981	195	93	2.1×

The win grows with the cached prefix length and is marginal at ~100 tokens, where constant per-request overhead dominates both bars. The NVIDIA ratios are smaller than Apple Silicon's at the same size because the RTX 3090 prefills so quickly in absolute terms that the fixed per-request overhead is a larger share of warm TTFT.

GPU Sharing / Frame-Time Impact

When inference runs in-process with a game renderer, the Atelico Compute Graphics Scheduler (CGS) schedules inference and rendering on a single shared hardware context instead of letting the OS context-switch between them. The table below renders two canonical test scenes (Sponza and San Miguel, both at 4K) five ways: the game alone, the three CGS scheduling modes, and an "Other" baseline where the engine runs in a separate CUDA context and contends with the game for the GPU.

All measurements are on an RTX 5080, with a LLaMA 3.2 3B inference workload running during render. Values are mean FPS, with the 1% low in parentheses.

Scene	Game only	CGS PrioritizeGraphics	CGS Balance	CGS PrioritizeCompute	Other (separate context)
Sponza @ 4K	6211 (1984)	5952 (1894)	6061 (1946)	4854 (1667)	531 (365)
San Miguel @ 4K	123 (108)	122 (108)	121 (108)	115 (59)	3 (2)

The three CGS modes keep the game at or near native FPS. Without CGS (right-most column), the WDDM driver context-switches between graphics and inference and the game collapses — to single-digit FPS on the heavy San Miguel scene.

Image Generation

Atelico IG is a small on-device text-to-image model (0.7B). The two FLUX rows are end-to-end wall-clock medians of 5 runs against Replicate's hosted endpoints (they include queue + inference + network round-trip and are not a kernel-for-kernel comparison); they are shown for context only.

Model	Backend / GPU	Image size	Steps	Seconds/image
Atelico IG	Metal · M1 Max	512×512	1	1.0
Atelico IG	Metal · M4	512×512	1	1.0
Atelico IG	Metal · iPhone 13	512×512	1	4.0
Atelico IG	CUDA · RTX 2080 Ti	1024×1024	1	0.60
Atelico IG	CUDA · RTX 3090	1024×1024	1	0.39
Atelico IG	CUDA · RTX 5080	1024×1024	1	0.27
FLUX schnell	Cloud (Replicate)	1024×1024	4	0.77
FLUX.2 [pro]	Cloud (Replicate)	1024×1024	50	10.76

Speech-to-Text (Whisper)

Realtime factor (RTF, higher is better) and word error rate (WER) per Whisper model size. RTF is the inverse of how long the engine takes to transcribe one second of audio, so a value of 30× means ~33 ms of compute per second of speech.

Apple Silicon · M1 Max (Metal)

Model	RTF	WER (%)
whisper-tiny.en	33.3×	3.2
whisper-base.en	20.0×	1.8
whisper-small.en	9.1×	3.0
distil-whisper-large-v3	7.1×	3.2
whisper-large-v3-turbo	6.3×	4.3
whisper-large-v3	2.2×	1.8

NVIDIA · RTX 3090 (CUDA)

Model	RTF	WER (%)
whisper-tiny.en	100.0×	0.7
whisper-base.en	100.0×	0.7
whisper-small.en	33.3×	1.8
whisper-large-v3-turbo	33.3×	1.8
distil-whisper-large-v3	33.3×	0.7
whisper-large-v3	11.1×	0.7

Text-to-Speech

Realtime factor (higher is better) for the two on-device TTS engines: Kokoro (high-quality neural, 54 voices) and Atelico TTS (lightweight, instant voice cloning).

Engine	Voice	Backend / GPU	RTF
Kokoro	af_heart	Metal · M1 Max	5.5×
Atelico TTS	alba	Metal · M1 Max	18.6×
Kokoro	af_heart	CUDA · RTX 3090	10.3×
Atelico TTS	alba	CUDA · RTX 3090	13.0×

Audio-to-Face

Audio-to-Face turns each chunk of spoken audio into 52 ARKit blendshape weights to drive a character's face. Cold first-frame is the delay from the first audio sample to the first blendshape output on a fresh load; warm first-frame is the same delay once the model is hot in memory; steady-state per-frame is the recurring cost while the character keeps talking.

Stage	Metal · M1 Max (ms)	CUDA · RTX 3090 (ms)
First-frame latency (cold)	121.6	30.6
First-frame latency (warm)	62.5	15.9
Steady-state per frame	2.63	0.95