LoRA Adapters

LoRA adapters let you specialize a base model at runtime without loading a separate copy. A single 1B–8B base can host many adapters and switch between them in milliseconds.

Two common patterns:

• Per-character adapters — one adapter per NPC, quest line, or language register. The base model handles language; the adapter shapes voice, style, and personality.
• Per-task adapters — one adapter per cognitive task: planning, decision-making, dialogue generation, summarization, intent classification. A single base model becomes a multi-skill backbone, with the active adapter selected per request based on what the game loop is currently asking for. Behaviour frameworks built on Atelico typically route a planner adapter, a decider adapter, and a dialogue adapter in turn during one NPC tick.

Atelico supports three on-disk adapter formats:

• Standard LoRA — the original Low-Rank Adaptation format. Each layer ships two factor matrices A and B (low-rank decomposition of the weight delta).
• LoRA-XS (arXiv:2405.17604) — inserts a small trainable r×r matrix R between two frozen SVD-derived factors of the base weight. Only R is trained, so the per-layer parameter count drops from r·(d_in + d_out) to just r² (256 floats per layer at rank 16, regardless of model size). On-disk adapters can be ~100× smaller than standard LoRA at comparable quality.
• Turbo LoRA — a standard LoRA adapter bundled with a stack of Medusa-style draft heads, trained jointly on one narrow task. The LoRA fits the task; the draft heads predict several future tokens at once so the model can decode them in a single forward. On structurally predictable outputs (JSON, structured extraction, fixed templates) this can multiply decode throughput. See Turbo LoRA below.

All three formats use the same load/unload/scale API — the engine auto-detects which one you've handed it.

Loading an Adapter

LoRA management lives in the SDK layer — every binding (Godot, Unity, Unreal, Python, C FFI, Rust SDK) exposes the same three operations: load, set scale, unload. The adapter directory must contain an adapter_config.json (HuggingFace PEFT layout) and the safetensors weight files.

Python example:

engine.lora_load(
    "in-memory::meta-llama/Llama-3.2-1B-Instruct-Q4_K_M",
    "/path/to/pirate-lora",
)

engine.lora_set_scale(
    "in-memory::meta-llama/Llama-3.2-1B-Instruct-Q4_K_M",
    0.7,  # blend at 70% strength
)

engine.lora_unload("in-memory::meta-llama/Llama-3.2-1B-Instruct-Q4_K_M")

Once loaded, request the adapted variant by appending ::adapter-name to the model id in your chat completion call (the adapter name is taken from the adapter_config.json metadata):

response = engine.chat_completion(
    model="in-memory::meta-llama/Llama-3.2-1B-Instruct-Q4_K_M::pirate",
    messages=[{"role": "user", "content": "Greet the tavern."}],
)

See the per-SDK API references for the exact method signatures in each language.

Adapter Format Detection

The engine reads adapter_config.json and scans the safetensors keys to decide which format you're loading:

1. If adapter_config.json sets peft_type to "LORA_XS" (case-insensitive), the adapter is treated as LoRA-XS.
2. Otherwise, the engine inspects the safetensors keys. Any tensor ending in .lora_R.weight (or the reference-implementation alias .default_lora_latent_mapping.weight) marks the adapter as LoRA-XS.
3. If neither signal is present, the adapter is treated as standard LoRA.

A Turbo LoRA adapter is a standard LoRA whose safetensors additionally carries medusa_head.* draft-head tensors. The engine detects those keys automatically while applying the LoRA delta and attaches the head stack — no separate flag or format marker is needed.

No flag, environment variable, or API change is required to use LoRA-XS or Turbo LoRA — drop the adapter into your asset store and load it like any other.

LoRA-XS On-Disk Variants

Two on-disk layouts are accepted for LoRA-XS adapters:

Variant	Tensors per layer	Notes
A (full)	lora_A, lora_B, lora_R	Self-contained. Matches the reference implementation.
B (R-only)	lora_R only	Smallest on-disk size. A and B are reconstructed at load time from the base weight via a deterministic randomized truncated SVD. The reconstruction is cached per (model, rank), so subsequent adapter swaps of the same rank are instant.

Variant B is the recommended layout for shipped game assets — a rank-16 R-only adapter on an 8B base model is well under 100 KB per layer.

Runtime Behavior

• One adapter is active per base model at a time. Loading a second adapter onto the same model replaces the first.
• lora_set_scale(model, α) blends the adapter at strength α (0.0 = base model only, 1.0 = full adapter, intermediate values interpolate).
• Switching adapters is in-memory; there is no GPU model reload.
• Inference throughput with an active adapter is within a few percent of the base model; the adapter forward path is a single fused matmul per attached projection.

Turbo LoRA

Turbo LoRA is a per-task fine-tuning recipe (from Predibase) that combines two existing techniques in a single adapter:

• A standard LoRA adapter (applied to the q/k/v/o/gate/up/down projections) fits the base model to one narrow task.
• A small stack of Medusa-style draft heads (Cai et al. 2024) — trained jointly with the LoRA — predicts several future tokens in parallel from a single hidden state.

Unlike a standard LoRA, which only changes what the model says, a Turbo LoRA also changes how fast it decodes. At inference the heads draft K-1 speculative tokens past the trunk's own next-token prediction, and a single batched verify forward decides which to accept. When the output is structurally predictable (JSON, structured extraction, fixed templates, code) the heads accept at a high rate and one multi-token forward emits several tokens of progress.

Turbo LoRA is per-task, not a general accelerator. The draft heads memorise correlations specific to the task's output distribution; on out-of-distribution prompts the accept rate collapses and you lose the speedup (and on a general-purpose SFT adapter the heads never pay off in the first place). Train and use a Turbo LoRA for the specific narrow task you ship.

Supported targets

Turbo LoRA's draft-head acceleration requires a float (non-quantized) trunk on a non-recurrent architecture:

Architecture	Turbo LoRA draft heads
Llama (float)	Supported
Qwen3 (float)	Supported
Gemma4 (float)	Supported
SmolLM3 (float)	Supported
Quantized / GGUF trunks	Not supported — Turbo LoRA requires a float trunk
Qwen 3.5 / 3.6 (hybrid DeltaNet, recurrent)	Rejected with an error

Loading a Turbo LoRA adapter onto a recurrent Qwen target (Qwen 3.5 / 3.6, which use DeltaNet linear attention) fails loudly rather than silently dropping the heads. On a recurrent trunk the draft heads cannot speed anything up — speculative self-speculation is structurally perf-neutral there (about 1× the autoregressive baseline) and is not bit-identical to autoregressive decode (the recurrent state amplifies tiny floating-point ordering differences). The error directs you to use a plain LoRA adapter (one with no medusa_head.* tensors) on these models, or to run Turbo LoRA on a non-recurrent target.

Training a Turbo LoRA

Training is a joint PyTorch + PEFT step driven by atelico-language/tools/train_turbo_lora.py. It ingests a JSON list of {"instruction", "output"} examples (or a HuggingFace dataset id) and exports a single safetensors file containing both the PEFT-format LoRA tensors and the medusa_head.{k}.linear.{weight,bias} head tensors, plus an adapter_config.json.

python atelico-language/tools/train_turbo_lora.py \
    --model <path-to-hf-model-dir> \
    --dataset <hf-id-or-local-json> \
    --max-samples 5000 --epochs 5 \
    --batch 4 --seq-len 256 \
    --num-heads 5 \
    --rank 16 --alpha 32 \
    --lora-lr 1e-3 --head-lr 1e-3 \
    --warmup-ratio 0.1 --lr-scheduler linear \
    --checkpoint-every 500 \
    --out adapters/my-turbo.safetensors \
    --log-every 50 --dtype bfloat16

Fixed-template outputs win. The instruction part of each example should be a templated wrapper around the variable input, not free-form prose — the heads learn the template trivially and spend their capacity on the output structure. Watch the per-head top1_head_k metric in the training log; if heads don't climb past ~80% on training batches, you won't see speedup at inference no matter how low the loss goes.

A worked end-to-end example — named-entity extraction (text → JSON) on the wikiann dataset — lives in atelico-language/examples/turbo-lora-ner/. It ships a dataset-prep script (prep_dataset.py) and a step-by-step README. The conceptual deep dive (joint loss, speedup math, per-architecture support, and a debug decision tree) is in atelico-language/docs/turbo-lora.md.

Loading and using a Turbo LoRA

A trained Turbo LoRA loads exactly like any other adapter — the same lora_load / lora_set_scale / lora_unload API. Point the loader at a directory containing the exported *.safetensors and its adapter_config.json; the engine detects the medusa_head.* tensors and attaches the draft-head stack automatically.

engine.lora_load(
    "in-memory::meta-llama/Llama-3.2-1B-Instruct",
    "/path/to/ner-turbo-lora",
)

response = engine.chat_completion(
    model="in-memory::meta-llama/Llama-3.2-1B-Instruct::ner",
    messages=[{"role": "user", "content": "Extract entities ..."}],
)

lora_unload removes both the LoRA delta and the attached draft heads. This is the correct teardown when hot-swapping adapters at runtime — it prevents stale heads from staying attached to a newly loaded, incompatible LoRA.

The speedup is hardware-sensitive: CUDA's batched verify forward is cheap relative to a single-token forward, so the multi-token acceptance translates directly into wall-clock gains. On Apple Silicon (Metal) the batched matmul is inefficient at small batch sizes, so even at a high accept rate the wall-clock gain is limited. Bench Turbo LoRA on CUDA to see its real numbers; the adapter and training are platform-agnostic.

See Also

• Models — list of supported base architectures
• NPC Dialogue — using per-character adapters to drive NPC personality