Why an on-device fallback engine?
Tuplet, our band management app for iOS and Android, ships an AI assistant that band members use to check availability, schedule rehearsals, and manage setlists. Our primary engine is Apple’s FoundationModels framework (Apple Intelligence, iOS 26+), which is excellent — but only on devices that support it. Devices on older iOS versions, ineligible hardware, or users who haven’t opted into Apple Intelligence still deserve an assistant.
So Tuplet runs a second, fully on-device engine backed by MLX Swift and small quantized Qwen3 models. Everything — inference, tool calling, and the ~600 MB to multi-GB model weights — lives on the phone. No inference bills, no server, no band data leaving the device.
This post covers the three pieces that made it work: an engine protocol that keeps both engines swappable, hand-rolled tool calling on top of MLX LM, and model delivery via a self-hosted Cloudflare R2 CDN plus Apple’s Background Assets framework.
One protocol, two engines
The assistant’s ViewModel knows nothing about which model is running. It depends on a four-method protocol:
protocol AssistantEngine {
var availability: AssistantAvailability { get }
func makeSession(bandName: String) async
func send(_ text: String) async throws -> AsyncThrowingStream<AssistantStreamEvent, Error>
func reset()
}
FoundationModelsEngine implements it with LanguageModelSession and owns all the Apple Intelligence availability rules (OS version, hardware eligibility, opt-in state). MLXEngine implements the same protocol with a local model. A single factory function picks the engine — release builds prefer FoundationModels and fall back to MLX — and swapping engines means changing that one function, not the UI.
The payoff of this inversion showed up immediately: every tool behind the assistant is defined once as an engine-agnostic “core” struct, and each engine wires those cores into its own tool-calling mechanism. Adding a tool to both engines is one core plus two small adapters.
Tool calling in MLX Swift, by hand
FoundationModels gives you @Generable schemas and generates tool declarations for you. MLX gives you a model and a chat template — the tool plumbing is yours to build. With MLXLMCommon it comes down to three pieces.
1. A Codable args struct per tool, shared with the engine-agnostic core:
struct SetlistToolCore: Sendable {
struct Args: Codable {
var operation: String?
var bandName: String?
var songId: String?
// ...
}
static let name = "setlist"
static let description = "View or manage the band's master setlist..."
func call(arguments: Args) async -> String { /* ... */ }
}
2. A JSON schema declaration so the chat template advertises the tool to the model:
Tool(
name: SetlistToolCore.name,
description: SetlistToolCore.description,
parameters: [
.required("operation", type: .string,
description: "Operation: list, search, eventFind, add, ..."),
.optional("bandName", type: .string,
description: "Band name. Omit to use the selected band."),
// ...
],
handler: { [setlist] args in
await setlist.call(arguments: args)
}
)
3. A dispatch table handed to ChatSession as toolDispatch. The session runs the full multi-turn loop automatically: the model emits a tool call, the dispatcher decodes the JSON into the typed Args, executes, and feeds the string result back for the next generation.
Two hard-won rules for small models making these calls:
- Make every parameter optional in the schema, validate in the handler. The Qwen3-0.6B model occasionally emits malformed arguments — we watched it call
setlistwith a createEvent-shaped payload (title,startDate,eventType) and nooperationat all. If a required property is missing from the declared schema, the decode throws and the whole turn dies. If everything is optional, the same call decodes fine and the tool returns a usage hint (“operation is required; to show or count songs use ‘list’”), which the model reads and self-corrects from in the same conversation. - Feed the model answers, not chores. When asked “how many songs are in the setlist?”, a 0.6B model won’t count. But if the
listoperation’s result ends withTotal: 12 songs (8 confirmed, 4 suggested), it only has to read a number aloud. Tool output should hand the model the sentence it needs to say.
We also pick the model by device: ≥ 8 GB of unified memory gets Qwen3-4B-4bit, smaller devices get Qwen3-0.6B-4bit (~0.6 GB — the smallest model whose tool calling we found reliable).
Delivering the weights: self-hosted R2, not Hugging Face
The first model-download path we shipped pulls from Hugging Face via huggingface-swift, and it works — but for a consumer app we didn’t want first-run downloads depending on HF’s availability, rate limits, or region performance. So we mirrored the models to a Cloudflare R2 bucket behind our own domain (assets.tuplet.app) and wrote a custom downloader that conforms to MLX LM’s downloader protocol.
Details that mattered:
- Pin the revision explicitly.
ModelConfiguration(id:)silently defaults to Hugging Face’smainref. Our R2 layout uses a version constant (v1), so the first on-device run 404’d looking for/main/config.json. Passingrevision:explicitly to the configuration fixed it — and gave us a version pin to bump when we re-quantize weights. - Immutable caching and range requests. R2 objects get
Cache-Control: immutable, and the CDN serves the multi-gigabyte safetensors efficiently via range requests — which is also how we cheaply verify a file exists before committing to a full download. - A tiny upload script with opinions. R2’s S3-compatible API forced lowercase object keys (the app expects them lowercased), needed
--s3-no-check-bucketbecause our scoped token can’tCreateBucket, and sets the cache header on every upload.
Download progress streams to the UI as a percentage; MLX’s fractionCompleted can exceed 1.0 when a download spans multiple files (we logged “102%”), so clamp it before it hits a ProgressView.
Background Assets: the prefetch pack path
An ~0.6 GB download on first assistant use is acceptable; a multi-gigabyte one is not a great first impression. Apple’s Background Assets framework solves this: an app can declare asset packs that the system downloads and keeps updated — including before first launch — via a small background extension.
We built the ExtensionKit-style download-manager extension and registered the 0.6B model as a prefetched asset pack backed by the same R2 CDN — deliberately only the small tier, so low-RAM devices never prefetch weights they can’t hold (the 4B model stays on-demand in-app). The extension itself is minimal: ManagedDownloaderExtension schedules every pack whose download policy applies to the install event, so the whole thing is a one-line struct. At session-creation time, the engine checks the pack first and only falls back to the in-app R2 download:
@available(iOS 26.4, *)
private static func packModelDirectoryIfCurrent() async -> URL? {
let manager = AssetPackManager.shared
let status = await manager.localStatus(ofAssetPackWithID: packID)
guard status.contains(.downloaded) else { return nil }
// The pack content directory doubles as the MLX model folder.
return try? manager.url(for: FilePath("."))
}
The beautiful part: MLX’s ModelConfiguration accepts a local directory, so a downloaded pack completely bypasses the downloader — the factory loads weights straight off disk.
The gotcha we hit in production is worth repeating: AssetPackManager.shared precondition-crashes if the app’s Info.plist lacks the BAAppGroupID key. We shipped one release without the embedded extension (and its Info.plist keys removed), and on iOS 26.4 devices the engine crashed the moment it touched AssetPackManager — before our graceful fallback could run. The fix is a one-line guard before any Background Assets API:
guard Bundle.main.object(forInfoDictionaryKey: "BAAppGroupID") != nil else {
return nil // pack path inert — ride the in-app R2 download
}
That makes the pack path structurally optional: present the extension and keys, and prefetch works; absent them, the app degrades to the CDN download instead of crashing.
What we’d tell ourselves at the start
- Put both engines behind one protocol and share tool logic in engine-agnostic cores. Every engine-specific API (and every engine-specific bug) stays contained.
- Small models need defensive schemas. Optional everything, validate in the handler, return self-correcting usage strings.
- Write tool output for the model, not the developer. Counts, dates in local time, and ready-to-speak sentences.
- Own your weights. An R2 mirror with a pinned revision costs almost nothing and removes a third party from first-run reliability.
- Background Assets is the right shape for big weights, but probe for the extension before touching it. One missing Info.plist key is a crash.
The result: on any device — Apple Intelligence or not — Tuplet’s assistant answers setlist questions, schedules rehearsals, and checks availability, with the model weights riding in a Background Assets pack or streaming from our own CDN. Fully on-device, fully ours.