| title | AI Agent Usage Guide |
|---|---|
| description | End-to-end recipes for building AI agents with the LEAP SDK — same patterns across iOS, macOS, Android, JVM, and native. |
This guide walks through the patterns for building a real AI agent — multi-turn conversation, function calling with tool dispatch, multimodal inputs, and a complete view-model wiring. The cross-platform pages cover individual APIs in depth — start here for the full picture, then drill into the dedicated references when you need details.
Downloader (LeapModelDownloader on Android · ModelDownloader on iOS / macOS · LeapDownloader on JVM/native)
↓
ModelRunner
↓
Conversation
↓
MessageResponse (streaming: Chunk · ReasoningChunk · FunctionCalls · AudioSample · Complete)
The ModelRunner owns native memory; the Conversation holds chat history; the MessageResponse stream delivers incremental output. Same shape on every platform.
| Concern | Where it lives |
|---|---|
| Install & set up the dependency | Quick Start |
| Load the model | Model Loading |
| Drive the streaming loop | Conversation & Generation |
| Define and dispatch tools | Function Calling |
| Force structured JSON output | Constrained Generation |
| Voice UX | Voice Assistant Widget |
| Hybrid on-device + cloud | OpenAI-Compatible Client |
| Desktop & native targets | Desktop & Native Platforms |
Every agent has the same shape: send a ChatMessage, iterate the response stream, dispatch each variant. Use the language's exhaustive switch — onEnum(of:) (Swift) or is checks against the sealed interface (Kotlin) — so the compiler errors if a new MessageResponse case is added.
The defining feature of an agent: the model emits FunctionCalls, you execute the tool, append the result as a tool-role message, and continue. The same pattern works on every platform.
while true {
var toolCalls: [LeapFunctionCall] = []
for try await response in workingConv.generateResponse(message: pending) {
switch onEnum(of: response) {
case .chunk(let c):
appendUI(c.text)
case .functionCalls(let payload):
toolCalls.append(contentsOf: payload.functionCalls)
case .complete:
break
case .reasoningChunk, .audioSample:
break
case .error(let err):
throw NSError(domain: "AgentLoop", code: 1, userInfo: [NSLocalizedDescriptionKey: err.message])
}
}
if toolCalls.isEmpty { break } // Agent is done
// Execute tools sequentially, append results, loop.
// (Swift's stdlib has no `asyncMap`; use a `for await` accumulation pass.)
var toolMessages: [ChatMessage] = []
for call in toolCalls {
let result = await runtimeDispatch(call)
toolMessages.append(ChatMessage(role: .tool, textContent: result))
}
let updatedHistory = workingConv.history + toolMessages
workingConv = workingConv.modelRunner.createConversationFromHistory(history: updatedHistory)
pending = ChatMessage(role: .user, textContent: "") // empty turn — let the model continue
}
}
```
while (true) {
val toolCalls = mutableListOf<LeapFunctionCall>()
workingConv.generateResponse(pending).collect { response ->
when (response) {
is MessageResponse.Chunk -> appendUI(response.text)
is MessageResponse.FunctionCalls -> toolCalls.addAll(response.functionCalls)
is MessageResponse.ReasoningChunk -> {}
is MessageResponse.AudioSample -> {}
is MessageResponse.Complete -> {}
is MessageResponse.Error -> throw response.throwable
}
}
if (toolCalls.isEmpty()) break
val toolMessages = toolCalls.map { call ->
val result = runtimeDispatch(call)
ChatMessage(
role = ChatMessage.Role.TOOL,
content = listOf(ChatMessageContent.Text(result))
)
}
val updatedHistory = workingConv.history + toolMessages
workingConv = workingConv.modelRunner.createConversationFromHistory(updatedHistory)
pending = ChatMessage(role = ChatMessage.Role.USER, content = listOf(ChatMessageContent.Text("")))
}
}
```
Define runtimeDispatch(_:) as your tool-call → result router: validate arguments, call the underlying implementation, JSON-encode the result. Register the corresponding LeapFunction definitions on the conversation before you start the loop — see Function Calling.
// Audio-capable model — WAV blob
let audioMessage = ChatMessage(
role: .user,
content: [.text("Transcribe."), ChatMessageContent.fromWAVData(wavData)],
reasoningContent: nil,
functionCalls: nil
)
// Audio-capable model — raw float32 PCM samples (no WAV re-encode)
let pcmMessage = ChatMessage(
role: .user,
content: [.text("How's my pronunciation?"),
ChatMessageContent.fromFloatSamples(samples, sampleRate: 16000)],
reasoningContent: nil,
functionCalls: nil
)
```
// Audio-capable model — WAV blob
val audioMessage = ChatMessage(
role = ChatMessage.Role.USER,
content = listOf(
ChatMessageContent.Text("Transcribe."),
ChatMessageContent.Audio(wavBytes)
)
)
// Audio-capable model — raw float32 PCM (no WAV re-encode)
val pcmMessage = ChatMessage(
role = ChatMessage.Role.USER,
content = listOf(
ChatMessageContent.Text("How's my pronunciation?"),
ChatMessageContent.AudioPcmF32(samples, sampleRate = 16000)
)
)
```
See Messages & Content for audio format requirements (WAV, mono, 16 kHz recommended) and helpers for recording from the microphone.
A ChatViewModel that loads the model, registers a tool, drives generation, and exposes streaming text to the UI.
@MainActor
final class ChatViewModel: ObservableObject {
@Published var responseText = ""
@Published var isLoading = false
@Published var isGenerating = false
@Published var errorMessage: String?
private let downloader: ModelDownloader = {
let caches = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask).first!.path
let modelsDir = (caches as NSString).appendingPathComponent("leap_models")
return ModelDownloader(config: LeapDownloaderConfig(saveDir: modelsDir))
}()
private var modelRunner: ModelRunner?
private var conversation: Conversation?
private var generationTask: Task<Void, Never>?
func loadModel() async {
isLoading = true
defer { isLoading = false }
do {
let runner = try await downloader.loadModel(
modelName: "LFM2.5-1.2B-Instruct",
quantizationType: "Q4_K_M"
)
modelRunner = runner
conversation = runner.createConversation(systemPrompt: "You are a helpful assistant.")
conversation?.registerFunction(weatherFunction)
} catch {
errorMessage = "Failed to load model: \(error.localizedDescription)"
}
}
func send(_ text: String) {
guard let conversation else { return }
generationTask?.cancel()
isGenerating = true
responseText = ""
generationTask = Task { [weak self] in
defer { Task { @MainActor in self?.isGenerating = false } }
do {
let userMessage = ChatMessage(role: .user, textContent: text)
let options = GenerationOptions()
.with(temperature: 0.3)
.with(minP: 0.15)
.with(repetitionPenalty: 1.05)
for try await response in conversation.generateResponse(
message: userMessage,
generationOptions: options
) {
await MainActor.run { self?.handle(response) }
}
} catch {
await MainActor.run { self?.errorMessage = "Generation failed: \(error.localizedDescription)" }
}
}
}
func stopGeneration() {
generationTask?.cancel()
}
@MainActor
private func handle(_ response: MessageResponse) {
switch onEnum(of: response) {
case .chunk(let c): responseText += c.text
case .reasoningChunk(let r): print("[thinking] \(r.reasoning)")
case .functionCalls(let f):
for call in f.functionCalls { Task { await dispatch(call) } }
case .audioSample(let a):
audioRenderer.enqueue(a.samples, sampleRate: Int(a.sampleRate))
case .complete(let c):
if let stats = c.stats {
print("\nFinished — \(stats.totalTokens) tokens at \(stats.tokenPerSecond) tok/s")
}
case .error(let err):
errorMessage = "Generation failed: \(err.message)"
}
}
}
```
class ChatViewModel(application: Application) : AndroidViewModel(application) {
private val downloader = LeapModelDownloader(application)
private var modelRunner: ModelRunner? = null
private var conversation: Conversation? = null
private var generationJob: Job? = null
private val _responseText = MutableStateFlow("")
val responseText: StateFlow<String> = _responseText.asStateFlow()
private val _isLoading = MutableStateFlow(false)
val isLoading: StateFlow<Boolean> = _isLoading.asStateFlow()
private val _isGenerating = MutableStateFlow(false)
val isGenerating: StateFlow<Boolean> = _isGenerating.asStateFlow()
private val _errorMessage = MutableStateFlow<String?>(null)
val errorMessage: StateFlow<String?> = _errorMessage.asStateFlow()
fun loadModel() {
viewModelScope.launch {
_isLoading.value = true
try {
val runner = downloader.loadModel(
modelName = "LFM2.5-1.2B-Instruct",
quantizationType = "Q4_K_M",
)
modelRunner = runner
conversation = runner.createConversation("You are a helpful assistant.").also {
it.registerFunction(weatherFunction)
}
} catch (e: Exception) {
_errorMessage.value = "Failed to load model: ${e.message}"
} finally {
_isLoading.value = false
}
}
}
fun send(text: String) {
generationJob?.cancel()
_responseText.value = ""
generationJob = viewModelScope.launch {
_isGenerating.value = true
val userMessage = ChatMessage(
role = ChatMessage.Role.USER,
content = listOf(ChatMessageContent.Text(text))
)
conversation?.generateResponse(
userMessage,
GenerationOptions.build {
temperature = 0.3f
minP = 0.15f
repetitionPenalty = 1.05f
},
)
?.onEach { handle(it) }
?.catch { e -> _errorMessage.value = "Generation failed: ${e.message}" }
?.onCompletion { _isGenerating.value = false }
?.collect()
}
}
fun stopGeneration() { generationJob?.cancel() }
private suspend fun handle(response: MessageResponse) {
when (response) {
is MessageResponse.Chunk -> _responseText.value += response.text
is MessageResponse.ReasoningChunk -> Log.d(TAG, "Reasoning: ${response.reasoning}")
is MessageResponse.FunctionCalls -> response.functionCalls.forEach { dispatch(it) }
is MessageResponse.AudioSample -> audioRenderer.enqueue(response.samples, response.sampleRate)
is MessageResponse.Complete -> Log.d(TAG, "Done: ${response.stats?.totalTokens} tokens")
is MessageResponse.Error -> _errorMessage.value = "Generation failed: ${response.message}"
}
}
override fun onCleared() {
super.onCleared()
generationJob?.cancel()
runBlocking(Dispatchers.IO) { modelRunner?.unload() }
}
companion object { private const val TAG = "ChatViewModel" }
}
```
- Always handle every
MessageResponsecase. Even if you only care about.chunkand.complete, give.functionCalls,.audioSample, and.reasoningChunkexplicit (empty) branches — otherwise an exhaustive switch will fail to compile when a new variant is added. - Cancel before re-issuing. Don't start a second
generateResponse(...)while one is in flight. Either cancel the previousTask/Job, or checkconversation.isGeneratingfirst. - Don't
runBlockingin production paths. It's fine inonCleared()for guaranteed cleanup (becauseviewModelScopeis already cancelled at that point). Anywhere else, it freezes the calling thread. - Use
cacheDir(Android) /cachesDirectory(iOS) for KV-cache reuse paths. They're regenerable — letting the OS reclaim them on storage pressure is the right semantics. See Model Loading → KV cache reuse. - Validate tool-call arguments before dispatching. The
arguments: Map<String, Any?>(Kotlin) /[String: Any?](Swift) shape is unsafe by design — defensively coerce types and apply business-level invariants. - Match the model's recommended sampling parameters. The LEAP bundle manifest (
sampling_parametersundergeneration_time_parametersin each<Quant>.jsonon LiquidAI/LeapBundles) carries defaults tuned per checkpoint for the llama.cpp engine the SDK runs. Overridingtemperatureand friends often hurts quality more than it helps — start from the manifest values rather than the HF model card defaults (the two can differ).
| Symptom | Likely cause / fix |
|---|---|
LeapModelLoadingException / LeapError.modelLoadingFailure |
Missing companion file for multimodal model (mmproj / audio decoder). Verify the manifest or pass explicit paths via loadSimpleModel(ModelSource(...)). |
| Model loads but generates gibberish | Wrong sampling parameters or wrong function-call parser for the model family. Check the model card; default to LFMFunctionCallParser for LFM models, HermesFunctionCallParser for Qwen3/Hermes. |
| "ZIP archive corrupted" on download | Network hiccup mid-download. LeapDownloader / LeapModelDownloader validates SHA-256, so a partial file fails the check. Remove the cache directory and retry. |
Generation hangs after cancel() |
Cancellation is cooperative — the engine checks between tokens. There's at most one extra token of slack. If it's longer, you may be missing a Job cancel or the stream is being awaited on a thread other than the one you're cancelling from. |
| Voice widget records silence | Missing microphone permission, or AVAudioSession/Android audio config not set to playAndRecord / mono / 16 kHz. See Voice Assistant Widget. |
K/N executable fails at start with dlsym@GLIBC_2.34 |
Runtime host's glibc is older than 2.34. Upgrade to Ubuntu 22.04+, Debian 12+, RHEL 9+, or build for an older runtime target. |
| Compile error "@Guide annotation missing" (Swift) | All properties on a @Generatable struct need a @Guide. Annotate every stored property. |
For deeper failure-mode coverage, see Utilities → Errors.