# NobodyWho
> NobodyWho runs LLMs locally in Kotlin, Swift, Python, Flutter, React Native, Expo and Godot. It uses llama.cpp and supports streaming chat, tool calling, embeddings, RAG, speech to text, text to speech, voice activity detection, offline inference, and GPU acceleration.
The binding sections use the latest released documentation. The main branch may document unreleased features. Main branch documentation is not included.
## Home
### Overview
Source: [https://docs.nobodywho.ooo/docs/](https://docs.nobodywho.ooo/docs/)
#### What is NobodyWho?
NobodyWho is a lightweight, open-source inference engine for running open-weights LLMs inside your software.
We provide a simple, efficient, offline and privacy forward way of interacting with LLMs. No infrastructure needed!
In short, if you want to run a LLM, integrate it with [tools](https://docs.nobodywho.ooo/python/tool-calling), configure its output,
enable real-time streaming of tokens, create embeddings, or synthesize speech, NobodyWho makes it easy.
All of this is enabled by [Llama.cpp](https://github.com/ggml-org/llama.cpp), while having nice, simple API.
No need to mess around with docker containers, GPU servers, API keys, etc. We make it easy to run local models in Kotlin, Swift, Python, Flutter, React Native, Expo and Godot!
#### Code documentation
If you already know the basics of LLMs, jump to the docs for your favorite language:
- [Python](https://docs.nobodywho.ooo/python/)
- [Kotlin](https://docs.nobodywho.ooo/kotlin/)
- [Swift](https://docs.nobodywho.ooo/swift/)
- [React Native/Expo](https://docs.nobodywho.ooo/react-native/)
- [Flutter](https://docs.nobodywho.ooo/flutter/)
- [Godot](https://docs.nobodywho.ooo/godot/)
Or install the skill so your AI coding agent knows everything about NobodyWho:
```bash
npx skills add https://github.com/nobodywho-ooo/nobodywho --skill nobodywho
```
#### Basic LLM concepts
If you are unfamiliar with the basics of LLMs or are just interested we also provide a simple introduction to the most important concepts you need to know in order to get the most out of NobodyWho.
---
### LLM Basics
Source: [https://docs.nobodywho.ooo/docs/llm-basics](https://docs.nobodywho.ooo/docs/llm-basics)
Our goal with NobodyWho is to make it easy to run local LLMs. For this reason we have made it possible to use NobodyWho with minimal knowledge of how LLM works. However you still need to know some basic concepts, so for these we provide some brief explanations. The concepts covered are tokens, context, samplers, tools, and thinking/reasoning.
#### Tokens
Tokens are the basic units that LLMs process. A token is typically a word, part of a word, or a punctuation mark. For example, "hello" is one token, while "understanding" might be split into two tokens: "understand" and "ing". It is worth noting that the vocabulary of tokens used is different for each model as it is defined during training.
When the model generates text, it produces one token at a time. This is why the default response object of NobodyWho is a stream of tokens and why you can read the response token-by-token.
#### Context
Context refers to all the text the model can "see" when generating a response. This includes:
- Previous messages in the conversation
- The current user prompt
- Any system instructions
Essentially the context acts as the models memory of the current conversation, available tools etc. This is important to remember as once your chosen model has been initialized most of your interactions with the model will happen through the context.
##### Context Size
Every model has a maximum context size (also called context window or context length), measured in tokens. Common sizes range from 2048 to 128,000 tokens.
Once you reach the context limit, you must either:
- Start a new conversation
- Remove old messages from the history
- Summarize earlier parts of the conversation
Currently NobodyWho resolves this issue automatically by removing old messages from the context.
Having a larger context allows for longer and more complex conversations, but it also slows down the response time, as the model has to process more tokens each time it generates a response.
#### CPU threads
When layers run on the CPU, inference is split across a pool of threads. Every operation in
the model ends at a barrier where all threads wait for the slowest one, so the pool is only as
fast as its weakest member. That makes "more threads" the wrong instinct:
- **Hyperthreads (SMT) don't add compute.** Two threads on one physical core contend for the
same vector units, so pairing them up mostly adds synchronisation overhead.
- **Efficiency cores are genuinely slower hardware.** On a big.LITTLE CPU — Apple silicon, most
phones, newer Intel — an E-core in the pool holds every P-core at the barrier until it
catches up.
So NobodyWho defaults to one thread per *performance* core rather than per logical CPU, the
same choice llama.cpp's own tools make. On a 12-core Mac with 8 P-cores and 4 E-cores, that
means 8 threads — and measured on that machine it is roughly twice as fast at generation as
using all 12:
| threads | prompt processing | generation |
|---|---|---|
| 8 (performance cores only) | 435 tok/s | 90 tok/s |
| 12 (every logical CPU) | 327 tok/s | 48 tok/s |
You normally shouldn't need to touch this. Two cases where you might: lowering it to leave CPU
headroom for other work (rendering a game, serving several models from one box), or raising it
if NobodyWho could not read your machine's topology and guessed low — it logs a warning when
that happens. Every binding exposes the option on its chat type; see your binding's chat docs.
Values above your CPU count are clamped, and offloading to the GPU makes the setting largely
irrelevant.
#### Samplers
LLMs don't output text directly. Instead, they generate a probability distribution over all possible next tokens. Since the model weights are static after training, this means that the same input tokens always generate the same distribution. Depending on the use case however, there are many possible ways of choosing a next token from this distribution. This is configured using a **sampler**. A **sampler** splits the process of choosing a next token into two parts: Shifting the distribution and Sampling the distribution.
##### Shifting the Distribution
Before sampling the distribution to get the next token, it is possible to adjust the distribution provided by the LLM to encourage certain behavior. Examples of these adjustments are:
- **Temperature**: Higher values make output more creative/random, lower values make it more focused/deterministic.
- **Top-k/Top-p**: Limit which tokens are considered, filtering out unlikely options
- **Penalties**: Lower the probalities of tokens already present in the context.
It is important to note that the steps in this part of the process can be chained. So it is possible to first apply a Temperature shift and then Top-k.
##### Sampling the distribution
Once the distribution has been shifted the next step is to actually sample the distribution. This can also be done a few different ways:
- **Dist**: Sample the distribution randomly
- **Greedy**: Always pick the most likely token (deterministic but sometimes repetitive)
- **Mirostat**: Advanced sampling presented in this [article](https://arxiv.org/abs/1904.09751)
Since this part actually chooses the next token, these cannot be chained.
NobodyWho also supports more advanced ways of configuraing a sampler, like for example follow a JSON Schema.
#### Speculative decoding (MTP)
Speculative decoding is a way of making inference faster without changing the model's outputs. A small, cheap "draft" model proposes the next few tokens, and the real target model verifies them in one batch. Tokens that the target would have picked anyway are accepted for free — otherwise the target's own choice is used. Same distribution, fewer sequential forward passes.
NobodyWho supports the **MTP** (Multi-Token Prediction) flavour used by Gemma 4: instead of running a separate draft model, MTP uses extra "heads" trained to predict several tokens ahead of the target. You download the MTP heads as a companion `.gguf` (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B), pass it as the draft model path when loading the model, and enable MTP when constructing the chat (see your binding's chat docs for the exact option).
Expect a large speedup on structured/deterministic output (code, JSON, math, tool calls) and a neutral to modest speedup on high-entropy prose. MTP is also highly hardware-dependent — you will get the most out of it when memory bandwidth is the biggest bottleneck during generation.
This means that on systems with unified memory (e.g. a Mac mini), the speedup might not be very big and you might even get worse performance. For this reason it is important to be mindful about both your hardware and your use case. The best option is of course to benchmark it yourself.
Enabling MTP adds around 5% to VRAM usage. It is off by default.
#### Tools
Tools (also called function calling) allow the LLM to request external actions. Instead of just generating text, the model can indicate it wants to:
- Search a database
- Perform a calculation
- Fetch data from an API
- Execute custom code
You define available tools, and the model decides when to use them based on the conversation. After a tool executes, you provide the result back to the model so it can continue the conversation.
This enables LLMs to go beyond pure text generation and interact with your application's functionality.
#### Thinking/Reasoning
Some models have been specifically trained to reason through complex tasks step-by-step before giving a final answer. You can check the model's Hugging Face page to find out whether it supports thinking.
Inside the LLM's raw output, the thinking/reasoning content usually appears between model-specific tags:
```
...reasoning steps...
...final answer...
```
Here are some tags examples:
- Qwen3: `...`
- Gemma 4: `<|channel>thought ...`
- Ministral: `[THINK]…[/THINK]`
These tags are model and template specific, so always check the model's chat template if you need to parse out the thinking content yourself.
---
### Model Selection
Source: [https://docs.nobodywho.ooo/docs/model-selection](https://docs.nobodywho.ooo/docs/model-selection)
Choosing the right language model can make or break your project. In general you want to go as small as possible while still having the capabilities you need for your application.
#### TL;DR
You can pick a model based on your needs on our [model selection page](https://www.nobodywho.ai/models/).
If you don't want to choose a chat model, pass `auto` as the model path to select one based on available memory.
If you just want a ~2GB chat model that works well, use:
```
huggingface:NobodyWho/Qwen_Qwen3-4B-GGUF/Qwen_Qwen3-4B-Q4_K_M.gguf
```
If you want something smaller and faster, use:
```
huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf
```
Pass these as the model path when creating a `Chat`, `Model`, `Encoder`, etc. NobodyWho will download the model automatically and cache it locally for future use.
#### Getting a model
NobodyWho can download models directly from Hugging Face. Instead of downloading a file manually, pass a `huggingface:` path where you'd normally pass a file path:
```
huggingface:owner/repo/filename.gguf
```
The model is downloaded once and cached locally — no internet connection is needed after the first load. `hf:` is also accepted as a shorthand.
You can also pass a full `https://` URL to download a model from any host.
Of course, you can still pass a local file path if you prefer to manage model files yourself.
We recommend starting with the models on our [Hugging Face page](https://huggingface.co/NobodyWho) since they are known to work well with NobodyWho.
Once you're more familiar, you can also try models from [Unsloth](https://huggingface.co/unsloth/models).
Broadly, almost any `.gguf` model on [Hugging Face](https://huggingface.co) should work, though some may fail due to formatting issues, like the LFM models, that we have patched [here](https://huggingface.co/collections/NobodyWho/lfm-25).
#### Understanding model file names
Model files follow a naming convention like this: `Qwen_Qwen3-0.6B-Q4_K_M.gguf`
Here's what each part means:
- `Qwen` the organization that trained the model.
- `Qwen3` the name of the model release.
- `0.6B` the parameter count in billions. This model has 0.6 billion (600 million) parameters.
- `Q4` the quantization level, i.e. the number of bits used per parameter.
- `K_M` details about the quantization technique. `S` is faster but less precise, `L` is slower but more precise, and `M` is a middle ground. You don't need to worry too much about this for now.
For chatting, you'll need an instruction-tuned GGUF file that includes a Jinja2 chat template in its metadata. This describes the vast majority of GGUF files available, so if you're unsure, just try it — NobodyWho will give you a descriptive error message if something isn't right.
For embeddings or cross-encoding, you'll need models specifically designed for those tasks, they are typically named accordingly. Although note that cross-encoding models are sometimes referred to as "reranking" models.
#### Quantization
Quantization refers to the practice of reducing the number of bits per weight.
This can make the model faster and smaller, with a relatively small loss in response quality.
Generally speaking, you can used models quantized down the Q4 or Q5 levels (4 or 5 bits per weight respectively),
while loosing barely any accuracy.
Look at the plot below to get a feel for how quantization levels differ.
It shows the models' ability to predict text on the y-axis versus the number of bits per weight on the x-axis.
In general, it's preferable to use a model with more parameters and fewer bits per parameter, as compared to a model with fewer parameters and more bits per parameter.
Your results may vary.
#### Estimating Memory Usage
The memory requirement of a model is roughly its parameter count multiplied by its quantization level.
Here's a few examples:
- 2B @ Q8 ~= 2GB
- 2B @ Q4 ~= 1GB
- 14B @ Q4 ~= 7GB
- 14B @ Q2 ~= 3.5GB
- ..and so on
#### Comparing Models
There are many places online for comparing benchmark scores of different LLMs, here's a few of them:
**[LLM-Stats.com](https://llm-stats.com/)**
- Includes filters for open models and small models.
- Compares recent models on a few different benchmarks.
**[OpenEvals on huggingface](https://huggingface.co/spaces/OpenEvals/find-a-leaderboard)**
- A collection of benchmark leaderboards in different domains.
- Includes both inaccessible proprietary models and open models.
Remember that you need an open model, in order to be able to find a GGUF download and run it locally (e.g. Gemma is open, but Gemini isn't).
---
*Need help choosing between specific models? Check our [community Discord](https://discord.gg/qhaMc2qCYB).*
---
### OpenAI-compatible server
Source: [https://docs.nobodywho.ooo/docs/server](https://docs.nobodywho.ooo/docs/server)
NobodyWho provides an **experimental** local server that implements the OpenAI Chat Completions API. It loads one model and requires no API key.
**Note:**
`nobodywho-server` is experimental! Not to be used for any production/sensitive loads.
#### Setup (for a coding agent)
Start the server:
```bash
uvx \
--from 'git+https://github.com/nobodywho-ooo/nobodywho.git#subdirectory=nobodywho/server' \
nobodywho-server \
--model hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf \
--name qwen
```
The server listens on `http://127.0.0.1:8888` by default.
To use with, for example [Pi](https://pi.dev/), load the [Pi extension](https://gist.github.com/duarteocarmo/39a3f1148e2e33c27fddbbef2d971deb):
```bash
NOBODYWHO_BASE_URL=http://127.0.0.1:8888/v1 \
pi -e 'git:https://gist.github.com/duarteocarmo/39a3f1148e2e33c27fddbbef2d971deb.git' \
--model nobodywho/qwen
```
And start working.
#### API
The API is very minimal for now:
- `GET /health`
- `GET /v1/models`
- `POST /v1/chat/completions`
```bash
curl http://127.0.0.1:8888/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen",
"messages": [{"role": "user", "content": "Say hello."}],
"stream": true
}'
```
#### Limitations
- single model
- sequential generations
- no auth of any kind
- no `required` tool choice (only `auto`, `none`, or a named function)
#### References
- [OpenAI API reference](https://developers.openai.com/api/reference/overview)
- [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create)
- [OpenAI streaming guide](https://platform.openai.com/docs/guides/streaming-responses?api-mode=chat)
#### Future enhancements
- Expose GGUF metadata through the API
- Publish an official NobodyWho Pi extension instead of a gist
---
## Kotlin
This section documents Kotlin 3.0.0, the latest released version.
### Chat
Source: [https://docs.nobodywho.ooo/kotlin/chat](https://docs.nobodywho.ooo/kotlin/chat)
Every interaction with your LLM starts with a `Chat` object.
#### Creating a Chat
The simplest way is using `Chat.fromPath`:
```kotlin
val chat = Chat.fromPath(modelPath = "./model.gguf")
```
This is a suspend function since loading a model can take a bit of time, but it won't block your UI thread.
Another way is to load the model separately and share it between multiple chats:
```kotlin
val model = Model.load(modelPath = "./model.gguf")
val chat1 = Chat(model = model)
val chat2 = Chat(model = model)
```
NobodyWho takes care of the separation, so your chat histories won't collide or interfere with each other.
You can pass `"auto"` as `modelPath` to select a chat model based on available memory.
#### Prompts and responses
The `Chat.ask()` function sends your message to the LLM, which then starts generating a response:
```kotlin
val chat = Chat.fromPath(modelPath = "./model.gguf")
val stream = chat.ask("Is water wet?")
```
The return type is a `TokenStream`. If you want to read the response as it generates, collect it as a Flow:
```kotlin
chat.ask("Is water wet?").asFlow().collect { token ->
print(token)
}
```
If you just want the complete response, call `completed()`:
```kotlin
val fullResponse = chat.ask("Is water wet?").completed()
```
All messages and responses are stored in the `Chat`, so the next `ask()` remembers the conversation.
#### Chat history
Inspect the messages inside the `Chat`:
```kotlin
val msgs = chat.getChatHistory()
println(msgs[0].content) // "Is water wet?"
```
Or set the history directly:
```kotlin
chat.setChatHistory(listOf(
Message.User(content = "What is water?")
))
```
#### System prompt
A system prompt guides the model's overall behavior. Some models ship with a built-in default.
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
systemPrompt = "You are a mischievous assistant!"
)
```
The system prompt persists until the chat context is reset.
#### Context
The context is the token window the LLM currently considers. Larger context means more computational overhead:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
contextSize = 4096u
)
```
The default is `4096`. When the context fills up during a conversation, NobodyWho automatically shrinks it by removing old messages (keeping the system prompt and first user message). You can check the maximum context size the model was trained with using `model.maxCtx` — setting `contextSize` above this value has no benefit.
To reset the context with a new system prompt and tools:
```kotlin
chat.resetContext(systemPrompt = "New system prompt", tools = listOf())
```
To just clear the history without changing settings:
```kotlin
chat.resetHistory()
```
To inspect how much of the context is currently in use, call `getStats()`:
```kotlin continuation
val stats = chat.getStats()
println("Using ${stats.contextUsed} of ${stats.contextSize} tokens")
```
#### CPU threads
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Efficiency cores end up pacing the whole thread pool, so using every
CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads) for the numbers.
Override it with `threadCount` when you want to leave CPU headroom for the rest of your app —
often a good idea on phones, where the big cores are also driving the UI:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
threadCount = 4u
)
```
Leave it unset — or pass `null` — to keep the detected default. Values above the device's CPU
count are clamped, and it has little effect when the model is offloaded to the GPU.
#### GPU
When loading a model, GPU acceleration is enabled by default:
```kotlin
val model = Model.load(modelPath = "./model.gguf", useGpu = true)
```
NobodyWho uses Vulkan on Linux/Windows and Metal on macOS for GPU acceleration.
#### Speculative decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target model verify several candidate tokens per forward pass. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
Load the model with a compatible draft-heads gguf (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B) and pass an `MtpConfig` when constructing the chat:
```kotlin
val model = Model.load(
modelPath = "./gemma-4-e2b.gguf",
draftModelPath = "./mtp-gemma-4-e2b.gguf"
)
// MtpConfig() uses the default drafter tuning; tune with MtpConfig(kMax = ..., pMin = ...)
val chat = Chat(model = model, mtp = MtpConfig())
```
`Chat.fromPath` accepts the same two parameters if you don't need to share the model:
```kotlin
val chat = Chat.fromPath(
modelPath = "./gemma-4-e2b.gguf",
draftModelPath = "./mtp-gemma-4-e2b.gguf",
mtp = MtpConfig()
)
```
Loading the draft heads adds around 5% to VRAM usage.
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
#### Template Variables
Chat templates are used internally by models to format conversation history. Template variables are boolean flags that control specific behaviors.
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
templateVariables = mapOf("enable_thinking" to true)
)
```
You can also modify them on an existing chat:
```kotlin
chat.setTemplateVariable("enable_thinking", false)
val variables = chat.getTemplateVariables()
```
##### Example: Qwen3 Reasoning
The Qwen3 model family supports `enable_thinking`, which controls whether the model shows its reasoning process before answering:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
templateVariables = mapOf("enable_thinking" to true)
)
val response = chat.ask("Solve this logic puzzle: ...").completed()
```
**Info:**
Template variables are model-specific. If a model's chat template doesn't use a specific variable, it will be ignored.
---
### Downloading models
Source: [https://docs.nobodywho.ooo/kotlin/downloading-models](https://docs.nobodywho.ooo/kotlin/downloading-models)
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to access gated/private models, how to observe a download in progress, and how to inspect what's already in the local cache.
#### Supported model path formats
The `modelPath` argument to `Model.load`, `Chat.fromPath`, and `Model.download` accepts:
| Form | Example | Notes |
| ---- | ------- | ----- |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
| Local path | `./model.gguf` | Used as-is |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs.
#### Android permissions
Android requires explicit approval, adding the internet permission to your app's `AndroidManifest.xml`:
```xml
```
NobodyWho does not declare this permission for you, so apps that only load models from local paths keep working without requesting any network access.
#### Downloading a gated model
Some HuggingFace models are either private or gated by a license that you need to accept. For both scenarios, you need to be authorized to download the model weights.
In that case, you can resort to manually accessing the model page through your web browser, getting the GGUF file downloaded and then pointing our chat instance to the path where you have stored it:
```kotlin
val chat = Chat.fromPath(modelPath = "./model.gguf")
```
Or you can use the `Model.download` function, where you can pass in the authorization token:
```kotlin
import ai.nobodywho.Model
import ai.nobodywho.Chat
val modelPath = Model.download(
modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
headers = mapOf("Authorization" to "Bearer your_hf_token")
)
val chat = Chat.fromPath(modelPath = modelPath)
```
The token can be generated in [your account settings](https://huggingface.co/settings/tokens).
#### Tracking download progress
When loading a remote model, pass an `onDownloadProgress` callback to observe the download. It receives `(downloadedBytes, totalBytes)` and is not called for cached or local files.
```kotlin
val model = Model.load(
modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total ->
println("$downloaded / $total bytes")
}
```
#### Inspecting the model cache
`getCachedModels()` returns every `.gguf` model in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `Model.download` and by `Chat.fromPath`'s `hf://` paths.
```kotlin
import ai.nobodywho.getCachedModels
for (model in getCachedModels()) {
println("${model.path} — ${model.size} bytes")
}
```
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/kotlin/embeddings-and-rag](https://docs.nobodywho.ooo/kotlin/embeddings-and-rag)
When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.
#### Understanding Embeddings
Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.
#### The Encoder
The `Encoder` converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).
We recommend [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf).
```kotlin
import ai.nobodywho.Encoder
val encoder = Encoder.fromPath(modelPath = "./embedding-model.gguf")
val embedding = encoder.encode("What is the weather like?")
println("Vector with ${embedding.size} dimensions")
```
Use `encodeBatch` for multiple texts. It returns embeddings in input order:
```kotlin
val texts = listOf(
"Paris is the capital of France.",
"Berlin is the capital of Germany."
)
val embeddings = encoder.encodeBatch(texts = texts)
```
##### Comparing Embeddings
Measure how similar two pieces of text are with cosine similarity:
```kotlin
import ai.nobodywho.Encoder
import ai.nobodywho.cosineSimilarity
val encoder = Encoder.fromPath(modelPath = "./embedding-model.gguf")
val query = encoder.encode("How do I reset my password?")
val doc1 = encoder.encode("You can reset your password in the account settings")
val doc2 = encoder.encode("The password requirements include 8 characters minimum")
val similarity1 = cosineSimilarity(query, doc1)
val similarity2 = cosineSimilarity(query, doc2)
println("Document 1 similarity: ${"%.3f".format(similarity1)}") // Higher score
println("Document 2 similarity: ${"%.3f".format(similarity2)}") // Lower score
```
#### The CrossEncoder for Better Ranking
While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents rather than comparing vectors.
```kotlin
import ai.nobodywho.CrossEncoder
val crossEncoder = CrossEncoder.fromPath(modelPath = "./reranker-model.gguf")
val query = "How do I install Python packages?"
val documents = listOf(
"Someone previously asked about Python packages",
"Use pip install package-name to install Python packages",
"Python packages are not included in the standard library"
)
val scores = crossEncoder.rank(query, documents)
println(scores) // [0.23, 0.89, 0.45] - second doc scores highest
```
##### Automatic Sorting
Use `rankAndSort` to get documents sorted by relevance:
```kotlin
val rankedDocs = crossEncoder.rankAndSort(query, documents)
for ((doc, score) in rankedDocs) {
println("[${"%.3f".format(score)}] $doc")
}
```
#### Building a RAG System
Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.
```kotlin
import ai.nobodywho.*
suspend fun main() {
val crossEncoder = CrossEncoder.fromPath(modelPath = "./reranker-model.gguf")
val knowledge = listOf(
"Our company offers a 30-day return policy for all products",
"Free shipping is available on orders over $50",
"Customer support is available via email and phone",
"We accept credit cards, PayPal, and bank transfers",
"Order tracking is available through your account dashboard"
)
fun searchKnowledge(query: String): String {
val ranked = runBlocking { crossEncoder.rankAndSort(query, knowledge) }
return ranked.take(3).joinToString("\n") { it.first }
}
val searchTool = Tool(
name = "search_knowledge",
description = "Search the knowledge base for relevant information",
function = ::searchKnowledge
)
val chat = Chat.fromPath(
modelPath = "./model.gguf",
systemPrompt = "You are a customer service assistant. Use the search_knowledge tool to find relevant information before answering.",
templateVariables = mapOf("enable_thinking" to false),
tools = listOf(searchTool)
)
val response = chat.ask("What is your return policy?").completed()
println(response)
}
```
#### Recommended Models
##### For Embeddings
- [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf) - Good balance of speed and quality (~25MB)
##### For Cross-Encoding (Reranking)
- [bge-reranker-v2-m3-Q8_0.gguf](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf) - Multilingual support with excellent accuracy
#### Best Practices
**Precompute embeddings**: If you have a fixed knowledge base, generate embeddings once and reuse them.
**Use embeddings for filtering**: For large collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank.
**Limit cross-encoder inputs**: Cross-encoders are more expensive than embeddings. Filter first with embeddings, then rerank.
---
### Getting started
Source: [https://docs.nobodywho.ooo/kotlin/](https://docs.nobodywho.ooo/kotlin/)
#### How do I get started?
First, add NobodyWho to your `build.gradle.kts`:
```kotlin
// Android
implementation("ai.nobodywho:nobodywho-android:2.2.0")
// Desktop JVM (Linux, macOS, Windows)
implementation("ai.nobodywho:nobodywho:2.2.0")
```
Next, pick a model. NobodyWho can download GGUF models directly from Hugging Face — just pass an `hf://` path. See [model selection](https://docs.nobodywho.ooo/docs/model-selection) for recommendations.
Then create a `Chat` and call `.ask`!
```kotlin
import ai.nobodywho.Chat
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
val chat = Chat.fromPath(
modelPath = "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// stream tokens as they are generated
chat.ask("Is water wet?").asFlow().collect { token ->
print(token)
}
// ...or get the entire response as a single string
val response = chat.ask("Is water wet?").completed()
println(response)
}
```
On Android, use `lifecycleScope` or `viewModelScope` instead of `runBlocking`.
This is a super simple example, but we believe that examples which do simple things, should be simple!
To get a full overview of the functionality provided by NobodyWho, simply keep reading.
---
### Sampling
Source: [https://docs.nobodywho.ooo/kotlin/sampling](https://docs.nobodywho.ooo/kotlin/sampling)
The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which you can freely configure to achieve better quality outputs or constrain outputs to a known format (e.g. JSON).
#### Sampler presets
To get a quick start, NobodyWho offers well-known presets. For example, to adjust the "creativity" of your model:
```kotlin
import ai.nobodywho.Chat
import ai.nobodywho.SamplerPresets
val chat = Chat.fromPath(
modelPath = "./model.gguf",
sampler = SamplerPresets.temperature(0.2f)
)
```
Setting `temperature` to `0.2` makes the distribution less flat, so the model favours more probable tokens.
The full list of presets:
```kotlin
object SamplerPresets {
fun default(): SamplerConfig
fun dry(): SamplerConfig
fun greedy(): SamplerConfig
fun json(): SamplerConfig
fun temperature(temperature: Float): SamplerConfig
fun topK(topK: Int): SamplerConfig
fun topP(topP: Float): SamplerConfig
// Constrain output to a specific format:
fun constrainWithJsonSchema(schema: String): SamplerConfig
fun constrainWithRegex(pattern: String): SamplerConfig
fun constrainWithGrammar(grammar: String): SamplerConfig
}
```
#### Structured output
One of the most useful features is constraining the model to produce structured output — this gives you a hard guarantee that the output matches a specific format.
##### Regular expressions
For simpler patterns, constrain the output with a regex:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
sampler = SamplerPresets.constrainWithRegex("yes|no")
)
val answer = chat.ask("Is the sky blue?").completed()
// answer is guaranteed to be exactly "yes" or "no"
```
##### JSON schema
Enforce any JSON output:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
sampler = SamplerPresets.json()
)
```
Or use a JSON schema for specific object shapes:
```kotlin
val schema = """
{
"type": "object",
"properties": {
"name": {"type": "string", "maxLength": 50},
"age": {"type": "integer"}
},
"required": ["name", "age"],
"additionalProperties": false
}
"""
val chat = Chat.fromPath(
modelPath = "./model.gguf",
sampler = SamplerPresets.constrainWithJsonSchema(schema)
)
val response = chat.ask("Give me a person with name and age.").completed()
// response is always valid JSON matching the schema
```
##### Custom grammars (advanced)
For cases where JSON schema and regex are not expressive enough, supply a custom grammar. `constrainWithGrammar` accepts both **Lark** syntax and **GBNF** (llama.cpp format).
**Lark syntax** (recommended):
```kotlin
val sampler = SamplerPresets.constrainWithGrammar("""
start: record (NEWLINE record)* NEWLINE?
record: field ("," field)*
field: /[^,"\n\r]+/
NEWLINE: /\r?\n/
""")
```
**GBNF syntax** (also accepted):
```kotlin
val sampler = SamplerPresets.constrainWithGrammar("""
file ::= record (newline record)* newline?
record ::= field ("," field)*
field ::= /[^,"\n\r]+/
newline ::= "\r\n" | "\n"
""")
```
**Info:**
The older `SamplerPresets.grammar()` method is deprecated. Use
`SamplerPresets.constrainWithGrammar()` instead — it accepts both Lark and GBNF strings.
#### Building custom samplers with the DSL
Sampler presets abstract away some control. For more advanced configurations — chaining samplers, tuning parameters — use the `buildSampler` DSL:
```kotlin
import ai.nobodywho.buildSampler
val sampler = buildSampler {
topK(40)
temperature(0.8)
minP(0.05)
dist()
}
val chat = Chat.fromPath(
modelPath = "./model.gguf",
sampler = sampler
)
```
##### Available sampling steps
Inside `buildSampler { }`, call any of the **shift steps** below (each reshapes the distribution), then one **terminal step** that picks the token. Most steps have defaults, so you only pass what you want to change.
Shift steps — call as many as you want, in order:
- `topK(40)` — keep only the 40 most likely tokens
- `topP(0.95)` — nucleus: keep the top tokens up to 95% of the probability mass
- `minP(0.05)` — drop tokens below 5% of the most likely token's probability
- `typicalP(0.9)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `xtc(0.5, 0.1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `temperature(0.8)` — below 1.0 = more focused, above 1.0 = more random
- `penalties(penaltyRepeat = 1.1)` — per-token repetition penalty (`penaltyRepeat` 1.0 = off)
- `dry()` — penalty for repeated *phrases* (its defaults are a good start)
- `seed(42)` — fix the RNG for reproducible output
- `grammar(...)` — deprecated; use the `constrainWith*` presets above
Terminal step — call at most one:
- `dist()` — pick a token with weighted randomness (used by default if you omit it)
- `greedy()` — always take the most likely token
- `mirostatV1()` / `mirostatV2()` — steer output "surprise" toward a target
`minKeep` (on the truncation steps) is the floor on how many tokens survive a cut.
For reproducible output, set the RNG seed with `seed(value)` anywhere in the chain.
It is consumed by every random sampler — `dist`, `mirostatV1`, `mirostatV2`, and the `xtc`
shift step. `greedy` ignores it. If unset, a default seed is used.
```kotlin
val sampler = buildSampler {
topK(40)
temperature(0.8)
seed(42)
dist()
}
```
You can also change the sampler on an existing chat:
```kotlin
val newSampler = buildSampler {
temperature(1.2)
topP(0.9)
dist()
}
chat.setSamplerConfig(newSampler)
```
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/kotlin/speech-to-text](https://docs.nobodywho.ooo/kotlin/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format.
```kotlin
import ai.nobodywho.SpeechToText
val stt = SpeechToText.load(source = "hf://onnx-community/whisper-base")
val text = stt.transcribeFile("recording.mp3").completed()
println(text)
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribePcm` is available:
```kotlin
val text = stt.transcribePcm(samples, sampleRate = 16000u).completed()
```
In order to make this work, the buffer needs to be mono i16 PCM samples. The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with classic Chat models, streaming is available, so the transcription can be consumed token by token:
```kotlin
stt.transcribeFile("recording.mp3").asFlow().collect { piece ->
print(piece)
}
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `source` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```kotlin
val stt = SpeechToText.load(
source = "hf://onnx-community/whisper-base",
quantization = "q4"
)
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, pass its ISO 639-1 code as `language` to skip detection and improve performance:
```kotlin
val stt = SpeechToText.load(
source = "hf://onnx-community/whisper-base",
language = "en"
)
```
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/kotlin/text-to-speech](https://docs.nobodywho.ooo/kotlin/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
```kotlin
import ai.nobodywho.TextToSpeech
import java.io.File
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
val tts = TextToSpeech.load(
source = "hf://NobodyWho/Kokoro-82M", // Hugging Face repo (hf://owner/repo) or local folder with the model files.
voice = "bf_emma", // Voice to use from the model.
language = "en-gb", // Language code for the input text.
)
// Generate WAV bytes for this sentence.
val wav = tts.synthesize(text = "Hello from NobodyWho!")
// Save the audio to a file.
File("out.wav").writeBytes(wav)
}
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, or a local directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/kotlin/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/kotlin/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```kotlin
val tts = TextToSpeech.load(
source = "hf://NobodyWho/Kokoro-82M",
voice = "bf_emma",
language = "en-gb",
)
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.0`.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```kotlin
val tts = TextToSpeech.load(
source = "hf://Supertone/supertonic-3",
language = "en",
)
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.05`.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Must be greater than `0`; defaults to `8`.
- `silenceDuration`: seconds of silence between long text chunks. Higher values add longer pauses. Must be `0` or higher; defaults to `0.3`.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```kotlin
val tts = TextToSpeech.load(
source = "hf://KevinAHM/pocket-tts-onnx",
voice = "alba",
language = "english_2026-04",
huggingfaceToken = "hf_...", // Alternative: set the HF_TOKEN environment variable.
)
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Defaults to `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Defaults to `int8`.
- `temperature`: controls how varied the speech sounds. Defaults to `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingfaceToken`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory or a custom source that NobodyWho cannot recognize:
```kotlin
import ai.nobodywho.TextToSpeechArchitecture
val tts = TextToSpeech.load(
source = "/path/to/local/kokoro-folder",
architecture = TextToSpeechArchitecture.KOKORO,
)
```
Supported architecture values are `TextToSpeechArchitecture.KOKORO` and `TextToSpeechArchitecture.SUPERTONIC`.
#### GPU
TextToSpeech uses GPU acceleration by default when available. Disable it with `device = TextToSpeechDevice.CPU`:
```kotlin
import ai.nobodywho.TextToSpeechDevice
val tts = TextToSpeech.load(
source = "hf://Supertone/supertonic-3",
device = TextToSpeechDevice.CPU,
)
```
#### Local model folder format
When `source` is a local directory, point it at the top-level model folder and pass the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/kotlin/tool-calling](https://docs.nobodywho.ooo/kotlin/tool-calling)
To give your LLM the ability to interact with the outside world, you will need tool calling.
**Info:**
Note that **not every model** supports tool calling. If the model does not have
such an option, it might not call your tools.
For reliable tool calling, we recommend trying the [Qwen](https://huggingface.co/collections/NobodyWho/qwen-3) family of models.
#### Declaring a tool
Create a tool by passing a function reference. NobodyWho uses Kotlin reflection to automatically inspect parameter names and types:
```kotlin
import ai.nobodywho.Tool
fun getWeather(city: String, unit: String): String {
return """{"temp": 22, "unit": "$unit"}"""
}
val weatherTool = Tool(
name = "get_weather",
description = "Get the current weather for a city",
function = ::getWeather
)
```
To let your LLM use the tool, pass it when creating `Chat`:
```kotlin
val chat = Chat.fromPath(
modelPath = "./model.gguf",
tools = listOf(weatherTool)
)
```
NobodyWho figures out the right tool calling format, inspects the names and types of the parameters, and configures the sampler.
Supported parameter types: `String`, `Int`, `Long`, `Double`, `Float`, `Boolean`, and collections like `List` or `Map`.
#### Multiple tools
Naturally, more tools can be defined and the model can chain calls:
```kotlin
fun getCurrentDir(): String {
return System.getProperty("user.dir") ?: "."
}
fun listFiles(path: String): String {
return java.io.File(path).list()?.joinToString(", ") ?: "empty"
}
fun getFileSize(filepath: String): String {
return "File size: ${java.io.File(filepath).length()} bytes"
}
val chat = Chat.fromPath(
modelPath = "./model.gguf",
tools = listOf(
Tool(name = "get_current_dir", description = "Gets the current directory", function = ::getCurrentDir),
Tool(name = "list_files", description = "Lists files in a directory", function = ::listFiles),
Tool(name = "get_file_size", description = "Gets the size of a file", function = ::getFileSize),
)
)
val response = chat.ask("What is the biggest file in my current directory?").completed()
println(response)
```
#### Suspend functions
Suspend functions work as tools too — just pass the reference the same way:
```kotlin
suspend fun fetchData(query: String): String {
return httpClient.get("https://api.example.com?q=$query").bodyAsText()
}
val fetchTool = Tool(
name = "fetch_data",
description = "Fetch data from an API",
function = ::fetchData
)
```
Your UI and other coroutines remain responsive while the tool executes.
#### Function scope limitation
The `Tool` class uses Kotlin reflection (`KFunction`) to introspect parameter names and types. This requires the function to be a **top-level function**, a **class method**, or a **companion object method**.
Local functions defined inside other functions or coroutines are **not supported** — the Kotlin compiler mangles their JVM signatures, which breaks reflection.
```kotlin
// ✅ Works — top-level function
fun ping(): String = "pong"
val pingTool = Tool(name = "ping", description = "Ping", function = ::ping)
// ✅ Works — companion object method
class MyTools {
companion object {
fun ping(): String = "pong"
}
}
val pingTool = Tool(name = "ping", description = "Ping", function = MyTools::ping)
// ❌ Does NOT work — local function inside a coroutine
suspend fun example() {
fun ping(): String = "pong" // This will fail at runtime
val tool = Tool(name = "ping", description = "Ping", function = ::ping)
}
```
#### Changing tools at runtime
You can change the available tools on an existing chat:
```kotlin
chat.setTools(listOf(newTool1, newTool2))
```
Or reset the context with new tools:
```kotlin
chat.resetContext(systemPrompt = "Updated prompt", tools = listOf(newTool))
```
#### Tool calling and the context
As with most things made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/kotlin/vision](https://docs.nobodywho.ooo/kotlin/vision)
Easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/kotlin/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/kotlin/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts:
1. Multimodal LLM that can consume image-tokens and/or audio-tokens
2. Projection model that converts images to image-tokens and/or audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 manage both!
Usually, the projection model includes `mmproj` in its name.
If you are unsure which ones to pick, try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true).
Load the projection model alongside the main model:
```kotlin
import ai.nobodywho.Model
import ai.nobodywho.Chat
val model = Model.load(
modelPath = "./multimodal-model.gguf",
projectionModelPath = "./mmproj.gguf"
)
val chat = Chat(model = model)
```
**Info:**
The language model and projection model must **fit** together, as they are trained together.
You can't take an arbitrary projection model and pair it with any LLM.
#### Composing a prompt
With the model configured, compose a multimodal prompt using `Prompt`:
```kotlin
import ai.nobodywho.Prompt
val response = chat.ask(Prompt(
Prompt.Text("Tell me what you see in the image and what you hear in the audio."),
Prompt.Image("./dog.png"),
Prompt.Audio("./sound.mp3"),
)).completed()
println(response) // It's a dog!
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/kotlin/voice-activity-detection](https://docs.nobodywho.ooo/kotlin/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart.
#### Streaming
The main use case for VAD is streaming audio into the model chunk by chunk. This is useful, for
example, to detect when to stop listening to the microphone and start running
transcription/answer generation.
To do that, we provide a `push` interface:
```kotlin
import ai.nobodywho.VoiceActivityDetection
import ai.nobodywho.VoiceActivityDetectionEvent
import ai.nobodywho.SpeechToText
val vad = VoiceActivityDetection.load(sampleRate = 16000u, source = "hf://onnx-community/silero-vad")
val stt = SpeechToText.load(source = "hf://onnx-community/whisper-base")
while (true) {
val chunk = readMic() // however you're reading from the microphone
if (vad.push(chunk) == VoiceActivityDetectionEvent.SPEECH_ENDED) break
}
val speech = vad.finish()
val transcription = stt.transcribePcm(speech, sampleRate = 16000u).completed()
println(transcription)
```
`VoiceActivityDetection` acts as a buffer: every `push()` call tells you the current state
(`SPEECH_STARTED`, `SPEECH_ENDED`, `SPEECH`, or `SILENCE`), so you can decide when to stop
listening. Once you do, `finish()` gives you back the buffered audio for just the speech segment,
and resets internal state so it's ready for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `segment()` method is for:
```kotlin
import ai.nobodywho.VoiceActivityDetection
import ai.nobodywho.SpeechToText
val vad = VoiceActivityDetection.load(sampleRate = 16000u, source = "hf://onnx-community/silero-vad")
val stt = SpeechToText.load(source = "hf://onnx-community/whisper-base")
val audio = readWavPcm("recording.wav") // a full recording as i16 PCM samples
for (speech in vad.segment(audio)) {
val transcription = stt.transcribePcm(speech, sampleRate = 16000u).completed()
println(transcription)
}
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance. For that,
we provide numerous params that you can tweak:
```kotlin
val vad = VoiceActivityDetection.load(
sampleRate = 16000u,
// VAD is currently fixed to Silero ONNX, but you can change the source.
source = "hf://onnx-community/silero-vad",
// Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
threshold = 0.5f,
// Determines the minimum duration classified as speech.
minSpeechDurationMs = 250u,
// Determines the minimum duration classified as silence.
minSilenceDurationMs = 250u,
// Determines how much audio to keep before the official SPEECH_STARTED, to avoid cutting off the start.
prerollDurationMs = 500u,
)
```
---
## Python
This section documents Python 2.0.0, the latest released version.
### Chat
Source: [https://docs.nobodywho.ooo/python/chat](https://docs.nobodywho.ooo/python/chat)
As you may have noticed in the [welcome guide](https://docs.nobodywho.ooo/python/), every interaction with your LLM starts by instantiating a `Chat` object.
In the following sections, we talk about which configuration options it has, and when to use them.
You can pass `"auto"` as the model path to select a chat model based on available memory.
#### Prompts and responses
The `Chat.ask()` function is central to NobodyWho. This function sends your message to the LLM, which then starts generating a response.
```python
from nobodywho import Chat, TokenStream
chat = Chat("./model.gguf")
response: TokenStream = chat.ask("Is water wet?")
```
The return type of `ask` is a `TokenStream`.
If you want to start reading the response as soon as possible, you can just iterate over the `TokenStream`.
Each token is either an individual word or fragments of a word.
```python continuation
for token in response:
print(token, end="", flush=True)
print("\n")
```
If you just want to get the complete response, you can call `TokenStream.completed()`.
This will block until the model is done generating its entire response.
```python continuation
full_response: str = response.completed()
```
All of your messages and the model's responses are stored in the `Chat` object, so the next time you call `Chat.ask()`, it will remember the previous messages.
#### Chat history
If you want to inspect the messages inside the `Chat` object, you can use `get_chat_history`.
```python continuation
msgs: list[dict] = chat.get_chat_history()
print(msgs[0]["content"]) # "Is water wet?"
```
Similarly, if you want to edit what messages are in the context, you can use `set_chat_history`:
```python continuation
chat.set_chat_history([{
"role": "user",
"content": "What is water?",
"assets": []
}])
```
#### System prompt
A system prompt is a special message put into the chat context, which should guide its overall behavior.
Some models ship with a built-in system prompt. If you don't specify a system prompt yourself, NobodyWho will fall back to using the model's default system prompt.
You can specify a system prompt when initializing a `Chat`:
```python
from nobodywho import Chat
chat = Chat("./model.gguf", system_prompt="You are a mischievous assistant!")
```
This `system_prompt` is then persisted until the chat context is `reset`.
#### Context
The context is the text window which the LLM currently considers. Specifically this is the number of tokens the LLM keeps in memory for your current conversation.
As bigger context size means more computational overhead, it makes sense to constrain it. This can be done with `n_ctx` setting, again at the time of creation:
```python
chat = Chat("./model.gguf", n_ctx=4096)
```
The default value is `4096`, however this is mainly useful for short and simple conversations. Choosing the right context size is quite important and depends heavily on your use case. You can check the maximum context size the model was trained with using `model.max_ctx()` — setting `n_ctx` above this value has no benefit.
Even with properly selected context size it might happen that you fill up your entire context during a conversation. When this happens, NobodyWho will shrink the context for you. Currently this is done by removing old messages (apart from the system prompt and the first user message) from the chat history, until the size reaches `n_ctx / 2`. The KV cache is also updated automatically. In the future we plan on adding more advanced methods of context shrinking.
Again, `n_ctx` is fixed to the `Chat` instance, so it is currently not possible to change the size after `Chat` is created. To reset the current context content, just call `.reset()` with the new system prompt and potentially changed tools.
```python continuation
chat.reset(system_prompt="New system prompt", tools=[])
```
If you don't want to change the already set defaults (`system_prompt`, `tools`), but only reset the context, then go for `reset_history`.
To inspect how much of the context is currently in use, call `.stats()`:
```python continuation
stats = chat.stats()
print(f"Using {stats.context_used} of {stats.context_size} tokens")
```
#### CPU threads
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Hyperthread siblings and efficiency cores end up pacing the whole
thread pool, so using every CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads)
for the numbers.
Override it with `n_threads` when you want to leave CPU headroom for other work, or when
NobodyWho could not read your machine's topology (it logs a warning if so):
```python
chat = Chat("./model.gguf", n_threads=4)
```
Leave it unset — or pass `None` — to keep the detected default. Values above your CPU count are
clamped, and it has little effect when the model is offloaded to the GPU.
#### Sharing model between contexts
There are scenarios where you would like to keep separate chat contexts (e.g. for every user of your app), but have only one model loaded. With plain `Chat` this is not possible.
For this use case, instead of the path to the `.gguf` model, you can pass in `Model` object, which can be shared between multiple `Chat` instances.
```python
from nobodywho import Chat, Model
model = Model('./model.gguf')
chat1 = Chat(model)
chat2 = Chat(model)
...
```
NobodyWho will then take care of the separation, such that your chat histories won't collide or interfere with each other, while having only one model loaded.
#### Asynchronous model loading
Loading a model into memory can take a few seconds - longer if you're using a really large model.
If you want to load the model without blocking execution of your application (e.g. to keep UI responsive), you can load the model asynchronously:
```python
import asyncio
from nobodywho import ChatAsync, Model
async def main():
model = await Model.load_model_async("./model.gguf")
chat = ChatAsync(model)
asyncio.run(main())
```
#### GPU
Instantiating `Model` is also useful, when enabling GPU acceleration. This can be done as:
```python
Model('./model.gguf', use_gpu_if_available=True)
```
So far, NobodyWho relies purely on [Vulkan](https://www.vulkan.org), however support
of more architectures is planned (for details check out our [issues](https://github.com/nobodywho-ooo/nobodywho/issues) or join us on [Discord](https://discord.gg/qhaMc2qCYB)).
#### Speculative decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target model verify several candidate tokens per forward pass. When it works this can give a significant speedup — but see the warning below before enabling it.
To use MTP, load the model with a compatible draft-heads gguf (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B) and pass an `MtpConfig` when constructing the chat:
```python notest
from nobodywho import Chat, Model, MtpConfig
model = Model("./gemma-4-e2b.gguf", draft_model_path="./mtp-gemma-4-e2b.gguf")
# MtpConfig() uses the default drafter tuning; tune with MtpConfig(k_max=..., p_min=...)
chat = Chat(model, mtp=MtpConfig())
```
Loading the draft heads adds around 5% to VRAM usage. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
#### Template Variables
Chat templates are used internally by models to format conversation history into the expected prompt format. Different models may support different template variables that control specific behaviors. Template variables are boolean flags passed to the chat template that can enable or disable certain features.
##### Using Template Variables
You can set template variables when creating a chat or modify them on existing instances:
```python
# Set template variables when creating a chat
chat = Chat("./model.gguf", template_variables={"enable_thinking": True})
```
You can also modify template variables on an existing chat instance:
```python continuation
# Set a single template variable
chat.set_template_variable("enable_thinking", True)
# Set multiple template variables at once
chat.set_template_variables({
"enable_thinking": True,
"verbose_mode": False
})
# Get current template variables
variables = chat.get_template_variables()
print(variables) # {"enable_thinking": True, "verbose_mode": False}
```
With the next message sent, the updated settings will be propagated to the model.
##### Example: Qwen3 and Qwen3.5 Reasoning
The Qwen3 and Qwen3.5 model families support the `enable_thinking` template variable, which controls whether the model should engage in explicit reasoning steps before answering:
```python
# Enable thinking mode for Qwen models
chat = Chat("./model.gguf", template_variables={"enable_thinking": True})
chat.ask("Solve this logic puzzle: ...")
```
When `enable_thinking` is enabled, these models will show their reasoning process before providing the final answer.
##### Model-Specific Variables
Different models may support different template variables depending on their chat template implementation. The available variables and their effects depend entirely on how the model's chat template is designed. Check your model's documentation to see which template variables are supported.
**Info:**
Note that template variables are model-specific. If a model's chat template doesn't use a specific variable, that variable will be ignored gracefully.
##### Backward Compatibility
For backward compatibility, the deprecated `allow_thinking` parameter is still available but internally sets the `enable_thinking` template variable:
```python
# Deprecated - use template_variables instead
chat = Chat("./model.gguf", allow_thinking=True)
chat.set_allow_thinking(True)
```
---
### Downloading models
Source: [https://docs.nobodywho.ooo/python/downloading-models](https://docs.nobodywho.ooo/python/downloading-models)
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to observe a download in progress, how to access gated/private models, and how to inspect what's already in the local cache.
#### Supported model path formats
The `model_path` argument to `Chat`, `download_model`, and friends accepts:
| Form | Example | Notes |
| ---- | ------- | ----- |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
| Local path | `./model.gguf` | Used as-is |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs.
#### Tracking download progress
When loading a remote model, pass an `on_download_progress` callback to observe the download. It receives `(downloaded_bytes, total_bytes)` and is not called for cached or local files. If you don't pass anything, NobodyWho prints a default terminal progress bar.
```python
from nobodywho import download_model
model_path = download_model(
'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
on_download_progress=lambda downloaded, total: print(f"{downloaded}/{total} bytes"),
)
```
#### Downloading a gated model
Some HuggingFace models are private or gated by a license you need to accept. In both cases you need to be authorized to download the model weights.
You can manually download the GGUF file via your web browser and then point `Chat` at the local path:
```python
from nobodywho import Chat
chat = Chat('./model.gguf')
```
Or use `download_model` with an `Authorization` header:
```python
from nobodywho import Chat, download_model
model_path = download_model(
'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
headers={ "Authorization": "Bearer your_hf_token" }
)
chat = Chat(model_path)
```
You can generate a HuggingFace token in [your account settings](https://huggingface.co/settings/tokens).
#### Inspecting the model cache
`get_cached_models` returns every `.gguf` model that lives in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `download_model` and by `Chat`'s `huggingface:` paths.
```python
from nobodywho import get_cached_models
for path, size in get_cached_models():
print(f"{path}: {size / 1024 / 1024:.1f} MiB")
```
- Paths are absolute.
- Sizes are in bytes.
- The list is empty if nothing has been downloaded yet.
- Raises `RuntimeError` if the cache directory cannot be read.
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/python/embeddings-and-rag](https://docs.nobodywho.ooo/python/embeddings-and-rag)
When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.
#### Understanding Embeddings
Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.
For example, "Schedule a meeting for next Tuesday" and "Book an appointment next week" would have very similar embeddings, despite using different words.
#### The Encoder
The `Encoder` object converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).
We recommend you first try [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf).
```python
from nobodywho import Encoder
encoder = Encoder('./embedding-model.gguf')
embedding = encoder.encode("What is the weather like?")
print(f"Vector with {len(embedding)} dimensions")
```
The resulting embedding is a list of floats (typically 384 or 768 dimensions depending on the model).
##### Batch Encoding
Use `encode_batch` when encoding multiple texts. It batches the work internally and returns one embedding per text in input order.
```python continuation
texts = [
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
]
embeddings = encoder.encode_batch(texts)
```
##### Comparing Embeddings
To measure how similar two pieces of text are, compare their embeddings using cosine similarity:
```python
from nobodywho import Encoder, cosine_similarity
encoder = Encoder('./embedding-model.gguf')
query = encoder.encode("How do I reset my password?")
doc1 = encoder.encode("You can reset your password in the account settings")
doc2 = encoder.encode("The password requirements include 8 characters minimum")
similarity1 = cosine_similarity(query, doc1)
similarity2 = cosine_similarity(query, doc2)
print(f"Document 1 similarity: {similarity1:.3f}") # Higher score
print(f"Document 2 similarity: {similarity2:.3f}") # Lower score
```
Cosine similarity returns a value between -1 and 1, where 1 means identical meaning and -1 means opposite meaning.
##### Practical Example: Finding Relevant Documents
```python
from nobodywho import Encoder, cosine_similarity
encoder = Encoder('./embedding-model.gguf')
# Your knowledge base
documents = [
"Python supports multiple programming paradigms including object-oriented and functional",
"JavaScript is primarily used for web development and runs in browsers",
"SQL is a domain-specific language for managing relational databases",
"Git is a version control system for tracking changes in source code"
]
# Pre-compute document embeddings in a batch
doc_embeddings = encoder.encode_batch(documents)
# Search query
query = "What language should I use for database queries?"
query_embedding = encoder.encode(query)
# Find the most relevant document
similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
best_idx = similarities.index(max(similarities))
print(f"Most relevant: {documents[best_idx]}")
print(f"Similarity score: {similarities[best_idx]:.3f}")
```
#### The CrossEncoder for Better Ranking
While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents to determine how well the document answers the query.
The key difference: embeddings compare vector similarity, while cross-encoders understand the relationship between query and document.
##### Why CrossEncoder Matters
Consider this example:
```
Query: "What are the office hours for customer support?"
Documents: [
"Customer asked: What are the office hours for customer support?",
"Support team responds: Our customer support is available Monday-Friday 9am-5pm EST",
"Note: Weekend support is not available at this time"
]
```
Using embeddings alone, the first document scores highest (most similar to the query) even though it provides no useful information. A cross-encoder correctly identifies that the second document actually answers the question.
##### Using CrossEncoder
```python
from nobodywho import CrossEncoder
# Download a reranking model like bge-reranker-v2-m3-Q8_0.gguf
crossencoder = CrossEncoder('./reranker-model.gguf')
query = "How do I install Python packages?"
documents = [
"Someone previously asked about Python packages",
"Use pip install package-name to install Python packages",
"Python packages are not included in the standard library"
]
# Get relevance scores for each document
scores = crossencoder.rank(query, documents)
print(scores) # [0.23, 0.89, 0.45] - second doc scores highest
```
##### Automatic Sorting
For convenience, use `rank_and_sort` to get documents sorted by relevance:
```python continuation
# Returns list of (document, score) tuples, sorted by score
ranked_docs = crossencoder.rank_and_sort(query, documents)
for doc, score in ranked_docs:
print(f"[{score:.3f}] {doc}")
```
This returns documents ordered from most to least relevant.
#### Building a RAG System
Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.
Here's a complete example building a customer service assistant with access to company policies:
```python
from nobodywho import Chat, CrossEncoder
# Initialize the cross-encoder for document ranking
crossencoder = CrossEncoder('./reranker-model.gguf')
# Your knowledge base
knowledge = [
"Our company offers a 30-day return policy for all products",
"Free shipping is available on orders over $50",
"Customer support is available via email and phone",
"We accept credit cards, PayPal, and bank transfers",
"Order tracking is available through your account dashboard"
]
# Create a tool that searches the knowledge base
from nobodywho import tool
@tool(description="Search the knowledge base for relevant information")
def search_knowledge(query: str) -> str:
# Rank all documents by relevance to the query
ranked = crossencoder.rank_and_sort(query, knowledge)
# Return top 3 most relevant documents
top_docs = [doc for doc, score in ranked[:3]]
return "\n".join(top_docs)
# Create a chat with access to the knowledge base
chat = Chat(
'./model.gguf',
system_prompt="You are a customer service assistant. Use the search_knowledge tool to find relevant information from our policies before answering customer questions.",
tools=[search_knowledge]
)
# The chat will automatically search the knowledge base when needed
response = chat.ask("What is your return policy?").completed()
print(response)
```
The LLM will call the `search_knowledge` tool, receive the most relevant documents, and use them to generate an accurate answer.
#### Async API
For non-blocking operations, use `EncoderAsync` and `CrossEncoderAsync`:
```python
import asyncio
from nobodywho import EncoderAsync, CrossEncoderAsync
async def main():
encoder = EncoderAsync('./embedding-model.gguf')
crossencoder = CrossEncoderAsync('./reranker-model.gguf')
# Generate embeddings asynchronously
embeddings = await encoder.encode_batch([
"What is the weather?",
"What is the return policy?",
])
# Rank documents asynchronously
query = "What is our refund policy?"
docs = ["Refunds processed within 5-7 business days", "No refunds on sale items", "Contact support to initiate refund"]
ranked = await crossencoder.rank_and_sort(query, docs)
for doc, score in ranked:
print(f"[{score:.3f}] {doc}")
asyncio.run(main())
```
#### Recommended Models
##### For Embeddings
- [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf) - Good balance of speed and quality (~25MB)
- Supports English text with 384-dimensional embeddings
##### For Cross-Encoding (Reranking)
- [bge-reranker-v2-m3-Q8_0.gguf](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf) - Multilingual support with excellent accuracy
#### Best Practices
**Precompute embeddings**: If you have a fixed knowledge base, generate embeddings once and reuse them. Don't re-encode the same documents repeatedly.
**Use embeddings for filtering**: When working with large document collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank them.
**Limit cross-encoder inputs**: Cross-encoders are more expensive than embeddings. Don't pass thousands of documents to `rank()` - filter first with embeddings.
**Choose appropriate context size**: The `n_ctx` parameter (default 2048) should match your model's recommended context size. Check the model documentation.
```python
# For longer documents, increase context size
encoder = Encoder('./embedding-model.gguf', n_ctx=4096)
crossencoder = CrossEncoder('./reranker-model.gguf', n_ctx=4096)
```
#### Complete RAG Example
Here's a full example showing a two-stage retrieval system:
```python
from nobodywho import Chat, Encoder, CrossEncoder, cosine_similarity, tool
# Initialize models
encoder = Encoder('./embedding-model.gguf')
crossencoder = CrossEncoder('./reranker-model.gguf')
# Large knowledge base
knowledge_base = [
"Python 3.11 introduced performance improvements through faster CPython",
"The Django framework is used for building web applications",
"NumPy provides support for large multi-dimensional arrays",
"Pandas is the standard library for data manipulation and analysis",
# ... 100+ more documents
]
# Precompute embeddings for all documents
doc_embeddings = encoder.encode_batch(knowledge_base)
@tool(description="Search the knowledge base for information relevant to the query")
def search(query: str) -> str:
# Stage 1: Fast filtering with embeddings
query_embedding = encoder.encode(query)
similarities = [
(doc, cosine_similarity(query_embedding, doc_emb))
for doc, doc_emb in zip(knowledge_base, doc_embeddings)
]
# Get top 20 candidates
candidates = sorted(similarities, key=lambda x: x[1], reverse=True)[:20]
candidate_docs = [doc for doc, _ in candidates]
# Stage 2: Precise ranking with cross-encoder
ranked = crossencoder.rank_and_sort(query, candidate_docs)
# Return top 3 most relevant
top_results = [doc for doc, score in ranked[:3]]
return "\n---\n".join(top_results)
# Create RAG-enabled chat
chat = Chat(
'./model.gguf',
system_prompt="You are a technical documentation assistant. Always use the search tool to find relevant information before answering programming questions.",
tools=[search]
)
# The chat automatically searches and uses retrieved documents
response = chat.ask("What Python libraries are best for data analysis?").completed()
print(response)
```
This two-stage approach combines the speed of embeddings with the accuracy of cross-encoders, making it efficient even for large knowledge bases.
---
### Getting started
Source: [https://docs.nobodywho.ooo/python/](https://docs.nobodywho.ooo/python/)
#### How do I get started?
First, install `nobodywho`.
```bash
pip install nobodywho
```
Or preferably:
```bash
uv add nobodywho
```
Next, pick a model. NobodyWho can download GGUF models directly from Hugging Face — just pass a `huggingface:` path. See [model selection](https://docs.nobodywho.ooo/docs/model-selection) for recommendations.
Then make a `Chat` object and call `.ask()`!
```python
from nobodywho import Chat
chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
response = chat.ask('Is water wet?')
# print each token as it is generated
for token in response:
print(token, end="", flush=True)
# ...or get the entire response as a single string
full_response = response.completed()
print(full_response)
```
This is a super simple example, but we believe that examples which do simple things, should be simple!
To get a full overview of the functionality provided by NobodyWho, simply keep reading.
---
### Logging and Troubleshooting
Source: [https://docs.nobodywho.ooo/python/logging-and-troubleshooting](https://docs.nobodywho.ooo/python/logging-and-troubleshooting)
The python bindings for NobodyWho integrate with python's standard `logging` utilities.
In short, to enable debug logs:
```python
import logging
logging.basicConfig(level=logging.DEBUG)
```
This can be useful for getting some insight into what the model is choosing to do and when.
For example when tool calls are made, when context shifting happens, etc.
---
### Sampling
Source: [https://docs.nobodywho.ooo/python/sampling](https://docs.nobodywho.ooo/python/sampling)
The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which using NobodyWho you can freely modify,
to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).
#### Sampler presets
To get a quick start, NobodyWho offers a couple of well-known presets, which you can quickly utilize.
For example, if you want to increase or decrease the "creativity" of your model, select our `temperature` preset:
```python
from nobodywho import SamplerPresets
Chat("./model.gguf", sampler=SamplerPresets.temperature(0.2))
```
Setting `temperature` to `0.2`, will then affect the sampler when choosing the next token, making the distribution less flat and therefore the model will favour more probable tokens.
To see the whole list of presets, check out the `SamplerPresets` class:
```python
class SamplerPresets:
def default() -> SamplerConfig: ...
def dry() -> SamplerConfig: ...
def greedy() -> SamplerConfig: ...
def json() -> SamplerConfig: ...
def temperature(temperature: float) -> SamplerConfig: ...
def top_k(top_k: int) -> SamplerConfig: ...
def top_p(top_p: float) -> SamplerConfig: ...
# Constrain output to a specific format:
def constrain_with_json_schema(schema: str) -> SamplerConfig: ...
def constrain_with_regex(pattern: str) -> SamplerConfig: ...
def constrain_with_grammar(grammar: str) -> SamplerConfig: ...
```
#### Structured output
One of the most useful features is constraining the model to produce structured output —
this gives you a hard guarantee that the output matches a specific format, rather than
relying on the model to get it right on its own.
##### Regular expressions
For simpler patterns, you can constrain the output with a regex:
```python
# Force the model to answer with exactly "yes" or "no"
chat = Chat('./model.gguf', sampler=SamplerPresets.constrain_with_regex(r"yes|no"))
answer = chat.ask("Is the sky blue?").completed()
```
##### JSON schema
In some use-cases it might be useful to let the LLM generate JSON output.
This could be done either in the simple way, just enforcing any JSON by the preset:
```python
Chat('./model.gguf', sampler=SamplerPresets.json())
```
Or utilizing JSON schemas to really force the LLM to give you the specific object shapes
that you want:
```python
import json
chat = Chat('./model.gguf', sampler=SamplerPresets.constrain_with_json_schema({
"type": "object",
"properties": {
"name": {"type": "string", "maxLength": 50},
"age": {"type": "integer"}
},
"required": ["name", "age"],
"additionalProperties": False
}))
response = chat.ask("Give me a person as JSON with name and age fields.").completed()
person = json.loads(response) # always valid JSON matching the schema
```
##### Custom grammars
For cases where JSON schema and regex are not expressive enough, you can supply a custom grammar.
`constrain_with_grammar` accepts both **Lark** syntax and **GBNF** (llama.cpp format) -
NobodyWho automatically converts GBNF to Lark before passing it to the inference engine.
**Lark syntax** (recommended):
```python
sampler = SamplerPresets.constrain_with_grammar("""
start: record (NEWLINE record)* NEWLINE?
record: field ("," field)*
field: /[^,\"\\n\\r]+/
NEWLINE: /\\r?\\n/
""")
```
**GBNF syntax** (also accepted):
```python
sampler = SamplerPresets.constrain_with_grammar("""
file ::= record (newline record)* newline?
record ::= field ("," field)*
field ::= /[^,"\\n\\r]+/
newline ::= "\\r\\n" | "\\n"
""")
```
See the [Lark documentation](https://lark-parser.readthedocs.io/en/latest/grammar.html) and the
[GBNF specification](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) for the
full grammar syntax.
**Info:**
The older `SamplerPresets.grammar()` method is deprecated. Use `SamplerPresets.constrain_with_grammar()` instead - it accepts both Lark and GBNF strings and should run faster!
#### Defining your own samplers
Sampler presets abstract away some control, that you might want - for example, if you
want to chain samplers, change more "advanced" parameters, etc. For that use case,
we provide `SamplerBuilder` class:
```python
from nobodywho import SamplerBuilder
Chat(
"./model.gguf",
sampler=SamplerBuilder()
.temperature(0.8)
.top_k(5)
.dist()
)
```
With `SamplerBuilder` you can chain multiple steps together and then select how do you
want to sample from the distribution. Keep in mind, that `SamplerBuilder` provides two
types of methods: ones which modify the distribution (returning again the instance of
`SamplerBuilder`) and ones which sample from the distribution (returning `SamplerConfig`).
So in order to have the sampler working properly and not giving you type errors, be careful
to always end the chain with one of the sampling steps (e.g. `dist`, `greedy`, `mirostat_v2`, etc.).
For reproducible output, set the RNG seed with `.seed(value)` anywhere in the chain.
It is consumed by every random sampler in the chain — `dist`, `mirostat_v1`, `mirostat_v2`,
and the `xtc` shift step. `greedy` ignores it. If unset, a default seed is used.
```python
sampler = SamplerBuilder().temperature(0.8).top_k(5).seed(42).dist()
```
##### Available sampling steps
Pick any of the **shift steps** below (each reshapes the token distribution), then finish with one **terminal step** that picks the token — exactly like the `.temperature(0.8).top_k(5).dist()` chain above.
Shift steps — add as many as you want, applied in order:
- `.top_k(40)` — keep only the 40 most likely tokens
- `.top_p(0.95, min_keep=1)` — nucleus: keep the top tokens up to 95% of the probability mass
- `.min_p(0.05, min_keep=1)` — drop tokens below 5% of the most likely token's probability
- `.typical_p(0.9, min_keep=1)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `.xtc(0.5, 0.1, min_keep=1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `.temperature(0.8)` — below 1.0 = more focused, above 1.0 = more random
- `.penalties(64, 1.1, 0.0, 0.0)` — per-token repetition penalty: `last_n, repeat, freq, present` (`repeat` 1.0 = off)
- `.dry(0.8, 1.75, 2, -1, ["\n"])` — penalty for repeated *phrases*: `multiplier, base, allowed_length, last_n, seq_breakers`
- `.seed(42)` — fix the RNG for reproducible output
- `.grammar(...)` — deprecated; use the `constrain_with_*` presets above
Terminal step — one of these turns the chain into a `SamplerConfig`, so finish with exactly one:
- `.dist()` — pick a token with weighted randomness (the usual choice)
- `.greedy()` — always take the most likely token
- `.mirostat_v1(5.0, 0.1, 100)` / `.mirostat_v2(5.0, 0.1)` — steer output "surprise" toward a target
`min_keep` is the floor on how many tokens survive a cut (`1` is fine).
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/python/speech-to-text](https://docs.nobodywho.ooo/python/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format.
```python notest
from nobodywho import SpeechToText
stt = SpeechToText(source="hf://onnx-community/whisper-base")
text = stt.transcribe_file("recording.mp3").completed()
print(text)
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribe_pcm` is available:
```python notest
text = stt.transcribe_pcm(samples, sample_rate=16000).completed()
```
In order to make this work, the buffer needs to be mono i16 PCM samples. The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with classic Chat models, streaming is available, so the transcription can be consumed token by token:
```python notest
for piece in stt.transcribe_file("recording.mp3"):
print(piece, end="", flush=True)
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `source` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```python notest
stt = SpeechToText(
source="hf://onnx-community/whisper-base",
quantization="q4",
)
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, pass its ISO 639-1 code as `language` to skip detection and improve performance:
```python notest
stt = SpeechToText(
source="hf://onnx-community/whisper-base",
language="en",
)
```
---
### Streaming & Async API
Source: [https://docs.nobodywho.ooo/python/streaming-and-async-api](https://docs.nobodywho.ooo/python/streaming-and-async-api)
Synchronously waiting for the full response to arrive can be costly. If your application
domain allows you to, you would ideally want to stream the tokens to the user as soon
as they arrive, and spend the time in between doing useful work, rather than just waiting.
#### Streaming tokens
Allowing streaming is super simple. Instead of calling the `.completed()` method, just iterate
over the response object:
```python
chat = Chat('./model.gguf')
response = chat.ask('How are you?')
for token in response:
print(token, end="", flush=True)
```
Still, bear in mind that for the individual tokens, you are waiting synchronously.
#### Async API
If you don't want to wait synchronously, swap out the `Chat` object for a `ChatAsync`. All of the API stays the same, so either you can opt for a full, completed message:
```python
import asyncio
from nobodywho import ChatAsync
async def main():
chat = ChatAsync('./model.gguf')
response = await chat.ask('How are you?').completed()
print(response)
asyncio.run(main())
```
Or again stream tokens:
```python
import asyncio
from nobodywho import ChatAsync
async def main():
chat = ChatAsync('./model.gguf')
response = chat.ask('How are you?')
async for token in response:
print(token, end="", flush=True)
asyncio.run(main())
```
Similarly, the other model types we support also implement async behaviour, so
you can go for `EncoderAsync` and `CrossEncoderAsync`, which are
both part of the [embeddings & rag functionality](https://docs.nobodywho.ooo/python/embeddings-and-rag), or
`SpeechToTextAsync`, part of the [speech-to-text functionality](https://docs.nobodywho.ooo/python/speech-to-text).
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/python/text-to-speech](https://docs.nobodywho.ooo/python/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
```python notest
from pathlib import Path
from nobodywho import TextToSpeech
tts = TextToSpeech(
source="hf://NobodyWho/Kokoro-82M", # Hugging Face repo ID or local folder with the model files.
voice="bf_emma", # Voice to use from the model.
language="en-gb", # Language code for the input text.
)
# Generate WAV bytes for this sentence.
wav = tts.synthesize("Hello from NobodyWho!")
# Save the audio to a file.
Path("out.wav").write_bytes(wav)
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, or a local directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/python/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/python/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```python notest
tts = TextToSpeech(
source="hf://NobodyWho/Kokoro-82M",
voice="bf_emma",
language="en-gb",
)
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.0`.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```python notest
tts = TextToSpeech(
source="hf://Supertone/supertonic-3",
language="en",
)
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.05`.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Must be greater than `0`; defaults to `8`.
- `silence_duration`: seconds of silence between long text chunks. Higher values add longer pauses. Must be `0` or higher; defaults to `0.3`.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```python notest
tts = TextToSpeech(
source="hf://KevinAHM/pocket-tts-onnx",
voice="alba",
language="english_2026-04",
huggingface_token="hf_...", # Alternative: set the HF_TOKEN environment variable.
)
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Defaults to `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Defaults to `int8`.
- `temperature`: controls how varied the speech sounds. Defaults to `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingface_token`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory or a custom source that NobodyWho cannot recognize:
```python notest
tts = TextToSpeech(
source="/path/to/local/kokoro-folder",
architecture="kokoro",
)
```
Supported architecture values are `kokoro`, `pocket-tts`, and `supertonic`.
#### GPU
TextToSpeech uses GPU acceleration by default when available. Disable it with `device="cpu"`:
```python notest
tts = TextToSpeech(
source="hf://Supertone/supertonic-3",
device="cpu",
)
```
#### Local model folder format
When `source` is a local directory, point it at the top-level model folder and pass the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/python/tool-calling](https://docs.nobodywho.ooo/python/tool-calling)
To give your LLM the ability to interact with the outside world, you will need tool calling.
**Info:**
Note that **not every model** supports tool calling. If the model does not have
such an option, it might not call your tools.
For reliable tool calling, we recommend trying the [Qwen](https://huggingface.co/Qwen/models) family of models.
#### Declaring a tool
A tool can be created from any (synchronous) python function, which returns a string.
To perform the conversion, you simply need to use the `@tool` decorator. To get
a good sense of how such a tool can look like, consider this geometry example:
```python
import math
from nobodywho import tool, Chat
@tool(description="Calculates the area of a circle given its radius")
def circle_area(radius: float) -> str:
area = math.pi * radius ** 2
return f"Circle with radius {radius} has area {area:.2f}"
```
As you can see, every `@tool` definition has to be complemented by a description
of what such tool does. To let your LLM use it, simply add it when creating `Chat`:
```python continuation
chat = Chat('./model.gguf', tools=[circle_area])
```
NobodyWho then figures out the right tool calling format, inspects the names and types of the parameters,
and configures the sampler.
Naturally, more tools can be defined and the model can chain the calls for them:
```python
import os
from pathlib import Path
from nobodywho import Chat, tool
@tool(description="Gets path of the current directory")
def get_current_dir() -> str:
return os.getcwd()
@tool(description="Lists files in the given directory", params={"path": "a relative or absolute path to a directory"})
def list_files(path: str) -> str:
files = [f.name for f in Path(path).iterdir() if f.is_file()]
return f"Files: {', '.join(files)}"
@tool(description="Gets the size of a file in bytes")
def get_file_size(filepath: str) -> str:
size = Path(filepath).stat().st_size
return f"File size: {size} bytes"
chat = Chat('./model.gguf', tools=[get_current_dir, list_files, get_file_size])
response = chat.ask('What is the biggest file in my current directory?').completed()
print(response) # The largest file in your current directory is `model.gguf`.
```
#### Providing parameter descriptions
When a tool call is declared, information about the description, the types and the parameters is provided to the model, so it knows it can use it. Crucially, also parameter names are provided.
If those are not enough, you can decide to provide additional information by the `params` parameter:
```python
from nobodywho import tool
@tool(
description="Given a longitude and latitude, gets the current temperature.",
params={
"lon": "Longitude - that is the vertical one!",
"lat": "Latitude - that is the horizontal one!"
}
)
def get_current_temperature(lon: str, lat: str) -> str:
...
```
These will be then appended to the information provided to model, so it can better navigate itself
when using the tool.
#### Pre-packaged tools
We ship NobodyWho with two packaged-in tools, which are general enough for mutliple use-cases - [monty](https://github.com/pydantic/monty) Python interpreter
and [bashkit](https://github.com/everruns/bashkit) Bash interpreter. Both of them should serve similar purpose - to give your small LLM a better chance to answer
questions requiring precise reasoning or some kind of computation, possibly on a big context.
The usage is straightforward. Start with importing either `python_tool` or `bash_tool` from `nobodywho`.
```python
from nobodywho import python_tool, bash_tool
chat = Chat('./model.gguf', tools=[python_tool(), bash_tool()])
```
Lastly, keep in mind that for most use-cases it is reasonable to constraint the tools with some limits regarding memory and computation time,
so that you don't end up executing infinite loop code. To solve this, `python_tool` provides `max_duration`, `max_memory` and `max_recursion_depth`
and `bash_tool` provides `max_commands`.
#### Tool calling and the context
As with everything made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/python/vision](https://docs.nobodywho.ooo/python/vision)
A picture is worth a thousand words (or at least a thousand tokens).
With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/python/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/python/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts for making this work:
1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 even manage both!
Usually, the projection model then includes `mmproj` in its name.
If you are unsure which ones to pick, or just want a reasonable default, you can try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true),
which can do both image and audio.
With the downloaded GGUFs, you can simply add the projection model as:
```python
from nobodywho import Model, Chat
model = Model("./vision-model.gguf", projection_model_path="./projection_model.gguf")
chat = Chat(
model, system_prompt="You are a helpful assistant, that can hear and see stuff!"
)
```
**Info:**
The language model and projection model have to **fit** together, as they are trained together!
Unfortunately you can't just take projection model and a LLM that you like and expect them
to work together.
#### Composing a prompt object
With the model configured, all that is left is to compose the prompt and send it to the model.
That is done through the `Prompt` object.
```python notest
from nobodywho import Audio, Image, Prompt, Text
prompt = Prompt([
Text("Tell me what you see in the image and what you hear in the audio."),
Image("./dog.png"),
Audio("./sound.mp3")
])
chat.ask(prompt).completed() # It's a dog and a penguin!
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/python/voice-activity-detection](https://docs.nobodywho.ooo/python/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart.
#### Streaming
The main use case for VAD is streaming audio into the model chunk by chunk. This is useful, for
example, to detect when to stop listening to the microphone and start running
transcription/answer generation.
To do that, we provide a `.push` interface:
```python notest
from nobodywho import VoiceActivityDetection, VoiceActivityDetectionEvent, SpeechToText
vad = VoiceActivityDetection(source="hf://onnx-community/silero-vad", sample_rate=16000)
stt = SpeechToText(source="hf://onnx-community/whisper-base")
while chunk := read_mic(): # assuming this is how you're reading from the microphone
if vad.push(chunk) == VoiceActivityDetectionEvent.SpeechEnded:
break
speech = vad.finish()
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)
```
`VoiceActivityDetection` acts as a buffer: every `.push()` call tells you the current state
(`SpeechStarted`, `SpeechEnded`, `Speech`, or `Silence`), so you can decide when to stop listening.
Once you do, `.finish()` gives you back the buffered audio for just the speech segment, and resets
internal state so it's ready for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `.segment` method is for:
```python notest
from nobodywho import VoiceActivityDetection, SpeechToText
vad = VoiceActivityDetection(source="hf://onnx-community/silero-vad", sample_rate=16000)
stt = SpeechToText(source="hf://onnx-community/whisper-base")
audio = read_wav_pcm("recording.wav") # a full recording as i16 PCM samples
for speech in vad.segment(audio):
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance.
For that, we provide numerous params that you can tweak:
```python
from nobodywho import VoiceActivityDetection
vad = VoiceActivityDetection(
# VAD is currently fixed to Silero ONNX, but you can change the source.
source="hf://onnx-community/silero-vad",
# Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
threshold=0.5,
# Determines the minimum duration classified as speech.
min_speech_duration_ms=250,
# Determines the minimum duration classified as silence.
min_silence_duration_ms=250,
# Determines how much audio to keep before the official "SpeechStarted", to avoid cutting off the start.
preroll_duration_ms=500,
)
```
---
## Swift
This section documents Swift 3.0.0, the latest released version.
### Chat
Source: [https://docs.nobodywho.ooo/swift/chat](https://docs.nobodywho.ooo/swift/chat)
As you may have noticed in the [welcome guide](https://docs.nobodywho.ooo/swift/), every interaction with your LLM starts by creating a `Chat` object.
In the following sections, we talk about which configuration options it has, and when to use them.
#### Creating a Chat
There are two main ways of creating a `Chat` object and the difference lies in when the model file is loaded.
The simplest way is using `Chat.fromPath`:
```swift
import NobodyWho
let chat = try await Chat.fromPath(modelPath: "/path/to/model.gguf")
```
The `modelPath` parameter accepts a local file path, a Hugging Face `hf://` URL, an `https://` URL, or `"auto"` to select a chat model based on available memory:
```swift
// From a Hugging Face repository
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// From an HTTPS URL
let chat = try await Chat.fromPath(
modelPath: "https://example.com/model.gguf"
)
```
When loading from a remote URL, you can track download progress with the `onDownloadProgress` callback:
```swift
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total in
print("Downloaded \(downloaded)/\(total) bytes")
}
```
This function is async since loading a model can take a bit of time, but this should not block any of your UI.
Another way to achieve the same thing is to load the model separately and then use the `Chat` constructor:
```swift
import NobodyWho
let model = try await Model.load(modelPath: "/path/to/model.gguf")
let chat = try Chat(model: model)
```
The `Model.load` function also supports `hf://` and `https://` URLs, as well as `onDownloadProgress`:
```swift
let model = try await Model.load(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total in
print("Downloaded \(downloaded)/\(total) bytes")
}
```
This allows for sharing the model between several `Chat` instances.
#### Prompts and responses
The `chat.ask()` function is central to NobodyWho. This function sends your message to the LLM, which then starts generating a response.
```swift
let chat = try await Chat.fromPath(modelPath: "/path/to/model.gguf")
let response = chat.ask("Is water wet?")
```
The return type of `ask` is a `TokenStream`, which conforms to `AsyncSequence`.
If you want to start reading the response as soon as possible, you can iterate over it using `for await`.
Each token is either an individual word or a fragment of a word.
```swift
for await token in response {
print(token, terminator: "")
}
```
If you just want to get the complete response, you can call `completed()`.
This will return the entire response string once the model is done generating.
```swift
let fullResponse = try await response.completed()
```
All of your messages and the model's responses are stored in the `Chat` object, so the next time you call `chat.ask()`, it will remember the previous messages.
#### Stopping generation
If you need to cancel the model's response while it is still generating (for example, when the user taps a "Stop" button), call `stopGeneration()`:
```swift
chat.stopGeneration()
```
This immediately stops token generation. Any tokens already produced are still available in the stream. The partial response is added to the chat history, so the conversation remains coherent. Note that `stopGeneration()` is a synchronous call and can be invoked from any thread.
#### Chat history
If you want to inspect the messages inside the `Chat` object, you can use `getChatHistory`.
```swift
let msgs = try await chat.getChatHistory()
print(msgs[0]) // The first message
```
Similarly, if you want to edit what messages are in the context, you can use `setChatHistory`:
```swift
try await chat.setChatHistory([
.message(role: .user, content: "What is water?", assets: [])
])
```
#### System prompt
A system prompt is a special message put into the chat context, which should guide its overall behavior.
Some models ship with a built-in system prompt. If you don't specify a system prompt yourself, NobodyWho will fall back to using the model's default system prompt.
You can specify a system prompt when creating a `Chat`:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
systemPrompt: "You are a mischievous assistant!"
)
```
This `systemPrompt` is then persisted until the chat context is reset.
#### Context
The context is the text window which the LLM currently considers. Specifically this is the number of tokens the LLM keeps in memory for your current conversation.
A bigger context size means more computational overhead, so it makes sense to constrain it. This can be done with the `contextSize` setting at creation time:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
contextSize: 4096
)
```
The default value is `4096`, however this is mainly useful for short and simple conversations. Choosing the right context size is quite important and depends heavily on your use case. You can check the maximum context size the model was trained with using `model.maxCtx` — setting `contextSize` above this value has no benefit.
Even with a properly selected context size it might happen that you fill up your entire context during a conversation. When this happens, NobodyWho will shrink the context for you. Currently this is done by removing old messages (apart from the system prompt and the first user message) from the chat history, until the size reaches `contextSize / 2`. The KV cache is also updated automatically.
To reset the current context content, call `resetContext()` with a new system prompt and potentially changed tools.
```swift
try await chat.resetContext(systemPrompt: "New system prompt", tools: [])
```
If you don't want to change the already set defaults (`systemPrompt`, `tools`), but only reset the context, then go for `resetHistory`.
To inspect how much of the context is currently in use, call `getStats()`:
```swift continuation
let stats = try await chat.getStats()
print("Using \(stats.contextUsed) of \(stats.contextSize) tokens")
```
#### CPU threads
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Efficiency cores end up pacing the whole thread pool, so using every
CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads) for the numbers.
Override it with `threadCount` when you want to leave CPU headroom for the rest of your app:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
threadCount: 4
)
```
Leave it unset — or pass `nil` — to keep the detected default. Values above the device's CPU
count are clamped, and it has little effect when the model is offloaded to the GPU.
#### Sharing model between contexts
There are scenarios where you would like to keep separate chat contexts (e.g. for every user of your app), but have only one model loaded. In this case you must load the model separately from creating the `Chat` instance.
```swift
let model = try await Model.load(modelPath: "/path/to/model.gguf")
let chat1 = try Chat(model: model)
let chat2 = try Chat(model: model)
```
NobodyWho will then take care of the separation, such that your chat histories won't collide or interfere with each other, while having only one model loaded.
#### GPU
When using `Model.load` or `Chat.fromPath` you have the option to disable GPU acceleration:
```swift
let model = try await Model.load(modelPath: "/path/to/model.gguf", useGpu: false)
```
By default `useGpu` is set to `true`, which uses Metal on Apple platforms.
#### Speculative decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target model verify several candidate tokens per forward pass. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
Load the model with a compatible draft-heads gguf (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B) and pass an `MtpConfig` when constructing the chat:
```swift
let model = try await Model.load(
modelPath: "/path/to/gemma-4-e2b.gguf",
draftModelPath: "/path/to/mtp-gemma-4-e2b.gguf"
)
// MtpConfig() uses the default drafter tuning; tune with MtpConfig(kMax: ..., pMin: ...)
let chat = try Chat(model: model, mtp: MtpConfig())
```
`Chat.fromPath` accepts the same two parameters if you don't need to share the model:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/gemma-4-e2b.gguf",
draftModelPath: "/path/to/mtp-gemma-4-e2b.gguf",
mtp: MtpConfig()
)
```
Loading the draft heads adds around 5% to VRAM usage.
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
#### Template Variables
Chat templates are used internally by models to format conversation history into the expected prompt format. Different models may support different template variables that control specific behaviors. Template variables are boolean flags passed to the chat template that can enable or disable certain features.
##### Using Template Variables
You can set template variables when creating a chat or modify them on existing instances:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
templateVariables: ["enable_thinking": true]
)
```
You can also modify template variables on an existing chat instance:
```swift
try await chat.setTemplateVariable(name: "enable_thinking", value: true)
let variables = try await chat.getTemplateVariables()
```
##### Example: Qwen3 and Qwen3.5 Reasoning
The Qwen3 and Qwen3.5 model families support the `enable_thinking` template variable, which controls whether the model should engage in explicit reasoning steps before answering:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
templateVariables: ["enable_thinking": true]
)
let response = chat.ask("Solve this logic puzzle: ...")
```
When `enable_thinking` is enabled, these models will show their reasoning process before providing the final answer.
**Info:**
Note that template variables are model-specific. If a model's chat template doesn't use a specific variable, that variable will be ignored gracefully.
---
### Downloading models
Source: [https://docs.nobodywho.ooo/swift/downloading-models](https://docs.nobodywho.ooo/swift/downloading-models)
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to observe a download in progress, how to access gated/private models, and how to inspect what's already in the local cache.
#### Supported model path formats
The `modelPath` argument to `Chat.fromPath` and `Model.downloadModel` accepts:
| Form | Example | Notes |
| ---- | ------- | ----- |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
| Local path | `./model.gguf` | Used as-is |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs.
#### Tracking download progress
When loading from a remote URL, pass a progress closure to `Chat.fromPath`. It receives `(downloaded, total)` byte counts and is not called for cached or local files.
```swift
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total in
print("Downloaded \(downloaded)/\(total) bytes")
}
```
#### Downloading a gated model
Some HuggingFace models are private or gated by a license you need to accept. In both cases you need to be authorized to download the model weights.
You can manually download the GGUF file via your web browser and then point `Chat.fromPath` at the local path:
```swift
let chat = try await Chat.fromPath(modelPath: "./model.gguf")
```
Or use `Model.downloadModel` with an `Authorization` header:
```swift
import NobodyWho
let modelPath = try await Model.downloadModel(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
headers: ["Authorization": "Bearer your_hf_token"]
)
let chat = try await Chat.fromPath(modelPath: modelPath)
```
You can generate a HuggingFace token in [your account settings](https://huggingface.co/settings/tokens).
#### Inspecting the model cache
`getCachedModels()` returns every `.gguf` model in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `Model.downloadModel` and by `Chat.fromPath`'s `hf://` paths.
```swift
import NobodyWho
let models = try getCachedModels()
for model in models {
print("\(model.path): \(model.size) bytes")
}
```
Each `CachedModel` has:
- `path: String` — absolute path to the cached `.gguf` file
- `size: UInt64` — size in bytes
The array is empty if nothing has been downloaded yet. The call throws if the cache directory cannot be read.
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/swift/embeddings-and-rag](https://docs.nobodywho.ooo/swift/embeddings-and-rag)
NobodyWho provides `Encoder` and `CrossEncoder` classes for building retrieval-augmented generation (RAG) pipelines entirely on-device.
#### Embeddings
An `Encoder` converts text into a numerical vector (embedding) that captures its semantic meaning. Texts with similar meanings will have similar embeddings.
```swift
import NobodyWho
let encoder = try await Encoder.fromPath(modelPath: "/path/to/embeddings.gguf", contextSize: 512, useGpu: true)
let embedding1 = try await encoder.encode("The cat sat on the mat")
let embedding2 = try await encoder.encode("A feline rested on the rug")
let embedding3 = try await encoder.encode("The stock market crashed today")
let similar = cosineSimilarity(a: embedding1, b: embedding2) // High similarity
let different = cosineSimilarity(a: embedding1, b: embedding3) // Low similarity
```
Use `encodeBatch` for multiple texts. It returns embeddings in input order:
```swift
let texts = [
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
]
let embeddings = try await encoder.encodeBatch(texts)
```
You can also create an encoder from an already-loaded model:
```swift
let model = try await Model.load(modelPath: "/path/to/embeddings.gguf", useGpu: true)
let encoder = Encoder(model: model, contextSize: 512)
```
#### Cross-Encoder for reranking
A `CrossEncoder` takes a query and a list of documents and scores each document by its relevance to the query. Unlike embeddings (which are computed independently), a cross-encoder processes the query and document together, giving more accurate relevance scores.
```swift
let crossEncoder = try await CrossEncoder.fromPath(modelPath: "/path/to/reranker.gguf", contextSize: 512, useGpu: true)
let query = "How do I reset my password?"
let documents = [
"Click 'Forgot Password' on the login page.",
"Our company was founded in 2020.",
"Contact support for account recovery.",
"The weather is sunny today.",
]
// Get raw similarity scores
let scores = try await crossEncoder.rank(query: query, documents: documents)
// Or get documents sorted by relevance (most relevant first)
let ranked = try await crossEncoder.rankAndSort(query: query, documents: documents)
for (document, score) in ranked {
print("\(score): \(document)")
}
```
#### Building a RAG pipeline
A typical RAG pipeline combines both tools:
1. **Index**: Use the `Encoder` to create embeddings for your document collection
2. **Retrieve**: When a user asks a question, embed the query and find the most similar documents using `cosineSimilarity`
3. **Rerank** (optional): Use the `CrossEncoder` to rerank the top candidates for better precision
4. **Generate**: Pass the relevant documents to a `Chat` as context in the system prompt
```swift
// 1. Embed your documents (do this once, store the results)
let encoder = try await Encoder.fromPath(modelPath: "/path/to/embeddings.gguf", contextSize: 512, useGpu: true)
let docs = ["Document 1...", "Document 2...", "Document 3..."]
let docEmbeddings = try await encoder.encodeBatch(docs)
// 2. Embed the query and find similar documents
let queryEmbedding = try await encoder.encode("What is the return policy?")
let similarities = docEmbeddings.map { cosineSimilarity(a: queryEmbedding, b: $0) }
// 3. Rerank the top results
let crossEncoder = try await CrossEncoder.fromPath(modelPath: "/path/to/reranker.gguf", contextSize: 512, useGpu: true)
let topDocs = // ... select top N by similarity
let ranked = try await crossEncoder.rankAndSort(query: "What is the return policy?", documents: topDocs)
// 4. Generate a response with context
let context = ranked.prefix(3).map { $0.0 }.joined(separator: "\n\n")
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
systemPrompt: "Answer based on the following documents:\n\n\(context)"
)
let response = try await chat.ask("What is the return policy?").completed()
```
---
### Getting started
Source: [https://docs.nobodywho.ooo/swift/](https://docs.nobodywho.ooo/swift/)
#### How do I get started?
Add NobodyWho to your project using Swift Package Manager. In Xcode, go to **File → Add Package Dependencies** and enter:
```
https://github.com/nobodywho-ooo/nobodywho-swift.git
```
Or add it to your `Package.swift`:
```swift
dependencies: [
.package(url: "https://github.com/nobodywho-ooo/nobodywho-swift.git", from: "2.1.0")
]
```
Models can be loaded from a local file path, a Hugging Face repository using `hf://` URLs, or any `https://` URL. If you don't have a specific model in mind, try [this one](https://huggingface.co/NobodyWho/Qwen_Qwen3-0.6B-GGUF/resolve/main/Qwen_Qwen3-0.6B-Q4_K_M.gguf). Read more about [model selection](https://docs.nobodywho.ooo/docs/model-selection).
```swift
import NobodyWho
// From a Hugging Face repository
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// From an HTTPS URL
let chat = try await Chat.fromPath(
modelPath: "https://huggingface.co/NobodyWho/Qwen_Qwen3-0.6B-GGUF/resolve/main/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
// From a local file
let chat = try await Chat.fromPath(modelPath: "/path/to/model.gguf")
```
Once you have a `Chat`, call `.ask` to get a response!
```swift
let response = try await chat.ask("Is water wet?").completed()
print(response) // Yes, indeed, water is wet!
```
This is a super simple example, but we believe that examples which do simple things, should be simple!
To get a full overview of the functionality provided by NobodyWho, simply keep reading.
#### Platform requirements
- **iOS**: iPhone 11 or newer with at least 4 GB of RAM. Requires iOS 15+.
- **macOS**: Apple Silicon or Intel Mac with at least 8 GB of RAM. Requires macOS 13+.
- **visionOS**: Apple Vision Pro. Requires visionOS 1.0+.
- **watchOS**: Requires watchOS 10+. CPU-only (Metal is not available). Due to limited memory on Apple Watch, only very small models are practical.
GPU acceleration is enabled by default using Metal on all Apple platforms.
#### Feedback & Contributions
We welcome your feedback and ideas!
- Bug Reports & Improvements: If you encounter a bug or have suggestions, please open an issue on our [Issues](https://github.com/nobodywho-ooo/nobodywho/issues) page.
- Feature Requests & Questions: For new feature requests or general questions, join the discussion on our [Discussions](https://github.com/nobodywho-ooo/nobodywho/discussions) page.
---
### Sampling
Source: [https://docs.nobodywho.ooo/swift/sampling](https://docs.nobodywho.ooo/swift/sampling)
The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which using NobodyWho you can freely modify, to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).
#### Sampler presets
To get a quick start, NobodyWho offers a couple of well-known presets, which you can quickly utilize.
For example, if you want to increase or decrease the "creativity" of your model, select our `temperature` preset:
```swift
import NobodyWho
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.temperature(0.2)
)
```
Setting `temperature` to `0.2` will affect the sampler when choosing the next token, making the distribution less flat and therefore the model will favour more probable tokens.
To see the whole list of presets, check out the `SamplerPresets` enum:
```swift
enum SamplerPresets {
static func `default`() -> SamplerConfig
static func dry() -> SamplerConfig
static func greedy() -> SamplerConfig
static func temperature(_ temperature: Float) -> SamplerConfig
static func topK(_ topK: Int32) -> SamplerConfig
static func topP(_ topP: Float) -> SamplerConfig
// Constrain output to a specific format:
static func constrainWithJsonSchema(_ schema: String) -> SamplerConfig
static func constrainWithRegex(_ pattern: String) -> SamplerConfig
static func constrainWithGrammar(_ grammar: String) -> SamplerConfig
}
```
#### Structured output
One of the most useful features is constraining the model to produce structured output —
this gives you a hard guarantee that the output matches a specific format, rather than
relying on the model to get it right on its own.
##### Regular expressions
For simpler patterns, you can constrain the output with a regex:
```swift
// Force the model to answer with exactly "yes" or "no"
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.constrainWithRegex("yes|no")
)
let answer = try await chat.ask("Is the sky blue?").completed()
```
##### JSON schema
In some use-cases it might be useful to let the LLM generate JSON output.
You can use a JSON schema to force the LLM to produce the exact object shape you need:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.constrainWithJsonSchema("""
{
"type": "object",
"properties": {
"name": { "type": "string", "maxLength": 50 },
"age": { "type": "integer" }
},
"required": ["name", "age"],
"additionalProperties": false
}
""")
)
let response = try await chat.ask("Give me a person as JSON with name and age fields.").completed()
// `response` is always valid JSON matching the schema
```
##### Custom grammars (advanced)
For cases where JSON schema and regex are not expressive enough, you can supply a custom grammar.
`constrainWithGrammar` accepts both **Lark** syntax and **GBNF** (llama.cpp format) —
NobodyWho automatically converts GBNF to Lark before passing it to the inference engine.
**Lark syntax** (recommended):
```swift
let sampler = SamplerPresets.constrainWithGrammar("""
start: record (NEWLINE record)* NEWLINE?
record: field ("," field)*
field: /[^,"\\n\\r]+/
NEWLINE: /\\r?\\n/
""")
```
**GBNF syntax** (also accepted):
```swift
let sampler = SamplerPresets.constrainWithGrammar("""
file ::= record (newline record)* newline?
record ::= field ("," field)*
field ::= /[^,"\\n\\r]+/
newline ::= "\\r\\n" | "\\n"
""")
```
See the [Lark documentation](https://lark-parser.readthedocs.io/en/latest/grammar.html) and the
[GBNF specification](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) for the
full grammar syntax.
**Info:**
The older `SamplerPresets.json()` and `SamplerPresets.grammar()` methods are deprecated.
Use `SamplerPresets.constrainWithJsonSchema()` for JSON output or
`SamplerPresets.constrainWithGrammar()` for custom grammars — the latter accepts both Lark and GBNF strings.
#### Defining your own samplers
Sampler presets abstract away some control that you might want - for example, if you
want to chain samplers, change more "advanced" parameters, etc. For that use case,
we provide the `SamplerBuilder` class:
```swift
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
sampler: SamplerBuilder().temperature(temperature: 0.8).topK(topK: 5).dist()
)
```
With `SamplerBuilder` you can chain multiple steps together and then select how you
want to sample from the distribution. Keep in mind that `SamplerBuilder` provides two
types of methods: ones which modify the distribution (returning again the instance of
`SamplerBuilder`) and ones which sample from the distribution (returning `SamplerConfig`).
So in order to have the sampler working properly, be careful
to always end the chain with one of the sampling steps (e.g. `dist()`, `greedy()`, `mirostatV2()`, etc.).
For reproducible output, set the RNG seed with `.seed(seed:)` anywhere in the chain.
It is consumed by every random sampler in the chain — `dist`, `mirostatV1`, `mirostatV2`,
and the `xtc` shift step. `greedy` ignores it. If unset, a default seed is used.
```swift
let sampler = SamplerBuilder().temperature(temperature: 0.8).topK(topK: 5).seed(seed: 42).dist()
```
##### Available sampling steps
Pick any of the **shift steps** below (each reshapes the token distribution), then finish with one **terminal step** that picks the token — exactly like the `.temperature(...).topK(...).dist()` chain above.
Shift steps — add as many as you want, applied in order:
- `.topK(topK: 40)` — keep only the 40 most likely tokens
- `.topP(topP: 0.95, minKeep: 1)` — nucleus: keep the top tokens up to 95% of the probability mass
- `.minP(minP: 0.05, minKeep: 1)` — drop tokens below 5% of the most likely token's probability
- `.typicalP(typP: 0.9, minKeep: 1)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `.xtc(xtcProbability: 0.5, xtcThreshold: 0.1, minKeep: 1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `.temperature(temperature: 0.8)` — below 1.0 = more focused, above 1.0 = more random
- `.penalties(penaltyLastN: 64, penaltyRepeat: 1.1, penaltyFreq: 0.0, penaltyPresent: 0.0)` — per-token repetition penalty (`penaltyRepeat` 1.0 = off)
- `.dry(multiplier: 0.8, base: 1.75, allowedLength: 2, penaltyLastN: -1, seqBreakers: ["\n"])` — penalty for repeated *phrases*
- `.seed(seed: 42)` — fix the RNG for reproducible output
- `.grammar(...)` — deprecated; use the `constrainWith*` presets above
Terminal step — one of these turns the chain into a `SamplerConfig`, so finish with exactly one:
- `.dist()` — pick a token with weighted randomness (the usual choice)
- `.greedy()` — always take the most likely token
- `.mirostatV1(tau: 5.0, eta: 0.1, m: 100)` / `.mirostatV2(tau: 5.0, eta: 0.1)` — steer output "surprise" toward a target
`minKeep` is the floor on how many tokens survive a cut (`1` is fine).
You can also change the sampler configuration on an existing chat instance:
```swift
let sampler = SamplerBuilder().temperature(temperature: 0.8).topK(topK: 5).dist()
try await chat.setSamplerConfig(sampler)
```
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/swift/speech-to-text](https://docs.nobodywho.ooo/swift/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format.
```swift
import NobodyWho
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")
let text = try await stt.transcribeFile(path: "recording.mp3").completed()
print(text)
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribePcm` is available:
```swift
let text = try await stt.transcribePcm(samples: samples, sampleRate: 16000).completed()
```
In order to make this work, the buffer needs to be mono i16 PCM samples (`[Int16]`). The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with classic Chat models, streaming is available, so the transcription can be consumed token by token:
```swift
for try await piece in stt.transcribeFile(path: "recording.mp3") {
print(piece, terminator: "")
}
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `source` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```swift
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base", quantization: "q4")
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, pass its ISO 639-1 code as `language` to skip detection and improve performance:
```swift
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base", language: "en")
```
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/swift/text-to-speech](https://docs.nobodywho.ooo/swift/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
```swift
import Foundation
import NobodyWho
let tts = try await TextToSpeech.load(
source: "hf://NobodyWho/Kokoro-82M", // Hugging Face repo (hf://owner/repo) or local folder with the model files.
voice: "bf_emma", // Voice to use from the model.
language: "en-gb" // Language code for the input text.
)
// Generate WAV bytes for this sentence.
let wav = try await tts.synthesize("Hello from NobodyWho!")
// Save the audio to a file.
try wav.write(to: URL(fileURLWithPath: "out.wav"))
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, or a local directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/swift/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/swift/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```swift
let tts = try await TextToSpeech.load(
source: "hf://NobodyWho/Kokoro-82M",
voice: "bf_emma",
language: "en-gb"
)
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.0`.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```swift
let tts = try await TextToSpeech.load(
source: "hf://Supertone/supertonic-3",
language: "en"
)
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.05`.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Must be greater than `0`; defaults to `8`.
- `silenceDuration`: seconds of silence between long text chunks. Higher values add longer pauses. Must be `0` or higher; defaults to `0.3`.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```swift
let tts = try await TextToSpeech.load(
source: "hf://KevinAHM/pocket-tts-onnx",
voice: "alba",
language: "english_2026-04",
huggingfaceToken: "hf_..." // Alternative: set the HF_TOKEN environment variable.
)
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Defaults to `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Defaults to `int8`.
- `temperature`: controls how varied the speech sounds. Defaults to `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingfaceToken`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory or a custom source that NobodyWho cannot recognize:
```swift
let tts = try await TextToSpeech.load(
source: "/path/to/local/kokoro-folder",
architecture: .kokoro
)
```
Supported architecture values are `.kokoro`, `.pocketTts`, and `.supertonic`.
#### GPU
TextToSpeech runs on CPU only on Apple platforms for now. Metal/CoreML acceleration may be added in the future.
Most apps should keep the default `device: .auto`. If you want to force CPU explicitly, pass `device: .cpu`:
```swift
let tts = try await TextToSpeech.load(
source: "hf://Supertone/supertonic-3",
device: .cpu
)
```
#### Local model folder format
When `source` is a local directory, point it at the top-level model folder and pass the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/swift/tool-calling](https://docs.nobodywho.ooo/swift/tool-calling)
To give your LLM the ability to interact with the outside world, you will need tool calling.
**Info:**
Note that **not every model** supports tool calling. If the model does not have
such an option, it might not call your tools.
For reliable tool calling, we recommend trying the [Qwen](https://huggingface.co/collections/NobodyWho/qwen-3) family of models.
#### The @DeclareTool macro
The easiest way to create a tool is with the `@DeclareTool` macro. Just annotate any function with a description, and NobodyWho generates the tool for you:
```swift
import NobodyWho
@DeclareTool("Calculates the area of a circle given its radius")
func circleArea(radius: Double) -> String {
let area = Double.pi * radius * radius
return "Circle with radius \(radius) has area \(String(format: "%.2f", area))"
}
// The macro generates `circleAreaTool` automatically
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
tools: [circleAreaTool]
)
```
The generated variable is named `Tool` — so `circleArea` becomes `circleAreaTool`.
Supported parameter types: `String`, `Int`, `Double`, `Float`, `Bool`, and collections like `[String]` or `[String: Int]`.
##### Async tools
The `@DeclareTool` macro also works with async functions. This is useful for tools that need to make network requests, database queries, or other asynchronous operations:
```swift
@DeclareTool("Search the knowledge base")
func search(query: String) async -> String {
let results = await knowledgeBase.search(query)
return results.joined(separator: "\n")
}
@DeclareTool("Get the current weather for a city")
func getWeather(city: String, unit: String) async -> String {
let data = await weatherAPI.fetch(city: city, unit: unit)
return "{\"temp\": \(data.temp), \"unit\": \"\(unit)\"}"
}
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
tools: [searchTool, getWeatherTool]
)
```
##### Scope limitation
`@DeclareTool` is a Swift peer macro, which means it can only introduce new declarations at **top-level** or **type-member** scope. It **does not work inside function bodies**:
```swift
// ✅ Works — top-level scope
@DeclareTool("Ping the server")
func ping() -> String { return "pong" }
class MyApp {
// ✅ Works — type-member scope
@DeclareTool("Get status")
static func status() -> String { return "ok" }
}
func example() {
// ❌ Does NOT work — local scope
@DeclareTool("Ping the server")
func ping() -> String { return "pong" }
_ = pingTool // error: cannot find 'pingTool' in scope
}
```
If you need to define a tool inside a function body (for example, to capture local variables), use the manual `Tool` initializer described below.
#### Creating tools manually
You can also create tools without the macro, using the `Tool` initializer directly. This works in any scope, including inside function bodies where the macro cannot be used. Each parameter is a `(name, jsonSchema)` tuple, and arguments from the LLM are passed positionally in the same order as the `parameters` array:
```swift
let circleAreaTool = Tool(
name: "circle_area",
description: "Calculates the area of a circle given its radius",
parameters: [("radius", #"{"type": "number"}"#)]
) { args in
let radius = args[0] as! Double
let area = Double.pi * radius * radius
return "Circle with radius \(radius) has area \(String(format: "%.2f", area))"
}
```
This is especially useful when you need to capture local state:
```swift
func runChat() async throws {
var callCount = 0
let pingTool = Tool(
name: "ping",
description: "Ping the server",
parameters: []
) { _ in
callCount += 1
return "pong"
}
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
tools: [pingTool]
)
let _ = try await chat.ask("Ping the server").completed()
print("Tool was called \(callCount) time(s)")
}
```
For async callbacks, there's a separate initializer:
```swift
let searchTool = Tool(
name: "search",
description: "Search the knowledge base",
parameters: [("query", #"{"type": "string"}"#)]
) { args async in
let query = args[0] as! String
return await knowledgeBase.search(query)
}
```
##### Parameter schema reference
Each parameter schema is a JSON Schema string. Common types:
| Swift type | JSON Schema |
|---|---|
| `String` | `#"{"type": "string"}"#` |
| `Int` | `#"{"type": "integer"}"#` |
| `Double`, `Float` | `#"{"type": "number"}"#` |
| `Bool` | `#"{"type": "boolean"}"#` |
| `[String]` | `#"{"type": "array", "items": {"type": "string"}}"#` |
| `[String: Int]` | `#"{"type": "object", "additionalProperties": {"type": "integer"}}"#` |
#### Multiple tools
Naturally, more tools can be defined and the model can chain the calls for them:
```swift
@DeclareTool("Gets path of the current directory")
func getCurrentDir() -> String {
return FileManager.default.currentDirectoryPath
}
@DeclareTool("Lists files in the given directory")
func listFiles(path: String) -> String {
let files = try? FileManager.default.contentsOfDirectory(atPath: path)
return (files ?? []).joined(separator: ", ")
}
@DeclareTool("Gets the size of a file in bytes")
func getFileSize(filepath: String) -> String {
let attrs = try? FileManager.default.attributesOfItem(atPath: filepath)
let size = attrs?[.size] as? Int ?? 0
return "File size: \(size) bytes"
}
let chat = try await Chat.fromPath(
modelPath: "/path/to/model.gguf",
tools: [getCurrentDirTool, listFilesTool, getFileSizeTool]
)
let response = try await chat
.ask("What is the biggest file in my current directory?")
.completed()
print(response)
```
#### Tool calling and the context
As with most things made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/swift/vision](https://docs.nobodywho.ooo/swift/vision)
A picture is worth a thousand words (or at least a thousand tokens).
With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/swift/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/swift/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts for making this work:
1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 even manage both!
Usually, the projection model includes `mmproj` in its name.
If you are unsure which ones to pick, or just want a reasonable default, you can try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true),
which can do both image and audio.
With the downloaded GGUFs, you can load them using `Chat.fromPath`:
```swift
import NobodyWho
let chat = try await Chat.fromPath(
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf",
systemPrompt: "You are a helpful assistant, that can hear and see stuff!"
)
```
Or load the model separately:
```swift
let model = try await Model.load(
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf"
)
let chat = try Chat(model: model, systemPrompt: "You are a helpful assistant.")
```
**Info:**
The language model and projection model have to **fit** together, as they are trained together!
Unfortunately you can't just take a projection model and an LLM that you like and expect them
to work together.
#### Composing a prompt object
With the model configured, all that is left is to compose the prompt and send it to the model.
Use `Prompt` to build prompts that mix text, images, and audio, then pass them to `chat.ask()`:
```swift
let prompt = Prompt([
Prompt.text("Tell me what you see in the image and what you hear in the audio."),
Prompt.image("/path/to/dog.png"),
Prompt.audio("/path/to/sound.mp3"),
])
let response = try await chat.ask(prompt).completed()
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/swift/voice-activity-detection](https://docs.nobodywho.ooo/swift/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart.
#### Streaming
The main use case for VAD is streaming audio into the model chunk by chunk. This is useful, for
example, to detect when to stop listening to the microphone and start running
transcription/answer generation.
To do that, we provide a `push` interface:
```swift
import NobodyWho
let vad = try await VoiceActivityDetection.load(sampleRate: 16000, source: "hf://onnx-community/silero-vad")
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")
while let chunk = readMic() { // however you're reading from the microphone
if try vad.push(chunk: chunk) == .speechEnded {
break
}
}
let speech = vad.finish()
let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
print(transcription)
```
`VoiceActivityDetection` acts as a buffer: every `push` call tells you the current state
(`.speechStarted`, `.speechEnded`, `.speech`, or `.silence`), so you can decide when to stop
listening. Once you do, `finish()` gives you back the buffered audio for just the speech segment,
and resets internal state so it's ready for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `segment` method is for:
```swift
import NobodyWho
let vad = try await VoiceActivityDetection.load(sampleRate: 16000, source: "hf://onnx-community/silero-vad")
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")
let audio = readWavPcm(path: "recording.wav") // a full recording as [Int16] PCM samples
for speech in try vad.segment(samples: audio) {
let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
print(transcription)
}
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance. For that,
we provide numerous params that you can tweak:
```swift
let vad = try await VoiceActivityDetection.load(
sampleRate: 16000,
// VAD is currently fixed to Silero ONNX, but you can change the source.
source: "hf://onnx-community/silero-vad",
// Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
threshold: 0.5,
// Determines the minimum duration classified as speech.
minSpeechDurationMs: 250,
// Determines the minimum duration classified as silence.
minSilenceDurationMs: 250,
// Determines how much audio to keep before the official .speechStarted, to avoid cutting off the start.
prerollDurationMs: 500
)
```
---
## React Native/Expo
This section documents React Native/Expo 3.0.0, the latest released version.
### Chat
Source: [https://docs.nobodywho.ooo/react-native/chat](https://docs.nobodywho.ooo/react-native/chat)
As you may have noticed in the [welcome guide](https://docs.nobodywho.ooo/react-native/), every interaction with your LLM starts by creating a `Chat` object.
In the following sections, we talk about which configuration options it has, and when to use them.
#### Creating a Chat
There are two main ways of creating a `Chat` object and the difference lies in when the model file is loaded.
The simplest way is using `Chat.fromPath`:
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({ modelPath: "/path/to/model.gguf" });
```
This function is async since loading a model can take a bit of time, but this should not block any of your UI.
Another way to achieve the same thing is to load the model separately and then use the `Chat` constructor:
```typescript
import { Model, Chat } from "react-native-nobodywho";
const model = await Model.load({ modelPath: "/path/to/model.gguf" });
const chat = new Chat({ model });
```
This allows for sharing the model between several `Chat` instances.
You can pass `"auto"` as `modelPath` to select a chat model based on available memory.
#### Prompts and responses
The `chat.ask()` function is central to NobodyWho. This function sends your message to the LLM, which then starts generating a response.
```typescript
const chat = await Chat.fromPath({ modelPath: "/path/to/model.gguf" });
const response = chat.ask("Is water wet?");
```
The return type of `ask` is a `TokenStream`.
If you want to start reading the response as soon as possible, you can iterate over the `TokenStream` using `for await`.
Each token is either an individual word or a fragment of a word.
```typescript
for await (const token of response) {
console.log(token);
}
```
If you just want to get the complete response, you can call `completed()`.
This will return the entire response string once the model is done generating.
```typescript
const fullResponse = await response.completed();
```
All of your messages and the model's responses are stored in the `Chat` object, so the next time you call `chat.ask()`, it will remember the previous messages.
#### Chat history
If you want to inspect the messages inside the `Chat` object, you can use `getChatHistory`.
```typescript
const msgs = await chat.getChatHistory();
console.log(msgs[0]); // The first message
```
Similarly, if you want to edit what messages are in the context, you can use `setChatHistory`:
```typescript
await chat.setChatHistory([
{ role: "user", content: "What is water?" },
]);
```
#### System prompt
A system prompt is a special message put into the chat context, which should guide its overall behavior.
Some models ship with a built-in system prompt. If you don't specify a system prompt yourself, NobodyWho will fall back to using the model's default system prompt.
You can specify a system prompt when creating a `Chat`:
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
systemPrompt: "You are a mischievous assistant!",
});
```
This `systemPrompt` is then persisted until the chat context is reset.
#### Context
The context is the text window which the LLM currently considers. Specifically this is the number of tokens the LLM keeps in memory for your current conversation.
A bigger context size means more computational overhead, so it makes sense to constrain it. This can be done with the `contextSize` setting at creation time:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
contextSize: 4096,
});
```
The default value is `4096`, however this is mainly useful for short and simple conversations. Choosing the right context size is quite important and depends heavily on your use case. You can check the maximum context size the model was trained with using `model.maxCtx` — setting `contextSize` above this value has no benefit.
Even with a properly selected context size it might happen that you fill up your entire context during a conversation. When this happens, NobodyWho will shrink the context for you. Currently this is done by removing old messages (apart from the system prompt and the first user message) from the chat history, until the size reaches `contextSize / 2`. The KV cache is also updated automatically.
To reset the current context content, call `resetContext()` with a new system prompt and potentially changed tools.
```typescript
await chat.resetContext({ systemPrompt: "New system prompt", tools: [] });
```
If you don't want to change the already set defaults (`systemPrompt`, `tools`), but only reset the context, then go for `resetHistory`.
To inspect how much of the context is currently in use, call `getStats()`:
```typescript continuation
const stats = await chat.getStats();
console.log(`Using ${stats.contextUsed} of ${stats.contextSize} tokens`);
```
#### CPU threads
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Efficiency cores end up pacing the whole thread pool, so using every
CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads) for the numbers.
Override it with `threadCount` when you want to leave CPU headroom for the rest of your app —
often a good idea on phones, where the big cores are also driving the UI:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
threadCount: 4,
});
```
Omit it to keep the detected default. Values above the device's CPU count are clamped, and it
has little effect when the model is offloaded to the GPU.
#### Sharing model between contexts
There are scenarios where you would like to keep separate chat contexts (e.g. for every user of your app), but have only one model loaded. In this case you must load the model separately from creating the `Chat` instance.
```typescript
import { Model, Chat } from "react-native-nobodywho";
const model = await Model.load({ modelPath: "/path/to/model.gguf" });
const chat1 = new Chat({ model });
const chat2 = new Chat({ model });
```
NobodyWho will then take care of the separation, such that your chat histories won't collide or interfere with each other, while having only one model loaded.
#### GPU
When using `Model.load` or `Chat.fromPath` you have the option to disable/enable GPU acceleration:
```typescript
const model = await Model.load({ modelPath: "/path/to/model.gguf", useGpu: false });
```
or
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
useGpu: false,
});
```
By default `useGpu` is set to `true`.
#### Speculative decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target model verify several candidate tokens per forward pass. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
Load the model with a compatible draft-heads gguf (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B) and pass an `mtp` config when constructing the chat:
```typescript
import { Model, Chat } from "react-native-nobodywho";
const model = await Model.load({
modelPath: "/path/to/gemma-4-e2b.gguf",
draftModelPath: "/path/to/mtp-gemma-4-e2b.gguf",
});
// `{}` uses the default drafter tuning; override fields with e.g. `{ kMax: 5 }`
const chat = new Chat({ model, mtp: {} });
```
`Chat.fromPath` accepts the same two options if you don't need to share the model:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/gemma-4-e2b.gguf",
draftModelPath: "/path/to/mtp-gemma-4-e2b.gguf",
mtp: {},
});
```
Loading the draft heads adds around 5% to VRAM usage.
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
#### Template Variables
Chat templates are used internally by models to format conversation history into the expected prompt format. Different models may support different template variables that control specific behaviors. Template variables are boolean flags passed to the chat template that can enable or disable certain features.
##### Using Template Variables
You can set template variables when creating a chat or modify them on existing instances:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
templateVariables: { enable_thinking: true },
});
```
You can also modify template variables on an existing chat instance:
```typescript
// Set a single template variable
await chat.setTemplateVariable("enable_thinking", true);
// Get current template variables
const variables = await chat.getTemplateVariables();
console.log(variables); // Map { "enable_thinking" => true }
```
With the next message sent, the updated settings will be propagated to the model.
##### Example: Qwen3 and Qwen3.5 Reasoning
The Qwen3 and Qwen3.5 model families support the `enable_thinking` template variable, which controls whether the model should engage in explicit reasoning steps before answering:
```typescript
const chat = await Chat.fromPath({
modelPath: '/path/to/model.gguf',
templateVariables: { enable_thinking: true },
});
const response = chat.ask("Solve this logic puzzle: ...");
```
When `enable_thinking` is enabled, these models will show their reasoning process before providing the final answer.
##### Model-Specific Variables
Different models may support different template variables depending on their chat template implementation. The available variables and their effects depend entirely on how the model's chat template is designed. Check your model's documentation to see which template variables are supported.
**Info:**
Note that template variables are model-specific. If a model's chat template doesn't use a specific variable, that variable will be ignored gracefully.
---
### Downloading models
Source: [https://docs.nobodywho.ooo/react-native/downloading-models](https://docs.nobodywho.ooo/react-native/downloading-models)
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to access gated/private models, how to observe a download in progress, and how to inspect what's already in the local cache.
#### Supported model path formats
The `modelPath` option to `Chat.fromPath` and `downloadModel` accepts:
| Form | Example | Notes |
| ---- | ------- | ----- |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
| Local path | `./model.gguf` | Used as-is |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs.
#### Android permissions (React Native only)
Loading a model from `hf://` or `https://` needs network access. The React Native app template already declares the internet permission in `android/app/src/main/AndroidManifest.xml`:
```xml
```
If your project has removed it, add it back. Apps that only load models from local paths need no network permission.
#### Downloading a gated model
Some HuggingFace models are private or gated by a license you need to accept. In both cases you need to be authorized to download the model weights.
You can manually download the GGUF file via your web browser, place it on the device, and then point `Chat.fromPath` at the local path:
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({ modelPath: "./model.gguf" });
```
Or use `downloadModel` with an `Authorization` header:
```typescript
import { downloadModel, Chat } from "react-native-nobodywho";
const modelPath = await downloadModel({
modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
headers: { Authorization: "Bearer your_hf_token" },
});
const chat = await Chat.fromPath({ modelPath });
```
You can generate a HuggingFace token in [your account settings](https://huggingface.co/settings/tokens).
#### Tracking download progress
When loading a remote model, pass an `onDownloadProgress` option to observe the download. It receives `(downloaded, total)` byte counts, is throttled to roughly 10 Hz with a guaranteed final emit on completion, and is not called for cached or local files.
```typescript
const chat = await Chat.fromPath({
modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
onDownloadProgress: (downloaded, total) => {
console.log(`${downloaded} / ${total} bytes`);
},
});
```
#### Inspecting the model cache
`getCachedModels()` returns every `.gguf` model in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `downloadModel` and by `Chat.fromPath`'s `huggingface:` paths.
```typescript
import { getCachedModels } from "react-native-nobodywho";
for (const model of getCachedModels()) {
console.log(`${model.path}: ${Number(model.size)} bytes`);
}
```
Each entry has:
- `path: string` — absolute path to the cached `.gguf` file
- `size: bigint` — size in bytes (the underlying Rust `u64`, exposed as JavaScript `bigint`)
The array is empty if nothing has been downloaded yet. The call throws if the cache directory cannot be read.
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/react-native/embeddings-and-rag](https://docs.nobodywho.ooo/react-native/embeddings-and-rag)
When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.
#### Understanding Embeddings
Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.
For example, "Schedule a meeting for next Tuesday" and "Book an appointment next week" would have very similar embeddings, despite using different words.
#### The Encoder
The `Encoder` object converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).
We recommend you first try [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf).
```typescript
import { Encoder } from "react-native-nobodywho";
const encoder = await Encoder.fromPath({
modelPath: "/path/to/embedding-model.gguf",
});
const embedding = await encoder.encode("What is the weather like?");
console.log(`Vector with ${embedding.length} dimensions`);
```
The resulting embedding is an array of numbers (typically 384 or 768 dimensions depending on the model).
##### Batch Encoding
Use `encodeBatch` for multiple texts. It returns embeddings in input order:
```typescript
const texts = [
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
];
const embeddings = await encoder.encodeBatch(texts);
```
##### Comparing Embeddings
To measure how similar two pieces of text are, compare their embeddings using cosine similarity:
```typescript
import { Encoder, cosineSimilarity } from "react-native-nobodywho";
const encoder = await Encoder.fromPath({
modelPath: "/path/to/embedding-model.gguf",
});
const query = await encoder.encode("How do I reset my password?");
const doc1 = await encoder.encode(
"You can reset your password in the account settings",
);
const doc2 = await encoder.encode(
"The password requirements include 8 characters minimum",
);
const similarity1 = cosineSimilarity(query, doc1);
const similarity2 = cosineSimilarity(query, doc2);
console.log(`Document 1 similarity: ${similarity1.toFixed(3)}`); // Higher score
console.log(`Document 2 similarity: ${similarity2.toFixed(3)}`); // Lower score
```
Cosine similarity returns a value between -1 and 1, where 1 means identical meaning and -1 means opposite meaning.
#### The CrossEncoder for Better Ranking
While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents to determine how well the document answers the query.
The key difference is that embeddings compare vector similarity, while cross-encoders understand the relationship between query and document, at a potentially larger computation cost.
##### Why CrossEncoder Matters
Consider this example:
```
Query: "What are the office hours for customer support?"
Documents: [
"Customer asked: What are the office hours for customer support?",
"Support team responds: Our customer support is available Monday-Friday 9am-5pm EST",
"Note: Weekend support is not available at this time"
]
```
Using embeddings alone, the first document scores highest (most similar to the query) even though it provides no useful information. A cross-encoder correctly identifies that the second document actually answers the question.
##### Using CrossEncoder
```typescript
import { CrossEncoder } from "react-native-nobodywho";
// Download a reranking model like bge-reranker-v2-m3-Q8_0.gguf
const crossencoder = await CrossEncoder.fromPath({
modelPath: "/path/to/reranker-model.gguf",
});
const query = "How do I install Python packages?";
const documents = [
"Someone previously asked about Python packages",
"Use pip install package-name to install Python packages",
"Python packages are not included in the standard library",
];
// Get relevance scores for each document
const scores = await crossencoder.rank(query, documents);
console.log(scores); // [0.23, 0.89, 0.45] - second doc scores highest
```
##### Automatic Sorting
For convenience, use `rankAndSort` to get documents sorted by relevance:
```typescript
// Returns list of [document, score] pairs, sorted by score
const rankedDocs = await crossencoder.rankAndSort(query, documents);
for (const [doc, score] of rankedDocs) {
console.log(`[${score.toFixed(3)}] ${doc}`);
}
```
This returns documents ordered from most to least relevant.
#### Building a RAG System
Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.
Here's a complete example building a customer service assistant with access to company policies:
```typescript
import { Chat, Tool, CrossEncoder } from "react-native-nobodywho";
// Initialize the cross-encoder for document ranking
const crossencoder = await CrossEncoder.fromPath({
modelPath: "/path/to/reranker-model.gguf",
});
// Your knowledge base
const knowledge = [
"Our company offers a 30-day return policy for all products",
"Free shipping is available on orders over $50",
"Customer support is available via email and phone",
"We accept credit cards, PayPal, and bank transfers",
"Order tracking is available through your account dashboard",
];
// Create a tool that searches the knowledge base
const searchKnowledgeTool = new Tool({
name: "search_knowledge",
description: "Search the knowledge base for relevant information",
parameters: [
{ name: "query", type: "string", description: "The search query" },
],
call: async (query: string) => {
const ranked = await crossencoder.rankAndSort(query, knowledge);
const topDocs = ranked
.slice(0, 3)
.map(([doc]) => doc);
return topDocs.join("\n");
},
});
// Create a chat with access to the knowledge base
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
systemPrompt:
"You are a customer service assistant. Use the search_knowledge tool to find relevant information from our policies before answering customer questions.",
tools: [searchKnowledgeTool],
});
// The chat will automatically search the knowledge base when needed
const response = await chat.ask("What is your return policy?").completed();
console.log(response);
```
The LLM will call the `search_knowledge` tool, receive the most relevant documents, and use them to generate an accurate answer.
#### Recommended Models
##### For Embeddings
- [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf) - Good balance of speed and quality (~25MB)
- Supports English text with 384-dimensional embeddings
##### For Cross-Encoding (Reranking)
- [bge-reranker-v2-m3-Q8_0.gguf](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf) - Multilingual support with excellent accuracy
#### Best Practices
**Precompute embeddings**: If you have a fixed knowledge base, generate embeddings once and reuse them. Don't re-encode the same documents repeatedly.
**Use embeddings for filtering**: When working with large document collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank them.
**Limit cross-encoder inputs**: Cross-encoders are more expensive than embeddings. Don't pass thousands of documents to `rank()` - filter first with embeddings.
---
### Getting started
Source: [https://docs.nobodywho.ooo/react-native/](https://docs.nobodywho.ooo/react-native/)
#### How do I get started?
First, install `react-native-nobodywho`.
```bash
# React Native
npm install react-native-nobodywho
# Expo
npx expo install react-native-nobodywho
```
##### React Native
No additional initialization step is required — the native module is loaded automatically when you first import from the package.
##### Expo
NobodyWho ships native code, so it does **not** run in [Expo Go](https://docs.expo.dev/get-started/set-up-your-environment/). You need a [development build](https://docs.expo.dev/develop/development-builds/introduction/).
###### • Continuous Native Generation (CNG) projects
In a managed project, `ios/` and `android/` are not committed. For the **first build**, run the command for the platform you are targeting (`run:ios` and `run:android` are alternatives — pick your target). It runs prebuild automatically since the native folders don't exist yet, installs pods, and builds NobodyWho in:
```bash
npx expo run:ios
# or
npx expo run:android
```
After you **upgrade** NobodyWho (or any other native dependency), the native folders already exist, so `run:*` would build against stale native code. Regenerate them from scratch, then rebuild:
```bash
npx expo prebuild --clean # regenerate ios/ and android/ from scratch
npx expo run:ios # then rebuild (or run:android)
```
###### • Bare projects
Here `ios/` and `android/` are committed. Autolinking registers the module on your next native build — just rebuild with `expo run:*` (or your existing native build). Do not run `prebuild --clean` here unless you intend to regenerate the native folders, since it overwrites manual native edits.
You can also build with [EAS Build](https://docs.expo.dev/build/introduction/) instead of building locally. No config plugin is required.
#### Pick a model
Now you are ready to pick a model. NobodyWho can download GGUF models directly from Hugging Face — just pass a `huggingface:` path. See [model selection](https://docs.nobodywho.ooo/docs/model-selection) for recommendations.
Then create a `Chat` object and call `.ask`!
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
});
const response = await chat.ask("Is water wet?").completed();
console.log(response); // Yes, indeed, water is wet!
```
This is a super simple example, but we believe that examples which do simple things, should be simple!
To get a full overview of the functionality provided by NobodyWho, simply keep reading.
#### Android requirements
If you use the x86_64 Android emulator for development, your app must set `minSdkVersion` to at least 31. This is due to a threading feature (ELF TLS) that the Rust runtime requires on x86_64. ARM64 devices (i.e. all real phones) work with any `minSdkVersion`.
No specific NDK version is required — NobodyWho ships prebuilt shared libraries, so your project's NDK version does not affect the Rust code.
#### Minimum recommended specs
- iOS: iPhone 11 or newer with at least 4 GB of RAM. We tested a Qwen3 0.6B (332 MB) on an iPhone X (iOS 16) and while it ran, performance was too slow to be practical.
- Android: Snapdragon 855 / Adreno 640 / 6 GB RAM or better. The same Qwen3 0.6B model performed notably better on a OnePlus 7 Pro (Android 12) than on the iPhone X tested above.
#### Feedback & Contributions
We welcome your feedback and ideas!
- Bug Reports & Improvements: If you encounter a bug or have suggestions, please open an issue on our [Issues](https://github.com/nobodywho-ooo/nobodywho/issues) page.
- Feature Requests & Questions: For new feature requests or general questions, join the discussion on our [Discussions](https://github.com/nobodywho-ooo/nobodywho/discussions) page.
---
### Sampling
Source: [https://docs.nobodywho.ooo/react-native/sampling](https://docs.nobodywho.ooo/react-native/sampling)
The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which using NobodyWho you can freely modify, to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).
#### Sampler presets
To get a quick start, NobodyWho offers a couple of well-known presets, which you can quickly utilize.
For example, if you want to increase or decrease the "creativity" of your model, select our `temperature` preset:
```typescript
import { Chat, SamplerPresets } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.temperature(0.2),
});
```
Setting `temperature` to `0.2` will affect the sampler when choosing the next token, making the distribution less flat and therefore the model will favour more probable tokens.
To see the whole list of presets, check out the `SamplerPresets` class:
```typescript
class SamplerPresets {
static default(): SamplerConfig;
static dry(): SamplerConfig;
static greedy(): SamplerConfig;
static json(): SamplerConfig;
static temperature(temperature: number): SamplerConfig;
static topK(topK: number): SamplerConfig;
static topP(topP: number): SamplerConfig;
// Constrain output to a specific format:
static constrainWithJsonSchema(schema: string): SamplerConfig;
static constrainWithRegex(pattern: string): SamplerConfig;
static constrainWithGrammar(grammar: string): SamplerConfig;
}
```
#### Structured output
One of the most useful features is constraining the model to produce structured output —
this gives you a hard guarantee that the output matches a specific format, rather than
relying on the model to get it right on its own.
##### Regular expressions
For simpler patterns, you can constrain the output with a regex. Both regex literals and strings are accepted:
```typescript
// Force the model to answer with exactly "yes" or "no"
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.constrainWithRegex(/yes|no/),
});
const answer = await chat.ask("Is the sky blue?").completed();
```
##### JSON schema
In some use-cases it might be useful to let the LLM generate JSON output.
This could be done either in the simple way, just enforcing any JSON by the preset:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.json(),
});
```
Or utilizing JSON schemas to really force the LLM to give you the specific object shapes
that you want:
```typescript
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
sampler: SamplerPresets.constrainWithJsonSchema({
type: "object",
properties: {
name: { type: "string", maxLength: 50 },
age: { type: "integer" },
},
required: ["name", "age"],
additionalProperties: false,
}),
});
const response = await chat.ask("Give me a person as JSON with name and age fields.").completed();
const person = JSON.parse(response); // always valid JSON matching the schema
```
##### Custom grammars (advanced)
For cases where JSON schema and regex are not expressive enough, you can supply a custom grammar.
`constrainWithGrammar` accepts both **Lark** syntax and **GBNF** (llama.cpp format) —
NobodyWho automatically converts GBNF to Lark before passing it to the inference engine.
**Lark syntax** (recommended):
```typescript
const sampler = SamplerPresets.constrainWithGrammar(`
start: record (NEWLINE record)* NEWLINE?
record: field ("," field)*
field: /[^,"\\n\\r]+/
NEWLINE: /\\r?\\n/
`);
```
**GBNF syntax** (also accepted):
```typescript
const sampler = SamplerPresets.constrainWithGrammar(`
file ::= record (newline record)* newline?
record ::= field ("," field)*
field ::= /[^,"\\n\\r]+/
newline ::= "\\r\\n" | "\\n"
`);
```
See the [Lark documentation](https://lark-parser.readthedocs.io/en/latest/grammar.html) and the
[GBNF specification](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) for the
full grammar syntax.
**Info:**
The older `SamplerPresets.grammar()` method is deprecated. Use
`SamplerPresets.constrainWithGrammar()` instead — it accepts both Lark and GBNF strings.
#### Defining your own samplers
Sampler presets abstract away some control that you might want - for example, if you
want to chain samplers, change more "advanced" parameters, etc. For that use case,
we provide the `SamplerBuilder` class:
```typescript
import { Chat, SamplerBuilder, SamplerConfig } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
sampler: new SamplerBuilder().temperature(0.8).topK(5).dist() as SamplerConfig,
});
```
With `SamplerBuilder` you can chain multiple steps together and then select how you
want to sample from the distribution. Keep in mind that `SamplerBuilder` provides two
types of methods: ones which modify the distribution (returning again the instance of
`SamplerBuilder`) and ones which sample from the distribution (returning `SamplerConfig`).
So in order to have the sampler working properly, be careful
to always end the chain with one of the sampling steps (e.g. `dist()`, `greedy()`, `mirostatV2()`, etc.).
For reproducible output, set the RNG seed with `.seed(value)` anywhere in the chain.
It is consumed by every random sampler in the chain — `dist`, `mirostatV1`, `mirostatV2`,
and the `xtc` shift step. `greedy` ignores it. If unset, a default seed is used.
```typescript
const sampler = new SamplerBuilder().temperature(0.8).topK(5).seed(42).dist() as SamplerConfig;
```
##### Available sampling steps
Pick any of the **shift steps** below (each reshapes the token distribution), then finish with one **terminal step** that picks the token — exactly like the `.temperature(0.8).topK(5).dist()` chain above.
Shift steps — add as many as you want, applied in order:
- `.topK(40)` — keep only the 40 most likely tokens
- `.topP(0.95, 1)` — nucleus: keep the top tokens up to 95% of the probability mass
- `.minP(0.05, 1)` — drop tokens below 5% of the most likely token's probability
- `.typicalP(0.9, 1)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `.xtc(0.5, 0.1, 1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `.temperature(0.8)` — below 1.0 = more focused, above 1.0 = more random
- `.penalties(64, 1.1, 0.0, 0.0)` — per-token repetition penalty: `penaltyLastN, penaltyRepeat, penaltyFreq, penaltyPresent` (`penaltyRepeat` 1.0 = off)
- `.dry(0.8, 1.75, 2, -1, ["\n"])` — penalty for repeated *phrases*: `multiplier, base, allowedLength, penaltyLastN, seqBreakers`
- `.seed(42)` — fix the RNG for reproducible output
- `.grammar(...)` — deprecated; use the `constrainWith*` presets above
Terminal step — one of these turns the chain into a `SamplerConfig`, so finish with exactly one:
- `.dist()` — pick a token with weighted randomness (the usual choice)
- `.greedy()` — always take the most likely token
- `.mirostatV1(5.0, 0.1, 100)` / `.mirostatV2(5.0, 0.1)` — steer output "surprise" toward a target
`minKeep` is the floor on how many tokens survive a cut (`1` is fine). Terminal steps return a `SamplerConfig`, so cast with `as SamplerConfig` as in the examples.
You can also change the sampler configuration on an existing chat instance:
```typescript
const sampler = new SamplerBuilder()
.temperature(0.8)
.topK(5)
.dist() as SamplerConfig;
await chat.setSamplerConfig(sampler);
```
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/react-native/speech-to-text](https://docs.nobodywho.ooo/react-native/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format.
```typescript
import { SpeechToText } from "react-native-nobodywho";
const stt = await SpeechToText.load({
source: "hf://onnx-community/whisper-base",
});
const text = await stt.transcribeFile("recording.mp3").completed();
console.log(text);
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribePcm` is available:
```typescript
const text = await stt.transcribePcm(samples, 16000).completed();
```
In order to make this work, the buffer needs to be mono i16 PCM samples (an `Int16Array` or a plain `number[]`). The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with classic Chat models, streaming is available, so the transcription can be consumed token by token:
```typescript
for await (const piece of stt.transcribeFile("recording.mp3")) {
process.stdout.write(piece);
}
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `source` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```typescript
const stt = await SpeechToText.load({
source: "hf://onnx-community/whisper-base",
quantization: "int8",
});
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, pass its ISO 639-1 code as `language` to skip detection and improve performance:
```typescript
const stt = await SpeechToText.load({
source: "hf://onnx-community/whisper-base",
language: "en",
});
```
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/react-native/text-to-speech](https://docs.nobodywho.ooo/react-native/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
```typescript
import { TextToSpeech } from "react-native-nobodywho";
const tts = await TextToSpeech.load({
source: "hf://NobodyWho/Kokoro-82M", // Hugging Face repo (hf://owner/repo) or local folder with the model files.
voice: "bf_emma", // Voice to use from the model.
language: "en-gb", // Language code for the input text.
});
// Generate WAV bytes for this sentence.
const wav = await tts.synthesize("Hello from NobodyWho!");
// wav is a Uint8Array containing WAV bytes.
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, or a local directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/react-native/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/react-native/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```typescript
const tts = await TextToSpeech.load({
source: "hf://NobodyWho/Kokoro-82M",
voice: "bf_emma",
language: "en-gb",
});
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.0`.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```typescript
const tts = await TextToSpeech.load({
source: "hf://Supertone/supertonic-3",
language: "en",
});
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.05`.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Must be greater than `0`; defaults to `8`.
- `silenceDuration`: seconds of silence between long text chunks. Higher values add longer pauses. Must be `0` or higher; defaults to `0.3`.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```typescript
const tts = await TextToSpeech.load({
source: "hf://KevinAHM/pocket-tts-onnx",
voice: "alba",
language: "english_2026-04",
huggingfaceToken: "hf_...", // Alternative: set the HF_TOKEN environment variable.
});
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Defaults to `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Defaults to `int8`.
- `temperature`: controls how varied the speech sounds. Defaults to `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingfaceToken`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory or a custom source that NobodyWho cannot recognize:
```typescript
const tts = await TextToSpeech.load({
source: "/path/to/local/kokoro-folder",
architecture: "kokoro",
});
```
Supported architecture values are `kokoro`, `pocket-tts`, and `supertonic`.
#### GPU
TextToSpeech uses GPU acceleration by default when available. Disable it with `device: "cpu"`:
```typescript
const tts = await TextToSpeech.load({
source: "hf://Supertone/supertonic-3",
device: "cpu",
});
```
#### Local model folder format
When `source` is a local directory, point it at the top-level model folder and pass the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/react-native/tool-calling](https://docs.nobodywho.ooo/react-native/tool-calling)
To give your LLM the ability to interact with the outside world, you will need tool calling.
**Info:**
Note that **not every model** supports tool calling. If the model does not have
such an option, it might not call your tools.
For reliable tool calling, we recommend trying the [Qwen](https://huggingface.co/collections/NobodyWho/qwen-3) family of models.
#### Declaring a tool
A tool is created by providing a name, description, parameter schemas, and a callback function.
Any regular function can be used as a tool — arguments from the LLM are passed positionally
in the same order as the `parameters` array:
```typescript
import { Tool } from 'react-native-nobodywho';
const circleArea = (radius: number): string => {
const area = Math.PI * radius * radius;
return `Circle with radius ${radius} has area ${area.toFixed(2)}`;
};
const circleAreaTool = new Tool({
name: 'circle_area',
description: 'Calculates the area of a circle given its radius',
parameters: [
{ name: 'radius', type: 'number', description: 'The radius of the circle' },
],
call: circleArea,
});
```
Every `Tool` needs a callback function, a name, a description of what the tool does, and a `parameters` array describing its inputs. Each parameter uses [JSON Schema](https://json-schema.org/) properties (`type`, `enum`, `description`, etc.) plus a `name` field. Arguments are passed to your function positionally in array order.
To let your LLM use it, simply add it when creating `Chat`:
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
tools: [circleAreaTool],
});
```
NobodyWho then figures out the right tool calling format and configures the sampler.
Naturally, more tools can be defined and the model can chain the calls for them:
```typescript
import { Chat, Tool } from "react-native-nobodywho";
const getCurrentDir = (): string => '/home/user/documents';
// In a real app, you'd read the filesystem here
const listFiles = (path: string): string =>
'Files: report.pdf, notes.txt, model.gguf';
// In a real app, you'd check the actual file size
const getFileSize = (filepath: string): string => 'File size: 1024 bytes';
const getCurrentDirTool = new Tool({
name: "get_current_dir",
description: "Gets path of the current directory",
parameters: [],
call: getCurrentDir,
});
const listFilesTool = new Tool({
name: "list_files",
description: "Lists files in the given directory",
parameters: [
{ name: "path", type: "string", description: "The path to the directory to list" },
],
call: listFiles,
});
const getFileSizeTool = new Tool({
name: "get_file_size",
description: "Gets the size of a file in bytes",
parameters: [
{ name: "filepath", type: "string", description: "The path to the file" },
],
call: getFileSize,
});
const chat = await Chat.fromPath({
modelPath: "/path/to/model.gguf",
tools: [getCurrentDirTool, listFilesTool, getFileSizeTool],
});
const response = await chat
.ask("What is the biggest file in my current directory?")
.completed();
console.log(response);
```
#### Tool calling and the context
As with most things made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/react-native/vision](https://docs.nobodywho.ooo/react-native/vision)
A picture is worth a thousand words (or at least a thousand tokens).
With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/react-native/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/react-native/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts for making this work:
1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 even manage both!
Usually, the projection model includes `mmproj` in its name.
If you are unsure which ones to pick, or just want a reasonable default, you can try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true),
which can do both image and audio.
With the downloaded GGUFs, you can load them using `Chat.fromPath`:
```typescript
import { Chat } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf",
systemPrompt: "You are a helpful assistant, that can hear and see stuff!",
});
```
Or load the model separately:
```typescript
import { Model, Chat } from "react-native-nobodywho";
const model = await Model.load({
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf",
});
const chat = new Chat({
model,
systemPrompt: "You are a helpful assistant.",
});
```
#### Composing a prompt object
With the model configured, all that is left is to compose the prompt and send it to the model.
Use `Prompt` to build prompts that mix text, images, and audio, then pass them to `chat.ask()`:
```typescript
import { Chat, Prompt } from "react-native-nobodywho";
const chat = await Chat.fromPath({
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf",
});
const response = await chat
.ask(
new Prompt([
Prompt.Text("Tell me what you see in the image and what you hear in the audio."),
Prompt.Image("/path/to/dog.png"),
Prompt.Audio("/path/to/sound.mp3"),
]),
)
.completed();
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/react-native/voice-activity-detection](https://docs.nobodywho.ooo/react-native/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart.
#### Streaming
The main use case for VAD is streaming audio into the model chunk by chunk. This is useful, for
example, to detect when to stop listening to the microphone and start running
transcription/answer generation.
To do that, we provide a `push` interface:
```typescript
import {
VoiceActivityDetection,
VoiceActivityDetectionEvent,
SpeechToText,
} from "react-native-nobodywho";
const vad = await VoiceActivityDetection.load({
sampleRate: 16000,
source: "hf://onnx-community/silero-vad",
});
const stt = await SpeechToText.load({ source: "hf://onnx-community/whisper-base" });
while (true) {
const chunk = readMic(); // however you're reading from the microphone
if (vad.push(chunk) === VoiceActivityDetectionEvent.SpeechEnded) break;
}
const speech = vad.finish();
const transcription = await stt.transcribePcm(speech, 16000).completed();
console.log(transcription);
```
`VoiceActivityDetection` acts as a buffer: every `push()` call tells you the current state
(`SpeechStarted`, `SpeechEnded`, `Speech`, or `Silence`), so you can decide when to stop listening.
Once you do, `finish()` gives you back the buffered audio for just the speech segment, and resets
internal state so it's ready for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `segment()` method is for:
```typescript
import { VoiceActivityDetection, SpeechToText } from "react-native-nobodywho";
const vad = await VoiceActivityDetection.load({
sampleRate: 16000,
source: "hf://onnx-community/silero-vad",
});
const stt = await SpeechToText.load({ source: "hf://onnx-community/whisper-base" });
const audio = readWavPcm("recording.wav"); // a full recording as i16 PCM samples
for (const speech of vad.segment(audio)) {
const transcription = await stt.transcribePcm(speech, 16000).completed();
console.log(transcription);
}
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance. For that,
we provide numerous params that you can tweak:
```typescript
const vad = await VoiceActivityDetection.load({
sampleRate: 16000,
// VAD is currently fixed to Silero ONNX, but you can change the source.
source: "hf://onnx-community/silero-vad",
// Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
threshold: 0.5,
// Determines the minimum duration classified as speech.
minSpeechDurationMs: 250,
// Determines the minimum duration classified as silence.
minSilenceDurationMs: 250,
// Determines how much audio to keep before the official SpeechStarted, to avoid cutting off the start.
prerollDurationMs: 500,
});
```
---
## Flutter
This section documents Flutter 3.0.0, the latest released version.
### Chat
Source: [https://docs.nobodywho.ooo/flutter/chat](https://docs.nobodywho.ooo/flutter/chat)
As you may have noticed in the [welcome guide](https://docs.nobodywho.ooo/flutter/), every interaction with your LLM starts by instantiating a `Chat` object.
In the following sections, we talk about which configuration options it has, and when to use them.w
#### Creating a Chat
There are two main ways of instantiating a `Chat` object and the difference lies in when the model file is loaded.
The simplest way is using `Chat.fromPath` like so:
```dart
final chat = await nobodywho.Chat.fromPath(modelPath: "./model.gguf");
```
This function is async since loading a model can take a bit of time, but this should not block the any of your UI.
Another way to achieve the same thing is to load the model seperately and then use the `Chat` constructor:
```dart
final model = await nobodywho.Model.load(modelPath: "./model.gguf");
final chat = nobodywho.Chat(model : model);
```
This allows for sharing the model between several `Chat` instances.
You can pass `"auto"` as `modelPath` to select a chat model based on available memory.
#### Prompts and responses
The `Chat.ask()` function is central to NobodyWho. This function sends your message to the LLM, which then starts generating a response.
```dart
import "dart:io"
final chat = await nobodywho.Chat.fromPath(modelPath: "./model.gguf");
final response = chat.ask("Is water wet?");
```
The return type of `ask` is a `TokenStream`.
If you want to start reading the response as soon as possible, you can just iterate over the `TokenStream`.
Each token is either an individual word or fragments of a word.
```dart continuation
await for (final token in response) {
print(token);
}
```
If you just want to get the complete response, you can call `TokenStream.completed()`.
This will return the entire response string once the model is done generating its entire response.
```dart continuation
final fullResponse = await response.completed();
```
All of your messages and the model's responses are stored in the `Chat` object, so the next time you call `Chat.ask()`, it will remember the previous messages.
#### Chat history
If you want to inspect the messages inside the `Chat` object, you can use `getChatHistory`.
```dart continuation
final msgs = await chat.getChatHistory();
print(msgs[0].content); // "Is water wet?"
```
Similarly, if you want to edit what messages are in the context, you can use `setChatHistory`:
```dart continuation
await chat.setChatHistory([
nobodywho.Message.user(content: "What is water?")
]);
```
#### System prompt
A system prompt is a special message put into the chat context, which should guide its overall behavior.
Some models ship with a built-in system prompt. If you don't specify a system prompt yourself, NobodyWho will fall back to using the model's default system prompt.
You can specify a system prompt when initializing a `Chat`:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
systemPrompt: "You are a mischievous assistant!"
);
```
This `systemPrompt` is then persisted until the chat context is `reset`.
#### Context
The context is the text window which the LLM currently considers. Specifically this is the number of tokens the LLM keeps in memory for your current conversation.
As bigger context size means more computational overhead, it makes sense to constrain it. This can be done with `contextSize` setting, again at the time of creation:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
contextSize: 4096
);
```
The default value is `4096`, however this is mainly useful for short and simple conversations. Choosing the right context size is quite important and depends heavily on your use case. You can check the maximum context size the model was trained with using `model.maxCtx` — setting `contextSize` above this value has no benefit.
Even with properly selected context size it might happen that you fill up your entire context during a conversation. When this happens, NobodyWho will shrink the context for you. Currently this is done by removing old messages (apart from the system prompt and the first user message) from the chat history, until the size reaches `contextSize / 2`. The KV cache is also updated automatically. In the future we plan on adding more advanced methods of context shrinking.
Again, `contextSize` is fixed to the `Chat` instance, so it is currently not possible to change the size after `Chat` is created. To reset the current context content, just call `resetContext()` with the new system prompt and potentially changed tools.
```dart continuation
await chat.resetContext(systemPrompt: "New system prompt", tools: []);
```
If you don't want to change the already set defaults (`systemPrompt`, `tools`), but only reset the context, then go for `resetHistory`.
To inspect how much of the context is currently in use, call `getStats()`:
```dart continuation
final stats = await chat.getStats();
print("Using ${stats.contextUsed} of ${stats.contextSize} tokens");
```
#### CPU threads
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Hyperthread siblings and efficiency cores end up pacing the whole
thread pool, so using every CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads)
for the numbers.
Override it with `threadCount` when you want to leave CPU headroom for the rest of your app —
often a good idea on phones, where the big cores are also driving the UI:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
threadCount: 4
);
```
Leave it unset to keep the detected default. Values above the device's CPU count are clamped,
and it has little effect when the model is offloaded to the GPU.
#### Sharing model between contexts
There are scenarios where you would like to keep separate chat contexts (e.g. for every user of your app), but have only one model loaded. In this case you must load the model
seperately from creating the `Chat` instance.
For this use case, instead of the path to the `.gguf` model, you can pass in `Model` object, which can be shared between multiple `Chat` instances.
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final model = await nobodywho.Model.load(modelPath: './model.gguf');
final chat1 = nobodywho.Chat(model: model);
final chat2 = nobodywho.Chat(model: model);
...
```
NobodyWho will then take care of the separation, such that your chat histories won't collide or interfere with each other, while having only one model loaded.
#### GPU
When instantiating `Model` or using `Chat.fromPath` you have the option to disable/enable GPU acceleration. This can be done as:
```dart
final model = await nobodywho.Model.load(modelPath: './model.gguf', useGpu: true);
```
or
```dart
final chat = await nobodywho.Chat.fromPath(modelPath: './model.gguf', useGpu : false);
```
By defualt `useGpu` is set to true.
So far, NobodyWho relies purely on [Vulkan](https://www.vulkan.org), however support
of more architectures is planned (for details check out our [issues](https://github.com/nobodywho-ooo/nobodywho/issues) or join us on [Discord](https://discord.gg/qhaMc2qCYB)).
#### Speculative decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target model verify several candidate tokens per forward pass. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
Load the model with a compatible draft-heads gguf (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B) and pass an `MtpConfig` when constructing the chat:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final model = await nobodywho.Model.load(
modelPath: "./gemma-4-e2b.gguf",
draftModelPath: "./mtp-gemma-4-e2b.gguf",
);
// MtpConfig() uses the default drafter tuning; tune with MtpConfig(kMax: ..., pMin: ...)
final chat = nobodywho.Chat(model: model, mtp: const nobodywho.MtpConfig());
```
`Chat.fromPath` accepts the same two parameters if you don't need to share the model:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: "./gemma-4-e2b.gguf",
draftModelPath: "./mtp-gemma-4-e2b.gguf",
mtp: const nobodywho.MtpConfig(),
);
```
Loading the draft heads adds around 5% to VRAM usage.
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
#### Template Variables
Chat templates are used internally by models to format conversation history into the expected prompt format. Different models may support different template variables that control specific behaviors. Template variables are boolean flags passed to the chat template that can enable or disable certain features.
##### Using Template Variables
You can set template variables when creating a chat or modify them on existing instances:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
templateVariables: {"enable_thinking": true}
);
```
You can also modify template variables on an existing chat instance:
```dart continuation
// Set a single template variable
await chat.setTemplateVariable("enable_thinking", true);
// Set multiple template variables at once
await chat.setTemplateVariables({
"enable_thinking": true,
"verbose_mode": false
});
// Get current template variables
final variables = await chat.getTemplateVariables();
print(variables); // {enable_thinking: true, verbose_mode: false}
```
With the next message sent, the updated settings will be propagated to the model.
##### Example: Qwen3 and Qwen3.5 Reasoning
The Qwen3 and Qwen3.5 model families support the `enable_thinking` template variable, which controls whether the model should engage in explicit reasoning steps before answering:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
templateVariables: {"enable_thinking": true}
);
final response = chat.ask("Solve this logic puzzle: ...");
```
When `enable_thinking` is enabled, these models will show their reasoning process before providing the final answer.
##### Model-Specific Variables
Different models may support different template variables depending on their chat template implementation. The available variables and their effects depend entirely on how the model's chat template is designed. Check your model's documentation to see which template variables are supported.
**Info:**
Note that template variables are model-specific. If a model's chat template doesn't use a specific variable, that variable will be ignored gracefully.
##### Backward Compatibility
For backward compatibility, the deprecated `allowThinking` parameter is still available but internally sets the `enable_thinking` template variable:
```dart
// Deprecated - use templateVariables instead
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
allowThinking: true
);
```
---
### Downloading models
Source: [https://docs.nobodywho.ooo/flutter/downloading-models](https://docs.nobodywho.ooo/flutter/downloading-models)
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to access gated/private models, how to observe a download in progress, and how to inspect what's already in the local cache.
#### Supported model path formats
The `modelPath` argument to `Chat.fromPath`, `downloadModel`, and friends accepts:
| Form | Example | Notes |
| ---- | ------- | ----- |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
| Local path | `./model.gguf` | Used as-is |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs.
#### Android permissions
Loading a model from `hf://` or `https://` needs network access. Add the internet permission to `android/app/src/main/AndroidManifest.xml`:
```xml
```
Things to note: Flutter's generated debug and profile manifests already include this permission for the development tooling, so downloads can succeed while you are debugging and then fail in a release build. Declaring it in the main manifest covers every build mode. Apps that only load models from local paths need no network permission.
#### Downloading a gated model
Some HuggingFace models are private or gated by a license you need to accept. In both cases you need to be authorized to download the model weights.
You can manually download the GGUF file via your web browser and then point `Chat.fromPath` at the local path:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
);
```
Or use `downloadModel` with an `Authorization` header:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final modelPath = await nobodywho.downloadModel(
modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
headers: {'Authorization': 'Bearer your_hf_token'},
);
final chat = await nobodywho.Chat.fromPath(modelPath: modelPath);
```
You can generate a HuggingFace token in [your account settings](https://huggingface.co/settings/tokens).
#### Tracking download progress
When loading a remote model, pass an `onDownloadProgress` callback to observe the download. It receives `(downloadedBytes, totalBytes)`, is throttled to roughly 10 Hz with a guaranteed final emit on completion, and is not called for cached or local files.
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
onDownloadProgress: (downloaded, total) {
print('$downloaded / $total bytes');
},
);
```
#### Inspecting the model cache
`getCachedModels` returns every `.gguf` model that lives in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `downloadModel` and by `Chat.fromPath`'s `huggingface:` paths. The call is synchronous.
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final models = nobodywho.getCachedModels();
for (final (path, size) in models) {
print('$path: ${size ~/ BigInt.from(1024 * 1024)} MiB');
}
```
- Paths are absolute.
- Sizes are `BigInt` byte counts (the underlying Rust `usize`).
- The list is empty if nothing has been downloaded yet.
- Throws if the cache directory cannot be read.
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/flutter/embeddings-and-rag](https://docs.nobodywho.ooo/flutter/embeddings-and-rag)
When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.
#### Understanding Embeddings
Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.
For example, "Schedule a meeting for next Tuesday" and "Book an appointment next week" would have very similar embeddings, despite using different words.
#### The Encoder
The `Encoder` object converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).
We recommend you first try [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf).
```dart
import 'dart:typed_data';
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
final embedding = await encoder.encode(text: "What is the weather like?");
print("Vector with ${embedding.length} dimensions");
```
The resulting embedding is a `Float32List` (typically 384 or 768 dimensions depending on the model).
##### Batch Encoding
Use `encodeBatch` for multiple texts. It returns a `List` in input order:
```dart continuation
final texts = [
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
];
final embeddings = await encoder.encodeBatch(texts: texts);
```
##### Comparing Embeddings
To measure how similar two pieces of text are, compare their embeddings using cosine similarity:
```dart
import 'dart:typed_data';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
final query = await encoder.encode(text: "How do I reset my password?");
final doc1 = await encoder.encode(text: "You can reset your password in the account settings");
final doc2 = await encoder.encode(text: "The password requirements include 8 characters minimum");
final similarity1 = nobodywho.cosineSimilarity(
a: query.toList(),
b: doc1.toList()
);
final similarity2 = nobodywho.cosineSimilarity(
a: query.toList(),
b: doc2.toList()
);
print("Document 1 similarity: ${similarity1.toStringAsFixed(3)}"); // Higher score
print("Document 2 similarity: ${similarity2.toStringAsFixed(3)}"); // Lower score
```
Cosine similarity returns a value between -1 and 1, where 1 means identical meaning and -1 means opposite meaning.
##### Practical Example: Finding Relevant Documents
```dart
import 'dart:typed_data';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
// Your knowledge base
final documents = [
"Python supports multiple programming paradigms including object-oriented and functional",
"JavaScript is primarily used for web development and runs in browsers",
"SQL is a domain-specific language for managing relational databases",
"Git is a version control system for tracking changes in source code"
];
// Pre-compute document embeddings in a batch
final docEmbeddings = await encoder.encodeBatch(texts: documents);
// Search query
final query = "What language should I use for database queries?";
final queryEmbedding = await encoder.encode(text: query);
// Find the most relevant document
double maxSimilarity = -1;
int bestIdx = 0;
for (int i = 0; i < docEmbeddings.length; i++) {
final similarity = nobodywho.cosineSimilarity(
a: queryEmbedding.toList(),
b: docEmbeddings[i].toList()
);
if (similarity > maxSimilarity) {
maxSimilarity = similarity;
bestIdx = i;
}
}
print("Most relevant: ${documents[bestIdx]}");
print("Similarity score: ${maxSimilarity.toStringAsFixed(3)}");
```
#### The CrossEncoder for Better Ranking
While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents to determine how well the document answers the query.
The key difference is that embeddings compare vector similarity, while cross-encoders understand the relationship between query and document, at a potentially larger computation cost.
##### Why CrossEncoder Matters
Consider this example:
```
Query: "What are the office hours for customer support?"
Documents: [
"Customer asked: What are the office hours for customer support?",
"Support team responds: Our customer support is available Monday-Friday 9am-5pm EST",
"Note: Weekend support is not available at this time"
]
```
Using embeddings alone, the first document scores highest (most similar to the query) even though it provides no useful information. A cross-encoder correctly identifies that the second document actually answers the question.
##### Using CrossEncoder
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
// Download a reranking model like bge-reranker-v2-m3-Q8_0.gguf
final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
final query = "How do I install Python packages?";
final documents = [
"Someone previously asked about Python packages",
"Use pip install package-name to install Python packages",
"Python packages are not included in the standard library"
];
// Get relevance scores for each document
final scores = await crossencoder.rank(query: query, documents: documents);
print(scores); // [0.23, 0.89, 0.45] - second doc scores highest
```
##### Automatic Sorting
For convenience, use `rankAndSort` to get documents sorted by relevance:
```dart continuation
// Returns list of (document, score) tuples, sorted by score
final rankedDocs = await crossencoder.rankAndSort(query: query, documents: documents);
for (final (doc, score) in rankedDocs) {
print("[${score.toStringAsFixed(3)}] $doc");
}
```
This returns documents ordered from most to least relevant.
#### Building a RAG System
Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.
Here's a complete example building a customer service assistant with access to company policies:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
Future main() async {
// Initialize the cross-encoder for document ranking
final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
// Your knowledge base
final knowledge = [
"Our company offers a 30-day return policy for all products",
"Free shipping is available on orders over \$50",
"Customer support is available via email and phone",
"We accept credit cards, PayPal, and bank transfers",
"Order tracking is available through your account dashboard"
];
// Create a tool that searches the knowledge base
final searchKnowledgeTool = nobodywho.Tool(
function: ({required String query}) async {
// Rank all documents by relevance to the query
final ranked = await crossencoder.rankAndSort(query: query, documents: knowledge);
// Return top 3 most relevant documents
final topDocs = ranked.take(3).map((e) => e.$1).toList();
return topDocs.join("\n");
},
name: "search_knowledge",
description: "Search the knowledge base for relevant information"
);
// Create a chat with access to the knowledge base
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
systemPrompt: "You are a customer service assistant. Use the search_knowledge tool to find relevant information from our policies before answering customer questions.",
templateVariables: {"enable_thinking": false},
tools: [searchKnowledgeTool]
);
// The chat will automatically search the knowledge base when needed
final response = await chat.ask("What is your return policy?").completed();
print(response);
}
```
The LLM will call the `search_knowledge` tool, receive the most relevant documents, and use them to generate an accurate answer.
#### Async Operations
In Flutter/Dart, all operations are asynchronous by default. There are no separate `EncoderAsync` or `CrossEncoderAsync` classes - the regular `Encoder` and `CrossEncoder` classes use async/await patterns:
```dart
import 'dart:typed_data';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
Future main() async {
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
// Generate embeddings asynchronously
final embedding = await encoder.encode(text: "What is the weather?");
// Rank documents asynchronously
final query = "What is our refund policy?";
final docs = [
"Refunds processed within 5-7 business days",
"No refunds on sale items",
"Contact support to initiate refund"
];
final ranked = await crossencoder.rankAndSort(query: query, documents: docs);
for (final (doc, score) in ranked) {
print("[${score.toStringAsFixed(3)}] $doc");
}
}
```
#### Recommended Models
##### For Embeddings
- [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf) - Good balance of speed and quality (~25MB)
- Supports English text with 384-dimensional embeddings
##### For Cross-Encoding (Reranking)
- [bge-reranker-v2-m3-Q8_0.gguf](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf) - Multilingual support with excellent accuracy
#### Best Practices
**Precompute embeddings**: If you have a fixed knowledge base, generate embeddings once and reuse them. Don't re-encode the same documents repeatedly.
**Use embeddings for filtering**: When working with large document collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank them.
**Limit cross-encoder inputs**: Cross-encoders are more expensive than embeddings. Don't pass thousands of documents to `rank()` - filter first with embeddings.
**Choose appropriate context size**: The `nCtx` parameter (default 4096) should match your model's recommended context size. Check the model documentation.
```dart
// For longer documents, increase context size
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
```
#### Complete RAG Example
Here's a full example showing a two-stage retrieval system:
```dart
import 'dart:typed_data';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
Future main() async {
// Initialize models
final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');
final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
// Large knowledge base
final knowledgeBase = [
"Python 3.11 introduced performance improvements through faster CPython",
"The Django framework is used for building web applications",
"NumPy provides support for large multi-dimensional arrays",
"Pandas is the standard library for data manipulation and analysis",
// ... 100+ more documents
];
// Precompute embeddings for all documents
final docEmbeddings = await encoder.encodeBatch(texts: knowledgeBase);
Future search({required String query}) async {
// Stage 1: Fast filtering with embeddings
final queryEmbedding = await encoder.encode(text: query);
final similarities = <(String, double)>[];
for (int i = 0; i < knowledgeBase.length; i++) {
final similarity = nobodywho.cosineSimilarity(
a: queryEmbedding.toList(),
b: docEmbeddings[i].toList()
);
similarities.add((knowledgeBase[i], similarity));
}
// Get top 20 candidates
similarities.sort((a, b) => b.$2.compareTo(a.$2));
final candidateDocs = similarities.take(20).map((e) => e.$1).toList();
// Stage 2: Precise ranking with cross-encoder
final ranked = await crossencoder.rankAndSort(query: query, documents: candidateDocs);
// Return top 3 most relevant
final topResults = ranked.take(3).map((e) => e.$1).toList();
return topResults.join("\n---\n");
}
final searchTool = nobodywho.Tool(
function: search,
name: "search",
description: "Search the knowledge base for information relevant to the query"
);
// Create RAG-enabled chat
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
systemPrompt: "You are a technical documentation assistant. Always use the search tool to find relevant information before answering programming questions.",
templateVariables: {"enable_thinking": false},
tools: [searchTool]
);
// The chat automatically searches and uses retrieved documents
final response = await chat.ask("What Python libraries are best for data analysis?").completed();
print(response);
}
```
This two-stage approach combines the speed of embeddings with the accuracy of cross-encoders, making it efficient even for large knowledge bases.
---
### Getting started
Source: [https://docs.nobodywho.ooo/flutter/](https://docs.nobodywho.ooo/flutter/)
#### How do I get started?
First, install `nobodywho`.
```bash
flutter pub add nobodywho
```
Next you need to import NobodyWho and we highly suggets you do this using the namespace `nobodywho` like so:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
```
since we have generic names such as `Model` and `Chat` in our package.
After you have imported the package it is very important that the next step is done correctly. As we dynamically link the rust binaries you must make
the following function call exactly once in your application!
```dart
await nobodywho.NobodyWho.init();
```
A call to any of the functions in NobodyWho will result in an error before `.init()` has been called.
However a second call to `.init()` will also result in an error, so you should be mindful about when you make this call.
We suggest you make it as early and as close to the root of your app as possible, as even though it is async it is a very fast operation.
With that setup done we can move on to the exiting stuff! We will in the rest of the docs that
you have imported NobodyWho using namespacing and that `.init()` has been called.
Now you are ready to pick a model. NobodyWho can download GGUF models directly from Hugging Face — just pass a `huggingface:` path. See [model selection](https://docs.nobodywho.ooo/docs/model-selection) for recommendations.
Then create a `Chat` object and call `.ask`!
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
final msg = await chat.ask('Is water wet?').completed();
print(msg); // Yes, indeed, water is wet!
```
This is a super simple example, but we believe that examples which do simple things, should be simple!
To get a full overview of the functionality provided by NobodyWho, simply keep reading. You can also have a look at our [flutter starter app repository](https://github.com/nobodywho-ooo/flutter-starter-example).
#### Minimum recommended specs
- iOS: iPhone 11 or newer with at least 4 GB of RAM. We tested a Qwen3 0.6B (332 MB) on an iPhone X (iOS 16) and while it ran, performance was too slow to be practical.
- Android: Snapdragon 855 / Adreno 640 / 6 GB RAM or better. The same Qwen3 0.6B model performed notably better on a OnePlus 7 Pro (Android 12) than on the iPhone X tested above.
#### Feedback & Contributions
We welcome your feedback and ideas!
- Bug Reports & Improvements: If you encounter a bug or have suggestions, please open an issue on our [Issues](https://github.com/nobodywho-ooo/nobodywho/issues) page.
- Feature Requests & Question: For new feature requests or general questions, join the discussion on our [Discussions](https://github.com/nobodywho-ooo/nobodywho/discussions) page.
---
### Sampling
Source: [https://docs.nobodywho.ooo/flutter/sampling](https://docs.nobodywho.ooo/flutter/sampling)
The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which using NobodyWho you can freely modify,
to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).
#### Sampler presets
To get a quick start, NobodyWho offers a couple of well-known presets, which you can quickly utilize.
For example, if you want to increase or decrease the "creativity" of your model, select our `temperature` preset:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
sampler: nobodywho.SamplerPresets.temperature(temperature: 0.2)
);
```
Setting `temperature` to `0.2`, will then affect the sampler when choosing the next token, making the distribution less flat and therefore the model will favour more probable tokens.
To see the whole list of presets, check out the `SamplerPresets` class:
```dart
class SamplerPresets {
static SamplerConfig defaultSampler();
static SamplerConfig dry();
static SamplerConfig greedy();
static SamplerConfig json();
static SamplerConfig temperature({required double temperature});
static SamplerConfig topK({required int topK});
static SamplerConfig topP({required double topP});
// Constrain output to a specific format:
static SamplerConfig constrainWithJsonSchema({required String schema});
static SamplerConfig constrainWithRegex({required String pattern});
static SamplerConfig constrainWithGrammar({required String grammar});
}
```
#### Structured output
One of the most useful features is constraining the model to produce structured output —
this gives you a hard guarantee that the output matches a specific format, rather than
relying on the model to get it right on its own.
##### Regular expressions
For simpler patterns, you can constrain the output with a regex:
```dart
// Force the model to answer with exactly "yes" or "no"
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
sampler: nobodywho.SamplerPresets.constrainWithRegex(pattern: r'yes|no'),
);
final answer = await chat.ask("Is the sky blue?").completed();
```
##### JSON schema
In some use-cases it might be useful to let the LLM generate JSON output.
Use `constrainWithJsonSchema` to enforce a specific JSON shape:
```dart
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
sampler: nobodywho.SamplerPresets.constrainWithJsonSchema(schema: {
'type': 'object',
'properties': {
'name': {'type': 'string', 'maxLength': 50},
'age': {'type': 'integer'},
},
'required': ['name', 'age'],
'additionalProperties': false,
}),
);
final response = await chat.ask("Give me a person as JSON with name and age fields.").completed();
final person = jsonDecode(response); // always valid JSON matching the schema
```
##### Custom grammars (advanced)
For cases where JSON schema and regex are not expressive enough, you can supply a custom grammar.
`constrainWithGrammar` accepts both **Lark** syntax and **GBNF** (llama.cpp format) —
NobodyWho automatically converts GBNF to Lark before passing it to the inference engine.
**Lark syntax** (recommended):
```dart
final sampler = nobodywho.SamplerPresets.constrainWithGrammar(grammar: """
start: record (NEWLINE record)* NEWLINE?
record: field ("," field)*
field: /[^,"\\n\\r]+/
NEWLINE: /\\r?\\n/
""");
```
**GBNF syntax** (also accepted):
```dart
final sampler = nobodywho.SamplerPresets.constrainWithGrammar(grammar: """
file ::= record (newline record)* newline?
record ::= field ("," field)*
field ::= /[^,"\\n\\r]+/
newline ::= "\\r\\n" | "\\n"
""");
```
See the [Lark documentation](https://lark-parser.readthedocs.io/en/latest/grammar.html) and the
[GBNF specification](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md) for the
full grammar syntax.
**Info:**
The older `SamplerPresets.grammar()` method is deprecated. Use
`SamplerPresets.constrainWithGrammar()` instead — it accepts both Lark and GBNF strings.
#### Defining your own samplers
Sampler presets abstract away some control, that you might want - for example, if you
want to chain samplers, change more "advanced" parameters, etc. For that use case,
we provide `SamplerBuilder` class:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final chat = await nobodywho.Chat.fromPath(
modelPath: "./model.gguf",
sampler: nobodywho.SamplerBuilder()
.temperature(temperature: 0.8)
.topK(topK: 5)
.dist()
);
```
With `SamplerBuilder` you can chain multiple steps together and then select how do you
want to sample from the distribution. Keep in mind, that `SamplerBuilder` provides two
types of methods: ones which modify the distribution (returning again the instance of
`SamplerBuilder`) and ones which sample from the distribution (returning `SamplerConfig`).
So in order to have the sampler working properly and not giving you type errors, be careful
to always end the chain with one of the sampling steps (e.g. `dist()`, `greedy()`, `mirostatV2()`, etc.).
For reproducible output, set the RNG seed with `.seed(seed: value)` anywhere in the chain.
It is consumed by every random sampler in the chain — `dist`, `mirostatV1`, `mirostatV2`,
and the `xtc` shift step. `greedy` ignores it. If unset, a default seed is used.
```dart
final sampler = nobodywho.SamplerBuilder()
.temperature(temperature: 0.8)
.topK(topK: 5)
.seed(seed: 42)
.dist();
```
##### Available sampling steps
Pick any of the **shift steps** below (each reshapes the token distribution), then finish with one **terminal step** that picks the token — exactly like the `.temperature(...).topK(...).dist()` chain above.
Shift steps — add as many as you want, applied in order:
- `.topK(topK: 40)` — keep only the 40 most likely tokens
- `.topP(topP: 0.95, minKeep: 1)` — nucleus: keep the top tokens up to 95% of the probability mass
- `.minP(minP: 0.05, minKeep: 1)` — drop tokens below 5% of the most likely token's probability
- `.typicalP(typP: 0.9, minKeep: 1)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `.xtc(xtcProbability: 0.5, xtcThreshold: 0.1, minKeep: 1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `.temperature(temperature: 0.8)` — below 1.0 = more focused, above 1.0 = more random
- `.penalties(penaltyLastN: 64, penaltyRepeat: 1.1, penaltyFreq: 0.0, penaltyPresent: 0.0)` — per-token repetition penalty (`penaltyRepeat` 1.0 = off)
- `.dry(multiplier: 0.8, base: 1.75, allowedLength: 2, penaltyLastN: -1, seqBreakers: ["\n"])` — penalty for repeated *phrases*
- `.seed(seed: 42)` — fix the RNG for reproducible output
- `.grammar(...)` — deprecated; use the `constrainWith*` presets above
Terminal step — one of these turns the chain into a `SamplerConfig`, so finish with exactly one:
- `.dist()` — pick a token with weighted randomness (the usual choice)
- `.greedy()` — always take the most likely token
- `.mirostatV1(tau: 5.0, eta: 0.1, m: 100)` / `.mirostatV2(tau: 5.0, eta: 0.1)` — steer output "surprise" toward a target
`minKeep` is the floor on how many tokens survive a cut (`1` is fine).
You can also change the sampler configuration on an existing chat instance:
```dart
final chat = await nobodywho.Chat.fromPath(modelPath: "./model.gguf");
final sampler = nobodywho.SamplerBuilder()
.temperature(temperature: 0.8)
.topK(topK: 5)
.dist();
await chat.setSamplerConfig(sampler);
```
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/flutter/speech-to-text](https://docs.nobodywho.ooo/flutter/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format.
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
// ... after NobodyWho.init().
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');
final text = await stt.transcribeFile('recording.mp3').completed();
print(text);
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribePcm` is available:
```dart continuation
final text = await stt.transcribePcm(samples, 16000).completed();
```
In order to make this work, the buffer needs to be mono i16 PCM samples. The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with classic Chat models, streaming is available, so the transcription can be consumed token by token:
```dart continuation
await for (final piece in stt.transcribeFile('recording.mp3')) {
stdout.write(piece);
}
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `source` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```dart
final stt = await nobodywho.SpeechToText.load(
source: 'hf://onnx-community/whisper-base',
quantization: 'q4',
);
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, pass its ISO 639-1 code as `language` to skip detection and improve performance:
```dart
final stt = await nobodywho.SpeechToText.load(
source: 'hf://onnx-community/whisper-base',
language: 'en',
);
```
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/flutter/text-to-speech](https://docs.nobodywho.ooo/flutter/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
```dart
import 'dart:io';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
// ... after NobodyWho.init().
final tts = await nobodywho.TextToSpeech.load(
source: 'hf://NobodyWho/Kokoro-82M', // Hugging Face repo (hf://owner/repo) or local folder with the model files.
voice: 'bf_emma', // Voice to use from the model.
language: 'en-gb', // Language code for the input text.
);
// Generate WAV bytes for this sentence.
final wav = await tts.synthesize(text: 'Hello from NobodyWho!');
// Save the audio to a file.
await File('out.wav').writeAsBytes(wav);
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, or a local directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/flutter/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/flutter/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```dart
// ... after NobodyWho.init().
final tts = await nobodywho.TextToSpeech.load(
source: 'hf://NobodyWho/Kokoro-82M',
voice: 'bf_emma',
language: 'en-gb',
);
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.0`.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```dart
// ... after NobodyWho.init().
final tts = await nobodywho.TextToSpeech.load(
source: 'hf://Supertone/supertonic-3',
language: 'en',
);
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Defaults to `1.05`.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Must be greater than `0`; defaults to `8`.
- `silenceDuration`: seconds of silence between long text chunks. Higher values add longer pauses. Must be `0` or higher; defaults to `0.3`.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```dart
final tts = await nobodywho.TextToSpeech.load(
source: 'hf://KevinAHM/pocket-tts-onnx',
voice: 'alba',
language: 'english_2026-04',
huggingfaceToken: 'hf_...', // Alternative: set the HF_TOKEN environment variable.
);
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Defaults to `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Defaults to `int8`.
- `temperature`: controls how varied the speech sounds. Defaults to `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingfaceToken`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory or a custom source that NobodyWho cannot recognize:
```dart
// ... after NobodyWho.init().
final tts = await nobodywho.TextToSpeech.load(
source: '/path/to/local/kokoro-folder',
architecture: .kokoro',
);
```
Supported architecture values are `kokoro`, `pocket-tts`, and `supertonic`.
#### GPU
TextToSpeech uses GPU acceleration by default when available. Disable it with `useGpu: false`:
```dart
// ... after NobodyWho.init().
final tts = await nobodywho.TextToSpeech.load(
source: 'hf://Supertone/supertonic-3',
useGpu: false,
);
```
#### Local model folder format
When `source` is a local directory, point it at the top-level model folder and pass the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/flutter/tool-calling](https://docs.nobodywho.ooo/flutter/tool-calling)
To give your LLM the ability to interact with the outside world, you will need tool calling.
**Info:**
Note that **not every model** supports tool calling. If the model does not have
such an option, it might not call your tools.
For reliable tool calling, we recommend trying the [Qwen](https://huggingface.co/Qwen/models) family of models.
#### Declaring a tool
A tool can be created from any Dart function that returns a `String` or `Future`.
To define one, pass the function to `Tool()` with a name and description. To get
a good sense of how such a tool can look like, consider this geometry example:
```dart
import 'dart:math' as math;
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final circleAreaTool = nobodywho.Tool(
name: "circle_area",
description: "Calculates the area of a circle given its radius",
function: ({ required double radius }) {
final area = math.pi * radius * radius;
return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
}
);
```
As you can see, every `Tool()` call needs a function, a name, and a description
of what such tool does. To let your LLM use it, simply add it when creating `Chat`:
```dart continuation
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
tools: [circleAreaTool]
);
```
NobodyWho then figures out the right tool calling format, inspects the names and types of the parameters,
and configures the sampler.
Naturally, more tools can be defined and the model can chain the calls for them:
```dart
import 'dart:io';
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final getCurrentDirTool = nobodywho.Tool(
name: "get_current_dir",
description: "Gets path of the current directory",
function: () => Directory.current.path
);
final listFilesTool = nobodywho.Tool(
name: "list_files",
description: "Lists files in the given directory.",
function: ({required String path}) {
final dir = Directory(path);
final files = dir.listSync()
.where((entity) => entity is File)
.map((file) => file.path.split('/').last)
.toList();
return "Files: ${files.join(', ')}";
},
parameterDescriptions : {"path" : "The path to directory you want list. Must be a valid path." }
);
final getFileSizeTool = nobodywho.Tool(
name: "get_file_size",
description: "Gets the size of a file in bytes.",
function: ({required String filepath}) async {
final file = File(filepath);
final size = await file.length();
return "File size: $size bytes";
},
parameterDescriptions : {"filepath" : "The path to file you wish to know the size of. Must be a valid path." }
);
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
tools: [getCurrentDirTool, listFilesTool, getFileSizeTool],
templateVariables: {"enable_thinking": false}
);
final response = await chat.ask('What is the biggest file in my current directory?').completed();
print(response); // The largest file in your current directory is `model.gguf`.
```
#### Pre-packaged tools
We ship NobodyWho with two packaged-in tools, which are general enough for multiple use-cases - [monty](https://github.com/pydantic/monty) Python interpreter
and [bashkit](https://github.com/everruns/bashkit) Bash interpreter. Both of them should serve similar purpose - to give your small LLM a better chance to answer
questions requiring precise reasoning or some kind of computation, possibly on a big context.
The usage is straightforward. Use the `Tool.python()` and `Tool.bash()` factory constructors:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final chat = await nobodywho.Chat.fromPath(
modelPath: './model.gguf',
tools: [nobodywho.Tool.python(), nobodywho.Tool.bash()],
);
```
Lastly, keep in mind that for most use-cases it is reasonable to constrain the tools with some limits regarding memory and computation time,
so that you don't end up executing infinite loop code. To solve this, `Tool.python()` provides `maxDuration`, `maxMemoryBytes` and `maxRecursionDepth`
and `Tool.bash()` provides `maxCommands`.
#### Tool calling and the context
As with everything made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/flutter/vision](https://docs.nobodywho.ooo/flutter/vision)
Easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/flutter/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/flutter/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts for making this work:
1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 even manage both!
Usually, the projection model then includes `mmproj` in its name.
If you are unsure which ones to pick, or just want a reasonable default, you can try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true),
which can do both image and audio.
With the downloaded GGUFs, you can simply add the projection model when loading the model:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
final model = await nobodywho.Model.load(
modelPath: "./multimodal-model.gguf",
projectionModelPath: "./mmproj.gguf",
);
final chat = nobodywho.Chat(
model: model,
systemPrompt: "You are a helpful assistant, that can hear and see stuff!",
);
```
**Info:**
The language model and projection model have to **fit** together, as they are trained together!
Unfortunately you can't just take projection model and a LLM that you like and expect them
to work together.
#### Composing a prompt object
With the model configured, all that is left is to compose the prompt and send it to the model.
That is done through `askWithPrompt`, which accepts a `Prompt` containing a list of `PromptPart` values.
```dart continuation
final response = await chat.askWithPrompt(nobodywho.Prompt([
nobodywho.TextPart("Tell me what you see in the image and what you hear in the audio."),
nobodywho.ImagePart("./dog.png"),
nobodywho.AudioPart("./sound.mp3"),
])).completed(); // It's a dog and a penguin!
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/flutter/voice-activity-detection](https://docs.nobodywho.ooo/flutter/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart.
#### Streaming
The main use case for VAD is streaming audio into the model chunk by chunk. This is useful, for
example, to detect when to stop listening to the microphone and start running
transcription/answer generation.
To do that, we provide a `push` interface:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
// ... after NobodyWho.init().
final vad = await nobodywho.VoiceActivityDetection.load(
sampleRate: 16000,
source: 'hf://onnx-community/silero-vad',
);
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');
while (true) {
final chunk = readMic(); // however you're reading from the microphone
if (vad.push(chunk) == nobodywho.VoiceActivityDetectionEvent.speechEnded) break;
}
final speech = vad.finish();
final transcription = await stt.transcribePcm(speech, 16000).completed();
print(transcription);
```
`VoiceActivityDetection` acts as a buffer: every `push()` call tells you the current state
(`speechStarted`, `speechEnded`, `speech`, or `silence`), so you can decide when to stop listening.
Once you do, `finish()` gives you back the buffered audio for just the speech segment, and resets
internal state so it's ready for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `segment()` method is for:
```dart
import 'package:nobodywho/nobodywho.dart' as nobodywho;
// ... after NobodyWho.init().
final vad = await nobodywho.VoiceActivityDetection.load(
sampleRate: 16000,
source: 'hf://onnx-community/silero-vad',
);
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');
final audio = readWavPcm('recording.wav'); // a full recording as i16 PCM samples
for (final speech in vad.segment(audio)) {
final transcription = await stt.transcribePcm(speech, 16000).completed();
print(transcription);
}
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance. For that,
we provide numerous params that you can tweak:
```dart
// ... after NobodyWho.init().
final vad = await nobodywho.VoiceActivityDetection.load(
sampleRate: 16000,
// VAD is currently fixed to Silero ONNX, but you can change the source.
source: 'hf://onnx-community/silero-vad',
// Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
threshold: 0.5,
// Determines the minimum duration classified as speech.
minSpeechDurationMs: 250,
// Determines the minimum duration classified as silence.
minSilenceDurationMs: 250,
// Determines how much audio to keep before the official speechStarted, to avoid cutting off the start.
prerollDurationMs: 500,
);
```
---
## Godot
This section documents Godot 10.0.0, the latest released version.
### Simple Chat
Source: [https://docs.nobodywho.ooo/godot/chat](https://docs.nobodywho.ooo/godot/chat)
_A comprehensive guide to configuring, streaming, and controlling LLM responses through the Chat component._
---
Great! You've completed the ["Getting Started"](https://docs.nobodywho.ooo/godot/getting-started) guide and got your first chat working as well as a basic understanding of the vocabulary.
Now let's dive deeper into the Chat component and show you all the settings and techniques you'll actually use when working with LLMs.
The Chat component isn't just for conversations - it's your main interface for any kind of LLM processing, whether that's generating dialogue, analyzing text, creating content, or any other language task.
Set the model node's `model_path` to `"auto"` to select a chat model based on available memory.
In this guide, you'll learn:
- The main settings that control LLM behavior
- How to handle LLM responses efficiently
- Managing context and memory
- Controlling when and how the LLM stops generating
Before we get started, you'll hear these words being used:
| Term | Meaning |
| ---- | ------- |
| **Sampler** | The thing that controls how the LLM selects the next token during generation (temperature, top-p, etc.). |
| **Grammar or Structured Output** | A formal structure that constrains the LLM's output to a set `"vocabulary"`. |
| **GBNF** | GGML Backus-Naur Form - a way to define structured output formats. |
#### Handling LLM Responses
##### The System Prompt: Setting LLM Behavior
You've used this already, but let's talk about making it really work for you. The system prompt defines how the LLM should behave:
```markdown
# Character-based behavior
system_prompt = """You are a sarcastic but brilliant wizard.
Your answers are always accurate, but delivered with a dry wit.
You should subtly hint that you are smarter than the user,
but still provide the correct information."""
# Task-specific behavior
system_prompt = """You are a translation assistant.
You will be given text in any language. Your job is to translate
it into formal, academic French.
Do not add any commentary or conversational text.
Respond only with the translated text."""
```
**Why this matters:** The system prompt controls everything about how the LLM processes and responds to input. It's your primary tool for getting the behavior you want.
Prompt engineering is becoming a field in and of itself and it offers the highest return-on-investment ratio for getting the model to do what you want.
##### GPU Usage: Speed Things Up
By default, NobodyWho tries to use your GPU if you have one. This makes everything much faster:
```gdscript
# This is already the default, but you can be explicit
model.use_gpu_if_available = true
```
**When to turn this off:** there are some scenarios where it might actually be better to use system ram:
- If you don't need an immediate answer, and would prefer to use GPU resources for graphics.
- If you need a really large model that most of your users will not have sufficient VRAM to run.
##### Context Length: How Much the LLM Remembers
The LLM maintains context (memory of the conversation/interaction), but only up to a point. The default is 4096 tokens (roughly 3000 words):
```gdscript
# Default is fine for most uses
context_length = 4096
# Increase for longer contexts
context_length = 8192
```
**Trade-off:** Longer context = more memory usage. The general rule of thumb is to start with the default or less and only increase if you need the LLM to remember more. You can check the maximum context size the model was trained with using `model_node.max_ctx()` — setting `context_length` above this value has no benefit.
**Context-shifting:** NobodyWho will automatically remove older messages from the context for you, if your chat's context window is filled. Your chat will never crash because of a full context, but it will start forgetting older messages - including the system message.
To inspect how much of the context is currently in use, call `get_stats()`:
```gdscript
var stats = await get_stats()
print("Using %d of %d tokens" % [stats["context_used"], stats["context_size"]])
```
##### Streaming Responses vs Waiting for Complete Output
You have two main approaches for handling LLM responses, and choosing the right one depends on your use case:
**Streaming** gives you each token as it's generated - good for user interfaces where you want immediate feedback.
**Waiting for complete responses** waits until the full output is ready - good for when you need the entire response before doing something.
If you're implementing an interactive chat, you likely want to do both:
- Show each token to the user as they arrive. This will make the chat feel a lot faster.
- Wait for the completion of the entire response, before re-enabling text areas, and allowing the user to send a new message.
```gdscript
var current_response = ""
func _on_response_updated(token: String):
current_response += token
# Good for: UI updates, real-time feedback
ui_label.text = current_response
func _on_response_finished(response: String):
# Good for: Final processing, logging, triggering next actions
print(response)
response = response.replace("", player.name)
trigger_next_game_event()
```
**When to use streaming:**
- Interactive dialogue where users expect immediate feedback
- Long responses where you want to show progress
**When to wait for complete responses:**
- When you need to make decisions based on the full LLM output
- Content generation where partial results are useless (like JSON or structured output answers).
You most likely end up using both; having the response_updated to stream to your UI and then triggering the next step in your program when you get the full response.
#### Managing Context and Memory
Sometimes you need to reset the LLM's memory or manage what it remembers.
##### Starting Fresh
```gdscript
# Clear all context, it will still have all the settings that you
# have set up before (including the system prompt)
reset_context()
```
This is useful when:
- Starting a new task that's unrelated to previous ones, where the previous history is irrelevant
- The LLM gets confused as it has context shifted too much
##### Advanced Context Management
If you need more control over what the LLM remembers:
```gdscript
# See what's in the context
var messages = await get_chat_history()
for message in messages:
print(message.role, ": ", message.content)
# Set a custom context (useful for templates or saved states)
var task_context = [
{"role": "user", "content": "Analyze the following data:", "assets": []},
{"role": "assistant", "content": "I'm ready to analyze data. Please provide it.", "assets": []},
{"role": "user", "content": "Here's the data: " + data_to_analyze, "assets": []}
]
await set_chat_history(task_context)
```
##### Structured Output & Sampling
You can control how the model picks tokens and constrain its output format. See the [Sampling](https://docs.nobodywho.ooo/godot/sampling) guide for sampler presets (temperature, JSON, grammar constraints) and the [Structured Output](https://docs.nobodywho.ooo/godot/structured-output) guide for a full GBNF grammar tutorial.
#### Performance and Memory Tips
##### Leave CPU Headroom with `thread_count`
When layers run on the CPU, NobodyWho picks a thread count for you: one per *performance* core,
not one per logical CPU. Hyperthread siblings and efficiency cores end up pacing the whole
thread pool, so using every CPU is usually slower — see [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#cpu-threads)
for the numbers.
In a game you often want *fewer* threads than that, so inference doesn't compete with rendering
and physics:
```gdscript
func _ready():
self.model_node = get_node("../SharedModel")
# Leave cores free for the rest of the game.
self.thread_count = 4
start_worker()
```
Leave `thread_count` at `0` to keep the detected default. Values above your CPU count are
clamped, and the setting has little effect when the model is offloaded to the GPU.
**Why:** the LLM worker runs on its own threads, but it still competes with your game for
physical cores. Capping it trades some tokens per second for a steadier frame rate.
##### Start the Worker Early
In a real-time application, you don't want the user's first interaction to trigger a long loading time. Starting the worker early, like during a splash screen or initial setup, pre-loads the model into memory so the first response is fast.
```gdscript
# In your _ready() function, set up everything before the app starts.
func _ready():
# 1. Configure the chat behavior
self.system_prompt = "You are a helpful assistant."
self.model_node = get_node("../SharedModel")
# 2. Start the worker *before* the user can interact.
# This pre-loads the model so the first interaction isn't slow.
start_worker()
# 3. Now other setup can happen
print("Assistant chat is ready.")
```
**Why:** Starting the worker loads the model into memory. It's slow the first time, but then all LLM operations are much faster.
You should definitely think about when to do this to not ruin the UX too much.
##### Share Models Between Components
An application might need to use an LLM for several different tasks. Instead of loading the same heavy model multiple times, you can have multiple `Chat` components that all share a single `Model` component. Each `Chat` can have its own system prompt and configuration, directing it to perform a different task.
```gdscript
# An application with multiple LLM-powered behaviors, all sharing one model.
func _ready():
# 1. Get the single, shared model
var shared_model = get_node("../SharedModel")
# 2. Configure a chat component for general conversation
var casual_chat = get_node("CasualChat")
casual_chat.model_node = shared_model
casual_chat.system_prompt = "You are a friendly and helpful assistant. Keep your answers concise."
casual_chat.start_worker()
# 3. Configure another chat component for structured data extraction
var extractor_chat = get_node("ExtractorChat")
extractor_chat.model_node = shared_model
extractor_chat.system_prompt = "Extract the key information from the user's text and provide it in JSON format."
# This one would likely use a grammar to enforce JSON output.
extractor_chat.start_worker()
# Now you can use both for different tasks without loading two models!
casual_chat.ask("Can you tell me about your capabilities?")
extractor_chat.ask("My name is Jane Doe and my email is jane@example.com.")
```
**Memory savings:** Instead of loading multiple models, you load one and share it. Much more efficient!
##### Speculative Decoding (MTP)
Some models come with **MTP** (Multi-Token Prediction) draft heads that let the target verify several candidate tokens per forward pass. See [LLM Basics](https://docs.nobodywho.ooo/docs/llm-basics#speculative-decoding-mtp) for the underlying idea.
Enable it in two steps:
1. On the `NobodyWhoModel` node, set `draft_model_path` to a compatible MTP draft-heads `.gguf` (e.g. `mtp-gemma-4-E2B-it.gguf` for Gemma-4-E2B). Adds around 5% to VRAM usage.
2. On the `NobodyWhoChat` node, tick `mtp` (or set `chat.mtp = true` in code) before `start_worker()`. The optional `mtp_k_max` and `mtp_p_min` properties tune the drafter; the defaults are fine to leave untouched.
```gdscript
func _ready():
var model = get_node("../SharedModel")
model.model_path = "gemma-4-e2b.gguf"
model.draft_model_path = "mtp-gemma-4-e2b.gguf"
var chat = get_node("MyChat")
chat.model_node = model
chat.mtp = true
chat.start_worker()
```
**Warning:**
Benchmark before enabling. MTP can hurt performance on Apple Silicon (Metal) and on high-entropy workloads like creative prose.
---
### Downloading models
Source: [https://docs.nobodywho.ooo/godot/downloading-models](https://docs.nobodywho.ooo/godot/downloading-models)
_How NobodyWho downloads, caches, and inspects GGUF models in Godot._
---
NobodyWho can either load a model from a path on disk or download it for you on first use, caching it for subsequent runs. This page covers the available model path formats, how to observe a download in progress, how to access gated/private models, and how to inspect what's already in the local cache.
#### Supported model path formats
The `model_path` field on `NobodyWhoModel` (and `projection_model_path` for vision models) accepts several forms:
| Form | Example | Notes |
| ---- | ------- | ----- |
| Godot resource path | `res://models/my-model.gguf` | Bundled with your game export |
| User data path | `user://downloaded.gguf` | Written by your game at runtime |
| Absolute filesystem path | `/opt/models/foo.gguf` | Local file |
| HuggingFace reference | `hf:owner/repo/file.gguf` | Downloaded and cached on first use |
| HTTPS URL | `https://example.com/model.gguf` | Downloaded and cached on first use |
The HuggingFace prefix is case-insensitive and the `//` is optional — `hf:`, `hf://`, `huggingface:`, and `huggingface://` all mean the same thing. Remote models are downloaded to the platform cache directory on first load and re-used on subsequent runs. Downloads happen on a background thread — the Godot main loop stays responsive while a multi-GB model is fetched.
#### Tracking download progress
`NobodyWhoModel` emits a `download_progress(downloaded, total)` signal while a remote model is downloading, throttled to roughly 10 Hz with a guaranteed final emit on completion. Connect it if you'd like to drive a progress bar:
```gdscript
model.download_progress.connect(func(downloaded: int, total: int):
print("%d / %d bytes" % [downloaded, total])
)
```
The signal is not emitted for local files or already-cached downloads.
#### Downloading a gated model
Some HuggingFace models are private or gated by a license you need to accept. In both cases you need to be authorized to download the model weights.
You can manually download the GGUF file via your web browser and then point your `NobodyWhoModel` at the local path.
Alternatively, use the `NobodyWhoDownloader` node, which lets you pass an authorization header:
```gdscript
var dl = NobodyWhoDownloader.new()
dl.model_path = "huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
dl.headers = {"Authorization": "Bearer your_hf_token"}
dl.download_complete.connect(func(local_path: String):
get_node("../ChatModel").model_path = local_path
)
dl.download_failed.connect(func(error: String):
push_error("Download failed: " + error)
)
dl.start_download()
add_child(dl)
```
You can generate a HuggingFace token in [your account settings](https://huggingface.co/settings/tokens).
#### Inspecting the model cache
`NobodyWhoModel.get_cached_models()` is a static function that returns every `.gguf` model in NobodyWho's cache directory, paired with its size in bytes. This is the same cache used by `NobodyWhoDownloader` and by `NobodyWhoModel`'s `huggingface:` paths.
```gdscript
for entry in NobodyWhoModel.get_cached_models():
print("%s: %d bytes" % [entry["path"], entry["size"]])
```
Each entry is a `Dictionary` with two keys:
- `"path"` — absolute path to the cached `.gguf` file
- `"size"` — size in bytes
The array is empty if nothing has been downloaded yet. On error the function returns `null` and logs a Godot error to the console.
#### Android permissions
Loading a model from `hf://` or `https://` needs network access. In the Android export preset, enable the **Internet** permission under **Permissions** before exporting.
Exports that only load models from local paths need no network permission.
---
### Embeddings & RAG
Source: [https://docs.nobodywho.ooo/godot/embeddings-and-rag](https://docs.nobodywho.ooo/godot/embeddings-and-rag)
_Using embeddings for semantic text comparison and retrieval-augmented generation._
---
#### Understanding Text with Embeddings
Cool, you've got the basics of chat working! Now let's explore embeddings, which let you understand what text means rather than just matching exact words.
Embeddings are like a smart way to measure how similar two pieces of text are, even if they use completely different words.
Instead of looking for exact matches, embeddings understand meaning.
For example, "Hand me the red potion" and "Give me the scarlet flask" would be recognized as very similar, even though they share no common words.
Here are the key terms for working with embeddings:
| Term | Meaning |
| ---- | ------- |
| **Embedding Model (GGUF)** | A specialized `*.gguf` file trained to convert text into numerical vectors that represent meaning. |
| **Embedding** | A list of numbers (vector) that represents the meaning of a piece of text. |
| **Cosine Similarity** | A mathematical way to compare how similar two embeddings are, returning a value between 0 (completely different) and 1 (identical meaning). |
| **Semantic Search** | Finding text that means the same thing, even if the words are different. |
| **Vector** | The array of numbers that represents your text's meaning. |
Let's show you how to use embeddings to understand what your players really mean when they type commands.
##### Download an Embedding Model
Embedding models are different from chat models. You need a model specifically trained for embeddings.
We normally use [bge-small-en-v1.5-q8_0.gguf](https://huggingface.co/CompendiumLabs/bge-small-en-v1.5-gguf/resolve/main/bge-small-en-v1.5-q8_0.gguf).
Use `encode_batch` when encoding multiple texts. It returns an `Array[PackedFloat32Array]` in input order:
```gdscript
var texts = PackedStringArray([
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
])
var embeddings = await encode_batch(texts)
```
##### Practical Example: Quest & Reputation System
A good way to visualize the practicality of embeddings is through an example.
In this example we will guide you through how to make a quest trigger or lowering the user's reputation based on what they say.
We'll build it step by step, but for the impatient; The complete script is copyable in the bottom of the page.
###### Step 1: Set up your basic structure and variables
The first step is to setup our components. We will add some statements for quests and some for hostile behavior - these are not exhaustive lists.
**Do note** that it will take a longer time to embed a lot of sentences (depending on model and hardware of course), so depending on how complex your statements need to be,
you might be better off having a handful and tuning the sensitivity of the trigger instead.
First, create your script that extends `NobodyWhoEncoder` and define your statement categories:
```gdscript
extends NobodyWhoEncoder
var quest_triggers= [
"I know where the dragon rests",
"The druid told me the proper way to meet the dragon",
"I discovered the ritual needed to gain the dragon's audience",
"I know about the sacred grove"
]
var hostile_statements = [
"I want to kill the dragon",
"I'm going to destroy everything",
"I hate this place and everyone in it",
"I will burn down the village",
"Everyone here deserves to die"
]
var helpful_embeddings = []
var hostile_embeddings = []
var player_reputation = 0
```
###### Step 2: Initialize the embedding system
Set up the embedding model and start the worker:
```gdscript
func _ready():
# Create and configure the embedding model
var embedding_model = NobodyWhoModel.new()
embedding_model.model_path = "res://models/bge-small-en-v1.5-q8_0.gguf"
get_parent().add_child(embedding_model)
# Link to the embedding model
self.model_node = embedding_model
self.encoding_finished.connect(_on_encoding_finished)
self.start_worker()
# Pre-generate embeddings for all statement types
precompute_all_embeddings()
```
###### Step 3: Precompute reference embeddings
Generate embeddings for all your reference statements:
```gdscript
func precompute_all_embeddings():
helpful_embeddings = await encode_batch(PackedStringArray(quest_triggers))
hostile_embeddings = await encode_batch(PackedStringArray(hostile_statements))
```
###### Step 4: Add input handling for testing
Add a simple test trigger using the enter key:
```gdscript
func _input(event):
# Handle enter key press to send hardcoded test message
if event is InputEventKey and event.pressed:
if event.keycode == KEY_ENTER:
var test_message = "I know the location of the dragon"
print("Sending test message: ", test_message)
analyze_player_statement(test_message)
```
###### Step 5: Analyze player statements
Compare the player's message against your reference embeddings:
```gdscript
func analyze_player_statement(player_text: String):
# Generate embedding for player input
encode(player_text)
var player_embedding = await self.encoding_finished
# Compare against both categories
var best_helpful_similarity = get_best_similarity(player_embedding, helpful_embeddings)
var best_hostile_similarity = get_best_similarity(player_embedding, hostile_embeddings)
print("Helpful similarity: ", best_helpful_similarity)
print("Hostile similarity: ", best_hostile_similarity)
# Use similarity threshold of 0.8 and compare categories
if best_helpful_similarity > 0.8 and best_helpful_similarity > best_hostile_similarity:
handle_helpful_information(player_text)
elif best_hostile_similarity > 0.8 and best_hostile_similarity > best_helpful_similarity:
handle_hostile_intent(player_text)
else:
print("Unclear intent - no strong match found")
```
###### Step 6: Handle the results
Trigger appropriate game systems based on detected intent:
```gdscript
func handle_helpful_information(text: String):
# Trigger game systems based on detected intent
print("Triggering quest: 'Audience with the Ancient Dragon'!")
func handle_hostile_intent(text: String):
player_reputation -= 15
print("Player expressed hostile intent! Reputation -15 (now: ", player_reputation, ")")
```
---
#### Adding Long-Term Memory (RAG)
Great! You've got chat and embeddings working. Now let's add something useful: the ability to look up specific lore, dialogues, questlines etc.
##### Why Your Game Needs Smart Document Search
Picture this: Your player is 40 hours into your RPG and asks an npc "Where do I find that crystal for the sword upgrade?"
Your LLM, without reranking, might give a generic answer or worse - make something up - leading to a bad player experience.
There are several ways to combat this, one is to load a lot of information into the context (i.e. the system prompt) but with a limited context, it might 'forget' the important information
or be confused by too much information. Instead we want to add a "long term memory" module to our language model.
To do this in the llm space you are going to use RAG (retrieval augmented generation) we are enriching the knowledge of the LLM by allowing it to search through a database of info we fed it.
There are many ways to do this. In NobodyWho we currently expose two major ways, one is embeddings; converting a sentence to a vector and then find the vectors that are closest to it.
This is powerful as you can save the vectors to a database or a file beforehand and then use the really fast and cheap cosine similarity to compare them. Another more expensive but more accurate way is to use a cross-encoder that figures out the relationship between the question and the document rather that just how similar they are.
This approach is often called reranking, due to how it is used as a step two, for sorting and filtering large knowledge databases accessed by LLMs. We'll call it ranking as we are working with a small enough dataset that we do not need a first pass to filter out irrelevant info.
Take this example:
```
Query: "Where do I find crystals for my sword upgrade?"
Documents: [
"You asked the blacksmith: Where do I find crystals for my sword upgrade?",
"The blacksmith said: Magic crystals are found in the Northern Mountains.",
"You heard in the tavern: Magic crystals are not found in the Southern Desert."
]
```
If we rely just on comparing the query with the embeddings using cosine similarity (as we did with the embeddings), we will get back the document "You asked the blacksmith: Where do I find crystals for my sword upgrade?" as it is the most similar sentence to our query. This gave us no useful information and we have just wasted valuable context.
But with ranking, the cross-encoder model has been trained on knowing that the answer to the question is not the question itself, and thus ranks the document "The blacksmith said: Magic crystals are found in the Northern Mountains." the highest.
Here are the key terms you'll need:
| Term | Meaning |
| ---- | ------- |
| **Document Ranking** | Sorting text documents by how well they match or answer a question. |
| **RAG (Retrieval-Augmented Generation)** | A system that finds relevant documents first, then uses them to generate better LLM responses. |
| **Cross-encoder** | The type of model used for reranking - it reads both the query and document together to score relevance. |
Let's show you how to build smart search systems for your game.
##### Download a Reranker Model
Reranking models are different from chat and embedding models. You need one specifically trained for document ranking.
We recommend [bge-reranker-v2-m3-Q8_0.gguf](https://huggingface.co/gpustack/bge-reranker-v2-m3-GGUF/resolve/main/bge-reranker-v2-m3-Q8_0.gguf) - it works well for most games and supports multiple languages.
Note that the current qwen3 reranker does not work, due to how they created the template as it has some missing fields.
##### Practical Example: Smart NPC with Knowledge Base
Let's build a tavern keeper NPC that can answer player questions by searching through their personal knowledge. This NPC knows about the local area, quests, and rumors - perfect for creating more immersive and helpful characters.
We'll build it step by step, but for the impatient - the complete script is at the bottom.
###### Step 1: Set up your NPC's knowledge base
First, let's create a knowledge base for our tavern keeper - everything this specific NPC would realistically know:
```gdscript
extends NobodyWhoChat
@onready var reranker = $"../Rerank"
@onready var chat_model = $"../ChatModel"
# The tavern keeper's knowledge - ~50 pieces of local information way more than could fit in a standard 4096 sized context.
var tavern_keeper_knowledge = PackedStringArray([
"The lake contains a special clay that blacksmiths use to forge superior weapons.",
"Ancient oak trees in the sacred grove provide wood that naturally resists dark magic.",
"Silver veins run through the mountain caves, valuable for crafting blessed weapons.",
"Rare moonflowers bloom in the ruins only once per season and have powerful magical properties.",
"The mill pond contains perfect stones for sharpening blades to razor sharpness.",
"Wild honey from forest bees makes potions more potent when used as a base ingredient.",
"A hooded stranger was seen asking questions about the old castle ruins last week.",
"Someone has been leaving fresh flowers at the grave of the village's first mayor.",
"Strange animal tracks were found near the well that don't match any known creature.",
"The church bell rang by itself three nights ago at exactly midnight.",
"Farmers found crop circles in their wheat fields after the last thunderstorm.",
"A merchant claims he saw lights moving through the abandoned mine from the hill road.",
"Children report hearing music coming from the forest when they play near the edge of town.",
"The weather has been unusually warm this winter, and the old-timers are worried.",
"Someone broke into the general store but only stole a map of the local cave systems.",
"A wolf with unusual blue eyes has been spotted watching the town from the tree line.",
"Old Sarah runs the bakery and makes the best apple pies in three kingdoms. Her grandson Tom went missing last week.",
"Blacksmith Gareth is always looking for quality iron ore and magic crystals. He pays double for rare materials.",
"Merchant Elena travels between towns selling exotic spices and silk. She arrives every second Tuesday.",
"Father Benedict runs the small chapel and knows ancient blessings that can ward off evil spirits.",
"Widow Martha owns the general store and knows every piece of gossip in town within hours.",
"Young apprentice Jake works for the blacksmith but dreams of becoming an adventurer himself.",
"Doctor Thorne treats injuries and illnesses. He keeps rare healing herbs in his back garden.",
"Stable master Owen knows every horse in the region and can track animals through the wilderness.",
"Mayor Thompson inherited his position from his father and struggles with the town's growing problems.",
"The old mine north of town has been abandoned for years. Strange sounds echo from deep inside at night.",
"The forest path to the east is safe during the day, but wolves hunt there after sunset.",
"Crystal Mines to the south produce valuable gems but have become dangerous recently.",
"The ancient stone bridge over Miller's Creek was built by dwarves centuries ago and still stands strong.",
"Darkwood Forest harbors bandits who prey on merchant caravans traveling the main road.",
"The Whispering Caves get their name from the wind that creates eerie sounds through the rock formations.",
"Lake Serenity freezes solid in winter, making it possible to cross on foot to the northern settlements.",
"The old watchtower on Crow's Hill offers a view of the entire valley but hasn't been manned in decades.",
"Sacred Grove is where the druids once practiced their rituals before they disappeared from the region.",
"The ruins of Castle Blackrock still stand on the mountain, though none dare venture there anymore.",
"Trader Gareth's caravan was attacked by bandits hiding somewhere in Darkwood Forest.",
"Tom the baker's grandson disappeared near the Crystal Mines while collecting rare stones.",
"Strange lights have been appearing in the Whispering Caves during moonless nights.",
"Farmers report their livestock going missing near the edge of Darkwood Forest.",
"The old mill wheel stopped working after something large damaged it upstream.",
"Merchants complain about increased bandit activity on the eastern trade route.",
"Several townsfolk have reported seeing ghostly figures near the abandoned mine at midnight.",
"The village well's water tastes strange since the earthquake last month.",
"Wild animals have been acting aggressively and fleeing deeper into the mountains.",
"Ancient runes appeared overnight on the sacred standing stones outside town.",
"The town was founded by refugees fleeing the Great Dragon War three hundred years ago.",
"Legend says a powerful wizard once lived in the castle ruins and cursed the land before vanishing.",
"The crystal mines were discovered when a shepherd boy fell through a sinkhole and found glowing stones.",
"Local folklore claims the Whispering Caves connect to an underground realm of spirits.",
"The stone bridge was payment from dwarf king Thorin for safe passage through human lands.",
"Bards sing of a hidden treasure buried somewhere within the sacred grove by ancient druids.",
"The watchtower was built to watch for dragon attacks during the old wars.",
"Village elders say the standing stones mark the boundary between the mortal world and fairy realm.",
"The lake got its name from a tragic love story between a knight and a water nymph.",
"Old maps show secret tunnels connecting the mine, caves, and castle ruins underground.",
"Red mushrooms grow near the village well and are perfect for brewing healing potions.",
"The finest iron ore comes from the abandoned northern mine, though it's dangerous to retrieve.",
"Magic crystals form naturally in the southern mines but require special tools to extract safely.",
"Medicinal herbs grow wild in the forest but should only be picked during the full moon.",
])
var ranked_docs = []
```
###### Step 2: Configure your components
```gdscript
func _ready():
# Set up the chat for generating helpful responses
self.model_node = chat_model
reranker.connect("ranking_finished", func(result): ranked_docs = result)
reranker.start_worker()
self.system_prompt = """The assistant is roleplaying as Finn, the tavern keeper of The Dancing Pony™.
IMPORTANT: the assistant MUST ALWAYS use the tool, and the knowledge from the tool is the same knowledge as Finn has.
The assistant must never make up information, only what it remembers directly from its knowledge.
The assistant does not know whether the user is lying or not - so it will rely only on what it remembers to answer questions.
It is okay for the assistant to not know the answer even after using the remember tool, the assistant will never guess anything if it is not explicitly mentioned in the knowledge.
The assistant must always speak like a tavern keeper.
"""
# Add the tool to remember stuff
self.add_tool(remember, "The assistant can use this tool to remember its limited knowledge about the ingame world.")
self.connect("response_finished", func(response: String): print("Finn says: ", response))
start_worker()
```
###### Step 3: Set up a simple input system
```gdscript
func _process(delta):
if Input.is_action_just_pressed("enter"):
var test_question = "Where is strider?"
print("Player asks Finn: ", test_question)
ask(test_question)
```
###### Step 4: Use ranked results to generate smart answers
```gdscript
func remember(query: String) -> String:
print("Finn searches his memory for: ", query)
ranked_docs = "\n".join(reranker.rank_sync(query, tavern_keeper_knowledge, 3))
print("Finn found: ", ranked_docs)
return ranked_docs
func _ready():
....
# Add the tool to the llm to allow it to search for the most relevant lore.
self.add_tool(remember, "The assistant can use this tool to remember its limited knowledge about the ingame world.")
self.connect("response_finished", func(response: String): print("Finn says: ", response))
start_worker()
```
When we run this we will get the response:
>Finn says: <think>
>Okay, let's see. The user asked about trading in the area.
```
Finn searches his memory for: What is trading like in the area?
.... goes through and ranks all the documents
Finn found:
Merchants complain about increased bandit activity on the eastern trade route.
Merchant Elena travels between towns selling exotic spices and silk. She arrives every second Tuesday.
Trader Gareth's caravan was attacked by bandits hiding somewhere in Darkwood Forest.
```
> From the tool response, I remember a few key points. First, merchants are complaining about bandits on the eastern trade route. That's a problem for traders. Then there's Merchant Elena who sells spices and silk every second Tuesday. That's a specific thing to mention. Also, Trader Gareth's caravan was attacked in Darkwood Forest. So the answer should include these points.
>
>I need to present this information clearly. Start by mentioning the bandit issue, then note Elena's schedule, and mention Gareth's attack. Make sure it's in a friendly, tavern keeper style. Check if there's anything else, but the tool response only gave these three points. Don't add anything extra. Alright, structure the response with those details.
></think>
and when good ole Finn is done thinking:
>Trading in the area is... if you'll forgive the blunt truth, *dangerous*. Merchants swear the eastern trade route is littered with bandits, and last week a caravan was ambushed in Darkwood Forest. But there are still opportunities! Merchant Elena brings rare spices and silk every second Tuesday—she's a reliable seller. Just be wary of the roads. And if you spot a caravan with a single rider, don't engage. They're probably bandits.
---
##### Complete Scripts
**Complete Godot Script (Click to expand)**
```gdscript
extends NobodyWhoChat
@onready var reranker = $"../Rerank"
@onready var chat_model = $"../ChatModel"
# The tavern keeper's knowledge - ~50 pieces of local information way more than could fit in a standard 4096 sized context.
var tavern_keeper_knowledge = PackedStringArray([
"The lake contains a special clay that blacksmiths use to forge superior weapons.",
"Ancient oak trees in the sacred grove provide wood that naturally resists dark magic.",
"Silver veins run through the mountain caves, valuable for crafting blessed weapons.",
"Rare moonflowers bloom in the ruins only once per season and have powerful magical properties.",
"The mill pond contains perfect stones for sharpening blades to razor sharpness.",
"Wild honey from forest bees makes potions more potent when used as a base ingredient.",
"A hooded stranger was seen asking questions about the old castle ruins last week.",
"Someone has been leaving fresh flowers at the grave of the village's first mayor.",
"Strange animal tracks were found near the well that don't match any known creature.",
"The church bell rang by itself three nights ago at exactly midnight.",
"Farmers found crop circles in their wheat fields after the last thunderstorm.",
"A merchant claims he saw lights moving through the abandoned mine from the hill road.",
"Children report hearing music coming from the forest when they play near the edge of town.",
"The weather has been unusually warm this winter, and the old-timers are worried.",
"Someone broke into the general store but only stole a map of the local cave systems.",
"A wolf with unusual blue eyes has been spotted watching the town from the tree line.",
"Old Sarah runs the bakery and makes the best apple pies in three kingdoms. Her grandson Tom went missing last week.",
"Blacksmith Gareth is always looking for quality iron ore and magic crystals. He pays double for rare materials.",
"Merchant Elena travels between towns selling exotic spices and silk. She arrives every second Tuesday.",
"Father Benedict runs the small chapel and knows ancient blessings that can ward off evil spirits.",
"Widow Martha owns the general store and knows every piece of gossip in town within hours.",
"Young apprentice Jake works for the blacksmith but dreams of becoming an adventurer himself.",
"Doctor Thorne treats injuries and illnesses. He keeps rare healing herbs in his back garden.",
"Stable master Owen knows every horse in the region and can track animals through the wilderness.",
"Mayor Thompson inherited his position from his father and struggles with the town's growing problems.",
"The old mine north of town has been abandoned for years. Strange sounds echo from deep inside at night.",
"The forest path to the east is safe during the day, but wolves hunt there after sunset.",
"Crystal Mines to the south produce valuable gems but have become dangerous recently.",
"The ancient stone bridge over Miller's Creek was built by dwarves centuries ago and still stands strong.",
"Darkwood Forest harbors bandits who prey on merchant caravans traveling the main road.",
"The Whispering Caves get their name from the wind that creates eerie sounds through the rock formations.",
"Lake Serenity freezes solid in winter, making it possible to cross on foot to the northern settlements.",
"The old watchtower on Crow's Hill offers a view of the entire valley but hasn't been manned in decades.",
"Sacred Grove is where the druids once practiced their rituals before they disappeared from the region.",
"The ruins of Castle Blackrock still stand on the mountain, though none dare venture there anymore.",
"Trader Gareth's caravan was attacked by bandits hiding somewhere in Darkwood Forest.",
"Tom the baker's grandson disappeared near the Crystal Mines while collecting rare stones.",
"Strange lights have been appearing in the Whispering Caves during moonless nights.",
"Farmers report their livestock going missing near the edge of Darkwood Forest.",
"The old mill wheel stopped working after something large damaged it upstream.",
"Merchants complain about increased bandit activity on the eastern trade route.",
"Several townsfolk have reported seeing ghostly figures near the abandoned mine at midnight.",
"The village well's water tastes strange since the earthquake last month.",
"Wild animals have been acting aggressively and fleeing deeper into the mountains.",
"Ancient runes appeared overnight on the sacred standing stones outside town.",
"The town was founded by refugees fleeing the Great Dragon War three hundred years ago.",
"Legend says a powerful wizard once lived in the castle ruins and cursed the land before vanishing.",
"The crystal mines were discovered when a shepherd boy fell through a sinkhole and found glowing stones.",
"Local folklore claims the Whispering Caves connect to an underground realm of spirits.",
"The stone bridge was payment from dwarf king Thorin for safe passage through human lands.",
"Bards sing of a hidden treasure buried somewhere within the sacred grove by ancient druids.",
"The watchtower was built to watch for dragon attacks during the old wars.",
"Village elders say the standing stones mark the boundary between the mortal world and fairy realm.",
"The lake got its name from a tragic love story between a knight and a water nymph.",
"Old maps show secret tunnels connecting the mine, caves, and castle ruins underground.",
"Red mushrooms grow near the village well and are perfect for brewing healing potions.",
"The finest iron ore comes from the abandoned northern mine, though it's dangerous to retrieve.",
"Magic crystals form naturally in the southern mines but require special tools to extract safely.",
"Medicinal herbs grow wild in the forest but should only be picked during the full moon.",
])
var ranked_docs = []
func _ready():
# Set up the chat for generating helpful responses
self.model_node = chat_model
reranker.connect("ranking_finished", func(result): ranked_docs = result)
reranker.start_worker()
self.system_prompt = """The assistant is roleplaying as Finn, the tavern keeper of The Dancing Pony™.
IMPORTANT: the assistant MUST ALWAYS use the tool, and the knowledge from the tool is the same knowledge as Finn has.
The assistant must never make up information, only what it remembers directly from its knowledge.
The assistant does not know whether the user is lying or not - so it will rely only on what it remembers to answer questions.
It is okay for the assistant to not know the answer even after using the remember tool, the assistant will never guess anything if it is not explicitly mentioned in the knowledge.
The assistant must always speak like a tavern keeper.
"""
# Add the tool to remember stuff
self.add_tool(remember, "The assistant can use this tool to remember its limited knowledge about the ingame world.")
self.connect("response_finished", func(response: String): print("Finn says: ", response))
start_worker()
func _process(delta):
if Input.is_action_just_pressed("enter"):
var test_question = "Where is strider?"
print("Player asks Finn: ", test_question)
ask(test_question)
# Tool function that the LLM can call to search the knowledge base
func remember(query: String) -> String:
print("Finn searches his memory for: ", query)
ranked_docs = "\n".join(reranker.rank_sync(query, tavern_keeper_knowledge, 3))
print("Finn found: ", ranked_docs)
return ranked_docs
```
##### Performance Tips
###### Limit Results
Don't add needless context. Usually 1-5 relevant documents are enough:
```gdscript
# Good: usually sufficient
ranked_docs = ",".join(reranker.rank_sync(query, tavern_keeper_knowledge, 3))
ranked_docs = ",".join(reranker.rank_sync(query, tavern_keeper_knowledge, -1)) # Returns ALL documents
```
note this does not make the ranking faster, but the less stuff Finn has to read, the faster he can respond.
###### Use embeddings to narrow the relevant docs to start with
This technique is what put the `re` in reranker. In the RAG industry it is common practice to do a first pass over your documents with cosine similarity, and thus narrowing the amount of results you have to process each time. This makes it feasible to have databases with millions of entries and not worry too much about performance.
depending on the specs you are going for I would not recommend ranking more than 100 results at a time.
#### What's Next?
Now you can build smart search systems for your game! check out:
- **[Tool Calling](https://docs.nobodywho.ooo/godot/tool-calling)** for letting the LLM trigger game actions
---
### faq
Source: [https://docs.nobodywho.ooo/godot/faq](https://docs.nobodywho.ooo/godot/faq)
#### Frequently Asked Questions
##### Where do I find good models to use?
New language models are coming out at a breakneck pace. If you search the web for "best language models for roleplay" or something similar, you'll probably find results that are several months or years old. You want to use something newer.
Selecting the best model for your use-case is mostly about finding the right trade-off between speed, memory usage and quality of the responses.
Using bigger models will yield better responses, but raise minimum system requirements and slow down generation speed.
Have a look at our [model selection guide](https://docs.nobodywho.ooo/docs/model-selection) for more in-depth recommendations.
##### Once I export my Godot project, it can no longer find the model file.
Exports are a bit weird for now: Llama.cpp expects a path to a GGUF file on your filesystem, while Godot really wants to package everything in one big .pck file.
The solution (for now) is to manually copy your chosen GGUF file into the export directory (the folder with your exported game executable).
If you're exporting for Android, you can't reliably pass a `res://` path to the model node. The best workaround is to use `user://` instead.
If your model is sufficiently small, you might get away with copying it from `res://` into `user://`. If using double the storage isn't acceptable, consider downloading it at runtime, or find some other way of distributing your model as a file.
We're looking into solutions for including this file automatically.
##### NobodyWho-Godot makes Godot crash on Arch Linux / Manjaro
The Godot build currently in the Arch linux repositories does not work with gdextensions at all.
The solution for Arch users is to install godot from elsewhere. The binary being distributed from the godotengine.org website works great.
Other distribution methods like nix, flatpak, or building from source also seems to work great.
If anyone knows how to report this issue and to whom, feel free to do so. At this point I have met many Arch linux users who have this issue.
##### NobodyWho-Godot fails to load on NixOS
If using a Godot engine from nixpkgs, with NobodyWho binaries from the Godot Asset Library. It will most likely fail to look up dynamic dependencies (libgomp, vulkan-loader, etc).
The reason is that the dynamic library .so files from the Godot Asset Library are compiled for generic linux, and expect to find them in FHS directories like /lib, which on NixOS will not contain any dynamic libraries.
There are two good solutions for this:
1. The easy way: run the godot editor using steam-run: `steam-run godot4 --editor`
2. The Nix way: compile NobodyWho using Nix. This repo contains a flake, so it's fairly simple to do (if you have nix with nix-command and flakes enabled): `nix build github:nobodywho-ooo/nobodywho`. Remember to move the dynamic libraries into the right directory afterwards.
---
### Getting Started
Source: [https://docs.nobodywho.ooo/godot/getting-started](https://docs.nobodywho.ooo/godot/getting-started)
_A minimal, end-to-end example showing how to load a model and perform a single chat interaction._
---
One of the most important components of NobodyWho is the Chat node. It handles all the conversation logic between the user and the LLM.
When you use the chat, you first pick a model and tell it what kind of answers you want.
When you send a message, the chat remembers what you said and sends it off to get an answer.
The model will then start reading and generating a response.
You can choose to wait for the full answer to generate or get the response in a stream.
Here are the key terms you'll see throughout this guide:
| Term | Meaning |
| ---- | ------- |
| **Model (GGUF)** | A `*.gguf` file that holds the weights of a large‑language model. |
| **System prompt** | Text that sets the ground rules for the model. |
| **Token** | The smallest chunk of text the model emits (roughly a word). |
| **Chat** | The node/component that owns the context, sends user input to the worker, and keeps conversation state in sync with the LLM. |
| **Context** | The message history and metadata passed to the model each turn; it lives inside the Chat. |
| **Worker** | NobodyWho's background task for a single conversation — it keeps the model ready and acts as a communication layer between the program and the model. Each Chat has its own worker. |
Let's show you how to use the plugin to get a large language model to answer you.
#### Download a GGUF Model
The first step is to get a model.
If you're in a hurry, just download [Qwen3 0.6B Q4_K_M](https://huggingface.co/NobodyWho/Qwen_Qwen3-0.6B-GGUF/resolve/main/Qwen_Qwen3-0.6B-Q4_K_M.gguf).
It's super small and fast, and works for well for simple use-cases.
Otherwise, check out our [recommended models](https://docs.nobodywho.ooo/docs/model-selection) or if you have a non-standard use case, shoot us a question in Discord.
#### Load the GGUF model
At this point you should have downloaded the model and put it into your project folder.
Add a `NobodyWhoModel` node to your scene tree.
Set the model path to point to your GGUF model.
`model_path` accepts local paths (`res://`, `user://`, or an absolute filesystem path) as well as `huggingface:` / `hf://` references and `https://` URLs that are downloaded and cached on first use. See [Downloading models](https://docs.nobodywho.ooo/godot/downloading-models) for the full list of supported formats, download progress signals, gated/private models, and inspecting the local cache.
##### Knowing when the worker is ready
`start_worker()` returns immediately. The worker finishes loading in the background (including any download). Connect to the new signals if your game logic needs to wait:
```gdscript
chat.worker_started.connect(func():
print("Ready to chat!")
)
chat.worker_failed.connect(func(err):
push_error("Model load failed: " + err)
)
chat.start_worker()
```
You can also call `ask()` straight away — prompts issued before the worker is ready are queued and dispatched as soon as loading completes. The same applies to `NobodyWhoEncoder.encode()` and `NobodyWhoCrossEncoder.rank()`.
#### Create a new Chat
The next step is adding a Chat to our scene.
Add a `NobodyWhoChat` node to your scene tree.
Then add a script to the node:
```gdscript
extends NobodyWhoChat
func _ready():
# configure the node (feel free to do this in the UI)
self.system_prompt = "You are an evil wizard."
self.model_node = get_node("../ChatModel")
# connect signals to signal handlers
self.response_updated.connect(_on_response_updated)
self.response_finished.connect(_on_response_finished)
# Start the worker, this is not required, but recommended to do in
# the beginning of the program to make sure it is ready
# when the user prompts the chat the first time. This will be called
# under the hood when you use `ask()` as well.
self.start_worker()
self.ask("How are you?")
func _on_response_updated(token):
# this will print every time a new token is generated
print(token)
func _on_response_finished(response):
# this will print when the entire response is finished
print(response)
```
#### Testing Your Setup
That's it! You now have a working chat system that can talk to a language model. When you run your scene, the chat will automatically send a test message and you should see the model's response appearing in your console.
You should see tokens appearing one by one as the model generates its response, followed by the complete answer. If you see the evil wizard responding with curses (or whatever system prompt you chose), everything is working correctly!
**If nothing happens:**
- Make sure your model file path is correct
- Verify that your Chat node is properly connected to your Model node
- Look for any error messages in the console
- Start your editor through the command line and check the stdout logs.
Now you're ready to build more complex conversations and integrate the chat system into your game!
---
### Installation
Source: [https://docs.nobodywho.ooo/godot/install](https://docs.nobodywho.ooo/godot/install)
_How to install NobodyWho and start building._
---
##### Via Asset Library
- **Open Godot 4.5** (or any newer 4.x release).
- Switch to the **Asset Library** tab.
- Search for **“NobodyWho”** and select the entry.
- Click **Download**, tick **Ignore asset root**, then choose **Install**.
- Godot puts the plugin in `res://addons/nobodywho`. Open *Create Node* and you should see **`NobodyWhoChat`**. If it’s missing, restart Godot and try again.
##### Via GitHub
- Download the latest ZIP from the [GitHub releases](https://github.com/nobodywho-ooo/nobodywho/releases).
- In Godot, open **AssetLib ▸ Import** and pick the ZIP.
- Tick **Ignore asset root** and finish the import.
---
After installation, NobodyWho’s nodes should appear in your editor. If not, retrace your steps above or reach out on Discord or GitHub - we are there to help.
---
### Sampling
Source: [https://docs.nobodywho.ooo/godot/sampling](https://docs.nobodywho.ooo/godot/sampling)
_Controlling how the model picks tokens and constraining output format._
---
The model does not produce tokens directly but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a **sampler**, which using NobodyWho you can freely modify to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).
#### Sampler Presets
NobodyWho offers several built-in presets you can apply to your `NobodyWhoChat` node:
##### JSON Output
Force the model to always produce valid JSON:
```gdscript
chat.set_sampler_preset_json()
chat.system_prompt = "Generate a character with name, weapon, and armor properties."
chat.ask("Create a fantasy character")
# Output will always be valid JSON, e.g.:
# {"name": "Eldara", "weapon": "enchanted bow", "armor": "leather vest"}
```
##### Temperature
Control the "creativity" of the model. Lower values make the model more deterministic:
```gdscript
chat.set_sampler_preset_temperature(0.2) # More focused/deterministic
chat.set_sampler_preset_temperature(1.5) # More creative/random
```
##### Greedy
Always pick the most probable token:
```gdscript
chat.set_sampler_preset_greedy()
```
#### Defining your own samplers
Presets cover the common cases, but when you want to chain multiple shift
steps, set a seed for reproducible output, or use Mirostat, build a sampler
with `NobodyWhoSamplerBuilder`:
```gdscript
var cfg = NobodyWhoSamplerBuilder.new() \
.top_k(40) \
.temperature(0.8) \
.dist()
chat.set_sampler_config(cfg)
```
`NobodyWhoSamplerBuilder` has two kinds of methods: **shift steps** that transform the
probability distribution (returning the builder for further chaining) and
**terminal steps** that finalize the chain into a `NobodyWhoSamplerConfig`. Always end
the chain with one of the terminals: `dist()`, `greedy()`, `mirostat_v1(...)`,
or `mirostat_v2(...)`.
For reproducible output, set the RNG seed anywhere in the chain. The seed is
consumed by every random sampler in the chain — `dist`, `mirostat_v1`,
`mirostat_v2`, and the `xtc` shift step. `greedy` ignores it. If unset, a
default seed is used.
```gdscript
var cfg = NobodyWhoSamplerBuilder.new() \
.top_k(40) \
.temperature(0.8) \
.seed(42) \
.dist()
chat.set_sampler_config(cfg)
```
##### Available Sampling Steps
Pick any of the **shift steps** below (each reshapes the token distribution), then finish with one **terminal step** that picks the token — exactly like the `.top_k(40).temperature(0.8).dist()` chain above.
Shift steps — add as many as you want, applied in order:
- `.top_k(40)` — keep only the 40 most likely tokens
- `.top_p(0.95, 1)` — nucleus: keep the top tokens up to 95% of the probability mass
- `.min_p(0.05, 1)` — drop tokens below 5% of the most likely token's probability
- `.typical_p(0.9, 1)` — keep tokens whose "surprise" is close to average, dropping both the too-predictable and the too-random ([locally typical sampling](https://arxiv.org/abs/2202.00666))
- `.xtc(0.5, 0.1, 1)` — "exclude top choices": occasionally drop the top tokens for more variety
- `.temperature(0.8)` — below 1.0 = more focused, above 1.0 = more random
- `.penalties(64, 1.1, 0.0, 0.0)` — per-token repetition penalty: `penalty_last_n, penalty_repeat, penalty_freq, penalty_present` (`penalty_repeat` 1.0 = off)
- `.dry(0.8, 1.75, 2, -1, ["\n"])` — penalty for repeated *phrases*: `multiplier, base, allowed_length, penalty_last_n, seq_breakers`
- `.seed(42)` — fix the RNG for reproducible output
Terminal step — end the chain with exactly one:
- `.dist()` — pick a token with weighted randomness (the usual choice)
- `.greedy()` — always take the most likely token
- `.mirostat_v1(5.0, 0.1, 100)` / `.mirostat_v2(5.0, 0.1)` — steer output "surprise" toward a target
`min_keep` is the floor on how many tokens survive a cut (`1` is fine).
#### Structured Output
One of the most powerful features is constraining the model to produce output in a specific format. This gives you a hard guarantee that the output matches your format, rather than relying on the model to get it right on its own.
##### Grammar Constraints
You can constrain the model's output using a GBNF grammar:
```gdscript
var grammar = """
root ::= greeting " " name
greeting ::= "Hello" | "Hi" | "Hey"
name ::= "World" | "Friend" | "There"
"""
chat.set_sampler_preset_constrain_with_grammar(grammar)
```
This makes it **impossible** for the model to generate anything outside your defined format.
For a comprehensive tutorial on writing GBNF grammars, including JSON generation, compact formats, and practical game examples, see the [Structured Output](https://docs.nobodywho.ooo/godot/structured-output) guide.
##### JSON Schema Constraints
Force the model to produce JSON matching a specific schema:
```gdscript
var schema = JSON.stringify({
"type": "object",
"properties": {
"name": {"type": "string"},
"level": {"type": "integer"},
"class": {"type": "string", "enum": ["Warrior", "Mage", "Rogue"]}
},
"required": ["name", "level", "class"]
})
chat.set_sampler_preset_constrain_with_json_schema(schema)
```
##### Regex Constraints
For simpler patterns, constrain the output with a regular expression:
```gdscript
# Force the model to answer with exactly "yes" or "no"
chat.set_sampler_preset_constrain_with_regex("yes|no")
```
#### Changing Samplers Mid-Conversation
You can change the sampler at any point during a conversation. The new sampler will take effect on the next `ask()` call:
```gdscript
# Start with free-form chat
chat.ask("Tell me about yourself")
var response = await chat.response_finished
# Switch to structured output for the next question
chat.set_sampler_preset_json()
chat.ask("Now describe your stats as JSON")
var json_response = await chat.response_finished
```
---
### Speech to Text
Source: [https://docs.nobodywho.ooo/godot/speech-to-text](https://docs.nobodywho.ooo/godot/speech-to-text)
To transcribe audio into text, NobodyWho provides an integration with the Whisper models in ONNX format, through the `NobodyWhoSpeechToText` node.
#### Quick start
Add a `NobodyWhoSpeechToText` node to your scene, then use it from a script:
```gdscript
extends Node
@onready var stt: NobodyWhoSpeechToText = $NobodyWhoSpeechToText
func _ready():
stt.model_path = "hf://onnx-community/whisper-base"
stt.start_worker()
await stt.worker_started
stt.transcribe_file("res://recording.mp3")
var text = await stt.transcription_finished
print(text)
```
If the audio is not coming from a file, but instead directly from a buffer, `transcribe_pcm` is available:
```gdscript
stt.transcribe_pcm(samples, 16000) # samples: PackedByteArray, sample_rate: int
var text = await stt.transcription_finished
```
In order to make this work, `samples` needs to be a `PackedByteArray` of mono, little-endian i16 PCM samples. The sample rate can be anything - NobodyWho resamples internally to what Whisper expects.
As with the Chat node, streaming is available: connect to `transcription_updated` to consume the transcription piece by piece as it's decoded, alongside `transcription_finished` for the full transcript once it's done:
```gdscript
extends Node
@onready var stt: NobodyWhoSpeechToText = $NobodyWhoSpeechToText
func _ready():
stt.transcription_updated.connect(_on_transcription_updated)
stt.transcription_finished.connect(_on_transcription_finished)
stt.model_path = "hf://onnx-community/whisper-base"
stt.start_worker()
await stt.worker_started
stt.transcribe_file("res://recording.mp3")
func _on_transcription_updated(piece: String):
$Label.text += piece
func _on_transcription_finished(text: String):
print("Done: ", text)
```
#### Supported models
NobodyWho only supports Whisper models in **ONNX** format. `model_path` is a Hugging Face repo (`hf://owner/repo`) or a local directory containing such a model, e.g. `hf://onnx-community/whisper-base`. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to pick a size that fits your accuracy and speed needs.
You can also pick a `quantization` variant of the model to download and load. Lower-precision variants are smaller and faster, but can lose some transcription accuracy. Supported values are `default`, `fp16`, `int8`, `uint8`, `bnb4`, `q4`, `q4f16`, and `quantized`. Defaults to `default`.
```gdscript
stt.model_path = "hf://onnx-community/whisper-base"
stt.quantization = "q4"
```
#### Improving performance
By default, Whisper auto-detects the spoken language, which costs a bit of extra processing. If you already know the language, set its ISO 639-1 code on `language` to skip detection and improve performance:
```gdscript
stt.model_path = "hf://onnx-community/whisper-base"
stt.language = "en"
```
---
### Structured Output
Source: [https://docs.nobodywho.ooo/godot/structured-output](https://docs.nobodywho.ooo/godot/structured-output)
_Getting reliable, structured responses from your models_
---
Congratulations - you have understood the basics of having a large language models generate text for you.
You are now ready for some more juicy and complex options.
Here are the key terms you should know:
| Term | Meaning |
| ---- | ------- |
| **GBNF** | GGML Backus-Naur Form - a way to define strict rules for output format |
| **Grammar** | The set of rules that define what valid output looks like |
| **Token** | A piece of text (word, punctuation, etc.) that the model generates, generally 1 to 4 characters long |
| **Encoder** | Translates text into tokens that the model can understand |
#### My model is so stupid that it can not even write json
Yeah, most models will fail to generate valid json at some point if you just ask it to.
But fret not dear friend, the solution you are looking for is called **STRUCTURED OUTPUT**.
It is pretty much what it claims to be; A system that constrains the model's vocabulary to one that you determine.
This can be useful for a myriad of things, from forcing the LLM to never use modern words, to using the LLM
as the engine for your own procedural generation dungeon room.
This section will take you through creating your own grammar that the model will have to use.
##### Why GBNF Beats Prompt Engineering
You've probably tried this before:
```
""" Please respond in JSON format with name, level, and class fields
Only use those fields.
Only use valid json.
All json attributes should have " around them.
Please do not deviate from the instructions.
You will lose 10 points if you use other fields than level, class and name.
Do not write a message just json.
If you do not respond in valid json I will lose my job and my kids will starve.
"""
```
And got back something like:
```
Sure! Here's a character: {"name": "Eldara", "level": 15, "class": Wizard} - hope this helps!
```
Notice the problems? Missing quotes around "Wizard", extra text before and after. Your JSON parser explodes. 💥
GBNF fixes this by making it **impossible** for the model to generate anything except the format you define:
```json
{"name": "Eldara", "level": 15, "class": "Wizard"}
```
Valid. Every. Single. Time.
#### Understanding GBNF Grammar Rules
##### The Absolute Basics
A GBNF grammar is made up of **rules**. Each rule says "this thing can be made from these parts":
```
rule-name ::= what-it-can-be
```
##### Your First Grammar: Hello World
Let's start with the simplest possible grammar:
```
root ::= "Hello World"
```
This says: "The output must be exactly the text 'Hello World'". That's it. The model can't say anything else.
Try this and the model will always output: `Hello World`
##### Adding Choices with `|`
What if we want some variety? Use `|` (pipe) to give options:
```
root ::= "Hello World" | "Hi there" | "Greetings"
```
Now the model can choose between these three options, but nothing else.
##### Building Blocks with Multiple Rules
Here's where it gets interesting. You can break things into smaller pieces:
```
root ::= greeting " " name
greeting ::= "Hello" | "Hi" | "Hey"
name ::= "World" | "Friend" | "There"
```
This creates outputs like:
- `Hello World`
- `Hi Friend`
- `Hey There`
The model picks one option from `greeting`, adds a space, then picks one option from `name`.
##### Character Classes
Instead of listing every letter, use character classes:
```
root ::= letter letter letter
letter ::= [a-z]
```
`[a-z]` means "any lowercase letter from a to z". This generates random 3-letter combinations like `cat`, `how`, `dog`.
so letter letter letter will make a three letter word
Common character classes:
- `[a-z]` - lowercase letters
- `[A-Z]` - uppercase letters
- `[0-9]` - digits
- `[a-zA-Z]` - any letter
- `[a-zA-Z0-9]` - letters and numbers
##### Repetitions
This quickly becomes tedious if you want to create either long words or just any word. This is where repetitions come in:
- `*` means "zero or more"
- `+` means "one or more"
- `?` means "optional (zero or one)"
- `{n}` means "exactly n times"
- `{n,}` means "at least n times"
- `{n,m}` means "at least n and at most m times"
```
root ::= letter+
```
This means "one or more lowercase letters" - so you get words like `hello`, `a`, `supercalifragilisticexpialidocious`.
```
root ::= [a-z]+ [0-9]*
```
This means "letters followed by optional numbers" - so you get `hello`, `test123`, `word`.
##### Building JSON Step by Step
Now that you have been tricked into learning the basics of regex, we should build a small JSON generator. Start simple:
```
root ::= "{" "}"
```
This only generates: `{}`
Add one field:
```
root ::= "{" "\"name\"" ":" string "}"
string ::= "\"" [a-zA-Z]+ "\""
```
This generates: `{"name":"Bob"}` (where Bob is any sequence of letters)
Add more fields:
```
root ::= "{" "\"name\"" ":" string "," "\"level\"" ":" number "}"
string ::= "\"" [a-zA-Z]+ "\""
number ::= [0-9]+
```
This generates: `{"name":"Alice","level":"25"}`
##### Making It Flexible
Use repetition to handle variable numbers of fields:
```
root ::= "{" pair ("," pair)* "}"
pair ::= word ":" word
word ::= "\"" [a-zA-Z]+ "\""
```
The `("," pair)*` means "zero or more additional pairs, each preceded by a comma". This generates:
- `{"name":"Bob"}`
- `{"name":"Alice","job":"Wizard"}`
- `{"name":"Charlie","job":"Knight","weapon":"Sword"}`
##### Whitespace: Making It Readable
Add optional whitespace to make output prettier:
```
root ::= "{" ws pair (ws "," ws pair)* ws "}"
pair ::= string ws ":" ws string
string ::= "\"" [a-zA-Z ]+ "\""
ws ::= [ \t\n]*
```
The `ws` rule means "whitespace" - zero or more spaces, tabs, or newlines. Now you get nicely formatted JSON.
##### Advanced: Specific Values
Control exactly what values are allowed:
```
root ::= "{" "\"class\"" ":" class-type "}"
class-type ::= "\"Warrior\"" | "\"Mage\"" | "\"Rogue\"" | "\"Cleric\""
```
This only allows those four specific classes - no hallucinated "Tank-operator" in your neolithic era game!
##### Nested Structures
Build complex nested data:
```
root ::= "{" "\"character\"" ":" character-object "}"
character-object ::= "{" "\"name\"" ":" string "," "\"stats\"" ":" stats-object "}"
stats-object ::= "{" "\"hp\"" ":" number "," "\"mp\"" ":" number "}"
string ::= "\"" [a-zA-Z ]+ "\""
number ::= [0-9]+
```
This creates nested JSON like:
```json
{"character":{"name":"Gandalf","stats":{"hp":"100","mp":"200"}}}
```
#### Performance Optimization: Compact Formats
Now that you understand GBNF with JSON, let's talk optimization. JSON is verbose and every token costs time. For high-performance applications, you can create much more compact formats.
##### Why Compact Formats Matter
**JSON Format:**
```json
{"name":"Gandalf","level":15,"class":"Mage","hp":100,"mp":80}
```
*60 characters, ~38 tokens*
**Compact Format:**
```
Gandalf|High|Mage|Low|High
```
*22 characters, ~10 tokens*
**That's ~4 times faster while maintaining the same information!**
##### Building Compact Formats
Start with pipe-separated values:
```
root ::= [A-Z][a-z]+ "|" [1-9][0-9]? "|" class-type
class-type ::= "Warrior" | "Mage" | "Rogue" | "Cleric"
```
This generates: `Gandalf|15|Mage` (semantically clear - no ambiguity about what "Mage" means!)
**Why not single letters?** If you used `"W" | "M" | "R" | "C"`, the LLM has no inherent knowledge that "M" means "Mage" rather than "Monk" or "Mercenary". The model generates tokens based on semantic understanding, not arbitrary mappings.
##### Different delimiters for different levels
Use different separators for different levels:
```
root ::= character ("|" character)*
character ::= [A-Z][a-z]+ ":" stats ":" equipment
stats ::= stats-range "," stats-range "," stats-range
stats-range ::= "low" | "medium" | "high"
equipment ::= weapon-type "," armor-type
weapon-type ::= "Sword" | "Axe" | "Staff" | "Dagger"
armor-type ::= "Leather" | "Robes" | "Chain" | "Plate"
```
This generates: `Gandalf:high,low,low:Staff,Robes|Aragorn:low,high,medium:Sword,Plate` which in JSON would be:
```json
[
{
"name": "Gandalf",
"stats": {
"hp": "high",
"mp": "low",
"level": "low"
},
"equipment": {
"weapon": "Staff",
"armor": "Robes"
}
},
{
"name": "Aragorn",
"stats": {
"hp": "low",
"mp": "high",
"level": "medium"
},
"equipment": {
"weapon": "Sword",
"armor": "Plate"
}
}
]
```
##### Semantic Soundness
One advantage of using JSON is the hints it gives the LLM.
If it sees `"name": "Gandalf"`, instead of just `Gandalf` it might be more inclined to generate a wizard class or give the character a staff.
The same goes for numbers, the llm does not inherently understand what a good number for a high level or mana pool is - but it understands high vs low.
When designing compact formats:
✅ **Good:** `"Warrior" | "Mage" | "Rogue"`
✅ **Good:** `"Sword" | "Staff" | "Dagger"`
✅ **Good:** `"Leather" | "Robes" | "Chain"`
✅ **Good:** `"Low" | "Medium" | "High"`
❌ **Bad:** `"WAR" | "MAG" | "ROG"` - abbreviated and potentially ambiguous
❌ **Bad:** `"W" | "M" | "R"` - arbitrary single letters
❌ **Bad:** `"1" | "2" | "3"` - numeric values
The LLM generates text based on semantic understanding. Use full words that align perfectly with how language models think about concepts.
You should additionally provide the right context and single or few shots prompting to make it more robust.
##### Underscores footgun
The GBNF format does not support `_`. According to the [the GBNF format documentation](https://github.com/ggml-org/llama.cpp/tree/master/grammars#json-schemas--gbnf), only lowercase characters and dashes are allowed for naming nonterminals.
#### Practical Example: Legendary Weapon Generator
Let's build a weapon generation system that creates legendary weapons for your RPG. We'll start simple and add complexity step by step, showing you how GBNF grammars work in practice.
##### Why Use GBNF for Weapon Generation?
Traditional random generators often create nonsensical combinations like "Flaming Sword of Ice", with 8 fire damage and a random generic backstory as well an ice ability. More advanced systems exist but they rely on lookup tables which can become tedious very quickly.
LLMs with GBNF understand semantic coherence - they'll generate "Flamebrand, Ancient Sword of Solar Wrath" instead.
Which has 8 fire damage, and a meaningful backstory based on how you got it
or the lore from your game as well as an ability that is chosen based on the backstory, damage and name.
##### Step 1: Dynamic Weapon Name Generator
Let's start with a weapon generator that builds weapon names:
**Grammar:**
```
root ::= weapon-name " (" weapon-type ")"
weapon-name ::= name-prefix name-suffix
name-prefix ::= "Flame" | "Frost" | "Shadow" | "Storm" | "Light" | "Dark"
name-suffix ::= "brand" | "fang" | "bane" | "call" | "ward" | "rend"
weapon-type ::= "Sword" | "Axe" | "Dagger" | "Staff" | "Bow" | "Hammer"
```
```gdscript
extends Node
@onready var model = $Model # Your NobodyWhoModel node
@onready var chat = $Chat # Your NobodyWhoChat node
func _ready():
# Configure the weapon generator
model.model_path = "res://models/your-model.gguf"
chat.model_node = model
chat.system_prompt = "You are a legendary weapon generator for a fantasy RPG."
# Start the worker so it's ready
chat.start_worker()
# Connect to handle responses
chat.response_finished.connect(_on_weapon_generated)
func _input(event):
if event is InputEventKey and event.pressed and event.keycode == KEY_SPACE:
generate_weapon()
func generate_weapon():
chat.set_sampler_preset_constrain_with_grammar(grammar_string)
# Reset context to avoid new weapons to be influenced by already generated ones.
chat.reset_context()
chat.ask("Generate a weapon:")
func _on_weapon_generated(weapon_name: String):
print(weapon_name)
# Here you could add the weapon to inventory, display it in UI, etc.
```
**Output examples:**
- `Flamebrand (Sword)`
- `Shadowfang (Dagger)`
- `Stormcall (Staff)`
- `Darkward (Bow)`
This is more or less just a random number generator, but more GPU expensive...
##### Step 2: Adding Weapon Stats
Let's add damage and abilities to make weapons more interesting for gameplay, this is where we deviate from a random weapon generator to a semantic weapon generator:
**Grammar:**
```
root ::= weapon-name " (" weapon-type ") - " damage-level " damage, " ability-name " ability. " backstory
weapon-name ::= name-prefix name-suffix
name-prefix ::= "Flame" | "Frost" | "Shadow" | "Storm" | "Light" | "Dark"
name-suffix ::= "brand" | "fang" | "bane" | "call" | "ward" | "rend"
weapon-type ::= "Sword" | "Axe" | "Dagger" | "Staff" | "Bow" | "Hammer"
damage-level ::= "Low" | "Medium" | "High" | "Legendary"
ability-name ::= "Flame Strike" | "Frost Bite" | "Shadow Step" | "Lightning Bolt" | "Healing Aura" | "Poison Cloud"
backstory ::= [a-zA-Z0-9 ]+ "."
```
Be careful not to add too many symbols in your backstory. If the model can not write a `.` it will increase the chance that it will end the sentence instead of writing paragraph upon paragraph of text.
```gdscript
func generate_weapon():
chat.set_sampler_preset_constrain_with_grammar(grammar_string)
# Reset context to avoid new weapons to be influenced by already generated ones.
chat.reset_context()
chat.ask("Generate a weapon:")
func _on_weapon_generated(weapon_data: String):
print(weapon_data)
```
**Output examples:**
- `Shadowfang (Sword) - Legendary damage, Shadow Step ability. Shadowfang is a legendary sword that was forged by the ancient shadow realm.`
See how the examples will match flame and brand to a sword, will give it the flame strike ability as well as a thematic backstory. It feels like there is intent behind the creation of this weapon.
##### Step 3: Enhanced Backstories
Let's expand the backstory system to allow for richer, more detailed weapon lore:
**Grammar:**
```
root ::= weapon-name " (" weapon-type ") - " damage-level " damage, " ability-name " ability. Story: " backstory
weapon-name ::= name-prefix name-suffix
name-prefix ::= "Flame" | "Frost" | "Shadow" | "Storm" | "Light" | "Dark"
name-suffix ::= "brand" | "fang" | "bane" | "call" | "ward" | "rend"
weapon-type ::= "Sword" | "Axe" | "Dagger" | "Staff" | "Bow" | "Hammer"
damage-level ::= "Low" | "Medium" | "High" | "Legendary"
ability-name ::= "Flame Strike" | "Frost Bite" | "Shadow Step" | "Lightning Bolt" | "Healing Aura" | "Poison Cloud"
backstory ::= [a-zA-Z0-9 ]{50,200} "."
```
When doing this we want to also inject some of our lore. We will borrow from Lord of the rings here - replace with your own lore.
```gdscript
func _ready():
# Configure the weapon generator
chat.model_node = model
chat.system_prompt = "Generate a weapon a backstory in the LOTR universe"
# ... rest of the setup
func generate_weapon():
chat.set_sampler_preset_constrain_with_grammar(grammar_string)
# Reset context to avoid new weapons to be influenced by already generated ones.
chat.reset_context()
chat.ask("The party just found a new weapon after travelling through the mines of Moria:")
func _on_weapon_generated(weapon_data: String):
print(weapon_data)
```
**Output examples:**
- `Shadowfang (Sword) - Legendary damage, Shadow Step ability. The sword is made from the dark shards that were once part of the Balrog`
- `Flamebrand (Sword) - High damage, Flame Strike ability. Backstory involves a fallen dwarf lord named Drakon who was corrupted by the Balrogs and used the sword to slay an enemy.`
##### Step 4: Compact Format for Performance
For games that generate many weapons or even very complex weapons, you want maximum efficiency. Let's create a compact pipe-separated format:
**Grammar:**
```
root ::= weapon-name "|" weapon-type "|" damage-level "|" ability-name "|" weight "|" throwable "|" damage-type "|" durability "|" rarity "|" enchantment "|" material "|" short-story
weapon-name ::= name-prefix name-suffix
name-prefix ::= "Flame" | "Frost" | "Shadow" | "Storm" | "Light" | "Dark"
name-suffix ::= "brand" | "fang" | "bane" | "call" | "ward" | "rend"
weapon-type ::= "Sword" | "Axe" | "Dagger" | "Staff" | "Bow" | "Hammer"
damage-level ::= "Low" | "Medium" | "High" | "Legendary"
ability-name ::= "Flame Strike" | "Frost Bite" | "Shadow Step" | "Lightning Bolt" | "Healing Aura" | "Poison Cloud"
weight ::= "Heavy" | "Light"
throwable ::= "Throwable" | "Non-throwable"
damage-type ::= "Sharp" | "Pierce" | "Blunt"
durability ::= "Fragile" | "Sturdy" | "Unbreakable"
rarity ::= "Common" | "Rare" | "Epic" | "Legendary"
enchantment ::= "Glowing" | "Humming" | "Pulsing" | "Silent"
material ::= "Steel" | "Mithril" | "Obsidian" | "Crystal"
backstory ::= [a-zA-Z0-9 ]{50,200} "."
```
**Note:** So-called "thinking" or "reasoning" models will strongly prefer to start every generation with a block of text inside `<think>` tags. If your grammar doesn't naturally allow the output to be prefixed with a "thinking" section like this, it will try to squeeze it into free-text sections (e.g. like the backstory section in the example above). If relying a lot on structured generation, you may prefer to use a "non-thinking" model. If you prefer to keep the "thinking" ability, you could begin your grammar with a section like `"<think>" [a-zA-Z0-9 ]{10,1000} "." "</think>` to allow it to get it's reasoning section out of the way.
Furthermore, the current implementation of GBNF has some performance issues with using specifc ranges (eg: word{10,20}) - so it might be smarter to have a non grammarized model generate the short story.
**Output examples:**
- `Flamebrand|Sword|High|Flame Strike|Heavy|Non-throwable|Sharp|Sturdy|Epic|Glowing|Steel|Forged by fire elementals in ancient volcano`
or with thinking models (demonstrating that it will squeeze in the "thinking" section wherever possible:
- `Shadowfang|Axe|Legendary|Shadow Step|Light|Throwable|Sharp|Sturdy|Epic|Silent|Steel|The Shadowfang is a legendary axe that is said to have been forged in the depths of the Shadowspire Mountains by the elusive Night Hunter.`
- `Stormcall|Staff|Legendary|Lightning Bolt|Light|Non-throwable|Blunt|Unbreakable|Legendary|Pulsing|Crystal|The user wants me to generate a short story for the weapon. I will think...`
---
This is quite a powerful system for procedural generation of anything being weapons, levels, questlines or whatever you can think of, and even better
You get to influence the generation meaningfully with the prompt that you send, while keeping the variety offered by the system.
This complete system generates weapons with all the attributes your game systems might need, from combat mechanics (damage type, weight) to visual effects (enchantment, material) and lore (story).
---
### Text to Speech
Source: [https://docs.nobodywho.ooo/godot/text-to-speech](https://docs.nobodywho.ooo/godot/text-to-speech)
Generate natural-sounding speech from text, ready to save as a WAV file or play back in your app.
#### Quick start
Add a `NobodyWhoTextToSpeech` node to your scene, then use it from a script:
```gdscript
extends Node
@onready var tts: NobodyWhoTextToSpeech = $NobodyWhoTextToSpeech
func _ready():
tts.source = "hf://NobodyWho/Kokoro-82M" # Hugging Face repo ID or local folder with the model files.
tts.voice = "bf_emma" # Voice to use from the model.
tts.language = "en-gb" # Language code for the input text.
tts.start_worker()
await tts.worker_started
# Generate WAV bytes for this sentence.
var result: Dictionary = await tts.synthesize("Hello from NobodyWho!")
if not result.ok:
push_error(result.error)
return
# Save the audio to a file.
var file = FileAccess.open("user://out.wav", FileAccess.WRITE)
file.store_buffer(result.wav)
```
#### Models and sources
NobodyWho supports three speech synthesis architectures, all in ONNX format:
- [Kokoro](https://github.com/hexgrad/kokoro), a lightweight 24 kHz speech synthesis model. Model page: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M).
- [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), a compact 24 kHz speech synthesis model. Model page: [`KevinAHM/pocket-tts-onnx`](https://huggingface.co/KevinAHM/pocket-tts-onnx).
- [Supertonic](https://github.com/supertone-inc/supertonic), a multi-stage speech synthesis model with voice styles. Model page: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3).
`source` can be a Hugging Face repo (`hf://owner/repo`) as shown above, a Godot path (`res://` or `user://`), or a local filesystem directory laid out the same way as that repo. See [Local model folder format](https://docs.nobodywho.ooo/godot/text-to-speech#local-model-folder-format) and [Architecture](https://docs.nobodywho.ooo/godot/text-to-speech#architecture) for setup details.
#### Kokoro
For Kokoro, set `voice` and `language` together. They must agree with the model's available voices.
```gdscript
tts.source = "hf://NobodyWho/Kokoro-82M"
tts.voice = "bf_emma"
tts.language = "en-gb"
```
Optional settings include:
- `voice`: voice to use from the model, e.g. `bf_emma`. See the [Kokoro voices folder](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main/voices) for the full list. Defaults to `bf_emma`.
- `language`: input language code. Supported values are listed on the [Kokoro model page](https://huggingface.co/NobodyWho/Kokoro-82M). Defaults to `en-gb`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Set `0` to use the architecture default.
#### Supertonic
For Supertonic, you can start with the default `voice` and `language`, or set them explicitly.
```gdscript
tts.source = "hf://Supertone/supertonic-3"
tts.language = "en"
```
Optional settings include:
- `voice`: voice style. Supported values are `M1` to `M5` and `F1` to `F5`. Defaults to `M1`.
- `language`: input language code. See the [Supertonic model page](https://huggingface.co/Supertone/supertonic-3#supported-languages) for the full list. Defaults to `en`.
- `speed`: speech speed multiplier. `1.0` is normal speed, lower values are slower, higher values are faster. Set `0` to use the architecture default.
- `steps`: denoising steps. Higher values can improve quality but are slower. Lower values are faster but can sound rougher. Set `0` to use the architecture default.
- `silence_duration`: seconds of silence between long text chunks. Higher values add longer pauses. Set `-1` to use the architecture default.
#### Pocket TTS
For Pocket TTS, start with the default voice and language, or set them explicitly.
```gdscript
tts.source = "hf://KevinAHM/pocket-tts-onnx"
tts.voice = "alba"
tts.language = "english_2026-04"
tts.huggingface_token = "hf_..." # Alternative: set the HF_TOKEN environment variable.
```
Optional settings include:
- `voice`: built-in voice name. See the [Pocket TTS voice catalogue](https://github.com/kyutai-labs/pocket-tts?tab=readme-ov-file#voices). Defaults to `alba`.
- `language`: language bundle name. See the [available language bundles](https://huggingface.co/KevinAHM/pocket-tts-onnx/tree/main/onnx). Defaults to `english_2026-04`.
- `steps`: quality steps. Higher values are slower. Set `0` to use the default of `1`.
- `precision`: `int8` for faster loading or `fp32` for higher quality. Empty uses `int8`.
- `temperature`: controls how varied the speech sounds. Set `-1` to use the default of `0.7`.
Pocket TTS voice states are gated in [`kyutai/pocket-tts`](https://huggingface.co/kyutai/pocket-tts). Accept its terms, then pass a [Hugging Face access token](https://huggingface.co/settings/tokens) with `huggingface_token`, or alternatively set the `HF_TOKEN` environment variable.
#### Architecture
`architecture` is the TextToSpeech model family behind a source. In most cases, you do not need to set it because NobodyWho can infer it by looking for "kokoro", "pocket-tts", or "supertonic" in the `source` string.
Set `architecture` when you use a local directory, Godot path, or a custom source that NobodyWho cannot recognize:
```gdscript
tts.source = "res://models/kokoro-folder"
tts.architecture = "kokoro"
```
Supported architecture values are `kokoro`, `pocket-tts`, and `supertonic`.
#### GPU
TextToSpeech uses GPU acceleration by default when available. Disable it with `device = "cpu"`:
```gdscript
tts.source = "hf://Supertone/supertonic-3"
tts.device = "cpu"
tts.start_worker()
```
#### Local model folder format
When `source` is a local directory or Godot path, point it at the top-level model folder and set the matching `architecture`.
Use the Hugging Face file browsers as the reference layouts:
- Kokoro: [`NobodyWho/Kokoro-82M`](https://huggingface.co/NobodyWho/Kokoro-82M/tree/main)
- Supertonic: [`Supertone/supertonic-3`](https://huggingface.co/Supertone/supertonic-3/tree/main)
For Supertonic, that top-level folder must include both the `onnx/` and `voice_styles/` directories. Download the model files with the same relative paths, then pass that folder as `source`.
---
### Tool Calling
Source: [https://docs.nobodywho.ooo/godot/tool-calling](https://docs.nobodywho.ooo/godot/tool-calling)
_Triggering actions from within the model._
---
Welcome to the tool calling page!
Now that you have some of the basics understood (if not, please read [Chat](https://docs.nobodywho.ooo/godot/chat)),
we can move on to adding one of the truly powerful and fun components to our model; Tool/Function Calling.
Tool calling is a way to give your model actions to perform in your game world.
The model can:
* Check data - "What's my health?"
* Change the world - "Open the north gate."
* Run helper logic - damage rolls, crafting math, random loot.
We'll start with a small and simple tool, add arguments, then increase accuracy using schema and adding constraints.
**Note that not all models support tool calling**
---
#### A simple tool
This is an example of how to give the model access to a function we have created that gets the player's current stats (health, mana, gold).
```gdscript
extends NobodyWhoChat
func get_player_stats() -> String:
var player = GameManager.get_local_player()
return JSON.stringify({
"health": player.health,
"mana": player.mana,
"gold": player.gold
})
func _ready():
add_tool(get_player_stats, "Returns the local player's health, mana, and gold.")
```
Ask "How hurt am I?" - the model calls your tool and answers with real numbers.
---
#### But I need arguments, you say:
Sure - that is possible, but only primitives are currently implemented in NobodyWho:
Allowed primitive types: `int`, `float`, `bool`, `String`/`string`, `Array`/`string[]`
Models operate with JSON as an abstract layer instead of using a specific language (like Godot) when calling tools.
When NobodyWho receives a function or a delegate it will deconstruct the name and parameters and use them
to construct a JSON schema that we can pass to the model.
In the example below the generated json will look something like this:
```json
{
"type": "object",
"properties": {
"amount": {
"type": "integer",
"description": ""
}
},
"required": ["amount"]
}
```
This is then used to construct a lazy-loadable gbnf grammar, so the models always pass the correct number and set of arguments.
A limitation of this is that we cannot extract the description from a given argument.
Therefore it might be advantageous to write your own schema for maximum precision.
```gdscript
func heal_player(amount: int) -> String:
GameManager.get_local_player().heal(amount)
return "Healed %d HP" % amount
add_tool(heal_player, "Heals the local player by a number of hit-points")
```
*Godot auto-builds the JSON schema from the type hints.*
Therefore you must ensure that all parameters are listed and return type is defined from the method.
---
#### Your model is now ready to interact with the world
Have the model open a door.
```gdscript
func open_door(door_id: String) -> String:
DoorManager.open(door_id)
return "Opened door %s" % door_id
add_tool(open_door, "Opens a door in the world by id")
chat.ask("can you open the door")
```
The model will pause any generation until the tool is completed.
---
#### Multiple Tools & Resetting
You can add as many tools as like, but you need to reset the context before they will be taken into account.
```gdscript
add_tool(get_player_stats, "Player stats")
add_tool(open_door, "Open a door")
reset_context()
```
---
#### But I don't want it to hallucinate random strings
Don't worry, we've got you.
As I mentioned before, we are using the OpenSchema specification, which goes like this:
```jsonschema
{
"type": "object",
"properties": {
"color": {
"type": "string",
"description": "A specific color for the button",
"enum": ["red", "blue", "green"]
}
},
"required": ["color"],
}
```
The type must always be an `object`, the properties are a dictionary of where the key is the parameter name, and the value describes the data for the parameter. Ie. type determines whether it is a string, a list or something else. Description describes how the parameter is used.
If the properties are not a part of the `required` list, the model will see them as optional parameter.
```gdscript
# `press_button_schema` holds the JSON shown above.
func press_button(color: String) -> String:
ButtonManager.press(color)
return "Pressed %s button" % color
add_tool_with_schema(press_button,
press_button_schema,
"Press one of the three coloured buttons (red, blue, green)")
```
Result: the model **cannot** request any color other than *red*, *blue*, or *green*. Use the same pattern for item rarities, quest tiers, etc...
**Heads-up** – NobodyWho turns that schema into a GBNF grammar using the open-source [`richardanaya/gbnf`](https://github.com/richardanaya/gbnf) converter. It currently supports the common bits: primitive types, `enum`, `required`, flat `oneOf`, and simple arrays. Exotic keywords (`minimum`, `pattern`, deeply-nested refs) may be ignored until the library grows.
---
A note on descriptions:
The description helps the model pick the right tool and pass the right arguments. Be explicit. Explain when to use the tool, explain what the tool does.
Bad: **"Door"**
Good: **"Use this function when the assistant is blocked or needs to close a door. This tool opens or closes the door with the given id, if -1 is given, the nearest door will be interacted with."**
---
#### Pre-packaged tools
We ship NobodyWho with two packaged-in tools, which are general enough for multiple use-cases - [monty](https://github.com/pydantic/monty) Python interpreter
and [bashkit](https://github.com/everruns/bashkit) Bash interpreter. Both of them should serve similar purpose - to give your small LLM a better chance to answer
questions requiring precise reasoning or some kind of computation, possibly on a big context.
The usage is straightforward. Use `add_python_tool()` and `add_bash_tool()`:
```gdscript
func _ready():
add_python_tool()
add_bash_tool()
```
Lastly, keep in mind that for most use-cases it is reasonable to constrain the tools with some limits regarding memory and computation time,
so that you don't end up executing infinite loop code. To solve this, `add_python_tool()` provides `max_duration_secs`, `max_memory_bytes` and `max_recursion_depth`
and `add_bash_tool()` provides `max_commands`.
---
### Multimodal Models
Source: [https://docs.nobodywho.ooo/godot/vision](https://docs.nobodywho.ooo/godot/vision)
A picture is worth a thousand words (or at least a thousand tokens).
With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.
**Info:**
This is about models that **natively** ingest images and audio - no transcription step involved.
That matters for audio in particular: the model hears the raw sound, not just words that were said,
so it can react to tone of voice, music, or other non-speech noises. If you only need to convert
speech to text, see [Speech-to-Text](https://docs.nobodywho.ooo/godot/speech-to-text) instead. If you need to generate spoken audio
from text, see [Text-to-Speech](https://docs.nobodywho.ooo/godot/text-to-speech).
#### Choosing a model
Not all models have built-in image and audio capabilities. Generally, you will
need two parts for making this work:
1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens
To find such a model, refer to the [HuggingFace Image-Text-to-Text](https://huggingface.co/models?pipeline_tag=image-text-to-text&library=gguf&sort=likes) section
and [Audio-Text-to-Text](https://huggingface.co/models?pipeline_tag=audio-text-to-text&sort=trending). Some models like Gemma 4 even manage both!
Usually, the projection model then includes `mmproj` in its name.
If you are unsure which ones to pick, or just want a reasonable default, you can try [Gemma 4](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf?download=true) with its [BF16 projection model](https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/mmproj-BF16.gguf?download=true),
which can do both image and audio.
With the downloaded GGUFs, you can set the projection model on your `NobodyWhoModel` node.
In the editor, set the `projection_model_path` property to point to your projection model file.
Alternatively, you can set it in GDScript:
```gdscript
$ChatModel.projection_model_path = "res://mmproj.gguf"
```
**Info:**
The language model and projection model have to **fit** together, as they are trained together!
Unfortunately you can't just take projection model and a LLM that you like and expect them
to work together.
#### Composing a prompt object
With the model configured, all that is left is to compose the prompt and send it to the model.
That is done through the `NobodyWhoPrompt` object.
```gdscript
extends NobodyWhoChat
func _ready():
self.model_node = get_node("../ChatModel")
self.system_prompt = "You are a helpful assistant, that can hear and see stuff!"
var prompt = NobodyWhoPrompt.new()
prompt.add_text("Tell me what you see in the image and what you hear in the audio.")
prompt.add_image("res://dog.png")
prompt.add_audio("res://sound.mp3")
ask(prompt)
var response = await response_finished # It's a dog and a penguin!
```
That should be it! Beware though, that consuming images and audio can quickly drain the context,
and larger context sizes may be needed for smooth usage.
---
### Voice Activity Detection
Source: [https://docs.nobodywho.ooo/godot/voice-activity-detection](https://docs.nobodywho.ooo/godot/voice-activity-detection)
Before running transcription on audio, it's important to know when to stop listening and start
transcribing (and answering). This can be done in a simple way (a silence timeout), but that can
often be quite clunky and feel slow. Voice activity detection uses a small model that understands
the shape of speech and can reliably tell speech and silence apart, through the
`NobodyWhoVoiceActivityDetection` node.
#### Streaming
Add a `NobodyWhoVoiceActivityDetection` node to your scene, then push audio chunks into it as they
arrive from wherever you're capturing them (e.g. an `AudioEffectCapture`):
```gdscript
extends Node
@onready var vad: NobodyWhoVoiceActivityDetection = $NobodyWhoVoiceActivityDetection
@onready var stt: NobodyWhoSpeechToText = $NobodyWhoSpeechToText
func _ready():
vad.speech_ended.connect(_on_speech_ended)
vad.start_worker()
await vad.worker_started
stt.start_worker()
await stt.worker_started
# Call this whenever you get a new chunk from your microphone capture setup.
func _on_mic_chunk(samples: PackedByteArray):
vad.push(samples)
func _on_speech_ended():
var speech = vad.finish()
stt.transcribe_pcm(speech, 16000)
var text = await stt.transcription_finished
print(text)
```
`NobodyWhoVoiceActivityDetection` acts as a buffer: every `push()` call emits `speech_started` or
`speech_ended` when it crosses a confirmed boundary. Once `speech_ended` fires, `finish()` gives
you back the buffered audio for just the speech segment, and resets internal state so it's ready
for the next turn.
#### Segmentation
Another use case is segmenting speech out of audio you already have. A good example is a long,
mostly-silent recording with a few short occurrences of speech you want to transcribe. That's what
the `segment()` method is for:
```gdscript
extends Node
@onready var vad: NobodyWhoVoiceActivityDetection = $NobodyWhoVoiceActivityDetection
@onready var stt: NobodyWhoSpeechToText = $NobodyWhoSpeechToText
func _ready():
vad.start_worker()
await vad.worker_started
stt.start_worker()
await stt.worker_started
var audio: PackedByteArray = ... # a full recording, interleaved little-endian i16 PCM samples
for speech in vad.segment(audio):
stt.transcribe_pcm(speech, 16000)
var text = await stt.transcription_finished
print(text)
```
#### Configuring sensitivity
We try to provide reasonable defaults to capture most situations. However, especially in the case
of voice activity detection, manual tuning is often needed to reach better performance. For that,
`NobodyWhoVoiceActivityDetection` exposes a few properties you can tweak, either in the editor or
from code — set them before calling `start_worker()`, since that's when they're read:
```gdscript
extends Node
@onready var vad: NobodyWhoVoiceActivityDetection = $NobodyWhoVoiceActivityDetection
func _ready():
# VAD is currently fixed to Silero ONNX, but you can change the model_path.
vad.model_path = "hf://onnx-community/silero-vad"
# Determines the sensitivity to what counts as speech. Moving it up will make the VAD stricter.
vad.threshold = 0.5
# Determines the minimum duration classified as speech.
vad.min_speech_duration_ms = 250
# Determines the minimum duration classified as silence.
vad.min_silence_duration_ms = 250
# Determines how much audio to keep before the official speech_started signal, to avoid cutting off the start.
vad.preroll_duration_ms = 500
vad.start_worker()
await vad.worker_started
```
---