The model does not produce tokens but rather a probability distribution over all possible tokens. We must then choose how to pick the next token from the distribution. This is the job of a sampler, which using NobodyWho you can freely modify, to achieve better quality outputs or constrain the outputs to some known format (e.g. JSON).

Sampler presets

To get a quick start, NobodyWho offers a couple of well-known presets, which you can quickly utilize. For example, if you want to increase or decrease the "creativity" of your model, select our temperature preset:

import { Chat, SamplerPresets } from "react-native-nobodywho";

const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  sampler: SamplerPresets.temperature(0.2),
});

Setting temperature to 0.2 will affect the sampler when choosing the next token, making the distribution less flat and therefore the model will favour more probable tokens.

To see the whole list of presets, check out the SamplerPresets class:

class SamplerPresets {
  static default(): SamplerConfig;
  static dry(): SamplerConfig;
  static greedy(): SamplerConfig;
  static json(): SamplerConfig;
  static temperature(temperature: number): SamplerConfig;
  static topK(topK: number): SamplerConfig;
  static topP(topP: number): SamplerConfig;

  // Constrain output to a specific format:
  static constrainWithJsonSchema(schema: string): SamplerConfig;
  static constrainWithRegex(pattern: string): SamplerConfig;
  static constrainWithGrammar(grammar: string): SamplerConfig;
}

Structured output

One of the most useful features is constraining the model to produce structured output — this gives you a hard guarantee that the output matches a specific format, rather than relying on the model to get it right on its own.

Regular expressions

For simpler patterns, you can constrain the output with a regex. Both regex literals and strings are accepted:

// Force the model to answer with exactly "yes" or "no"
const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  sampler: SamplerPresets.constrainWithRegex(/yes|no/),
});
const answer = await chat.ask("Is the sky blue?").completed();

JSON schema

In some use-cases it might be useful to let the LLM generate JSON output. This could be done either in the simple way, just enforcing any JSON by the preset:

const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  sampler: SamplerPresets.json(),
});

Or utilizing JSON schemas to really force the LLM to give you the specific object shapes that you want:

const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  sampler: SamplerPresets.constrainWithJsonSchema({
    type: "object",
    properties: {
      name: { type: "string" },
      age:  { type: "integer" },
    },
    required: ["name", "age"],
  }),
});
const response = await chat.ask("Give me a person.").completed();
const person = JSON.parse(response); // always valid JSON matching the schema

Custom grammars (advanced)

For cases where JSON schema and regex are not expressive enough, you can supply a custom grammar. constrainWithGrammar accepts both Lark syntax and GBNF (llama.cpp format) — NobodyWho automatically converts GBNF to Lark before passing it to the inference engine.

Lark syntax (recommended):

const sampler = SamplerPresets.constrainWithGrammar(`
    start: record (NEWLINE record)* NEWLINE?
    record: field ("," field)*
    field: /[^,"\\n\\r]+/
    NEWLINE: /\\r?\\n/
`);

GBNF syntax (also accepted):

const sampler = SamplerPresets.constrainWithGrammar(`
    file   ::= record (newline record)* newline?
    record ::= field ("," field)*
    field  ::= /[^,"\\n\\r]+/
    newline ::= "\\r\\n" | "\\n"
`);

See the Lark documentation and the GBNF specification for the full grammar syntax.

Defining your own samplers

Sampler presets abstract away some control that you might want - for example, if you want to chain samplers, change more "advanced" parameters, etc. For that use case, we provide the SamplerBuilder class:

import { Chat, SamplerBuilder } from "react-native-nobodywho";

const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  sampler: new SamplerBuilder().temperature(0.8).topK(5).dist() as SamplerConfig,
});

With SamplerBuilder you can chain multiple steps together and then select how you want to sample from the distribution. Keep in mind that SamplerBuilder provides two types of methods: ones which modify the distribution (returning again the instance of SamplerBuilder) and ones which sample from the distribution (returning SamplerConfig). So in order to have the sampler working properly, be careful to always end the chain with one of the sampling steps (e.g. dist(), greedy(), mirostatV2(), etc.).

You can also change the sampler configuration on an existing chat instance:

const sampler = new SamplerBuilder()
  .temperature(0.8)
  .topK(5)
  .dist() as SamplerConfig;

await chat.setSamplerConfig(sampler);