Multimodal Models

Easily provide image and audio information directly to a multimodal LLM.

info

This is about models that natively ingest images and audio - no transcription step involved. That matters for audio in particular: the model hears the raw sound, not just words that were said, so it can react to tone of voice, music, or other non-speech noises. If you only need to convert speech to text, see Speech-to-Text instead. If you need to generate spoken audio from text, see Text-to-Speech.

Choosing a model

Not all models have built-in image and audio capabilities. Generally, you will need two parts:

1. Multimodal LLM that can consume image-tokens and/or audio-tokens
2. Projection model that converts images to image-tokens and/or audio to audio-tokens

To find such a model, refer to the HuggingFace Image-Text-to-Text section and Audio-Text-to-Text. Some models like Gemma 4 manage both! Usually, the projection model includes mmproj in its name.

If you are unsure which ones to pick, try Gemma 4 with its BF16 projection model.

Load the projection model alongside the main model:

import ai.nobodywho.Model
import ai.nobodywho.Chat

val model = Model.load(
    modelPath = "./multimodal-model.gguf",
    projectionModelPath = "./mmproj.gguf"
)
val chat = Chat(model = model)

info

The language model and projection model must fit together, as they are trained together. You can't take an arbitrary projection model and pair it with any LLM.

Composing a prompt

With the model configured, compose a multimodal prompt using Prompt:

import ai.nobodywho.Prompt

val response = chat.ask(Prompt(
    Prompt.Text("Tell me what you see in the image and what you hear in the audio."),
    Prompt.Image("./dog.png"),
    Prompt.Audio("./sound.mp3"),
)).completed()
println(response) // It's a dog!

That should be it! Beware though, that consuming images and audio can quickly drain the context, and larger context sizes may be needed for smooth usage.

Media in a message list

Prompt is for ask(). When you pass a whole conversation to complete(), the same interleaving is expressed as a list of content parts — the shape the OpenAI and Anthropic libraries use:

chat.complete(listOf(
    Message.User(
        ContentPart.Text("Tell me what you see in the image."),
        ContentPart.Image("./dog.png"),
        ContentPart.Text("Answer in one word."),
    )
)).completed()

Parts are ContentPart.Text, ContentPart.Image and ContentPart.Audio. The order is the order the model sees them in. A plain string stays valid wherever content is accepted (Message.User("hello")), so text-only messages need no change.

Media in a saved conversation

getChatHistory() records the file path of every image and audio clip in the content part it belongs to, so a conversation containing media can be saved and replayed later. complete() re-reads each file, so the model sees the images and audio again rather than a conversation with holes in it. The files therefore have to still be where they were — if one cannot be read, complete() throws instead of quietly answering without it.