Multimodal Models

A picture is worth a thousand words (or at least a thousand tokens). With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.

info

This is about models that natively ingest images and audio - no transcription step involved. That matters for audio in particular: the model hears the raw sound, not just words that were said, so it can react to tone of voice, music, or other non-speech noises. If you only need to convert speech to text, see Speech-to-Text instead. If you need to generate spoken audio from text, see Text-to-Speech.

Choosing a model

Not all models have built-in image and audio capabilities. Generally, you will need two parts for making this work:

1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens

To find such a model, refer to the HuggingFace Image-Text-to-Text section and Audio-Text-to-Text. Some models like Gemma 4 even manage both! Usually, the projection model then includes mmproj in its name.

If you are unsure which ones to pick, or just want a reasonable default, you can try Gemma 4 with its BF16 projection model, which can do both image and audio.

With the downloaded GGUFs, you can set the projection model on your NobodyWhoModel node. In the editor, set the projection_model_path property to point to your projection model file. Alternatively, you can set it in GDScript:

$ChatModel.projection_model_path = "res://mmproj.gguf"

info

The language model and projection model have to fit together, as they are trained together! Unfortunately you can't just take projection model and a LLM that you like and expect them to work together.

Composing a prompt object

With the model configured, all that is left is to compose the prompt and send it to the model. That is done through the NobodyWhoPrompt object.

extends NobodyWhoChat

func _ready():
    self.model_node = get_node("../ChatModel")
    self.system_prompt = "You are a helpful assistant, that can hear and see stuff!"

    var prompt = NobodyWhoPrompt.new()
    prompt.add_text("Tell me what you see in the image and what you hear in the audio.")
    prompt.add_image("res://dog.png")
    prompt.add_audio("res://sound.mp3")

    ask(prompt)
    var response = await response_finished  # It's a dog and a penguin!

That should be it! Beware though, that consuming images and audio can quickly drain the context, and larger context sizes may be needed for smooth usage.