Skip to main content

Embeddings & RAG

When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.

Understanding Embeddings​

Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.

For example, "Schedule a meeting for next Tuesday" and "Book an appointment next week" would have very similar embeddings, despite using different words.

The Encoder​

The Encoder object converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).

We recommend you first try bge-small-en-v1.5-q8_0.gguf.

import { Encoder } from "react-native-nobodywho";

const encoder = await Encoder.fromPath({
  modelPath: "/path/to/embedding-model.gguf",
});
const embedding = await encoder.encode("What is the weather like?");
console.log(`Vector with ${embedding.length} dimensions`);

The resulting embedding is an array of numbers (typically 384 or 768 dimensions depending on the model).

Comparing Embeddings​

To measure how similar two pieces of text are, compare their embeddings using cosine similarity:

import { Encoder, cosineSimilarity } from "react-native-nobodywho";

const encoder = await Encoder.fromPath({
  modelPath: "/path/to/embedding-model.gguf",
});

const query = await encoder.encode("How do I reset my password?");
const doc1 = await encoder.encode(
  "You can reset your password in the account settings",
);
const doc2 = await encoder.encode(
  "The password requirements include 8 characters minimum",
);

const similarity1 = cosineSimilarity(query, doc1);
const similarity2 = cosineSimilarity(query, doc2);

console.log(`Document 1 similarity: ${similarity1.toFixed(3)}`); // Higher score
console.log(`Document 2 similarity: ${similarity2.toFixed(3)}`); // Lower score

Cosine similarity returns a value between -1 and 1, where 1 means identical meaning and -1 means opposite meaning.

The CrossEncoder for Better Ranking​

While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents to determine how well the document answers the query.

The key difference is that embeddings compare vector similarity, while cross-encoders understand the relationship between query and document, at a potentially larger computation cost.

Why CrossEncoder Matters​

Consider this example:

Query: "What are the office hours for customer support?"
Documents: [
    "Customer asked: What are the office hours for customer support?",
    "Support team responds: Our customer support is available Monday-Friday 9am-5pm EST",
    "Note: Weekend support is not available at this time"
]

Using embeddings alone, the first document scores highest (most similar to the query) even though it provides no useful information. A cross-encoder correctly identifies that the second document actually answers the question.

Using CrossEncoder​

import { CrossEncoder } from "react-native-nobodywho";

// Download a reranking model like bge-reranker-v2-m3-Q8_0.gguf
const crossencoder = await CrossEncoder.fromPath({
  modelPath: "/path/to/reranker-model.gguf",
});

const query = "How do I install Python packages?";
const documents = [
  "Someone previously asked about Python packages",
  "Use pip install package-name to install Python packages",
  "Python packages are not included in the standard library",
];

// Get relevance scores for each document
const scores = await crossencoder.rank(query, documents);
console.log(scores); // [0.23, 0.89, 0.45] - second doc scores highest

Automatic Sorting​

For convenience, use rankAndSort to get documents sorted by relevance:

// Returns list of [document, score] pairs, sorted by score
const rankedDocs = await crossencoder.rankAndSort(query, documents);

for (const [doc, score] of rankedDocs) {
  console.log(`[${score.toFixed(3)}] ${doc}`);
}

This returns documents ordered from most to least relevant.

Building a RAG System​

Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.

Here's a complete example building a customer service assistant with access to company policies:

import { Chat, Tool, CrossEncoder } from "react-native-nobodywho";

// Initialize the cross-encoder for document ranking
const crossencoder = await CrossEncoder.fromPath({
  modelPath: "/path/to/reranker-model.gguf",
});

// Your knowledge base
const knowledge = [
  "Our company offers a 30-day return policy for all products",
  "Free shipping is available on orders over $50",
  "Customer support is available via email and phone",
  "We accept credit cards, PayPal, and bank transfers",
  "Order tracking is available through your account dashboard",
];

// Create a tool that searches the knowledge base
const searchKnowledgeTool = new Tool({
  name: "search_knowledge",
  description: "Search the knowledge base for relevant information",
  parameters: [
    { name: "query", type: "string", description: "The search query" },
  ],
  call: async (query: string) => {
    const ranked = await crossencoder.rankAndSort(query, knowledge);
    const topDocs = ranked
      .slice(0, 3)
      .map(([doc]) => doc);
    return topDocs.join("\n");
  },
});

// Create a chat with access to the knowledge base
const chat = await Chat.fromPath({
  modelPath: "/path/to/model.gguf",
  systemPrompt:
    "You are a customer service assistant. Use the search_knowledge tool to find relevant information from our policies before answering customer questions.",
  tools: [searchKnowledgeTool],
});

// The chat will automatically search the knowledge base when needed
const response = await chat.ask("What is your return policy?").completed();
console.log(response);

The LLM will call the search_knowledge tool, receive the most relevant documents, and use them to generate an accurate answer.

For Embeddings​

For Cross-Encoding (Reranking)​

Best Practices​

Precompute embeddings: If you have a fixed knowledge base, generate embeddings once and reuse them. Don't re-encode the same documents repeatedly.

Use embeddings for filtering: When working with large document collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank them.

Limit cross-encoder inputs: Cross-encoders are more expensive than embeddings. Don't pass thousands of documents to rank() - filter first with embeddings.