Skip to main content

Embeddings & RAG

When you want your LLM to search through documents, understand semantic similarity, or build retrieval-augmented generation (RAG) systems, you'll need embeddings and cross-encoders.

Understanding Embeddings​

Embeddings convert text into vectors (lists of numbers) that capture semantic meaning. Texts with similar meanings have similar vectors, even if they use different words.

For example, "Schedule a meeting for next Tuesday" and "Book an appointment next week" would have very similar embeddings, despite using different words.

The Encoder​

The Encoder object converts text into embedding vectors. You'll need a specialized embedding model (different from chat models).

We recommend you first try bge-small-en-v1.5-q8_0.gguf.

from nobodywho import Encoder

encoder = Encoder('./embedding-model.gguf')
embedding = encoder.encode("What is the weather like?")
print(f"Vector with {len(embedding)} dimensions")

The resulting embedding is a list of floats (typically 384 or 768 dimensions depending on the model).

Comparing Embeddings​

To measure how similar two pieces of text are, compare their embeddings using cosine similarity:

from nobodywho import Encoder, cosine_similarity

encoder = Encoder('./embedding-model.gguf')

query = encoder.encode("How do I reset my password?")
doc1 = encoder.encode("You can reset your password in the account settings")
doc2 = encoder.encode("The password requirements include 8 characters minimum")

similarity1 = cosine_similarity(query, doc1)
similarity2 = cosine_similarity(query, doc2)

print(f"Document 1 similarity: {similarity1:.3f}")  # Higher score
print(f"Document 2 similarity: {similarity2:.3f}")  # Lower score

Cosine similarity returns a value between -1 and 1, where 1 means identical meaning and -1 means opposite meaning.

Practical Example: Finding Relevant Documents​

from nobodywho import Encoder, cosine_similarity

encoder = Encoder('./embedding-model.gguf')

# Your knowledge base
documents = [
    "Python supports multiple programming paradigms including object-oriented and functional",
    "JavaScript is primarily used for web development and runs in browsers",
    "SQL is a domain-specific language for managing relational databases",
    "Git is a version control system for tracking changes in source code"
]

# Pre-compute document embeddings
doc_embeddings = [encoder.encode(doc) for doc in documents]

# Search query
query = "What language should I use for database queries?"
query_embedding = encoder.encode(query)

# Find the most relevant document
similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
best_idx = similarities.index(max(similarities))

print(f"Most relevant: {documents[best_idx]}")
print(f"Similarity score: {similarities[best_idx]:.3f}")

The CrossEncoder for Better Ranking​

While embeddings work well for initial filtering, cross-encoders provide more accurate relevance scoring. They directly compare a query against documents to determine how well the document answers the query.

The key difference: embeddings compare vector similarity, while cross-encoders understand the relationship between query and document.

Why CrossEncoder Matters​

Consider this example:

Query: "What are the office hours for customer support?"
Documents: [
    "Customer asked: What are the office hours for customer support?",
    "Support team responds: Our customer support is available Monday-Friday 9am-5pm EST",
    "Note: Weekend support is not available at this time"
]

Using embeddings alone, the first document scores highest (most similar to the query) even though it provides no useful information. A cross-encoder correctly identifies that the second document actually answers the question.

Using CrossEncoder​

from nobodywho import CrossEncoder

# Download a reranking model like bge-reranker-v2-m3-Q8_0.gguf
crossencoder = CrossEncoder('./reranker-model.gguf')

query = "How do I install Python packages?"
documents = [
    "Someone previously asked about Python packages",
    "Use pip install package-name to install Python packages",
    "Python packages are not included in the standard library"
]

# Get relevance scores for each document
scores = crossencoder.rank(query, documents)
print(scores)  # [0.23, 0.89, 0.45] - second doc scores highest

Automatic Sorting​

For convenience, use rank_and_sort to get documents sorted by relevance:

# Returns list of (document, score) tuples, sorted by score
ranked_docs = crossencoder.rank_and_sort(query, documents)

for doc, score in ranked_docs:
    print(f"[{score:.3f}] {doc}")

This returns documents ordered from most to least relevant.

Building a RAG System​

Retrieval-Augmented Generation (RAG) combines document search with LLM generation. The LLM uses retrieved documents to ground its responses in your knowledge base.

Here's a complete example building a customer service assistant with access to company policies:

from nobodywho import Chat, CrossEncoder

# Initialize the cross-encoder for document ranking
crossencoder = CrossEncoder('./reranker-model.gguf')

# Your knowledge base
knowledge = [
    "Our company offers a 30-day return policy for all products",
    "Free shipping is available on orders over $50",
    "Customer support is available via email and phone",
    "We accept credit cards, PayPal, and bank transfers",
    "Order tracking is available through your account dashboard"
]

# Create a tool that searches the knowledge base
from nobodywho import tool

@tool(description="Search the knowledge base for relevant information")
def search_knowledge(query: str) -> str:
    # Rank all documents by relevance to the query
    ranked = crossencoder.rank_and_sort(query, knowledge)
    
    # Return top 3 most relevant documents
    top_docs = [doc for doc, score in ranked[:3]]
    return "\n".join(top_docs)

# Create a chat with access to the knowledge base
chat = Chat(
    './model.gguf',
    system_prompt="You are a customer service assistant. Use the search_knowledge tool to find relevant information from our policies before answering customer questions.",
    tools=[search_knowledge]
)

# The chat will automatically search the knowledge base when needed
response = chat.ask("What is your return policy?").completed()
print(response)

The LLM will call the search_knowledge tool, receive the most relevant documents, and use them to generate an accurate answer.

Async API​

For non-blocking operations, use EncoderAsync and CrossEncoderAsync:

import asyncio
from nobodywho import EncoderAsync, CrossEncoderAsync

async def main():
    encoder = EncoderAsync('./embedding-model.gguf')
    crossencoder = CrossEncoderAsync('./reranker-model.gguf')
    
    # Generate embeddings asynchronously
    embedding = await encoder.encode("What is the weather?")
    
    # Rank documents asynchronously
    query = "What is our refund policy?"
    docs = ["Refunds processed within 5-7 business days", "No refunds on sale items", "Contact support to initiate refund"]
    ranked = await crossencoder.rank_and_sort(query, docs)
    
    for doc, score in ranked:
        print(f"[{score:.3f}] {doc}")

asyncio.run(main())

For Embeddings​

For Cross-Encoding (Reranking)​

Best Practices​

Precompute embeddings: If you have a fixed knowledge base, generate embeddings once and reuse them. Don't re-encode the same documents repeatedly.

Use embeddings for filtering: When working with large document collections (1000+ documents), use embeddings to narrow down to the top 50-100 candidates, then use a cross-encoder to rerank them.

Limit cross-encoder inputs: Cross-encoders are more expensive than embeddings. Don't pass thousands of documents to rank() - filter first with embeddings.

Choose appropriate context size: The n_ctx parameter (default 2048) should match your model's recommended context size. Check the model documentation.

# For longer documents, increase context size
encoder = Encoder('./embedding-model.gguf', n_ctx=4096)
crossencoder = CrossEncoder('./reranker-model.gguf', n_ctx=4096)

Complete RAG Example​

Here's a full example showing a two-stage retrieval system:

from nobodywho import Chat, Encoder, CrossEncoder, cosine_similarity, tool

# Initialize models
encoder = Encoder('./embedding-model.gguf')
crossencoder = CrossEncoder('./reranker-model.gguf')

# Large knowledge base
knowledge_base = [
    "Python 3.11 introduced performance improvements through faster CPython",
    "The Django framework is used for building web applications",
    "NumPy provides support for large multi-dimensional arrays",
    "Pandas is the standard library for data manipulation and analysis",
    # ... 100+ more documents
]

# Precompute embeddings for all documents
doc_embeddings = [encoder.encode(doc) for doc in knowledge_base]

@tool(description="Search the knowledge base for information relevant to the query")
def search(query: str) -> str:
    # Stage 1: Fast filtering with embeddings
    query_embedding = encoder.encode(query)
    similarities = [
        (doc, cosine_similarity(query_embedding, doc_emb))
        for doc, doc_emb in zip(knowledge_base, doc_embeddings)
    ]
    # Get top 20 candidates
    candidates = sorted(similarities, key=lambda x: x[1], reverse=True)[:20]
    candidate_docs = [doc for doc, _ in candidates]
    
    # Stage 2: Precise ranking with cross-encoder
    ranked = crossencoder.rank_and_sort(query, candidate_docs)
    
    # Return top 3 most relevant
    top_results = [doc for doc, score in ranked[:3]]
    return "\n---\n".join(top_results)

# Create RAG-enabled chat
chat = Chat(
    './model.gguf',
    system_prompt="You are a technical documentation assistant. Always use the search tool to find relevant information before answering programming questions.",
    tools=[search]
)

# The chat automatically searches and uses retrieved documents
response = chat.ask("What Python libraries are best for data analysis?").completed()
print(response)

This two-stage approach combines the speed of embeddings with the accuracy of cross-encoders, making it efficient even for large knowledge bases.