Build a Private RAG Assistant with Ollama and Qdrant
Many teams want an AI assistant that answers questions about their own documents, but cannot send those documents to a third-party API. Retrieval-augmented generation (RAG) with a local model solves this. I built one for a multi-tenant SaaS platform using Ollama and the Qdrant vector database, and this post explains the architecture and the pieces that matter.
Key takeaways
- RAG means: find the most relevant text first, then ask the model to answer using only that text.
- Ollama runs open models locally, and Qdrant stores and searches the embeddings.
- Chunking quality and tenant isolation matter more than the model you pick.
- Index on a schedule, and always return sources with the answer.
How RAG works
A plain language model only knows what it was trained on. RAG adds a retrieval step:
- Index. Split your documents into chunks, turn each chunk into an embedding (a list of numbers that captures meaning) and store it in a vector database.
- Retrieve. When a user asks a question, embed the question and find the closest chunks.
- Generate. Send the question and those chunks to the model with an instruction to answer from them.
The stack
- Ollama serves models over a local HTTP API. I used Qwen 2.5 for answers and an embedding model such as
nomic-embed-textfor vectors. - Qdrant stores vectors and supports filtering by metadata, which is how you keep tenants apart.
- A Node.js worker extracts text from files, chunks it and writes to Qdrant.
Step 1: create a collection
The vector size must match your embedding model. For nomic-embed-text it is 768.
PUT http://localhost:6333/collections/knowledge
{
"vectors": { "size": 768, "distance": "Cosine" }
}Step 2: embed and store chunks
async function embed(text) {
const res = await fetch("http://localhost:11434/api/embed", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ model: "nomic-embed-text", input: text }),
});
const data = await res.json();
return data.embeddings[0];
}
async function indexChunk({ id, text, tenantId, source }) {
await fetch("http://localhost:6333/collections/knowledge/points", {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
points: [{ id, vector: await embed(text), payload: { text, tenantId, source } }],
}),
});
}Step 3: retrieve and answer
async function ask(question, tenantId) {
const vector = await embed(question);
const search = await fetch("http://localhost:6333/collections/knowledge/points/search", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
vector,
limit: 5,
with_payload: true,
filter: { must: [{ key: "tenantId", match: { value: tenantId } }] },
}),
});
const { result } = await search.json();
const context = result.map((hit) => hit.payload.text).join("\n---\n");
const chat = await fetch("http://localhost:11434/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "qwen2.5",
stream: false,
messages: [
{
role: "system",
content:
"Answer only from the provided context. If it is not there, say you do not know.",
},
{ role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
],
}),
});
const data = await chat.json();
return { answer: data.message.content, sources: result.map((hit) => hit.payload.source) };
}The filter on tenantId is the important line. It guarantees one customer's question can never retrieve another customer's documents.
What makes it work in production
- Chunk by meaning, not by length. Split on headings and paragraphs, keep chunks to a few hundred words and overlap them slightly so answers are not cut in half.
- Extract text properly. PDF and DOCX files need real text extraction before chunking. Re-extract when a file changes.
- Index on a schedule. A nightly job over files, knowledge-base articles and announcements keeps the assistant current without slowing users down.
- Always return sources. Showing which documents an answer came from builds trust and makes mistakes easy to spot.
- Tell the model to admit gaps. An instruction to answer only from context reduces made-up answers.
When to use a hosted model instead
Local models trade some answer quality and speed for privacy and cost control. If your data is not sensitive, a hosted model may be simpler. If it is, the same pipeline works with either, because only the chat call changes.
I use this pattern in Dashify, and I offer AI and LLM integration for products that need it.
- RAG
- Ollama
- Qdrant
- LLM
- AI assistant