How to Build a RAG Pipeline: A Step-by-Step Tutorial for 2026

Step 1. What you are building, and what you need first

Retrieval-augmented generation is not a model feature you switch on. It is a pipeline: split documents into chunks, embed each chunk, store the vectors, find the closest ones to a question, paste them into a prompt, and let a model write the answer from them. Almost every RAG failure happens in the joins between those steps, not inside any one of them. For the conceptual grounding first, start with what RAG is and why it exists.

By the end of Step 6 you will have a working pipeline over a folder of Markdown files that answers questions with citations. Steps 7 to 9 turn that demo into something you can point real users at.

Prerequisites

• Python 3.10 or newer, required by both the text splitter and the local Voyage models.
• A Voyage AI API key for embeddings and reranking, and an Anthropic API key for generation.
• A folder of documents you actually care about. Resist the toy corpus; retrieval problems only appear on real text with real ambiguity.

Every version below was checked on PyPI in July 2026. Pin them, because all four libraries move fast.

~~~bash

python -m venv .venv

source .venv/bin/activate # Windows: .venv\Scripts\activate

pip install anthropic==0.119.0 voyageai==0.5.0 chromadb==1.5.9 langchain-text-splitters==1.1.2

export ANTHROPIC_API_KEY=sk-ant-...

export VOYAGE_API_KEY=pa-...

~~~

Both SDKs read those environment variables automatically, so no key ever needs to appear in your source.

Step 2. Load and chunk the documents

Chunking is the decision that quietly determines your ceiling. Retrieval can only return chunks you created, so a bad split puts the right answer permanently out of reach no matter how good your embedding model is.

~~~python

from pathlib import Path

from langchain_text_splitters import RecursiveCharacterTextSplitter

DOCS_DIR = Path('docs')

splitter = RecursiveCharacterTextSplitter(

chunk_size=1200,

chunk_overlap=150,

separators=['\n\n', '\n', '. ', ' ', ''],

length_function=len,

)

paths = sorted(DOCS_DIR.rglob('*.md'))

records = []

for path in paths:

text = path.read_text(encoding='utf-8')

for i, chunk in enumerate(splitter.split_text(text)):

records.append({

'id': f'{path.stem}-{i}',

'text': chunk,

'metadata': {

'source': str(path),

'title': path.stem,

'chunk_index': i,

},

})

print(f'{len(records)} chunks from {len(paths)} files')

~~~

RecursiveCharacterTextSplitter walks the separator list in order, trying paragraph breaks before line breaks before sentences before words, so it keeps related text together where it can. Its default list is paragraph, line, space, empty string; adding a sentence separator gives it one more sensible place to cut.

The honest version of the chunk size tradeoff. There is no correct chunk size, only a tradeoff you are choosing, usually without measuring it:

• Small chunks (200 to 500 characters) produce precise embeddings, because a short passage is about one thing. They also strand context. A chunk saying this limit does not apply to enterprise plans is useless when the sentence naming the limit sits in the previous chunk.
• Large chunks (2000 characters and up) carry their own context, but one vector must then represent the average meaning of everything inside. Mix three topics into a chunk and its embedding lands in the empty space between all three, matching none well.
• Overlap is insurance against cutting mid-thought, and it is not free: 15% overlap means roughly 15% more rows, 15% more embedding spend, and near-duplicate hits eating slots in your top-k. Deduplicate by source and position before building the prompt.

Roughly 1000 to 1500 characters with 10 to 15% overlap is a defensible start for prose. Treat it as the number you change once Step 8 gives you evidence.

Step 3. Embed the chunks

An embedding turns text into a fixed-length list of floats where distance approximates meaning. If that sentence is doing unexplained work, read our guide to embedding models first.

This guide uses Voyage AI, the provider Anthropic points to in its own docs, since Anthropic does not serve an embeddings endpoint. The Voyage 4 family launched on 15 January 2026 as voyage-4-large, voyage-4, voyage-4-lite and the open-weight voyage-4-nano, all with a 32,000-token context window and Matryoshka-trained output dimensions of 256, 512, 1024 (the default) and 2048.

~~~python

import voyageai

vo = voyageai.Client() # reads VOYAGE_API_KEY

EMBED_MODEL = 'voyage-4'

BATCH_SIZE = 128

def embed_documents(texts):

vectors = []

for start in range(0, len(texts), BATCH_SIZE):

batch = texts[start:start + BATCH_SIZE]

result = vo.embed(texts=batch, model=EMBED_MODEL, input_type='document')

vectors.extend(result.embeddings)

return vectors

def embed_query(text):

result = vo.embed(texts=[text], model=EMBED_MODEL, input_type='query')

return result.embeddings[0]

vectors = embed_documents([r['text'] for r in records])

print(len(vectors), len(vectors[0])) # rows, dimensions (1024 by default)

~~~

Three details that bite people:

• input_type is asymmetric and it matters. Embed stored passages with document and the incoming question with query. Voyage prepends different internal prompts for each. Getting this backwards degrades results silently, with no error.
• Batch, but stay inside the limits. Voyage accepts up to 1,000 texts per request, with a per-request token ceiling that varies by model: 320K for voyage-4, 120K for voyage-4-large, 1M for voyage-4-lite. A batch of 128 chunks sits comfortably inside all of them, and each response carries a total_tokens field so you can log real spend instead of guessing.
• The embedding model is a one-way door. Vectors from different models are not comparable, so changing model or output dimension later means re-embedding the whole corpus. Voyage 4 is unusual here: all four family members share one embedding space, so you can pair a cheaper model for bulk documents with a stronger one for queries.

Step 4. Store the vectors

Chroma is the primary path here because it runs as an embedded library with no server to operate, which keeps this tutorial about RAG rather than about infrastructure.

~~~python

import chromadb

client = chromadb.PersistentClient(path='./chroma')

collection = client.create_collection(

name='handbook',

configuration={'hnsw': {'space': 'cosine'}},

)

ADD_BATCH = 500

for start in range(0, len(records), ADD_BATCH):

window = records[start:start + ADD_BATCH]

collection.add(

ids=[r['id'] for r in window],

documents=[r['text'] for r in window],

embeddings=vectors[start:start + ADD_BATCH],

metadatas=[r['metadata'] for r in window],

)

print(f'indexed {len(records)} chunks')

~~~

Notes on the calls above. PersistentClient writes to a directory on disk and reloads it next run; the in-memory chromadb.Client() from the quickstart loses everything when the process exits, which is a common cause of a mysteriously empty index. Chroma defaults to squared L2, so setting space to cosine is deliberate: Chroma's own docs recommend cosine for text embeddings. Because you pass both documents and embeddings, Chroma stores them as-is rather than re-embedding. Use create_collection on the first run and get_or_create_collection to make the script re-runnable.

The vector database comparison goes deeper on operational differences. For a first pipeline, the store is rarely your bottleneck.

Step 5. Retrieve

~~~python

def retrieve(question, k=8, where=None):

query_vector = embed_query(question)

kwargs = {

'query_embeddings': [query_vector],

'n_results': k,

'include': ['documents', 'metadatas', 'distances'],

}

if where:

kwargs['where'] = where

result = collection.query(**kwargs)

hits = []

for doc, meta, dist in zip(

result['documents'][0],

result['metadatas'][0],

result['distances'][0],

):

hits.append({'text': doc, 'metadata': meta, 'distance': dist})

return hits

for hit in retrieve('what is the refund window for annual plans?', k=5):

print(round(hit['distance'], 4), hit['metadata']['source'])

~~~

Chroma returns results grouped per query, so a single question yields nested lists and you index with [0] everywhere. Asking for distances explicitly is worth the extra characters: ranking without visible scores is debugging blindfolded.

On the similarity metric. Voyage embeddings are normalised to unit length, so cosine similarity, dot product and Euclidean distance produce identical rankings. The choice changes the numbers, not the order. Lower distance is better. Resist turning distances into a confidence percentage in your UI: the useful threshold is corpus-specific and only findable by inspecting your own scored examples.

On top-k. Small k risks missing the answer; large k floods the prompt with near-misses the model must then ignore. Retrieve wide and narrow later: pull 30 to 50 candidates here and let Step 7 cut them to the handful the model reads.

Step 6. Generate the answer

~~~python

import anthropic

llm = anthropic.Anthropic() # reads ANTHROPIC_API_KEY

SYSTEM = (

'You answer questions using only the numbered sources supplied by the user. '

'Cite the sources you relied on inline as [1], [2] and so on. '

'If the sources do not contain the answer, say so plainly. Never fill a gap '

'from your own knowledge.'

)

def build_context(hits):

blocks = []

for i, hit in enumerate(hits, start=1):

source = hit['metadata']['source']

body = hit['text']

blocks.append(f'[{i}] source: {source}\n{body}')

return '\n\n'.join(blocks)

def answer(question, k=8):

hits = retrieve(question, k=k)

prompt = f'Sources:\n\n{build_context(hits)}\n\nQuestion: {question}'

response = llm.messages.create(

model='claude-opus-4-8',

max_tokens=2000,

system=SYSTEM,

thinking={'type': 'adaptive'},

messages=[{'role': 'user', 'content': prompt}],

)

text = ''.join(b.text for b in response.content if b.type == 'text')

return text, hits

reply, sources = answer('what is the refund window for annual plans?')

print(reply)

~~~

Claude Opus 4.8 has a 1M-token context window and a 128K max output, per Anthropic's model docs. Two API details are easy to get wrong. First, response.content is a list of typed blocks, not a string, so filter on block type before touching .text; with adaptive thinking on, the list also contains thinking blocks. Second, on Opus 4.8 thinking is off unless you ask for it. For long answers use llm.messages.stream(...) and read the final message from the stream, which avoids HTTP timeouts at large max_tokens values.

The system prompt does more work than it looks. Numbering the sources gives the model something concrete to cite, and explicit permission to say I do not know is the cheapest defence against confident invention.

Step 7. Make it good

A baseline pipeline is right often enough to demo and wrong often enough to be dangerous. Four upgrades, in the order they usually pay off.

Reranking (fixes: the right chunk is retrieved but ranked eighth). A cross-encoder reads the query and each candidate together, so it judges relevance far more precisely than a dot product between two independently computed vectors. Retrieve wide, rerank, keep a few.

~~~python

def retrieve_reranked(question, k_dense=40, k_final=6):

candidates = retrieve(question, k=k_dense)

reranked = vo.rerank(

query=question,

documents=[c['text'] for c in candidates],

model='rerank-2.5',

top_k=k_final,

)

results = []

for item in reranked.results:

hit = dict(candidates[item.index])

hit['relevance_score'] = item.relevance_score

results.append(hit)

return results

~~~

Voyage lists rerank-2.5 and the faster rerank-2.5-lite, both with 32,000-token context. Each result carries index (the position in the list you sent), document and relevance_score. This is the highest-leverage single change in most pipelines, and the one most often skipped.

Hybrid search (fixes: exact terms that embeddings blur). Dense vectors are weak on rare tokens: error codes, SKUs, surnames, version strings. Keyword retrieval is strong on exactly those and weak on paraphrase. Run both and fuse the rankings rather than the scores, since scores from different systems are not comparable:

~~~python

def reciprocal_rank_fusion(ranked_id_lists, k=60):

scores = {}

for ranked_ids in ranked_id_lists:

for rank, doc_id in enumerate(ranked_ids, start=1):

scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)

return sorted(scores, key=scores.get, reverse=True)

~~~

That function is deliberately store-agnostic: hand it the ranked ID lists from your dense and keyword searches and it returns one merged ordering. Chroma supports full-text filtering today via where_document with the $contains, $not_contains, $regex and $not_regex operators, and has since added sparse vector search with BM25 plus built-in reciprocal rank fusion. Check Chroma's current docs for exact class names before wiring that path; it is newer than the rest of this pipeline.

Metadata filtering (fixes: correct answer, wrong document). Vector similarity has no idea the 2023 policy was superseded. Filters do:

~~~python

hits = retrieve(

'what is the refund window?',

k=8,

where={'title': 'billing-policy-2026'},

)

~~~

Store everything you might ever filter on at ingest time; backfilling metadata means re-indexing.

Query rewriting (fixes: follow-ups and terse queries). What about annual? is unretrievable on its own. Rewrite it into a standalone query first, using a small fast model so the extra hop costs almost nothing:

~~~python

REWRITE = (

'Rewrite the user message as a single standalone search query. '

'Resolve pronouns and implied context from the conversation. '

'Return only the query, with no preamble.'

)

def rewrite_query(history, question):

response = llm.messages.create(

model='claude-haiku-4-5',

max_tokens=200,

system=REWRITE,

messages=[{'role': 'user', 'content': f'{history}\n\nMessage: {question}'}],

)

return ''.join(b.text for b in response.content if b.type == 'text').strip()

~~~

Haiku 4.5 costs 1 dollar per million input tokens against 5 for Opus 4.8 and does not support adaptive thinking, so the thinking parameter is correctly absent here.

One more cost lever: Anthropic's prompt caching bills cache reads at 0.1x the base input rate and 5-minute writes at 1.25x, with a 1,024-token minimum cacheable prefix on Opus 4.8. Retrieved chunks change every query, so cache the stable system prompt and place retrieved context after it.

Step 8. Evaluate it

Until you measure, every change is superstition. Build a golden set of 30 to 50 real questions with known answers and the document each answer lives in. Thirty catches regressions but will not resolve small differences, so treat close scores as ties.

RAG evaluation splits into retrieval quality and generation quality, and the split tells you where to work:

Ragas 0.4.3 implements all four. Its current interface takes an evaluator LLM and scores one example directly:

~~~python

from ragas.metrics.collections import Faithfulness

scorer = Faithfulness(llm=evaluator_llm) # see the Ragas docs for llm_factory setup

result = scorer.score(

user_input='what is the refund window for annual plans?',

response=reply,

retrieved_contexts=[h['text'] for h in sources],

)

print(result.value)

~~~

Ragas supports OpenAI, Anthropic and Google clients through its llm_factory helper; configure yours from its docs rather than copying a provider out of a tutorial. These metrics are themselves LLM-judged, so they are noisy, they cost money, and you should sanity-check them against human judgement on a few examples before trusting the aggregate. Our guide to AI evals covers building that harness properly. Run the set before and after every change: retrieval metrics move first, generation metrics follow.

Step 9. Common failure modes

• Bad chunking. Symptom: answers that trail off, or cite a chunk holding half the story. Cause: a splitter cutting through tables, lists or code. Fix: split on structure first, cap size second.
• No reranker. Symptom: the right document sits in the top 20 but never the top 3. Cause: single-vector similarity is a coarse signal. Fix: retrieve 40, rerank to 6.
• Stale index. Symptom: confident, well-cited answers to last quarter's policy. Cause: nothing re-embeds changed files. Fix: store a content hash per chunk and re-embed only what moved, on a named schedule.
• Context overflow. Symptom: the model ignores material demonstrably in the prompt. Cause: too many chunks, or duplicates crowding out the answer. A 1M-token window is not permission to fill it. Fix: fewer, deduplicated chunks after reranking.
• No citations. Symptom: you cannot tell whether an answer came from your documents or the model's memory. Cause: nothing in the prompt requires attribution. Fix: number sources, demand inline citations, log retrieved IDs with every response.
• Silent retrieval failure. Symptom: fluent nonsense. Cause: retrieval returned nothing useful and the model answered anyway. Fix: compare the top distance against a threshold calibrated on your data and refuse to answer below it.

Most of these look identical to a user, which is why systematic debugging of irrelevant results beats guessing.

How This Guide Was Built

Every import path, class name, method signature and parameter here was checked against primary documentation in July 2026. Nothing was written from memory, and nothing here has been run as a production system or benchmarked by us.

• anthropic 0.119.0 (PyPI, 23 July 2026). Message shape, adaptive thinking and content-block handling checked against Anthropic's official Python SDK documentation.
• voyageai 0.5.0 (PyPI, 10 July 2026). Client, embed and rerank signatures, input_type values, model names, per-request limits and unit-normalisation checked against the Voyage embeddings, reranker, pricing and FAQ pages, plus the Voyage 4 launch post of 15 January 2026.
• chromadb 1.5.9 (PyPI). PersistentClient, create_collection, the hnsw space setting, add and query parameters, the nested result shape and where_document operators checked against the Chroma docs on clients, collection configuration, adding data, query and get and full-text search, plus the sparse vector search announcement.
• langchain-text-splitters 1.1.2 (PyPI, 16 April 2026). Import path, constructor arguments and split_text checked against LangChain's splitter documentation.
• ragas 0.4.3 (PyPI). Metric names and the current scorer interface checked against the Ragas metrics reference and faithfulness pages.
• Claude facts (model ID, 1M context window, 128K max output, 5 and 25 dollars per million tokens for Opus 4.8, 1 and 5 for Haiku 4.5) from the Anthropic models overview; caching multipliers and the 1,024-token minimum from the prompt caching docs. Anthropic's own embeddings page is why this guide uses a third-party embedding provider.

Anything that could not be verified from primary documentation was left out rather than guessed. That is why the Ragas evaluator-LLM setup is referenced instead of reproduced, and why Chroma's newer sparse-vector API is described but not coded.

Conclusion

A RAG pipeline is a few hundred lines of glue, and the glue is where the quality lives. Get something working end to end in an afternoon with the versions pinned above, then spend your real effort on what moves the numbers: chunking that respects document structure, a reranker between retrieval and generation, metadata you can filter on, and a golden set that tells you whether any of it helped.

Steps 2 through 6 are deliberately boring, and that is the point. Boring pipelines are debuggable, and a debuggable pipeline you measure beats a clever one you do not.

This is an editorial synthesis of official library documentation, package registries, and vendor pricing pages; see our [methodology](/methodology). Verify current details with each project.