Technology Guide · 2025

RAG System Explained

Q: What is RAG (Retrieval-Augmented Generation)?

Retrieval-Augmented Generation (RAG) is an AI architecture that combines a retrieval system with a large language model. Instead of relying solely on information baked into the model during training, a RAG system retrieves relevant documents from an external knowledge base at query time and passes them to the LLM as context. This enables the LLM to answer questions about private, recent, or specialised data it was never trained on.

Q: What is a vector database and why is it used in RAG?

A vector database stores data as high-dimensional numerical vectors (embeddings) and enables fast similarity search — finding the most semantically similar documents to a query, not just keyword matches. In a RAG system, documents are converted to embeddings and stored in a vector database. When a user asks a question, the question is also converted to an embedding, and the vector database retrieves the most similar document chunks by comparing embedding distances.

Q: What is the difference between RAG and fine-tuning?

Fine-tuning trains the LLM itself on your specific data, baking knowledge into the model weights. RAG retrieves your data at query time and passes it as context without modifying the model. RAG is better for dynamic or frequently updated data, large knowledge bases, and when you need source citations. Fine-tuning is better for teaching the model a specific style, format, or domain-specific reasoning pattern. Most production AI systems use RAG for knowledge and fine-tuning for behaviour.

Q: What is chunking in a RAG system?

Chunking is the process of splitting large documents into smaller pieces before converting them to embeddings. Since LLMs have context window limits and embedding models work best on shorter texts, documents must be divided into chunks of typically 200 to 1000 tokens. The chunking strategy significantly impacts retrieval quality — chunks that are too small lose context, chunks that are too large dilute relevance.

Q: Which vector database should I use for a RAG system?

For most SaaS applications and early-stage RAG systems, pgvector (PostgreSQL extension) is the recommended starting point — it runs inside your existing Supabase database at no extra cost. For high-scale production systems with millions of vectors, Pinecone (managed, simple API) or Qdrant (open-source, self-hostable) are the leading purpose-built options. Weaviate is preferred for multi-modal RAG systems.

Q: How do you evaluate a RAG system?

RAG systems are evaluated across three dimensions: retrieval quality (did the system retrieve the right documents?), answer faithfulness (is the answer grounded in the retrieved documents?), and answer relevance (does the answer actually address the question?). Tools like RAGAS provide automated evaluation metrics. Human evaluation of a golden test set is also essential, particularly for domain-specific knowledge bases.

A complete guide to Retrieval-Augmented Generation — what RAG is, how it works step by step, vector databases, embeddings, chunking strategies, retrieval methods, evaluation, and real-world use cases.

▸ Direct Answer

RAG (Retrieval-Augmented Generation) is an AI architecture that lets an LLM answer questions about your private or specialised data by retrieving relevant documents at query time and passing them as context. Instead of retraining the model, you build a searchable knowledge base from your documents, retrieve the most relevant chunks when a user asks a question, and pass them to Claude or GPT-4o to generate a grounded answer with citations.

RAG vs Fine-Tuning

Better for dynamic data

Typical Latency

1–3 seconds E2E

Best Vector DB

pgvector (start here)

Best LLM

Claude Sonnet

RAG vs Fine-Tuning

Why RAG Instead of Fine-Tuning?

Both approaches enable LLMs to work with domain-specific knowledge. Understanding the tradeoffs is critical to choosing the right architecture for your use case.

🔍

RAG

Retrieval-Augmented Generation

Use When

✓Data changes frequently — documents are updated, added, or removed

✓Knowledge base is large — thousands to millions of documents

✓You need citations and source transparency in answers

✓You want to add new knowledge without retraining

✓Multiple knowledge bases needed for different users

Avoid When

✗Teaching the model a very specific writing style or tone

✗Memorising a small fixed set of structured facts

✗Reasoning patterns that require deep model understanding

🧠

Fine-Tuning

Training the Model on Your Data

Use When

✓Teaching the model a specific output format or style

✓Domain-specific reasoning that requires deep subject understanding

✓Small, stable, well-structured datasets (under 10,000 examples)

✓Consistent persona or brand voice across all outputs

Avoid When

✗Large or frequently changing knowledge bases

✗When you need source citations for answers

✗When data privacy means you cannot send documents to the LLM

▸ Key Insight

Most production AI systems use both — RAG for knowledge retrieval and fine-tuning for output style and reasoning behaviour. Start with RAG. Add fine-tuning later only if RAG alone cannot achieve the quality you need.

How It Works

RAG Pipeline — Step by Step

A RAG system has two phases: Ingestion (processing documents into a searchable knowledge base) and Retrieval + Generation (answering queries using that knowledge base).

Steps 1–3: Ingestion (run once)Steps 4–6: Query time (run per user question)

📄

01Ingestion

Document Loading & Chunking

Raw documents (PDFs, Word docs, web pages, database records) are loaded, cleaned, and split into smaller chunks. Each chunk becomes one searchable unit in the knowledge base.

▸Load documents from any source — files, URLs, databases, APIs

▸Clean and normalise text — remove headers, footers, formatting artefacts

▸Split into chunks of 200–1000 tokens with overlap (50–100 tokens) to preserve context across boundaries

▸Add metadata to each chunk — source document, page number, section title, date

🔢

02Ingestion

Embedding Generation

Each text chunk is converted into a vector embedding — a list of hundreds of numbers that encodes the semantic meaning of the text in a way that enables similarity comparison.

▸Pass each chunk through an embedding model (OpenAI text-embedding-3-small, or Cohere embed-v3)

▸Model outputs a vector of 768–3072 dimensions representing semantic meaning

▸Semantically similar texts produce vectors that are mathematically close together

▸Store both the raw text and its embedding vector, linked to the source document metadata

🗄️

03Ingestion

Vector Storage

The embeddings and their corresponding text chunks are stored in a vector database that supports fast approximate nearest-neighbour (ANN) search.

▸Insert embeddings into vector store (pgvector, Pinecone, Qdrant, Weaviate)

▸Create a vector index (IVFFlat or HNSW) for fast similarity search at scale

▸Store chunk text and metadata alongside the vector for retrieval

▸Implement update strategy — re-embed and replace when source documents change

❓

04Retrieval

Query Embedding

When a user asks a question, it is converted into an embedding using the same model used for documents. This creates a point in the same vector space as the knowledge base.

▸Receive user query as plain text

▸Pass query through the same embedding model used at ingestion time

▸Generate a query vector of the same dimensions as document vectors

▸Optionally rewrite or expand the query to improve retrieval (HyDE, query expansion)

🔍

05Retrieval

Similarity Search

The query vector is compared against all stored document vectors to find the most semantically similar chunks — the documents most likely to contain the answer.

▸Perform vector similarity search (cosine similarity or dot product) against the vector store

▸Retrieve top-k most similar chunks (typically k = 3–10)

▸Optionally apply metadata filters — date range, document type, user-specific access

▸Optionally combine with keyword search (BM25) for hybrid retrieval

✨

06Generation

Context Assembly & Generation

The retrieved chunks are assembled into a prompt with the user's question and passed to the LLM, which generates a grounded answer based on the provided context.

▸Rank and select the most relevant retrieved chunks to fit within the LLM context window

▸Assemble a prompt: system instructions + retrieved context + user question

▸Call the LLM (Claude, GPT-4o) with the assembled prompt

▸Return the generated answer along with source citations from the retrieved chunks

Chunking

Chunking Strategies

How you split your documents into chunks is one of the most impactful decisions in a RAG system. Poor chunking is the most common reason RAG systems retrieve irrelevant content.

Fixed-Size Chunking

Split documents into chunks of exactly N tokens, with optional overlap between consecutive chunks.

Chunk Size:500–1000 tokens

Overlap:50–100 tokens

BEST FOR

General purpose, simple documents

AVOID

Documents where semantic units span unpredictable lengths

Sentence / Paragraph Chunking

Split at natural sentence or paragraph boundaries to preserve semantic coherence within each chunk.

Chunk Size:Variable (1–5 sentences)

Overlap:1 sentence

BEST FOR

Articles, books, documentation

AVOID

Technical documents with long continuous code blocks

Semantic Chunking

Use an embedding model to detect semantic shifts in the text and split at meaning boundaries rather than character count.

Chunk Size:Variable

Overlap:None needed

BEST FOR

Long documents with distinct topic sections

AVOID

Simple, uniform documents (adds unnecessary complexity)

Hierarchical Chunking

Create chunks at multiple granularities — document summary, section summaries, and paragraph-level chunks. Retrieve at multiple levels.

Chunk Size:Multiple sizes

Overlap:N/A

BEST FOR

Large documents needing both broad context and specific retrieval

AVOID

Simple Q&A use cases (over-engineering)

Retrieval

Retrieval Methods

How you retrieve relevant documents from your knowledge base determines the quality of the answers your RAG system produces. For most production systems, hybrid retrieval is the recommended approach.

Dense Retrieval (Vector Search)

Convert query to embedding, find nearest vectors using cosine similarity or dot product. Captures semantic meaning — 'car' and 'automobile' are understood as related.

Strengths

✓Semantic understanding

✓Handles synonyms and paraphrasing

✓Language-agnostic

Weaknesses

✗Can miss exact keyword matches

✗Embedding model quality matters significantly

Sparse Retrieval (BM25 / Keyword)

Traditional keyword-based retrieval using TF-IDF or BM25 scoring. Finds documents containing the exact terms in the query.

Strengths

✓Exact keyword matching

✓Fast

✓No ML model required

✓Handles product codes, names, IDs well

Weaknesses

✗No semantic understanding

✗Misses synonyms

✗Sensitive to spelling variations

Hybrid Retrieval (Dense + Sparse)

Combine both vector search and keyword search, then merge results using Reciprocal Rank Fusion (RRF) or a learned reranker. Best of both worlds.

Strengths

✓Highest overall retrieval quality

✓Handles both semantic and exact queries

✓Recommended for production

Weaknesses

✗More complex to implement

✗Slightly higher latency

✗Two indices to maintain

Reranking

After initial retrieval, pass top-20 candidates to a cross-encoder reranker model that scores each document-query pair more accurately than the initial embedding search.

Strengths

✓Significantly improves precision

✓Cross-encoder understands query-document relationship

✓Works with any retrieval method

Weaknesses

✗Adds latency (extra model call)

✗Additional cost

✗Only applied to a small candidate set

Vector Databases

Choosing a Vector Database

Start with pgvector. Move to a purpose-built database only when you have validated your RAG system and are scaling beyond 1 million vectors.

pgvector

PostgreSQL Extension

Most SaaS apps, existing Supabase users

Scalability:Up to ~1M vectors comfortably

Cost:Free (runs in your existing DB)

Self-hostable:Yes

✓No additional infrastructure

✓SQL-native queries

✓Supabase integration built-in

✓Full ACID compliance

✗Not purpose-built for vectors

✗Slower than dedicated DBs at very high scale

Pinecone

Managed Vector DB

Production scale, teams wanting managed infra

Scalability:Billions of vectors

Cost:Free tier, then $70/mo+

Self-hostable:No

✓Purpose-built for vectors

✓Fastest managed option

✓Simple REST API

✓Automatic scaling

✗Vendor lock-in

✗No SQL

✗More expensive at scale

Qdrant

Open-Source / Cloud

Teams wanting open-source + self-hosting

Scalability:Hundreds of millions of vectors

Cost:Free self-hosted, Cloud from $25/mo

Self-hostable:Yes

✓Open source

✓Self-hostable

✓High performance

✓Rich filtering API

✗More complex setup

✗Smaller community than Pinecone

Weaviate

Open-Source / Cloud

Multi-modal RAG (text + images)

Scalability:Billions of objects

Cost:Free self-hosted, Cloud from $25/mo

Self-hostable:Yes

✓Multi-modal support

✓GraphQL API

✓Module system for embedding models

✓Good documentation

✗Heavier resource usage

✗More complex architecture

Use Cases

Real-World RAG Applications

RAG is one of the highest-ROI AI implementations for businesses in 2025. Here are the most common and impactful use cases.

📚

Internal Knowledge Base Q&A

Answer employee questions about internal policies, SOPs, HR documents, and product specs — without requiring them to search through shared drives.

HR policy chatbotEngineering runbook assistantCompany wiki Q&A

🎯

Customer Support Automation

Answer customer questions using your product documentation, FAQ, and past support tickets — reducing support volume and response time.

SaaS help centre botE-commerce FAQ assistantTechnical support copilot

⚖️

Legal & Compliance Research

Search through contracts, regulations, case law, and compliance documents to answer specific questions with precise citations.

Contract review assistantRegulatory compliance checkerLegal research tool

🏥

Healthcare & Medical Q&A

Answer clinical questions from a knowledge base of medical literature, treatment protocols, and drug databases — always with source citations for verification.

Clinical decision supportDrug interaction checkerPatient education bot

💼

Sales & Product Intelligence

Give sales teams instant access to product specs, competitive battlecards, pricing history, and customer case studies during prospect conversations.

Sales enablement copilotCompetitive intelligence botProposal assistant

💻

Code & Technical Documentation

Search through your entire codebase, API docs, and technical specifications to answer developer questions and generate contextually correct code.

Internal API assistantCodebase Q&ADev onboarding copilot

Recommended Stack

4Byte's RAG Technology Stack

Our recommended technology choices for building production RAG systems in 2025, based on shipping 45+ AI-powered products.

Layer

Recommended

Alternative

Notes

Embedding Model

OpenAI text-embedding-3-small

Cohere embed-v3-english

3-small is the best cost/performance ratio. 1,536 dimensions. $0.02/1M tokens.

Vector Database

pgvector (via Supabase)

Pinecone (managed) / Qdrant (self-hosted)

pgvector for most SaaS apps. Pinecone for >1M vectors at production scale.

LLM for Generation

Claude Sonnet (claude-sonnet-4-6)

GPT-4o

Claude's 200k context handles large retrieved contexts. Best instruction following.

Orchestration

n8n (workflow) or custom Python

LangChain / LlamaIndex

n8n for no-code RAG pipelines. Python for custom retrieval logic and reranking.

Document Loading

LlamaIndex SimpleDirectoryReader

Unstructured.io

Handles PDF, DOCX, HTML, CSV, Markdown. Unstructured.io for complex PDFs.

Reranker

Cohere Rerank v3

BGE-Reranker (open source)

Reranking improves precision by ~20%. Only needed for >5k document knowledge bases.

Evaluation

RAGAS

DeepEval

RAGAS provides context recall, precision, faithfulness, and relevancy scores automatically.

Evaluation

How to Evaluate Your RAG System

RAG systems require systematic evaluation across retrieval quality, generation faithfulness, and answer relevance. Use RAGAS to automate evaluation — manual checking alone does not scale.

Context Recall

> 0.85RAGAS

Did the retrieval system find all the relevant documents needed to answer the question?

Context Precision

> 0.75RAGAS

Of the retrieved documents, what proportion were actually relevant? (Low = too much noise)

Answer Faithfulness

> 0.90RAGAS

Is the generated answer grounded in the retrieved context? (Low = hallucination risk)

Answer Relevancy

> 0.85RAGAS

Does the generated answer actually address the question that was asked?

Latency (P95)

< 3 secondsCustom logging

End-to-end time from user query to generated answer — including retrieval and LLM call

Retrieval Latency

< 100msDB monitoring

Time for the vector search step alone — should be fast even at scale

▸ Evaluation Workflow

Build a golden test set of 20–50 question-answer pairs

Run your RAG system on all test questions

Score outputs with RAGAS (context recall, precision, faithfulness, relevancy)

Identify failing categories and tune chunking, retrieval k, or system prompt

Built by 4Byte

How 4Byte builds RAG systems

We have built RAG systems for customer support automation, internal knowledge bases, legal document search, and sales intelligence tools. Our default stack is Claude Sonnet + pgvector (Supabase) + Python for custom retrieval logic.

pgvector in Supabase handles 95% of RAG use cases at startup and growth stage. We move to Pinecone or Qdrant only when vector count exceeds 1 million or when query latency becomes a bottleneck.

Claude Sonnet for generation — 200k context, best instruction following

pgvector + Supabase as default vector store — no extra infra

Hybrid retrieval (dense + BM25) for production systems

RAGAS evaluation before and after every system change

45+

AI Systems Shipped

Including RAG & agent builds

pgvector

Default Vector Store

Inside Supabase, no extra cost

28 days

RAG System Delivery

From kickoff to production

≤ 4hrs

Response Time

On all new enquiries

FAQ

Frequently Asked Questions

What is RAG (Retrieval-Augmented Generation)?+

Retrieval-Augmented Generation (RAG) is an AI architecture that combines a retrieval system with a large language model. Instead of relying solely on information baked into the model during training, a RAG system retrieves relevant documents from an external knowledge base at query time and passes them to the LLM as context. This enables the LLM to answer questions about private, recent, or specialised data it was never trained on.

What is a vector database and why is it used in RAG?+

A vector database stores data as high-dimensional numerical vectors (embeddings) and enables fast similarity search — finding the most semantically similar documents to a query, not just keyword matches. In a RAG system, documents are converted to embeddings and stored in a vector database. When a user asks a question, the question is also converted to an embedding, and the vector database retrieves the most similar document chunks by comparing embedding distances.

What is the difference between RAG and fine-tuning?+

Fine-tuning trains the LLM itself on your specific data, baking knowledge into the model weights. RAG retrieves your data at query time and passes it as context without modifying the model. RAG is better for dynamic or frequently updated data, large knowledge bases, and when you need source citations. Fine-tuning is better for teaching the model a specific style, format, or domain-specific reasoning pattern. Most production AI systems use RAG for knowledge and fine-tuning for behaviour.

What is chunking in a RAG system?+

Chunking is the process of splitting large documents into smaller pieces before converting them to embeddings. Since LLMs have context window limits and embedding models work best on shorter texts, documents must be divided into chunks of typically 200 to 1000 tokens. The chunking strategy significantly impacts retrieval quality — chunks that are too small lose context, chunks that are too large dilute relevance.

Which vector database should I use for a RAG system?+

For most SaaS applications and early-stage RAG systems, pgvector (PostgreSQL extension) is the recommended starting point — it runs inside your existing Supabase database at no extra cost. For high-scale production systems with millions of vectors, Pinecone (managed, simple API) or Qdrant (open-source, self-hostable) are the leading purpose-built options. Weaviate is preferred for multi-modal RAG systems.

How do you evaluate a RAG system?+

RAG systems are evaluated across three dimensions: retrieval quality (did the system retrieve the right documents?), answer faithfulness (is the answer grounded in the retrieved documents?), and answer relevance (does the answer actually address the question?). Tools like RAGAS provide automated evaluation metrics. Human evaluation of a golden test set is also essential, particularly for domain-specific knowledge bases.

Related Resources

Continue Learning

TemplateAI Agent Planning TemplateView resource →ComparisonOpenAI vs ClaudeView resource →ComparisonSupabase vs FirebaseView resource →Automationn8n vs MakeView resource →

Open for new projects

Ready to build a RAG system?

We design and build production RAG systems for customer support, internal knowledge bases, legal search, and sales intelligence. Claude + pgvector + hybrid retrieval — shipped in 28 days. Free strategy call, response in under 4 hours.

Book a Free Strategy Call →AI Agent Planning Template →

📞100% Free Strategy Call

⚡Response in ≤ 4 Hours

🛡️No Obligation

🚀45+ Products Shipped

▸ Keep exploring

Explore the Learn hub

Curated learning paths organized by funnel stage — TOFU, MOFU, and BOFU.

Explore→

Read the Blog

Practical writing on shipping SaaS, AI, and software products.

Explore→

Talk to 4Byte

Book a free strategy call and get a scoped plan in 48 hours.

Explore→