RAG Tutorial: Build a Reliable Retrieval-Augmented Generation Workflow

Overview

RAG combines information retrieval with generation. When a user asks a question, the application first searches a collection of indexed documents, then supplies relevant passages to a language model as context for the response. The model can use that context instead of relying only on information encoded during training.

A typical RAG pipeline has seven stages:

1. Collect: Identify the documents, records, or approved data sources the application may use.
2. Prepare: Extract text, remove avoidable noise, preserve useful structure, and attach metadata.
3. Chunk: Divide content into passages that are large enough to retain meaning but focused enough to retrieve accurately.
4. Embed: Convert chunks into vector representations for semantic search.
5. Retrieve: Search the index for passages relevant to the user’s question, optionally combining vector and keyword search.
6. Generate: Give the selected context and instructions to the language model.
7. Evaluate and maintain: Measure retrieval and answer quality, then update the index as the underlying information changes.

The central design principle is to keep retrieval and generation observable as separate stages. If an answer is wrong, you should be able to determine whether the system retrieved the wrong passage, supplied insufficient context, misunderstood the passage, or generated an unsupported claim. A broader discussion of system components is available in this RAG application architecture guide.

Checklist by scenario

Before building the index

• Define the answer boundary. Decide which questions the system should answer and which it should decline. A support knowledge base, internal policy assistant, and research tool may require different source rules.
• List authoritative sources. Record the owner, update frequency, access permissions, and expected format for each source. Do not treat every available document as equally trustworthy.
• Choose a document identity scheme. Give each source a stable identifier and retain fields such as title, URL, section, version, publication date, and access scope.
• Plan removal and replacement. The ingestion process should be able to remove outdated content and replace a document without creating duplicate or conflicting chunks.
• Protect sensitive data. Apply access controls before content reaches the retrieval index. Filtering retrieved results after generation is too late if restricted text has already entered the model context.

During document preparation

• Extract text with structure intact. Preserve headings, lists, tables where practical, and relationships between labels and values. A plain-text extraction that loses section context can weaken retrieval.
• Remove repetitive noise. Headers, footers, navigation links, cookie notices, and duplicated boilerplate can consume context and produce irrelevant matches.
• Normalize carefully. Fix encoding problems and inconsistent whitespace, but do not flatten meaningful punctuation, identifiers, code, or formatting.
• Attach metadata to every chunk. Include the source identifier, document title, section path, version, timestamp, and any filtering attributes needed at query time.
• Keep a traceable relationship to the source. Each chunk should be linkable to its original document and location so the application can display citations or support review.

When choosing chunking rules

• Start with semantic boundaries. Headings, paragraphs, procedures, and complete list items are usually better boundaries than arbitrary character counts.
• Keep related qualifications together. Exceptions, definitions, conditions, and notes should remain with the rule they qualify whenever possible.
• Use overlap deliberately. A modest overlap can preserve meaning across boundaries, but excessive overlap increases index size and may return duplicate evidence.
• Test different chunk sizes. The best setting depends on document type, query length, model context limits, and whether answers require a single fact or a multi-step explanation.
• Store parent context when needed. A small matching passage may need its heading or neighboring passages added before generation. Separate retrieval granularity from generation context if that produces clearer answers.

When implementing retrieval and generation

• Make the query explicit. Decide whether to search the user’s original question, a rewritten query, or several query variants. Preserve the original question for auditing.
• Use metadata filters first where appropriate. Tenant, product, language, date, permission, or document type filters can prevent semantically similar but unauthorized or irrelevant results.
• Set a retrieval limit based on evidence quality. More passages do not automatically produce better answers. Inspect whether additional results add support or merely add noise.
• Consider hybrid search. Vector search helps with conceptual similarity, while keyword or lexical search can help with exact names, error codes, identifiers, and uncommon terms.
• Require grounded behavior. The generation instructions should tell the model to use the supplied evidence, distinguish uncertainty, and avoid presenting unsupported information as fact.
• Return citations with stable references. A citation should identify the source and, when possible, the section or passage used. Do not display a citation that the application cannot trace back to retrieved context.

Before production release

• Create a representative test set. Include common questions, ambiguous questions, questions with no answer in the corpus, terminology variations, and questions that should be refused or redirected.
• Evaluate retrieval separately. Check whether the expected evidence appears in the retrieved results before judging the generated response.
• Evaluate the final answer. Review correctness, completeness, citation alignment, clarity, and whether the answer makes claims beyond the evidence.
• Log the pipeline. Retain query versions, filters, retrieved identifiers, scores where useful, model configuration, latency, token usage, and final output subject to your data-handling requirements.
• Define operational fallbacks. Decide what happens when retrieval returns nothing, sources conflict, the index is unavailable, or the model cannot confidently answer.

What to double-check

RAG quality often depends on details that are easy to overlook during an initial prototype. Use the following review before changing a workflow or shipping a new version.

Document and index checks

• Are the newest approved documents present?
• Have deleted or superseded documents been removed from search?
• Do chunk metadata and access filters match the application’s authorization model?
• Can every retrieved chunk be traced to a human-readable source?
• Are tables, code samples, formulas, and lists represented in a way the model can interpret?

Retrieval checks

• Do exact-match questions retrieve exact terms, identifiers, and product names?
• Do paraphrased questions retrieve the same underlying evidence?
• Does a filter exclude content that should be available, or allow content that should not be available?
• Are duplicate chunks crowding out distinct supporting passages?
• Does the system behave sensibly when the question is outside the knowledge base?

Answer and citation checks

• Does the response answer the user’s actual question rather than merely summarize retrieved text?
• Does every important factual claim have appropriate support?
• Do citations point to the passages that support the claims they follow?
• Does the answer preserve important conditions, dates, exceptions, and scope?
• Does the model clearly state when the available evidence is incomplete or conflicting?

For a repeatable testing process, separate retrieval metrics from generation review and keep a fixed evaluation set alongside newly reported failures. The RAG evaluation checklist and this prompt testing and evaluation framework provide useful companion reading.

Common mistakes

Indexing everything without defining scope

A large corpus can create conflicting results, stale answers, and unclear ownership. Start with a bounded collection whose authority and update process are understood. Expand only when test results show that the current corpus cannot answer important questions.

Treating chunk size as a universal setting

There is no single ideal chunk size for every source. Policies, API references, transcripts, and technical documentation have different structures. Compare chunking strategies against representative queries instead of selecting a setting by convention.

Assuming semantic search replaces exact search

Embeddings are useful for meaning-based similarity, but exact identifiers and rare terms may need lexical matching. A hybrid vector search workflow is often worth testing when users search for version numbers, error messages, ticket IDs, or product codes.

Adding more context to solve every problem

Overloading the prompt with retrieved passages can make the answer less focused and increase processing cost. Improve filtering, ranking, chunk boundaries, and query formulation before simply increasing the context window.

Measuring only whether an answer sounds good

A fluent response can still be unsupported or incomplete. Review evidence retrieval, factual grounding, citation accuracy, refusal behavior, latency, and resource use. Keep failed examples as regression tests rather than relying on occasional manual spot checks.

Ignoring source updates

A RAG application is dependent on the lifecycle of its documents. Without versioning, scheduled ingestion, deletion handling, and monitoring, a well-designed prototype can gradually produce answers based on obsolete material.

When to revisit

Review this RAG workflow before a major planning cycle, after changing the model or embedding configuration, and whenever the underlying documents or user questions change materially. Also revisit it when you add a new tenant, data source, language, document format, or permission rule.

Use a practical maintenance cadence:

• After each significant content update: confirm ingestion, replacement, deletion, metadata, and citation links.
• After a model, embedding, or search change: rerun the fixed evaluation set and compare retrieval and answer results with the previous version.
• After a reported failure: classify the issue as ingestion, chunking, retrieval, authorization, prompting, generation, or user-interface failure, then add a regression case.
• During regular operational reviews: inspect unanswered questions, low-confidence responses, latency, token usage, duplicate retrievals, and stale sources.
• Before expanding scope: verify that the evaluation set, access model, fallback behavior, and ownership process cover the new data.

A dependable RAG system is maintained as a workflow, not installed as a one-time feature. Keep the corpus governed, retrieval measurable, prompts versioned, and failures visible. When you make the next change, run this checklist from source preparation through final citations rather than validating only the generated text.

For related implementation decisions, compare fine-tuning, prompting, and RAG, review AI API integration practices, and account for retrieval and model usage when estimating application costs with the AI app cost calculator inputs guide.