RAG Application Architecture: A Step-by-Step Guide to Retrieval-Augmented Generation

Overview

Retrieval-augmented generation, or RAG, combines search with language generation. Instead of asking a model to answer entirely from its trained knowledge, an application retrieves relevant content from a controlled source and places that content in the model’s context. The model then uses the retrieved material to produce an answer, summary, or action.

A typical RAG architecture has six stages:

1. Ingestion: collect documents and associated metadata.
2. Processing: clean, normalize, and divide documents into useful passages.
3. Indexing: create embeddings or other searchable representations and store them.
4. Retrieval: find candidate passages for a user query.
5. Generation: provide the selected context and instructions to an LLM.
6. Evaluation and monitoring: measure retrieval quality, answer quality, latency, cost, and failure patterns.

The most important architectural principle is separation of concerns. Document processing should be independently testable from retrieval. Retrieval should be measurable without relying only on the final answer. Generation should make its evidence visible where appropriate. This separation makes it easier to identify whether a poor response came from missing data, weak chunking, an ineffective search query, an unsuitable model, or an instruction problem.

RAG is not automatically the right solution for every knowledge task. A stable, narrow behavior may be better handled with prompting or fine-tuning, while frequently changing or organization-specific information is often a strong candidate for retrieval. For a structured comparison, see Fine-Tuning vs Prompting vs RAG.

What to track

1. Source and ingestion health

Track which sources are included, when they were last synchronized, and whether each ingestion run completed successfully. Useful fields include document ID, source location, version or modification time, content type, access scope, and processing status. A document that exists in the source system but never reaches the index is an ingestion failure, not a retrieval failure.

Also monitor duplicates, unsupported formats, empty files, extraction errors, and documents that exceed your processing limits. Keep enough metadata to remove or reprocess a document without rebuilding the entire index.

2. Chunking quality

Chunk size and boundaries directly affect retrieval. Chunks that are too small can lose the context needed to interpret a statement. Chunks that are too large may contain several unrelated topics and consume more context than necessary. Prefer boundaries that reflect the source, such as headings, paragraphs, list groups, code blocks, or policy sections, rather than splitting every document at an arbitrary character count.

Record the chunking method and configuration as part of the index version. If chunk size, overlap, cleaning rules, or parsing logic changes, you should be able to identify which index produced a response.

3. Embeddings and index coverage

Track the embedding model or representation used, the index version, the number of indexed chunks, and the percentage of source content successfully represented. A change in embedding configuration can alter search behavior even when the documents have not changed.

Do not treat a populated vector database as proof of good coverage. Test whether known questions retrieve the passage that contains the answer. For some applications, combine semantic search with keyword or metadata filters. Exact names, product identifiers, error codes, and legal phrases may benefit from lexical matching rather than vector similarity alone.

4. Retrieval and reranking

Measure retrieval separately from generation. For a test question with a known answer, record whether the relevant passage appears in the top results and how its rank changes over time. Track the number of candidates retrieved, the final number passed to the model, filtering decisions, and retrieval latency.

Reranking can improve the ordering of candidate passages, but it adds processing time and another component to operate. Compare retrieval with and without reranking on a representative test set. The right choice depends on whether the quality improvement justifies the added latency and complexity.

5. Citations and answer grounding

If your application displays citations, store the relationship between the answer and the source passages used to generate it. Check that citations point to the correct document and location, not merely to a loosely related source. A citation that cannot support the associated claim can create more risk than no citation.

Evaluate whether answers stay within the retrieved evidence. Your prompt can instruct the model to distinguish supported statements from uncertainty, but instructions are not a substitute for testing. Include questions whose answers are absent from the index and verify that the system can acknowledge insufficient information instead of filling the gap with an unsupported response.

6. Operational performance

Monitor end-to-end latency as well as the time spent in ingestion, embedding, search, reranking, and generation. Track token usage, failed requests, timeouts, cache behavior, and cost-related inputs. The AI App Cost Calculator Inputs guide provides a useful way to organize these variables.

Log enough information to debug a request without storing sensitive content unnecessarily. A trace may include the query, index version, retrieved document IDs, scores, prompt version, model identifier, latency, and outcome. Apply access controls and retention rules appropriate to the data handled by the application.

Cadence and checkpoints

A RAG system should be reviewed on more than one schedule because different parts change at different rates.

• Per request: capture retrieval results, citations, latency, failures, and model or prompt versions. Sample or redact content where necessary.
• Daily or per ingestion run: check synchronization status, processing failures, document counts, duplicate rates, and index freshness.
• Weekly: review unanswered questions, user corrections, low-confidence responses, empty retrievals, and unusually slow requests.
• Monthly: run a fixed evaluation set covering common questions, difficult questions, absent-answer cases, citation accuracy, and access-control boundaries.
• Quarterly: review chunking, embedding choices, retrieval strategy, reranking, model selection, prompt versions, and infrastructure costs.
• After a material change: evaluate immediately after changing source systems, parsers, index settings, prompts, models, permissions, or context limits.

Keep a small, versioned evaluation set rather than relying only on production anecdotes. Include real failure patterns after removing confidential information, along with deliberately designed edge cases. The RAG Evaluation Checklist can help organize pre-release and recurring checks. For broader prompt and output testing, use a repeatable LLM evaluation workflow.

How to interpret changes

A drop in answer quality does not identify the cause by itself. Diagnose the pipeline in order.

1. Check source freshness: confirm that the expected document was ingested and that obsolete versions were handled correctly.
2. Check extraction and chunking: inspect the indexed text for missing tables, broken headings, truncated sections, or chunks without useful context.
3. Check retrieval: determine whether the relevant passage appears among the candidates. If it does not, investigate the query, filters, embeddings, lexical search, and reranking.
4. Check context assembly: verify that the best passages survive deduplication, ranking, token limits, and permission filters.
5. Check generation: inspect the prompt, model behavior, answer format, and refusal or uncertainty instructions.

Interpret metrics together. Higher retrieval recall may increase context size and latency. More citations may not mean better grounding if citations are weakly matched. A lower average response time may reflect aggressive truncation that harms answer quality. Review representative traces and examples alongside aggregate measurements.

When comparing model or index changes, hold the evaluation set and test procedure constant. Change one major variable at a time where practical, record the expected trade-off, and preserve the previous version so that you can roll back. A model benchmark can inform a decision, but application-specific retrieval and grounding tests should carry greater weight for a RAG workflow.

When to revisit

Revisit this architecture whenever the information landscape or user task changes. A monthly review is appropriate for systems with active content updates, while a quarterly architecture review may be sufficient for relatively stable internal knowledge. Do not wait for a scheduled review after a major model release, source migration, parser change, security incident, permission change, or unexpected increase in unanswered questions.

Use the review to answer five practical questions:

• Are the current sources complete, fresh, and correctly permissioned?
• Do chunks preserve the structure and context users need?
• Does retrieval consistently surface evidence for the evaluation set?
• Do generated answers accurately reflect that evidence and show useful citations?
• Are latency, reliability, and cost still appropriate for the application?

Finish each review with a short change log: what changed, which tests were run, what trade-offs were accepted, and when the next review is due. A RAG application becomes easier to maintain when its data, index, prompts, models, and evaluation results are treated as versioned parts of one AI development workflow rather than as isolated configuration.