RAG for Document AI

An ML team builds a retrieval augmented generation system over 50,000 legal contracts. Retrieval works fine on clean, structured queries. Then a lawyer asks: which contracts have automatic renewal clauses with less than 30 days notice? The system returns three contracts. The actual answer is 47. The chunks were too large. The renewal clause was split across a chunk boundary. The retrieval step worked as designed. The problem was the design itself.

This scenario happens in every organization that deploys RAG. Not always with contracts, but the shape is the same. You build a pipeline. You test it on good data. Then production sends a query that the system was not designed to answer, and everything breaks in a specific, preventable way.

Document AI and RAG are not the same thing. But document AI without RAG is incomplete. This guide walks through what RAG actually means for documents, why it requires different engineering than general RAG, and how to build pipelines that don't fail at the contract clause boundary.

TL;DR

Retrieval augmented generation feeds an LLM with relevant document chunks to reduce hallucination and ground answers in real data. But standard RAG chunking and embedding strategies often fail on documents because they ignore structure, metadata, and domain-specific boundaries. Enterprise document RAG requires semantic chunking aware of document type, metadata-preserving vector stores, and hybrid retrieval combining dense and sparse search. The 87 percent accuracy gains in clinical decision support, 49 percent market growth, and three-to-six-month ROI timelines show why document teams now treat RAG as essential, not optional.

What RAG means for document AI (and what it doesn't)

Retrieval augmented generation is a pattern: take a query, search a knowledge base for relevant context, and feed that context to an LLM alongside the query. The LLM then generates an answer grounded in the retrieved facts rather than its training data.

For document AI, RAG solves a specific problem. Suppose you have an invoice processing pipeline that extracts vendor name, total amount, and line items from incoming invoices. An LLM fine-tuned on sample invoices can do this. But what if you want to answer questions like "Of all invoices paid to vendors in the Northeast, how many include a recurring fee?" Fine-tuning doesn't scale. You need retrieval.

RAG is not fine-tuning. Fine-tuning rewrites the model's weights based on your data. RAG leaves the model alone and instead gives it access to external documents at inference time. The trade-off: fine-tuning is slower to implement but persistent; RAG is fast to iterate but depends entirely on retrieval quality.

For documents, RAG also means different stakes. A chatbot using RAG can hallucinate and the user will notice. A contract analysis RAG that misses a liability cap clause exposes your company to legal risk. An insurance claims RAG that overlooks a coverage exclusion denies a justified claim. Document RAG pipelines need higher precision than general RAG, which changes how you design chunking, embedding, and retrieval.

Why document AI needs RAG differently than general RAG

Standard RAG assumes text is largely homogeneous. A news article, a research paper, a Wikipedia entry. You chunk it, embed it, and retrieve it.

Documents violate this assumption in every direction. An invoice has structure: line items, subtotals, tax amounts, payment terms. A contract has legal clauses that span multiple pages and reference other sections. A tax form has predefined fields and interdependent calculations. Medical records mix structured codes and unstructured clinical notes.

When a generic chunking algorithm splits a 10-page contract into equal 512-token chunks, it routinely splits clauses. The indemnification clause is in chunk 7, the liability cap is in chunk 8. A query for "liability terms" might retrieve chunk 8. But the LLM loses the context of what indemnification applies to because the clause boundary was arbitrary.

Metadata also matters in ways it does not for general text. A financial document's date, counterparty, and currency are not just annotations. They are query filters. You do not want to retrieve a clause from a 2018 contract when answering a question about 2024 renewal terms. Generic RAG systems have no language for this.

Finally, documents scale differently. Thousands of invoices, tens of thousands of contracts, millions of scanned claims forms. Each document has dozens of fields and hundreds of pages. Your vector database indexes billions of chunks. Query latency matters. So does cost. Dense vector search on billions of chunks is not cheap.

Document RAG needs to answer structural questions, preserve metadata, handle multi-document queries, and scale to real enterprise volumes. Generic RAG tools, built for knowledge bases of web articles, do not handle these constraints.

How to build a RAG pipeline for document AI

Building a document RAG pipeline involves six stages. Each stage has choices that compound downstream. Get any stage wrong and your renewal clause disappears into a chunk boundary.

Step 1 - Document processing and structuring

Before you chunk anything, the documents must be readable and classified.

Start with document ingestion. If your documents are PDFs, you need optical character recognition to extract text. If they are images, same problem. OCR quality directly affects downstream quality. A scanner error at this stage propagates through chunking and embedding.

Next, document classification. A contract is not an invoice. A medical record is not a tax form. You cannot chunk all of them the same way. Docsumo's intelligent document processing pipeline performs this classification automatically using machine learning. The classifier looks at the document and returns a label: "invoice," "contract," "claim form," "medical record."

Then comes pre-processing: normalize whitespace, remove boilerplate, identify headers and footers. Many OCR outputs are messy. "New York" becomes "New y ork". Tables get scrambled. Pre-processing cleans this up so that chunking algorithms have well-formed text to work with.

This stage is unglamorous but critical. Garbage in, garbage out applies to every ML pipeline. It applies triply to RAG pipelines where retrieval depends on text quality.

Step 2 - Chunking strategy for document types

This is where most document RAG pipelines fail.

A naive approach is fixed-size chunking: split every document into 512-token chunks, no exceptions. It is easy to implement. It is also wrong for documents.

Why? Consider a contract with an indemnification clause in section 4.2:

> "Vendor shall indemnify and hold harmless Client from all third-party claims arising from Vendor's performance of Services, provided that Vendor's liability under this indemnification shall not exceed the total fees paid in the preceding 12 months."

The liability cap is three lines. The indemnification obligation is two lines. If a fixed-size chunking algorithm sets chunk size at 10 lines, it might split this clause into chunks 7 and 8. A query for "What is Vendor's maximum liability?" might retrieve only chunk 8 without the context of what that cap applies to.

Better approaches:

• Semantic chunking: Split documents at logical boundaries (paragraphs, sections, clauses) rather than token counts. Requires NLP parsing to identify these boundaries.
• Structure-aware chunking: Use document metadata (headers, tables, lists) to inform splits. A legal clause that spans pages stays intact in one chunk.
• Hierarchical chunking: Create chunks at multiple granularities. A clause might be a chunk. So might the entire section. Retrieve at the appropriate level based on the query.
• Adaptive chunking: Let the document type guide chunk size. Invoices can use fixed-size chunks because they are uniform. Contracts need semantic chunking because clause boundaries vary.

Research data supports semantic approaches. Adaptive chunking in clinical decision support achieved 87 percent accuracy versus 50 percent for fixed-size chunking. In a cross-domain evaluation of 36 different chunking methods, semantic and structure-aware approaches consistently outperformed fixed-size splitting.

For document RAG, use semantic or structure-aware chunking. Document type matters: invoices are structured, so smaller fixed chunks work. Contracts are unstructured and need semantic boundaries. Insurance claims are semi-structured and need hybrid approaches. Tools like document annotation help identify optimal chunk boundaries for your document types.

Step 3 - Embedding model selection

An embedding model converts text (or chunks) into vectors. "Indemnification clause" becomes a 768-dimensional vector. Vectors from similar texts cluster together in vector space.

Choices:

• Open source models: BERT, all-MiniLM-L6-v2 (fast, small, fine-tunable).
• API-based models: OpenAI text-embedding-3-large, Vertex AI text-embedding-004 (higher quality, managed, more expensive).
• Domain-specific models: BioBERT for medical documents, FinBERT for financial documents (specialized but harder to maintain).

For documents, embedding model choice interacts with chunk size. A large embedding model (1536 dimensions) captures richer semantic meaning but costs more to compute and store. A small model (384 dimensions) is cheap but misses nuance.

Consider also that chunking strategy influences embedding effectiveness. The research on embedding and chunking interaction shows they are not independent. A domain-specific embedding model might mitigate damage from weak chunking. But weak embeddings cannot salvage bad chunks.

Start with a general-purpose model like Vertex AI text-embedding-004 or OpenAI text-embedding-3-small. If query relevance is poor, benchmark alternatives. Domain-specific models help if your documents are highly specialized (medical, legal, financial). Understanding how to handle unstructured data is critical when selecting embedding models, as different document types have different vocabulary and structure.

Step 4 - Vector store design

Once you embed chunks, you store the vectors in a vector database (or vector index). Examples: Chroma, Weaviate, Pinecone, Milvus.

Design decisions:

• Indexing method: Most vector stores use approximate nearest neighbor (ANN) indexing like HNSW. Exact nearest neighbor is slow at scale.
• Metadata filtering: Store chunk metadata (document ID, document date, page number, source section) separately. Enable filtering at retrieval time. This is critical for documents. You want to retrieve invoices only from 2024, or clauses only from the current contract version.
• Replication and backup: Documents are business critical. Your vector store must be queryable, not just searchable. Design for uptime.
• Dimensionality: Embedding dimensionality affects storage and latency. A 768-dimensional embedding costs less than 1536-dimensional. But if your embedding model produces 1536 dimensions, you store 1536.

For enterprises with thousands of documents, storage costs matter. An organization indexing 50,000 contracts at 100 chunks each with 768-dimensional embeddings uses about 3.8 terabytes of storage (rough: 50k * 100 * 768 * 4 bytes). At typical cloud rates, this costs thousands of dollars monthly. Smart document processing workflows that deduplicate and pre-filter documents can cut these costs significantly.

Use metadata filtering to reduce retrieval scope. If a query specifies a date range or document type, filter before vector search. This shrinks the index you search from billions to millions of chunks, cutting latency and cost. This is where intelligent document processing shines: extracted metadata enables precise filtering in RAG systems.

Step 5 - Retrieval strategy (dense, sparse, hybrid)

There are three ways to find relevant chunks: dense (vector) search, sparse (keyword) search, or both.

• Dense search (vector similarity): Compute embedding for the query, find chunks with similar embeddings. Good at capturing semantic meaning. Poor at exact phrase matching.
• Sparse search (BM25, TF-IDF): Index chunks as bags of words. Retrieve chunks where query keywords appear. Good at exact matching. Poor at semantic nuance.
• Hybrid: Combine both. Retrieve top-k dense results and top-k sparse results, merge, rerank. Takes more computation but captures both semantic and exact-match relevance.

For documents, hybrid is often necessary. A lawyer asks: "Does this contract automatically renew?" A dense search might miss the clause because "automatic renewal" uses different vocabulary across contracts. A sparse search for "automatic renewal" will find the exact phrase. A hybrid approach tries both.

Benchmark your retrieval strategy on your document types. A test set of 100 queries on a sample of documents reveals which strategy works. Document RAG pipelines that skip this step often end up retrieving chunks that are semantically similar but contextually wrong. Building a data labeling process early helps create these evaluation sets.

Step 6 - Generation and grounding

The final step: take the retrieved chunks and feed them to an LLM alongside the query.

Prompt engineering matters. A naive prompt: "Answer this question based on the following documents: [chunks] [query]."

A better prompt:

1. Instructs the LLM to only use information from the chunks.

2. Asks the LLM to cite its source (chunk ID, document, page).

3. Tells the LLM to say "I don't know" if the chunks don't answer the question.

Grounding means linking the LLM's answer back to source chunks. If the LLM says "Vendor's liability cap is 12 months of fees," it should cite the contract clause ID. This lets humans verify the answer and catches hallucinations. For organizations using invoice processing or other specialized workflows, grounded generation is essential to maintain audit trails and regulatory compliance.

Also design for graceful degradation. If retrieval returns no relevant chunks, what does the LLM do? A good pipeline returns a confidence score and allows users to escalate to manual review.

Where document RAG pipelines break and how to fix it

Most document RAG failures follow predictable patterns.

1. Failure: "Chunk boundaries split key clauses or field groups."

Diagnosis: Fixed-size chunking applied to unstructured documents. Query returns irrelevant chunks because context split across boundaries.

Fix: Switch to semantic or structure-aware chunking. Parse document structure and respect logical boundaries.

2. Failure: "Dense retrieval misses exact phrase matches that humans would find."

Diagnosis: The query contains a specific phrase, but dense embedding search doesn't weight exact matches. Semantic similarity alone is insufficient.

Fix: Use hybrid retrieval combining dense and sparse search, or boost BM25 scores for exact phrase matches.

3. Failure: "Vector search returns chapters from the wrong document."

Diagnosis: Metadata not filtering at retrieval time. All 50,000 contracts indexed in one vector space with no document-level partitioning.

Fix: Add metadata filtering. Retrieve from document or document set first, then vector search within that scope.

4. Failure: "LLM cites chunks but the citation contradicts the retrieved text."

Diagnosis: Prompt does not enforce grounding. LLM generates plausible-sounding answers without checking retrieved chunks.

Fix: Rewrite prompt to require explicit citation and verification against sources. Measure citation accuracy.

5. Failure: "Retrieval latency exceeds acceptable thresholds."

Diagnosis: Searching across billions of embeddings without filtering or approximate indexing. Dense retrieval on raw vector space is slow.

Fix: Partition vector store by document type or date range. Use ANN indexing (HNSW). Filter metadata before vector search.

6. Failure: "Performance degrades as more documents are indexed."

Diagnosis: Embedding model or chunking strategy does not scale to the corpus size. Drift occurs as data composition changes.

Fix: Monitor retrieval quality as you add documents. Benchmark periodically. If quality drops, revisit chunking or embedding model.

How Docsumo feeds structured data into RAG pipelines

Docsumo's platform extracts structured data from documents using AI-powered document extraction. But structure and RAG are complementary, not opposed.

Docsumo processes invoices and extracts vendor name, line items, and totals. It processes contracts and identifies and classifies clauses. It processes insurance claims and extracts policyholder info, claim details, and supporting documents.

For a RAG pipeline, this structure serves two purposes:

First, it informs chunking. Docsumo's document classification capability tells the RAG pipeline which chunking strategy to use. An invoice gets fixed-size line-item chunks. A contract gets semantic clause-level chunks.

Second, the extracted fields become metadata. Docsumo extracts the contract signature date; this date becomes a metadata field in the vector store. A query can filter: "Show me renewal clauses from contracts signed after January 2024." The RAG system searches only within that subset.

Docsumo also supports data labeling and annotation for custom document types. If you process proprietary forms, you can annotate sample documents and train Docsumo's extraction model. The extracted fields then populate metadata for RAG retrieval. For complex scenarios, AI Assist accelerates the annotation process, allowing you to build RAG-ready datasets faster.

This integration reduces hallucination. An LLM answering "What is the total invoice amount?" can retrieve the extracted total field directly instead of inferring from raw text. Extracted data is ground truth. RAG over raw chunks is probabilistic.

FAQs

Should we RAG every document or only long ones?

RAG is most valuable where knowledge bases are large (1000+ documents), queries are open-ended, and hallucination is risky. Small document collections (under 100) do not justify the complexity. Long documents (100+ pages) almost always need RAG because context windows are finite. Medium documents (10-30 pages) benefit if you need to ask multiple questions. Ask: Can an LLM fit the full document in context? If yes, skip RAG. If no, use it.

How many chunks are too many?

Retrieval quality degrades as the pool of irrelevant chunks grows. With millions of chunks, even good embeddings might retrieve noise. Solutions: (1) Partition by document type or metadata. (2) Use hybrid retrieval to combine dense and sparse. (3) Rerank retrieved chunks before feeding to LLM. Typical: retrieve top-20 or top-50, then rerank to top-3 or top-5 for generation. This keeps latency and cost reasonable.

Do we need to fine-tune after adding RAG?

Not usually. RAG replaces some benefits of fine-tuning (grounding in real data, reducing hallucination). Fine-tuning on top of RAG helps if: (1) Your documents are highly specialized and require domain vocabulary the base model lacks. (2) You want to enforce a specific output format or reasoning pattern. Otherwise, iterate on chunking and retrieval. Quality gains come faster than fine-tuning.

What embedding model is best for legal documents?

Start with Vertex AI text-embedding-004 or OpenAI text-embedding-3-small. Both are general-purpose and high-quality. If legal vocabulary is not captured well (benchmark on your documents), try domain-specific models like LegalAI embeddings or a fine-tuned BERT variant. Domain models help but add operational overhead. Many legal teams find general models sufficient if chunking and retrieval are sound.

How do we measure RAG quality?

Metrics: (1) Retrieval precision: Of the top-k retrieved chunks, how many are actually relevant? (2) Retrieval recall: Of all relevant chunks in the corpus, how many are in the top-k? (3) Generation quality: Human judges rate LLM answers on accuracy, citation correctness, and confidence. Build a labeled evaluation set: 50-100 queries with known answers and relevant documents. Evaluate retrieval and generation separately. Use these metrics to benchmark pipeline changes.