Document Embeddings: Why Keyword Search Fails and What Works Instead

A legal team searches their contract database for "indemnification obligations related to data breaches." Keyword search returns 14 contracts. It misses 31 that use "liability," "cybersecurity incidents," and "hold harmless" to describe the same obligations. Their search engine found exact keyword matches. It found nothing else. This is the vocabulary problem that has blocked semantic understanding in document search for decades.

Document embeddings solve it.

TL;DR

Document embeddings convert documents into dense numerical vectors that capture meaning, not just keywords. Unlike keyword search, embeddings find documents with similar content even when the wording differs completely. Legal teams find related contracts by concept. Support teams locate similar cases. Analysts identify duplicate reports. The cost tradeoff is real: you pay for computation upfront to generate embeddings, and you need a vector database to store and query them. The payoff is speed and accuracy on large, complex collections where traditional search fails. Semantic search with embeddings is particularly powerful in knowledge management systems.

What are document embeddings?

An embedding is a numerical vector that represents meaning. Think of it as a compressed summary of a document's semantic content. When you generate an embedding, you're converting a document into, say, 768 or 1536 numbers arranged in a specific order. Documents with similar meanings get embeddings close to each other in vector space. Documents with different meanings get embeddings far apart.

The key difference from keyword search: keywords are exact matches. You search for "indemnification" and the system returns documents containing that word. Embeddings capture relationships between concepts. They understand that "indemnification," "liability," and "hold harmless" all express similar legal ideas, even though the words are completely different.

Embeddings emerge from transformer-based neural networks like BERT and Sentence Transformers. These models are trained on massive amounts of text to learn how language works. When you feed a document into one of these models, the network processes it layer by layer, learning linguistic patterns, semantic relationships, and conceptual connections. The final layer outputs a vector: your embedding.

The quality of an embedding depends on three things: the model's architecture, the training data it was built on, and how you apply it to your specific documents. This is why intelligent document processing platforms increasingly use embeddings as a core component of their semantic understanding layer.

Why keyword search fails on complex document collections

Keyword search works fine for simple queries on small, consistent datasets. When your document collection grows, when terminology varies, or when meaning matters more than exact wording, keyword search hits a wall.

1. First, there is the vocabulary problem. Organizations use different terms to express the same concept. A medical dataset might contain "myocardial infarction," "MI," and "heart attack." A financial dataset might use "illiquidity event," "liquidity crisis," and "cash freeze" interchangeably. Keyword search will miss whole categories of relevant documents if the searcher doesn't guess the right terminology.
2. Second, synonymy and related concepts create recall problems. Searching for "contract termination" won't find documents about "agreement dissolution" or "relationship cessation," even though they mean essentially the same thing. A human reviewer knows these are related. A keyword index does not.
3. Third, keyword search becomes slow on massive collections. If you have 500,000 documents and you search for "data privacy," a keyword index still has to find and rank every document containing that exact phrase. With embeddings, you compute the search query's embedding once, then find the nearest vectors in a vector database using approximate nearest neighbor search. The result: answers in milliseconds instead of seconds, even on millions of documents.

The legal team example illustrates all three problems at once. The database contained 45 contracts discussing data breach liability. Keyword search found only 14 because the search terms didn't match the terminology in 31 contracts. An embedding-based search would have found all 45 because it understands that "indemnification," "liability," and "hold harmless" all relate to the same legal concept. This is one reason why modern document classification systems rely on semantic embeddings rather than rules.

How document embeddings work

Tokenisation and encoding

Documents don't feed directly into embedding models. They must first be converted into tokens. Tokenisation breaks text into small units: words, subwords, or characters depending on the tokenizer. A BERT tokenizer might convert "indemnification obligations" into ["in", "dem", "ni", "fi", "cation", "ob", "li", "gations"].

Each token gets mapped to a numerical ID. The model then processes this sequence of IDs, learning the relationships between tokens and accumulating semantic meaning as it passes through the network's layers. By the final layer, the model has processed all the context and relationships, and it outputs a vector.

The output vector's size varies by model. BERT outputs 768 dimensions. OpenAI's text-embedding-3 outputs 1536 by default. Smaller models output 384 or 512. More dimensions capture more nuance but require more storage and computation.

Embedding model architectures

Three families of models dominate production document embedding use cases.

BERT and its variants like RoBERTa were among the first transformer models adapted for embeddings. BERT outputs a representation for each token, so you typically average all token embeddings or use a special classification token as your document embedding. BERT works well for domain adaptation and fine-tuning because it's small enough to retrain on custom data.

Sentence Transformers (SBERT) were specifically designed for sentence and document-level embeddings. They use a Siamese network architecture that learns to map sentences with similar meanings close together and sentences with different meanings far apart. This makes them more effective for semantic similarity tasks than raw BERT. Sentence Transformers support both dense retrieval and sparse retrieval patterns.

Domain-specific models are trained on specialized corpora. BGE-M3, released in 2024, supports dense, sparse, and multi-vector retrieval in a single framework. Cohere's embed-v4 is specifically trained to handle spelling errors, formatting inconsistencies, mixed content types, and scanned handwriting, making it particularly suited for enterprise documents that often contain OCR artifacts or messy formatting. NVIDIA's NV-Embed-v2, released in October 2024, is fine-tuned on Mistral-7B for large-scale enterprise retrieval workloads. This matters especially for document scanning software that must handle noisy OCR output. Choosing between these models depends on your document types, latency requirements, and cost constraints.

Chunking strategies for long documents

Embedding models have maximum token limits. BERT accepts up to 512 tokens. Longer models accept 8,000 or more. A 50-page contract exceeds any of these limits.

Chunking divides long documents into smaller pieces that fit within the model's context window. A simple approach chunks documents into fixed-size overlapping windows, perhaps 400 tokens with 50-token overlap. This preserves context across chunk boundaries. A more sophisticated approach identifies semantic boundaries and chunks intelligently: splitting at paragraph breaks or section boundaries rather than arbitrarily cutting through sentences.

Overlapping chunks are important. If a key concept spans chunk boundaries, non-overlapping chunks will miss it. The overlap tradeoff is direct: more overlap means more chunks, more embeddings to generate, and higher computational cost.

When you retrieve results, you get chunks, not whole documents. You then either return the chunks directly to the user or aggregate them to return full documents. Many production systems return the top 5-10 chunks and let the user navigate back to source documents.

Vector storage and similarity search

Once you have embeddings, you store them in a vector database: Pinecone, Weaviate, Milvus, or pg-vector for Postgres-backed systems. The database maintains indices that make similarity search fast.

The most common similarity metric is cosine similarity. It measures the angle between two vectors, ranging from -1 (opposite) to 1 (identical). In practice, cosine similarity > 0.8 usually indicates meaningful similarity; > 0.9 suggests near-duplicates. The choice of threshold determines recall and precision tradeoffs.

Large collections can't compare the query embedding to every document embedding individually. Instead, vector databases use approximate nearest neighbor (ANN) algorithms like HNSW or IVF to find similar embeddings much faster. The tradeoff is a small accuracy loss in exchange for massive speed gains.

When you query, you embed the query, then ask the vector database for the top K most similar embeddings. The database returns indices and similarity scores. You fetch the associated document chunks and return them to the user.

Choosing the right embedding approach for your document types

Model selection depends on four factors: accuracy requirements, latency budgets, cost constraints, and document characteristics.

Gemini Embedding 2 currently leads production benchmarks with 1605 ELO score on head-to-head model matchups and is the safest choice if your organization already uses Google Cloud services. For most RAG and semantic search applications, e5-small or e5-base-instruct offer the best value: they achieve 100% Top-5 accuracy while requiring less computational overhead than larger models. These choices directly affect the performance of document AI software and retrieval systems.

Cohere's embed-v4 is specifically designed for enterprise environments where documents are messy. If your document collection includes scanned contracts, handwritten notes, OCR output, or mixed formatting, Cohere's model outperforms general-purpose models because it was trained explicitly on this type of noise. Cohere also offers virtual private cloud deployment and on-premises options, which matter for regulated industries handling sensitive data.

Domain-specific fine-tuning is an option if you have a sufficiently large labeled dataset (typically 1,000+ examples). Fine-tuning adapts a pre-trained model to your specific vocabulary and document types. This improves accuracy but requires infrastructure and expertise to manage fine-tuning pipelines. Most organizations start with pre-trained models and only move to fine-tuning after validating that the generic models underperform on their specific task.

Cost considerations are straightforward: API-based services like OpenAI and Cohere charge per token. Self-hosted open-source models like BGE-M3 or e5 have zero per-token cost but require you to maintain infrastructure (GPUs or CPUs) and manage latency and scaling yourself. The breakeven point depends on your query volume and document size.

For sensitive or proprietary documents, self-hosting is often required. No document content leaves your infrastructure; embeddings are computed on your hardware or a private cloud instance. This eliminates data privacy concerns but requires technical investment. Enterprise document automation platforms often provide this option for organizations with stringent data residency requirements.

How Docsumo uses document embeddings

Docsumo's platform integrates embeddings across multiple capabilities. When you upload documents to Docsumo for intelligent document processing, the platform first applies document classification powered by semantic understanding. Rather than using brittle keyword rules, the classification engine understands document meaning and routes incoming documents to the right processing workflow. This is embedding-based classification working behind the scenes.

For knowledge management, Docsumo's document AI software enables semantic search across your entire document collection. You ask a natural language question, and the platform converts it to an embedding and searches your documents by meaning, not keywords. This is particularly powerful when you're working across documents created by different teams or organizations using inconsistent terminology.

The document scanning software generates OCR text from scanned pages, and embeddings help the platform understand what each page contains semantically, enabling better extraction accuracy and field mapping. When documents contain sections with similar content across different files, embeddings identify these similarities, allowing Docsumo to apply extraction rules more intelligently. Understanding which pages are related through semantic similarity rather than exact text matching is a key advantage.

For enterprise deployments, the intelligent document processing platform uses embeddings as part of its agentic document extraction. The system embeds each document, understands its semantic content, and makes intelligent decisions about which extraction rules to apply and how to validate extracted data. This is more flexible and accurate than rule-based extraction alone.

Docsumo's document classification feature uses embeddings to group documents by actual content similarity, not just by file type or naming patterns. This means misnamed files are still routed correctly, and documents with non-standard structures are handled appropriately. When documents require AI-powered data extraction, embeddings help the system understand structural patterns and content relationships that rule-based approaches would miss.

 FAQs

How much does it cost to generate embeddings for my document collection?

Cost depends on the approach. If you use an API service, you pay per token. OpenAI charges approximately $0.10 per 1 million input tokens. A 10,000-document collection with an average of 2,000 tokens per document (accounting for chunking) would cost roughly $2-3. But that's a one-time cost. Subsequent searches have minimal cost unless you're regenerating embeddings frequently. If you self-host an open-source model, there's zero per-token cost, but you pay for infrastructure: GPU instances on AWS cost $0.50-$5 per hour depending on size. The breakeven point is roughly 1-5 million documents, depending on your query frequency and infrastructure costs. When evaluating costs, consider whether a full document AI platform might provide better overall economics than building embeddings infrastructure from scratch.

Can I use embeddings offline, on my own infrastructure?

Yes. Open-source models like BGE-M3, e5, and Sentence Transformers can be downloaded and run on your own servers or private cloud instances. The only cost is infrastructure. This is essential for regulated industries and organizations handling classified or sensitive data that can't be sent to third-party APIs.

What if my documents are very short (like tweets or support tickets) or very long (like academic papers or legal filings)?

Short documents often work better with embedding models designed for short text or sentence transformers. For very long documents, chunking becomes critical. Academic papers and legal filings need careful chunking strategies that preserve semantic context. Overlapping chunks and intelligent boundary detection (splitting at section breaks rather than arbitrary word counts) both improve retrieval quality.

How do I handle domain-specific terminology that generic embedding models don't understand?

Pre-trained models work reasonably well even on specialized terminology because they learn semantic relationships from a broad corpus. However, if your domain uses highly specialized terms or abbreviations, fine-tuning improves performance. You can also use domain-specific models like those built for biomedical or legal documents, or combine embeddings with keyword search as a hybrid approach: embed for semantic similarity, then rerank results using keyword relevance.

What's the main tradeoff with embeddings versus keyword search?

Keyword search is simple and requires no additional infrastructure; you just index your documents. Embeddings require computation upfront (generating vectors for every document chunk), storage space (vector databases), and query-time computation (embedding the search query). The payoff is accuracy and speed on complex document collections. For small, simple collections where keyword search works, embeddings are overkill. For large collections with terminology variance or semantic complexity, embeddings pay for themselves through better search results and faster queries.