Document embeddings: what they are, how they work and which model to use

Document embeddings are numerical vectors that represent what a document means, so documents with similar content sit close together even when they use different words. An embedding model turns each document, or each chunk of it, into a vector, and a vector database stores those vectors for semantic search, clustering, classification and similarity matching. That lets search find related documents by concept, where keyword search finds only exact word matches.

This guide covers how embeddings compare with keyword search, how they're made and searched, and how to choose a model.

What are document embeddings?#

An embedding model reads text and outputs a fixed-length list of numbers, often 384 to 3,072 of them. The model learned from large amounts of text which words and phrases mean similar things, so "indemnification", "liability" and "hold harmless" land near each other, while unrelated documents land far apart. Embedding quality depends on the model, what it was trained on, and how you apply it to your documents.

• Semantic search

  Find passages by meaning, not exact wording.

• Clustering

  Group similar reports, tickets or contracts without labeling them first.

• Classification

  Train a light classifier on embeddings to sort documents by type or topic.

• Duplicate detection

  Spot near-identical documents that differ in formatting or wording.

• Retrieval for RAG

  Fetch the right chunks for a language model to answer from.

Keyword search vs semantic search#

Keyword search matches the words in the query. Say a legal team searches its contracts for "indemnification obligations related to data breaches": a keyword index returns contracts that use those words and misses those that say "hold harmless" or "cybersecurity incident" for the same obligation. Semantic search embeds the query and returns the nearest documents by meaning, so it finds both.

Neither wins everywhere: semantic search can miss an exact invoice or policy number that keyword search finds at once. Most production systems run both and merge the results, which is called hybrid search.

How document embeddings work#

Tokenize and encode. The model splits text into tokens (words or word pieces), processes them through its layers and outputs one vector per chunk. Sizes vary: BERT-base produces 768 dimensions, and OpenAI's text-embedding-3-small and -large produce 1,536 and 3,072 by default, which can be shortened to save storage.

Chunk long documents. Every model has an input limit: 512 tokens for BERT, 8,192 for BGE-M3 and OpenAI's text-embedding-3 models. A 50-page contract has to be split.

1. Split at natural boundariesSections, headings or paragraphs, rather than a fixed word count that cuts sentences in half.
2. Overlap the chunksA little overlap, such as 50 tokens on a 400-token chunk, keeps ideas that cross a boundary; more overlap means more vectors to store.
3. Keep the sourceStore each chunk's document ID and page, so a result links back to where it came from.
4. Return chunks, then documentsShow the top 5 to 10 chunks, or group them by document.

Store and search. Vectors go into a vector database such as Pinecone, Weaviate, Milvus or pgvector for Postgres. Similarity is usually cosine similarity, and approximate nearest neighbor indexes such as HNSW or IVF keep queries fast across millions of vectors, trading a little accuracy for speed. What score counts as "similar" depends on the model, so set thresholds on pairs from your own documents.

Which embedding model works best for documents?#

• BERT-style encoders

  Small enough to fine-tune on your own labeled data; you pool token vectors into one document vector.

• Sentence Transformers (SBERT)

  A Python library of models trained so that similar sentences and passages get similar vectors; open and self-hostable.

• BGE-M3

  An open model (MIT license) for dense, sparse and multi-vector retrieval in 100+ languages, with inputs up to 8,192 tokens.

• API models

  OpenAI's text-embedding-3 models, and Cohere's Embed v4, which embeds text, images and mixed PDFs with a 128,000-token context.

• Test on your documents and queriesBuild 50 to 100 real queries with known right answers and compare which model ranks them highest, using recall@k or mean reciprocal rank (MRR).
• Match context length to your chunksA short input limit forces smaller chunks.
• Check languagesIf your documents aren't all in English, test each language you have.
• Decide where it runsSelf-host an open model when documents can't leave your infrastructure; use an API when they can.
• Price it at your volumePer-token API fees against the cost of running your own servers, including every time you re-embed.
• Plan for OCR textScanned documents need OCR first, and OCR errors carry into the embeddings, so check text quality before blaming the model.

Fine-tune only after a pre-trained model has proven weak on your task and you have enough labeled examples.

The bottom line#

Embeddings let software find and group documents by meaning, which keyword search can't do alone. Chunk carefully, test models on your own queries, and keep keyword search in the mix for exact values.

Docsumo is an intelligent document processing platform, not an embeddings tool: it extracts checked fields and tables from business documents such as invoices and bank statements, and exact values like those can sit alongside the text you embed. See document classification and agentic document extraction, or book a demo.

Frequently asked questions#