Text recognition algorithms: 10 methods behind modern OCR

Text recognition algorithms are the methods OCR systems use to find text in an image and read it. Modern OCR splits the job into text detection, text recognition and decoding. Detection algorithms such as CTPN, EAST, CRAFT and DBNet find where the text is. Recognition models read each region. They include CRNN, attention decoders and transformers such as TrOCR. In decoding, CTC and beam search turn model outputs into words. Open-source engines package these algorithms for day-to-day use. These engines include Tesseract (LSTM since version 4), PaddleOCR, EasyOCR and docTR.

How modern text recognition works

Take an invoice that arrives as a phone photo, tilted a few degrees and lit unevenly. Preprocessing straightens and cleans the image first. Then a detector such as EAST or CRAFT draws a box around every line of text on the page. The vendor's address, each line item and the total all get a box, wherever they appear in the photo.

Each box then goes to a recognizer. A CRNN reads the pixels inside a box as a sequence. It's trained with CTC, so it outputs the characters without ever being told where one letter ends and the next begins. A transformer such as TrOCR can do the same job on a stamped or handwritten line. Decoding turns those per-step outputs into the most likely string for each box.

Older engines matched character shapes against templates, and trained networks like these have replaced that approach since the mid-2010s. Either way, what comes out is still loose text, not a field. Nothing yet has said which line is the total or checked it against the line items above it.

Four of the methods below are detectors and three are recognizers. CTC and word beam search are the training and decoding methods recognizers rely on. The last method on the list reads documents rather than lines.

Text detection algorithms

Detectors find where the text is and return boxes or polygons for the recognizer to read.

• 1. CTPN (2016)

  The Connectionist Text Proposal Network links a row of thin, fixed-width text proposals with a recurrent network. Good on horizontal lines such as printed documents; weak on rotated or curved text.

• 2. EAST (2017)

  The Efficient and Accurate Scene Text detector predicts rotated boxes or quadrilaterals at every pixel in one fully convolutional pass. It's fast, which made it popular for real-time and mobile scanning.

• 3. CRAFT (2019)

  Character Region Awareness for Text detection finds each character and the links between characters, then groups them into words. Working at the character level helps with curved, irregular and widely spaced text.

• 4. DBNet (2019)

  Differentiable Binarization segments text regions and learns the binarization threshold during training, so it's both accurate and fast. Open-source engines such as docTR use it as a detector.

Text recognition algorithms

Recognizers read the text inside each region. CRNN-style models are trained and decoded with CTC; attention and transformer models decode the text one step at a time.

• 5. CRNN (2015)

  The Convolutional Recurrent Neural Network extracts visual features with convolutional layers, models the character sequence with LSTM layers and trains end to end with CTC. It reads a whole word or line at once and became the standard baseline.

• 6. CTC

  Connectionist Temporal Classification lets a network output a character sequence without knowing where each character sits in the image. It does this by allowing "blank" outputs and merging repeats. Tesseract's LSTM engine, introduced in version 4, uses the same kind of sequence recognition.

• 7. Attention-based encoder-decoders

  Instead of CTC, an attention mechanism focuses on one part of the image for each character it generates. They handle irregular and curved text better, often after a rectification step that straightens it.

• 8. TrOCR (2021) and PARSeq (2022)

  Transformer recognizers. TrOCR pairs a pre-trained image transformer with a pre-trained text transformer and reads printed and handwritten text. PARSeq reads with or without language context and is a strong scene text recognizer.

• 9. Word beam search

  A CTC decoder that keeps words to a dictionary while still allowing numbers and punctuation between them. It improves word accuracy on handwriting when the vocabulary is known.

Understanding documents, not just text

10. Layout-aware and OCR-free document models

Reading characters doesn't tell you which number is the invoice total. LayoutLMv3 (2022) combines text, position and image in one model. It's trained for tasks such as field extraction and document classification. Donut (2021) goes further. It's an OCR-free transformer that reads a document image and outputs structured data directly. Large vision-language models now do similar things at a higher cost per page.

Text recognition algorithms compared

Open-source OCR engines

For a hands-on guide to the most common one, see Tesseract OCR.

How to choose

What we'd do. A paper's benchmark score describes the test set its authors picked, not the invoices, statements or leases your team reads. Before settling on Tesseract, PaddleOCR, EasyOCR, docTR or a paid API, run the candidates on a sample of your own documents. Count what a person actually had to correct, instead of relying on the number in the paper. That's the figure that predicts what happens once it's handling real volume.

From text recognition to document data

For lending, insurance and finance teams, text alone isn't the target. What matters is the right value in the right field. Think of the closing balance on a bank statement, the total on an invoice or the limits on an ACORD certificate. That takes recognition plus layout understanding, validation and review. Docsumo's document AI combines them to reach 99% field-level accuracy on 250+ document types. For how accuracy is measured, see OCR accuracy.

What the papers don't test

Detectors such as EAST, CRAFT and DBNet find the text; recognizers such as CRNN with CTC and transformers read it; decoders clean up what's left. Every one of those methods was scored in a paper against images its own authors chose. Those images weren't the invoices, statements or forms your team will process. Add layout-aware models, validation and human review on top of whichever pipeline you choose. That way, the output is data you can trust rather than just text.

Book a demo to see it on your own documents, or start a free trial.

Frequently asked questions