Image segmentation: types, techniques, tools and document uses

Image segmentation splits an image into regions. It does so by labeling each pixel. A box only tells you a car is somewhere in the picture. Segmentation marks which pixels are the car, which are the road and which are the sky. Doctors use it on scans, and self-driving cars use it on the road. Satellite analysis and retail use it too. In document processing, it separates text from tables and signatures before they're read.

What is image segmentation in image processing?#

A computer sees an image as a grid of pixel values. Segmentation groups those pixels into regions that mean something, such as an organ, a vehicle, a building or a paragraph. The output is a mask, an image the same size as the original where each pixel carries a label.

Segmentation sits next to two other computer vision tasks.

• Image classification gives one label for the whole image ("this is an invoice").
• Object detection draws a bounding box around each object.
• Segmentation labels every pixel, capturing exact shapes and boundaries.

Types of image segmentation#

A 2019 paper by Kirillov and colleagues defined panoptic segmentation to combine the other two.

Image segmentation techniques#

Classic techniques

• Thresholding. Pixels on one side of a brightness value are marked as the object (foreground), and the rest as background. For dark text on a light page, the dark pixels are the object. Simple and fast, and still the standard first step on scanned pages. Otsu's method picks the threshold automatically.
• Edge-based. Detects sharp changes in intensity (for example with the Canny detector) and treats them as object boundaries.
• Region-based. Starts from a few chosen pixels and keeps adding neighboring pixels that look similar. Another version splits and joins regions. The watershed algorithm is a common example.
• Clustering. Groups pixels by color or texture with algorithms such as k-means, without labeled training data.

Here is thresholding on a scanned page with OpenCV, the usual first step before OCR.

import cv2

img = cv2.imread("scanned_page.png", cv2.IMREAD_GRAYSCALE)
img = cv2.GaussianBlur(img, (5, 5), 0)

# Otsu picks the threshold; THRESH_BINARY_INV makes text white on black
_, mask = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

# Merge characters into text blocks, then find each block's outline
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (25, 5))
blocks = cv2.dilate(mask, kernel, iterations=2)
contours, _ = cv2.findContours(blocks, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
print(f"{len(contours)} text regions found")

Deep learning techniques

Most production segmentation now uses neural networks trained on labeled masks.

• U-Net (2015) is a neural network that first shrinks an image to find what is in it, then expands it back to label each pixel. It was designed for biomedical images and is still widely used when there are few labeled training images.
• Mask R-CNN (2017) extends an object detector to predict a mask for each detected object, making it a standard choice for instance segmentation.
• Transformer-based models handle semantic, instance and panoptic segmentation in one architecture.
• Promptable foundation models. Meta's Segment Anything family segments objects from a prompt, without training on your specific objects. A prompt can be a click or a box. Since SAM 3 in November 2025, it can also be a short text phrase or example image.

Tools and libraries#

• OpenCV: open-source computer vision library with thresholding, edge detection and watershed, plus contour finding, for Python, C++ and Java.
• scikit-image: Python library with classic segmentation algorithms.
• PyTorch and TensorFlow: deep learning frameworks for training U-Net, Mask R-CNN and transformer models.
• MATLAB Image Processing Toolbox: interactive segmentation tools, common in research.
• Annotation tools such as VGG Image Annotator and Labelbox, for drawing the masks you train on. See image classification.

How to measure segmentation quality#

• Intersection over Union (IoU): the overlap between predicted and true mask divided by their combined area. 1.0 is a perfect match.
• Dice coefficient: twice the overlap divided by the total size of both masks; common in medical imaging.
• Pixel accuracy: share of pixels labeled correctly. Easy to read, but misleading when one class dominates, such as a page that's mostly white background.
• Mean IoU across classes and, for panoptic tasks, panoptic quality (PQ).

Where image segmentation is used#

• Document processing

  Splitting pages into text, tables, figures, signatures and stamps.

• Medical imaging

  Outlining organs and tumors in CT and MRI scans for diagnosis and treatment planning.

• Satellite and aerial imagery

  Mapping land use, crops, flooding and buildings.

• Retail

  Isolating products for catalogs and visual search.

• Manufacturing

  Finding defects on parts and surfaces.

• Autonomous driving

  Separating the road and lanes from vehicles and pedestrians.

Image segmentation in document processing#

A single page can mix typed text, handwriting, tables, logos, stamps, checkboxes and signatures. Reading it well starts with segmentation.

1. Separate foreground from backgroundSo faint or noisy scans can be read.
2. Find layout regionsHeaders, paragraphs, key-value blocks, tables and figures. This sets the reading order for multi-column pages.
3. Segment tables into cellsSo each value lands in the right row and column, even when a table runs across pages.
4. Separate the special partsThe tool checks whether a signature is present. Stamps and handwriting go to a model built for them.

Document AI platforms do this as part of extraction. Docsumo, for example, reads fields and tables from documents such as bank statements and invoices. It joins tables that run across pages into one, and reports 99% field-level accuracy on 250+ document types. See Docsumo Document AI and our guide to table extraction from complex PDFs.

We'd treat segmentation as one step inside a document tool, not as something to shop for on its own. Document AI, document intelligence and IDP (intelligent document processing) are different names for the same kind of software. We wouldn't pick vendors by the name they use. We'd run the same files through each tool, scans and long tables included, and compare what comes out.

Challenges in image segmentation#

• Labeled data. Pixel-level masks are slow to draw. Pre-trained and promptable models reduce how many you need.
• Boundaries and small objects. Thin structures and overlaps are the hardest to get right, along with tiny objects.
• Domain shift. Images unlike the training set cause errors. A model trained on clean images struggles with blur, glare or new layouts.
• Compute. Deep models need GPUs to train, and sometimes to run at speed.
• Evaluation. Pick metrics that reflect the task: IoU on table cells matters more than overall pixel accuracy on a mostly blank page.

When a threshold is enough and when you need a model#

Image segmentation labels every pixel so software can tell where one thing ends and another begins. Classic methods like thresholding still do useful work, especially on documents. Deep learning and promptable models handle complex scenes. For documents, segmentation lets a system read tables and signatures correctly, even on multi-column layouts.

Book a demo with a few of your own documents, or start a free trial.

Frequently asked questions#