What is data labeling? Types, process, techniques and quality checks
For teams building or buying machine learning, especially for documents: what data labeling is, how a labeling project runs, which techniques cut the effort, and how to measure label quality with a working example.
Key takeaways
- Data labeling is adding the correct answer (a label) to raw data such as images, text, audio or documents, so a supervised machine learning model can learn from it.
- Labels range from a single class for a whole item to boxes, spans, table cells and relations inside it.
- A labeling project runs through schema, guidelines, labeling, quality control and iteration; the guidelines matter as much as the labelers.
- Model-assisted labeling, active learning and weak supervision cut the manual work; LLMs can draft labels that people confirm.
- Check quality with inter-annotator agreement (for example Cohen's kappa) and a held-out gold set.
On this page
Data labeling is the process of adding the correct answer, called a label, to raw data such as images, text, audio or documents, so that a supervised machine learning model can learn from it. A labeled invoice might carry the class "invoice" and a box around the total with its value; a labeled email might carry the category "claim submission". Models learn patterns from thousands of these examples, then apply them to new, unlabeled data.
This guide covers the types of labels, how a labeling project runs, techniques that cut the manual effort, how to measure label quality (with a short code example), and what labeling looks like for document AI today.
Why data labeling matters#
A supervised model can only be as good as its labels. Inconsistent or wrong labels teach the model inconsistent or wrong answers, and no amount of model tuning fixes that. Labeled data also does a second job: a held-out labeled set is how you measure accuracy before and after deployment.
Types of data labels#
| Data | Label type | Example |
|---|---|---|
| Images and scanned pages | Class, bounding box, polygon, pixel mask | Page is a bank statement; box around the signature |
| Text | Class, entity span, relation, sentiment | Email is a complaint; "March 1, 2026" is the effective date |
| Documents | Document type, key-value fields, tables and cells, reading order | Invoice total, statement transactions, ACORD policy rows |
| Audio | Transcript, speaker, event | Call transcript with speaker turns |
| Tabular and time series | Outcome or class per record | Transaction is fraudulent or not |
How a data labeling project works#
- Define the task and schema. What will the model predict, and what exactly counts as each label? Write down fields, classes and formats.
- Collect representative data. Cover every source, layout and quality level the model will see in production, including hard cases.
- Write guidelines. Explain each label with examples and edge cases. Update them whenever labelers disagree.
- Choose a tool. Open-source tools such as Label Studio support image boxes, text spans and OCR-style labeling; document AI platforms have labeling built in.
- Label a pilot batch. Have two or more people label the same sample, compare, and fix the guidelines.
- Label at scale with quality control. Spot-check batches, track agreement, and route disagreements to an expert.
- Train, evaluate and iterate. Measure the model on a held-out gold set, find where it's weak, and label more of those cases.
Data labeling techniques#
| Technique | How it works | Best for |
|---|---|---|
| Manual labeling | People label every item | Small datasets, gold sets, high-stakes labels |
| Model-assisted labeling | A model pre-labels; people correct | Large datasets once a first model exists |
| Active learning | The model picks the items it's least sure about for people to label | Getting the most accuracy per labeled item |
| Weak or programmatic supervision | Rules and heuristics label data automatically, then are combined and cleaned | Large volumes where rules capture much of the signal |
| LLM-assisted labeling | A language model drafts labels from instructions; people review | Text and document tasks with clear definitions |
| Crowdsourcing | Many non-expert workers label simple items | Simple tasks with no sensitive data |
For financial and personal documents, crowdsourcing is usually ruled out by privacy requirements; labeling stays with trained staff or a vetted vendor under a data agreement.
How to measure label quality#
The standard check is inter-annotator agreement: two people label the same items and you measure how often they agree beyond chance. Cohen's kappa does this for two annotators. This example uses scikit-learn (tested with Python 3.11 and scikit-learn 1.9):
from sklearn.metrics import cohen_kappa_score
# Document types assigned to the same 10 pages by two annotators
annotator_a = ["invoice", "invoice", "statement", "pay_stub", "invoice",
"statement", "pay_stub", "invoice", "statement", "invoice"]
annotator_b = ["invoice", "invoice", "statement", "pay_stub", "statement",
"statement", "pay_stub", "invoice", "statement", "invoice"]
agreement = sum(a == b for a, b in zip(annotator_a, annotator_b)) / len(annotator_a)
kappa = cohen_kappa_score(annotator_a, annotator_b)
print(f"Raw agreement: {agreement:.0%}")
print(f"Cohen's kappa: {kappa:.2f}")
Output:
Raw agreement: 90% Cohen's kappa: 0.84
Kappa is lower than raw agreement because it discounts agreement expected by chance. On the widely used Landis and Koch (1977) scale, values above 0.8 indicate almost perfect agreement. Low kappa on a class usually means its definition is unclear: fix the guideline, then relabel.
Also keep a gold set labeled by experts and check each labeler against it, and track agreement per class, since overall numbers hide weak spots.
Challenges in data labeling#
- Cost and time. Manual labeling is slow. Use model-assisted labeling and active learning to focus effort.
- Consistency. Different people interpret the same rule differently. Guidelines, calibration rounds and agreement checks help.
- Edge cases. Rare cases matter most in production and are rarest in data. Seek them out deliberately.
- Privacy. Documents contain personal and financial data. Limit access, use secure tools and keep data within approved environments.
- Drift. New layouts and sources appear. Keep labeling a sample of production data over time.
Data labeling for document AI#
Document models need labels for document types, fields, tables and sometimes relations, which makes document labeling more detailed than simple classification. The good news is that far less of it is needed than a few years ago. Pre-trained models already cover common documents such as bank statements, invoices and pay stubs, and reviewer corrections in production become new labels automatically.
Docsumo covers 250+ document types with pre-trained models, and custom models can be trained for documents specific to your organization, with 99% field-level accuracy. See the platform and pricing, or read document annotation for a document-specific guide.
The bottom line#
Labels are the ground truth a model learns from and is measured against. Invest in a clear schema and guidelines, use models and active learning to reduce manual work, and measure agreement so you know your labels, and therefore your model, can be trusted.
Frequently asked questions#
What is data labeling in machine learning?
It's the process of attaching the correct output to each training example, such as "this page is an invoice" or "this box is the total amount", so a supervised model can learn the mapping from input to output.
What is the difference between data labeling and data annotation?
The terms are often used interchangeably. Annotation sometimes means richer markup (boxes, spans, relations), while labeling can mean assigning a class. For documents, see document annotation.
How do you measure data labeling quality?
Have multiple people label the same sample and measure agreement, for example with Cohen's kappa, and compare labels against a gold set labeled by experts. Low agreement usually means the guidelines need work.
Can AI label data automatically?
Partly. Pre-trained models and LLMs can propose labels, and active learning picks the examples most worth a person's time. People still need to review, especially for edge cases and high-stakes fields.
How much labeled data do I need?
It depends on the task and model. Fine-tuning a pre-trained model can work with hundreds of examples per class; training from scratch needs far more. Start small, measure, and label more where the model is weakest.