Data ingestion: types, process, tools and how to ingest documents
For data engineers, analysts and operations teams: what data ingestion is, how batch, streaming and micro-batch compare, a step-by-step process, the tools, and what changes when the source is PDFs and scans.
Key takeaways
- Data ingestion is moving data from where it's created, such as applications, databases, files, event streams and documents, into a system where it can be stored and used.
- There are three main patterns: batch (scheduled loads), streaming (continuous, event by event) and micro-batch (small, frequent loads).
- A typical pipeline runs identify sources, extract, validate, transform, load and monitor. In ELT, raw data is loaded first and transformed in the warehouse.
- Tools range from managed connectors for SaaS apps and databases to streaming platforms like Apache Kafka and serverless ETL like AWS Glue.
- Documents such as invoices and bank statements need an extra step: OCR and extraction turn each file into structured records before they can be ingested.
On this page
Data ingestion is the process of moving data from where it's created, such as applications, databases, files, event streams and documents, into a system where it can be stored, processed and analyzed, such as a data warehouse, data lake or operational database. It's the first stage of any data pipeline: nothing downstream works if the data doesn't arrive complete, on time and in a usable shape.
This guide covers the types of ingestion, the steps in a pipeline, tools, best practices and challenges, and what's different when the source is documents.
What is data ingestion?#
Ingestion covers everything between a source and the first place the data lands for use:
- Sources: SaaS applications, production databases, APIs, log files, IoT devices, spreadsheets, and documents such as invoices, bank statements and forms.
- Destinations: data warehouses, data lakes, lakehouses, operational databases and business applications.
It overlaps with, but isn't the same as, data extraction (pulling specific data from a source) and ETL (one pattern for moving and reshaping data).
Types of data ingestion#
| Type | How it works | Good for | Trade-offs |
|---|---|---|---|
| Batch | Loads data on a schedule, such as hourly or nightly | Reporting, reconciliation, stable sources | Data is only as fresh as the last run |
| Streaming (real time) | Processes each event as it happens | Fraud detection, monitoring, live dashboards | More complex and costly to run |
| Micro-batch | Loads small batches every few seconds or minutes | Near-real-time needs without full streaming | A middle ground on both freshness and cost |
A related choice is full vs incremental loading: re-copying everything each time, or only what changed since the last run, often using timestamps or change data capture (CDC).
How data ingestion works#
- Identify sources. List each source, its format, update frequency and owner.
- Extract. Pull data through APIs, database queries, CDC, file transfers or event streams.
- Validate. Check required fields, types, ranges and duplicates, and quarantine records that fail.
- Transform. Standardize formats, such as dates, currencies and IDs, and map fields to the target schema. In ELT, this happens after loading, inside the warehouse.
- Load. Write to the destination, in batches or as a stream.
- Monitor. Track volumes, freshness, failures and schema changes, and alert when something drifts.
Data ingestion tools#
- Managed connectors such as Fivetran and Airbyte replicate data from SaaS apps and databases into warehouses with little code.
- Apache Kafka is an open-source event streaming platform for high-volume, real-time pipelines; Kafka Connect adds source and sink connectors.
- AWS Glue is a serverless data integration service for discovering, preparing and moving data on AWS. Other clouds have equivalents.
- Orchestrators such as Apache Airflow schedule and monitor batch pipelines.
- Document processing platforms turn PDFs, scans and email attachments into structured records that can join the rest of your pipeline.
Ingesting documents#
Databases and APIs deliver data that's already structured. Documents don't. A bank statement, invoice or ACORD form arrives as a PDF, scan or photo, often by email, and the data inside it isn't usable until it's extracted. Document ingestion adds steps:
- Collect documents from email inboxes, upload portals, cloud folders and APIs.
- Classify and split each file into document types.
- Extract fields and tables with OCR and AI models.
- Validate the values, within each document and across related ones.
- Review exceptions with a person.
- Deliver structured JSON through an API or webhook, or load it into your warehouse or system of record.
Docsumo handles this pipeline for documents such as bank statements, invoices and ACORD forms, reports 99% field-level accuracy on 250+ document types, and delivers results through an API, webhooks and connectors.
Best practices#
- Define the target schema first so every source maps to the same model.
- Validate at the edge. Catch bad records before they reach the warehouse, and keep them for review instead of dropping them.
- Make loads idempotent. Re-running a job shouldn't create duplicates.
- Track lineage. Record where each record came from, and for documents, link each value to its page.
- Watch for schema drift. Alert when a source adds, removes or renames a field.
- Secure data in transit and at rest, and limit who can access raw data. For documents with personal or financial data, choose tools with SOC 2 Type 2, HIPAA and GDPR coverage.
Common challenges#
- Data quality. Missing values, duplicates and inconsistent formats. Fix with validation rules and quarantine queues.
- Volume and velocity. High-throughput streams need tools built for them.
- Many sources. Each new source means a new connector, mapping and test.
- Unstructured sources. Documents, emails and images need extraction before they can be ingested.
- Compliance. Personal and financial data must be handled under the regulations that apply to your business.
The bottom line#
Data ingestion gets data from where it's created to where it's used. Choose batch, streaming or micro-batch by how fresh the data must be, validate early, and monitor everything. When your sources include documents, add an extraction step so they arrive as structured records like everything else. To see document ingestion on your files, book a demo. For related reading, see what is data extraction.
Frequently asked questions#
What is data ingestion in simple terms?
It's the process of collecting data from its sources and bringing it into one place, such as a data warehouse, data lake or application database, so it can be analyzed or used.
What is the difference between data ingestion and ETL?
Ingestion is the broader job of getting data from sources into a destination. ETL (extract, transform, load) is one way to do it, where data is cleaned and reshaped before loading. See data extraction vs data ingestion.
What is the difference between batch and streaming ingestion?
Batch ingestion loads data on a schedule, such as nightly. Streaming ingestion processes each event as it happens, within seconds. Batch is simpler and cheaper; streaming suits fraud detection, monitoring and other time-critical uses.
How do you ingest data from PDFs and scanned documents?
Run the documents through OCR and a data extraction model to turn each one into structured fields and tables, validate the output, then load it like any other source. Intelligent document processing does these steps in one pipeline.