Building an invoice parser that actually works in production is harder than it looks. The demo is easy, you feed it a clean PDF, it extracts the invoice number and total amount, everyone's impressed. Then you try to process 8,000 real invoices from 200 different vendors, and suddenly 99% accurate becomes 70% usable.
This is the story of how we built a production-grade invoice parser from scratch, the mistakes we made, and what we learned along the way.
Starting with the right dataset
Most teams make the mistake of training on synthetic data or a handful of sample invoices. We decided to go straight to the source, the Kaggle high-quality invoice images for OCR dataset with 8,181 real invoice images. This meant we were not training on perfect documents. We were training on the messy reality: scanned invoices, varying templates, different languages, handwriting, poor scan quality.
The dataset was 1.14GB of raw images. Processing it required OCR extraction for every single invoice, a process that took 4+ hours even with parallel processing. But it gave us something synthetic data never could: real-world variance that our model needed to learn to handle.
The quality of your training data determines the quality of your production performance. There is no shortcut around this. Real invoices, with all their messiness, taught our model things that synthetic data simply could not.
The OCR challenge
Invoice OCR is deceptively hard. A clean digital PDF is straightforward. A scanned invoice from a vendor who uses a non-standard template, with text at angles, overlapping fields, and handwritten notes, that is where OCR engines struggle.
We implemented a tiered OCR approach: AWS Textract for high-quality documents when credentials are available, Tesseract as a robust offline fallback, and PyPDF2 for PDFs with embedded text.
The key insight: no single OCR engine wins across all invoice types. A router that selects the right engine based on document quality and type consistently outperforms any single engine.
The ensemble extraction approach
We did not rely on a single extraction method. We built an ensemble system that combines three approaches: regex patterns for high-confidence fields like invoice numbers, dates, and amounts; ML-based entity extraction using spaCy for the messier semantic fields; and heuristic rules for field relationships, if you find a PO number, look for vendor info nearby, that kind of thing.
This ensemble approach gave us 88.7% accuracy on our test set, significantly better than any single method alone. The patterns catch the obvious cases. The ML handles the nuanced ones. The heuristics catch edge cases the others miss.
Production-ready features
A demo invoice parser and a production invoice parser are different animals. Production needs batch processing with progress tracking, JSON and CSV export, per-field confidence scoring, error handling that does not crash on malformed invoices, and real-time monitoring dashboards.
We built all of this. The batch processing alone was crucial, processing 8,000 invoices one at a time would have taken days. With parallel batch processing, we could handle thousands in hours.
The training progress monitoring
One of the most valuable things we built was the training progress dashboard. Processing 8,181 invoices with OCR takes time, we needed to know exactly where we were. It shows current progress, percentage complete, estimated time remaining, and processing status in real time.
This is not just nice to have, it is essential for long-running processes. When something goes wrong at invoice #4,000, you want to know immediately, not discover it hours later.
What we would do differently
Looking back, there are things we would change. We would start with a smaller subset of the dataset before scaling up rather than processing everything at once. We would implement progress monitoring from day one, not after the first painful long run that teaches you why you need it. We would build per-field confidence thresholds from the start rather than a single global one. And we would add vendor-specific training earlier in the process instead of treating it as a later optimization.
The production reality
Our invoice parser is now in production, handling real invoices from real vendors. It is not perfect, no extraction system is. But it is reliable, it is fast, and it is improving over time as we capture human corrections and feed them back into the training pipeline.
The difference between a demo and production is the difference between it works on this one invoice and it works on thousands of invoices from hundreds of vendors. That gap is where the real engineering work happens, and that is exactly where we focused our effort.