OCR Benchmark Focusing on Automation
OCR/Document extraction field has seen lot of action recently with releases like Mixtral OCR, Andrew Ng's agentic document processing etc. Also there are several benchmarks for OCR, however all testing for something slightly different which make good comparison of models very hard. To give an example, some models like mixtral-ocr only try to convert a document to markdown format. You have to use another LLM on top of it to get the final result. Some VLM’s directly give structured information like key fields from documents like invoices, but you have to either add business rules on top…
OCR Benchmark Focusing on Automation launched on March 12, 2025 with 58 votes, #507 of 7,215 launches that month and more than 90% of that year's launches. The site was not reachable when checked in September 2026.
In plain words
OCR Benchmark Focusing on Automation is a benchmarking tool designed to address inconsistencies in evaluating optical character recognition and document extraction models. It assesses the actual automation rate achievable when processing documents, unlike existing benchmarks that test individual components in isolation. The tool targets teams evaluating document processing solutions, particularly those working with invoices and structured data extraction. It aims to provide standardized comparison across different OCR approaches, from markdown conversion models to vision language models that extract specific fields.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
OCR/Document extraction field has seen lot of action recently with releases like Mixtral OCR, Andrew Ng's agentic document processing etc. Also there are several benchmarks for OCR, however all testing for something slightly different which make good comparison of models very hard. To give an example, some…
Does a similar job
all alternatives →- BVBenchmarking VLMs vs. Traditional OCR2025 · getomni.ai · ▲146
Vision models have been gaining popularity as a replacement for traditional OCR. Especially with Gemini 2.0 becoming cost competitive with the cloud platforms. We've been continuously evaluating different models since we released the Zerox package last year (https://github.com/getomni-ai/zerox). And we wanted to put some numbers behind it. So we’re open sourcing our internal OCR benchmark + evaluation datasets. Full writeup + data explorer here: https://getomni.ai/ocr-benchmark Github: https://github.com/getomni-ai/benchmark Huggingface:…
Zerox – Document OCR with GPT-mini2024 · github.com · ▲246This started out as a weekend hack with gpt-4-mini, using the very basic strategy of "just ask the ai to ocr the document". But this turned out to be better performing than our current implementation of Unstructured/Textract. At pretty much the same cost. I've tested almost every variant of document OCR over the past year, especially trying things like table / chart extraction. I've found the rules based extraction has always been lacking. Documents are meant to be a visual representation after all. With weird layouts, tables, charts, etc. Using a vision model just make sense! In…
PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o2024 · github.com · ▲191I've developed a Python API service that uses GPT-4o for OCR on PDFs. It features parallel processing and batch handling for improved performance. Not only does it convert PDF to markdown, but it also describes the images within the PDF using captions like `[Image: This picture shows 4 people waving]`. In testing with NASA's Apollo 17 flight documents, it successfully converted complex, multi-oriented pages into well-structured Markdown. The project is open-source and available on GitHub. Feedback is welcome.
Qwen-2.5-32B is now the best open source OCR model2025 · github.com · ▲211Last week was big for open source LLMs. We got: - Qwen 2.5 VL (72b and 32b) - Gemma-3 (27b) - DeepSeek-v3-0324 And a couple weeks ago we got the new mistral-ocr model. We updated our OCR benchmark to include the new models. We evaluated 1,000 documents for JSON extraction accuracy. Major takeaways: - Qwen 2.5 VL (72b and 32b) are by far the most impressive. Both landed right around 75% accuracy (equivalent to GPT-4o’s performance). Qwen 72b was only 0.4% above 32b. Within the margin of error. - Both Qwen models passed mistral-ocr (72.2%), which is specifically trained for OCR. - Gemma-3…
OCR pipeline for ML training (tables, diagrams, math, multilingual)2025 · github.com · ▲170Hi HN, I’ve been working on an OCR pipeline specifically optimized for machine learning dataset preparation. It’s designed to process complex academic materials — including math formulas, tables, figures, and multilingual text — and output clean, structured formats like JSON and Markdown. Some features: • Multi-stage OCR combining DocLayout-YOLO, Google Vision, MathPix, and Gemini Pro Vision • Extracts and understands diagrams, tables, LaTeX-style math, and multilingual text (Japanese/Korean/English) • Highly tuned for ML training pipelines, including dataset generation and…
Ocrbase – pdf → .md/.json document OCR and structured extraction APIJan 2026 · github.com · ▲99📄 PDF/IMG ->.MD/JSON Document OCR API for PaddleOCR and GLMOCR. Self-hostable. - majcheradam/ocrbase
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, March 2025
the whole month →
Mimic Human Research & Save Findings in AI Knowledge Base
AI · 2025 · sider.ai
