TLDR;<\/strong> We were overpaying for OCR, so we compared flagship models with cheaper and older models by creating a new, curated dataset including standard documents you’d find in real-world industry.<\/p>\n We\u2019ve been looking at OCR \/ document extraction workflows and kept seeing the same pattern:<\/p>\n Too many teams are either stuck in legacy OCR pipelines, or are overpaying badly for LLM calls by defaulting to the newest\/ biggest model.<\/p>\n We put together a curated set of 42 standard documents<\/strong> and ran every model 10 times under identical conditions; 7,560 total calls. Main takeaway: for standard OCR, smaller and older models match premium accuracy at a fraction of the cost.<\/p>\n We track pass^n (reliability at scale), cost-per-success, latency, and critical field accuracy.<\/p>\n All documents are non-redacted due to synthetic data. Yet, all documents are real-world representative because their information density is similar, only the actual data content is synthetic.<\/p>\n Dataset Hugginface:<\/strong> https:\/\/huggingface.co\/datasets\/Timokerr\/OCR_baseline<\/a><\/p>\n Benchmark Harness Repo: https:\/\/github.com\/ArbitrHq\/ocr-mini-bench<\/a><\/p>\n Curious whether this matches what others here are seeing.<\/p>\n<\/div>\n submitted by \/u\/TimoKerre <\/a> \n
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
