{"id":37453,"date":"2025-12-26T18:27:59","date_gmt":"2025-12-26T17:27:59","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/struggling-to-extract-data-from-1500-mixed-scanned-digital-pdfs-tesseract-ocr-and-vision-llms-all-failing-need-advice\/"},"modified":"2025-12-26T18:27:59","modified_gmt":"2025-12-26T17:27:59","slug":"struggling-to-extract-data-from-1500-mixed-scanned-digital-pdfs-tesseract-ocr-and-vision-llms-all-failing-need-advice","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/struggling-to-extract-data-from-1500-mixed-scanned-digital-pdfs-tesseract-ocr-and-vision-llms-all-failing-need-advice\/","title":{"rendered":"Struggling To Extract Data From 1,500+ Mixed Scanned\/digital PDFs. Tesseract, OCR, And Vision LLMs All Failing. Need Advice."},"content":{"rendered":"

<\/p>\n

\n

Hi everyone,<\/p>\n

I am working on my thesis and I have a dataset of about 1,500 PDF reports from the DGHS (Health Services). I need to extract specific table rows (District-wise Dengue stats) from them.<\/p>\n

The Problem:<\/strong> The PDFs are a nightmare mix. Some are digital with selectable text, but many are low-quality scans or photos of paper reports. The fonts are often garbled (mojibake) when extracted as text, and the layout changes slightly between years.<\/p>\n

What I have tried so far (and why it failed):<\/strong><\/p>\n

    \n
  1. Tesseract OCR:<\/strong> It struggled hard with the Bengali\/English mix and the table borders. The output was mostly noise.<\/li>\n
  2. Standard PDF scraping (pdfplumber\/PyPDF):<\/strong> Works on the digital files, but returns garbage characters (e.g., Kg\u2021dvU<\/code> instead of “Chittagong”) due to bad font encoding in the source files.<\/li>\n
  3. Ollama (Llama 3.1 & MiniCPM-V):<\/strong>\n