Hot AI News

Intelligent document processing: How software turns scans and PDFs into data you can trust

Intelligent document processing turns scans and PDFs into structured fields, and attaches a confidence score so a person can check the doubtful ones. How it works, where it came from, and its limits.

Intelligent document processing: How software turns scans and PDFs into data you can trust
Photo illustration: Aiholics

Intelligent document processing is software that reads a scan, photo or PDF and returns usable data instead of a picture of text. It finds the words, works out how they relate (this is a field name, that is its value, these cells form a table), and says how sure it is about each answer.

A patient form that becomes a spreadsheet row

Picture a clinic’s paper intake form, scanned as a PDF. A person sees “Name: Jane Doe” and knows that Jane Doe is the name. To a computer, the scan is just pixels.

A document processing service turns those pixels into text, then into pairs such as Name and Jane Doe, then into a row in a database. That is the example AWS uses for its new agent skill, described in our report on Amazon Textract Core as an agent skill: an agent turns an intake form into structured data and marks unclear fields for staff to check.

Advertisements

Three layers: Reading letters, reading layout, reading meaning

A pen display shows a document divided into layout regions and converted into structured fields, with one field highlighted for review.
Photo illustration: Aiholics

The first layer is optical character recognition, or OCR. It detects printed (and sometimes handwritten) characters and returns lines and words with their position on the page.

The second layer is layout. Plain OCR output is a stream of text, and a table read left to right across the whole page makes no sense. Layout analysis groups text into key-value pairs, checkboxes and table cells.

The third layer is meaning. Amazon documents a QUERIES feature in its AnalyzeDocument API: the caller asks a plain-language question, such as what the patient’s date of birth is, and gets an answer back with its own score. Specialized extractors for invoices, receipts and identity documents sit at this level too.

Confidence scores decide who looks at what

Every extracted item comes with a number saying how likely the model thinks it is right. Amazon’s Textract FAQ describes a score from 0 to 100 for each extracted element, along with a bounding box showing where the item sits on the page.

The score is what makes automation safe to use. AWS suggests setting thresholds: accept high-confidence values automatically and send low-confidence ones to human review. Its own example is a rule that flags any extracted value scoring below 95 percent.

That threshold is a business decision, not a technical constant. A low bar saves labor but lets more errors through. A high bar catches more mistakes but sends more work to people.

From separate steps to models that see text and layout together

Early systems ran OCR first and rule-based code second, with templates for each form layout. A new vendor’s invoice meant a new template.

A shift came in 2019 and 2020. Researchers at Microsoft Research Asia, with university collaborators, posted LayoutLM: Pre-training of Text and Layout for Document Image Understanding on arXiv at the end of 2019. It extended the BERT language model with two-dimensional position information, so the model learns from where words sit, not only from what they say.

The authors pre-trained it on more than 6 million scanned documents from the IIT-CDIP collection. They reported gains on form understanding, receipt understanding and document image classification. Later models added image features as well. The idea of learning from text and layout together now underlies most commercial document extraction, though vendors rarely publish the internals of their own models.

Advertisements

Who offers it, and how agents plug in

The major cloud providers sell this as a service. Amazon Textract is AWS’s version. Its developer guide lists text, forms, tables, query responses and signatures as outputs, with layout analysis as a further option. It accepts PDF, PNG, JPEG and TIFF files.

Until recently, a team that wanted an AI agent to use such a service had to write its own integration. The newer approach packages instructions and helper code as a reusable skill, which the agent loads when it meets a document. In AWS’s announcement, the skill steers the agent to the right API for each document and bundles a parser that turns raw output into text in reading order plus structured fields. It is available through the AWS MCP server or the AWS Agent Toolkit on GitHub, so agents such as Kiro, Claude Code and Cursor can use it. The skill adds no charge; standard Textract pricing applies to the pages processed.

Where it still goes wrong

Language coverage is uneven. According to the Textract FAQ, printed text is detected in more than 40 languages, but form key-value extraction covers six: English, French, German, Italian, Portuguese and Spanish. Handwriting, invoices, receipts, identity documents and Queries are listed as English only.

Image quality matters. AWS advises supplying images of at least 150 DPI (dots per inch) and says table results are best when tables are visually separated from the surrounding page. A skewed phone photo of a crumpled receipt is harder than a clean scan.

A confidence score is also not a guarantee. It is the model’s own estimate, and a confidently wrong value can pass a threshold. For high-stakes fields, such as dosages, amounts and identity numbers, a review step or a cross-check against another source still makes sense.

One more limit is cost. Pricing is per page, so an agent that sends a long document repeatedly, or sends every page when one would do, can quietly multiply the bill.

Advertisements

What to watch next

Three things are worth following. The first is how many agent platforms ship ready-made document skills, and whether results stay consistent between them. The second is whether vendors publish accuracy figures by document type and language, so thresholds can be set from evidence instead of habit. The third is the line between classic extraction services and general-purpose multimodal models that read a page directly, and how each handles confidence reporting. For now, the pattern is stable: extract, score, and hand the doubtful fields to a person.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like