Confidence scores are only useful if something acts on them

A client called about an invoice we had processed for them. The total was wrong by a factor of ten. A smudged decimal point, and the model had read 4,850.00 as 48,500.00.

Here is the part that stung. Their accounting system had already paid it.

My platform takes a PDF and a schema and returns structured data. Invoices, delivery orders, statements. When it crashes, someone retries it and nothing bad happens. When it quietly returns the wrong number, money moves.

I could not make the model stop misreading smudged decimals. What I could do was make it tell me when it was unsure, and put a person in front of those cases before anything downstream saw them.

Ask for confidence in the schema

If you request structured output against a JSON Schema, the confidence score belongs in that schema alongside the value. Both OpenAI’s structured outputs and the equivalent constrained-decoding features elsewhere will hold the model to the shape.

{
  "type": "object",
  "properties": {
    "invoice_total": {
      "type": "object",
      "properties": {
        "value":      { "type": "number" },
        "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
        "source_page":{ "type": "integer" }
      },
      "required": ["value", "confidence", "source_page"]
    }
  }
}

Do not read too much into the number. When a model says 0.9 it does not mean it will be right nine times out of ten. It is not a probability, it is a hunch.

But it is a useful hunch. The fields it marks 0.4 really are wrong far more often than the ones it marks 0.95. That ranking is all a threshold needs.

The source_page field pulls its weight too. Without it a reviewer opens a ten-page PDF and starts hunting. With it they land on the page and look at one spot.

Four outcomes, not two

The mistake I made first was treating this as pass or fail. Extraction has more states than that, and collapsing them sends the wrong work to the wrong place.

Outcome	Condition	Route to
Accepted	All required fields at or above threshold	Downstream automatically
Needs review	Any required field below threshold	Human queue
Missing data	Field absent from the document entirely	Missing-data branch, not review
Failed	Extraction or mapping threw	Retry, then engineering

Splitting missing from uncertain matters more than it looks. “The model is unsure what the total is” and “this document has no total” need different human actions, and mixing them trains reviewers to skim.

Failed is separate again because a reviewer cannot fix a 500 from the OCR service.

Where do you set the threshold?

Against labelled documents, and not before you have some.

Take a sample of documents representative of what you actually receive, extract them, and record the model’s confidence next to whether the value was correct. Then read off the curve. Raising the threshold catches more errors and sends more work to humans; lowering it does the reverse. Where you land depends entirely on the cost of a wrong value getting through.

Two things are worth knowing before you tune. The threshold is per document type, because a clean digital PDF and a photographed receipt do not share an error profile. And it drifts, because document mixes change and models get swapped. Mine is configuration, not a constant, and it gets revisited.

Google’s human-in-the-loop documentation for Document AI describes the same routing shape, which is some comfort that this is the conventional design and not a local invention.

The queue is a product, not a table

The goal of a review queue is to make review fast, not to eliminate it. That distinction changes what you build.

A reviewer should see the extracted value, the confidence, and the region of the source page it came from, side by side, with the field focused and the cursor in it. Confirm is one key. Correct is type-and-enter. If a reviewer has to open the original PDF in another tab and search for the number, you have built a table with extra steps.

Two rules that came out of watching people use mine. Sort by confidence ascending, so the most likely errors are cleared first and a half-finished queue still has value. And keep corrections, because the difference between what the model said and what the human typed is the highest-quality evaluation data you will ever get for free.

The part that has no technical fix

If almost everything passes the threshold, reviewers stop reading and start clicking confirm. The queue becomes theatre and the errors flow through with human approval attached, which is worse than having no queue because now there is a name on the mistake.

I do not have a clean answer to this. What helps: keep the queue small enough to stay meaningful, sample the auto-accepted stream periodically to check the threshold is still honest, and watch review time per item. When it drops below a couple of seconds, people have stopped looking.

Sources

• Human-in-the-loop for Document AI, Google Cloud documentation
• Structured outputs, OpenAI platform documentation
• Use Amazon Augmented AI for human review, AWS documentation