# Confidence scores are only useful if something acts on them

An LLM that returns a per-field confidence score has told you nothing until a threshold routes the low ones to a person. How to design the routing, calibrate the threshold, and build a review queue people can actually clear.

By Elson Tan (https://elsontan.com)
Published: 2026-01-27
Reading time: 4 min
Tags: LLM, Documents, Architecture
Canonical: https://elsontan.com/blog/confidence-scores-and-human-review/

> Ask the model for a confidence score per extracted field, then route anything below threshold to a human queue instead of returning it silently. The threshold is calibrated against labelled documents, not guessed, and the review interface should reduce a decision to confirm or correct in seconds.

---
A client called about an invoice we had processed for them. The total was wrong by a factor of ten.
A smudged decimal point, and the model had read 4,850.00 as 48,500.00.

Here is the part that stung. Their accounting system had already paid it.

My platform takes a PDF and a schema and returns structured data. Invoices, delivery orders,
statements. When it crashes, someone retries it and nothing bad happens. When it quietly returns
the wrong number, money moves.

I could not make the model stop misreading smudged decimals. What I could do was make it tell me
when it was unsure, and put a person in front of those cases before anything downstream saw them.

## Ask for confidence in the schema

If you request structured output against a JSON Schema, the confidence score belongs in that
schema alongside the value. Both OpenAI's
[structured outputs](https://platform.openai.com/docs/guides/structured-outputs) and the equivalent
constrained-decoding features elsewhere will hold the model to the shape.

```json
{
  "type": "object",
  "properties": {
    "invoice_total": {
      "type": "object",
      "properties": {
        "value":      { "type": "number" },
        "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
        "source_page":{ "type": "integer" }
      },
      "required": ["value", "confidence", "source_page"]
    }
  }
}
```

Do not read too much into the number. When a model says `0.9` it does not mean it will be right
nine times out of ten. It is not a probability, it is a hunch.

But it is a useful hunch. The fields it marks `0.4` really are wrong far more often than the ones
it marks `0.95`. That ranking is all a threshold needs.

The `source_page` field pulls its weight too. Without it a reviewer opens a ten-page PDF and starts
hunting. With it they land on the page and look at one spot.

## Four outcomes, not two

The mistake I made first was treating this as pass or fail. Extraction has more states than that,
and collapsing them sends the wrong work to the wrong place.

| Outcome | Condition | Route to |
| --- | --- | --- |
| Accepted | All required fields at or above threshold | Downstream automatically |
| Needs review | Any required field below threshold | Human queue |
| Missing data | Field absent from the document entirely | Missing-data branch, not review |
| Failed | Extraction or mapping threw | Retry, then engineering |

Splitting missing from uncertain matters more than it looks. "The model is unsure what the total
is" and "this document has no total" need different human actions, and mixing them trains reviewers
to skim.

Failed is separate again because a reviewer cannot fix a `500` from the OCR service.

## Where do you set the threshold?

Against labelled documents, and not before you have some.

Take a sample of documents representative of what you actually receive, extract them, and record
the model's confidence next to whether the value was correct. Then read off the curve. Raising the
threshold catches more errors and sends more work to humans; lowering it does the reverse. Where
you land depends entirely on the cost of a wrong value getting through.

Two things are worth knowing before you tune. The threshold is per document type, because a clean
digital PDF and a photographed receipt do not share an error profile. And it drifts, because
document mixes change and models get swapped. Mine is configuration, not a constant, and it gets
revisited.

Google's [human-in-the-loop documentation for Document AI](https://cloud.google.com/document-ai/docs/hitl)
describes the same routing shape, which is some comfort that this is the conventional design and
not a local invention.

## The queue is a product, not a table

The goal of a review queue is to make review fast, not to eliminate it. That distinction changes
what you build.

A reviewer should see the extracted value, the confidence, and the region of the source page it
came from, side by side, with the field focused and the cursor in it. Confirm is one key. Correct
is type-and-enter. If a reviewer has to open the original PDF in another tab and search for the
number, you have built a table with extra steps.

Two rules that came out of watching people use mine. Sort by confidence ascending, so the most
likely errors are cleared first and a half-finished queue still has value. And keep corrections,
because the difference between what the model said and what the human typed is the highest-quality
evaluation data you will ever get for free.

## The part that has no technical fix

If almost everything passes the threshold, reviewers stop reading and start clicking confirm. The
queue becomes theatre and the errors flow through with human approval attached, which is worse than
having no queue because now there is a name on the mistake.

I do not have a clean answer to this. What helps: keep the queue small enough to stay meaningful,
sample the auto-accepted stream periodically to check the threshold is still honest, and watch
review time per item. When it drops below a couple of seconds, people have stopped looking.

## Sources

- [Human-in-the-loop for Document AI](https://cloud.google.com/document-ai/docs/hitl), Google Cloud documentation
- [Structured outputs](https://platform.openai.com/docs/guides/structured-outputs), OpenAI platform documentation
- [Use Amazon Augmented AI for human review](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-use-augmented-ai-a2i-human-review-loops.html), AWS documentation