How are confidence scores calculated in DocPath LLM?
Issue Description:
How are confidence scores calculated in DocPath LLM?
Resolution:
Understanding Confidence Scores in DocPath LLM
DocPath a fine-tuned large language model (LLM) for information extraction from documents
This article explains how confidence scores are calculated during data extraction using the DocPath LLM in Document Understanding. These scores help determine the reliability of extracted values from documents processed by the model.
The confidence score for each extracted field is derived from the probability distribution generated by the model. Unlike some generative models that may not natively produce such scores, DocPath’s architecture supports their generation, offering a probabilistic estimate of how certain the model is about its predictions. These scores are not absolute indicators of correctness but serve as a useful guide to assess the trustworthiness of extracted data.
For each field, the model evaluates multiple candidates values and assigns a score based on how confidently it predicts the correct one. While LLMs don’t output a single confidence score for an entire field, they do generate token-level probabilities. These are derived from raw, unnormalized scores (logits) for every possible token in the vocabulary, which are then normalized into probabilities.
To compute a field-level confidence score, DocPath aggregates the probabilities of all tokens that make up the extracted value (e.g., “INV”, “-”, and “123”). This aggregation is designed to be sensitive to uncertainty—even a single low-confidence token will reduce the overall score. Additionally, the calculation adjusts for the length of the field to ensure longer text isn’t unfairly penalized.
The result is a holistic confidence score that reflects the model’s certainty in the extracted value, helping users make informed decisions about downstream processing or validation.
If confidence scores are unexpectedly low:
- Ensure the document layout matches the training data.
- Check for OCR quality issues (e.g., blurry scans, poor contrast).
- Validate that the field label and context are consistent across samples.
- Retrain the model with more representative data if needed.