IXP model with DU framework | Need help to validate the table field confidence scpre

Help Document Understanding
1

Hi Team,

I have created a project in IXP for an unstructured document. Model training/prompting is completed.

Now I am trying to validate confidence score for each field in the field group (represented as table and columns in Extraction).

I am having difficulties in extracting the confidence score for the fields.

If anyone have done the similar thing, please share your experience or solution.

Thanks.

2

@Rohit_More

I believe this is what you are looking for

cheers

3

Hi @Rohit_More ,
Try these below steps once:

  1. After extraction, inspect the Document Data / Extraction Results output.
  2. Loop through the field group rows and access each field’s Confidence property.
  3. Use Write Line, Log Message, or the debugger to inspect the extracted JSON/object structure and locate the confidence values.
  4. If you’re using a Generative Extractor or Prompt-based model, note that some fields may not expose confidence scores in the same way as traditional ML extractors. In such cases, check the extractor output metadata or API response.
  5. A common approach is to convert the extraction result to JSON and inspect the confidence attributes for each column in the table.

Thanks

4

Hi @Jasmin_Bar, thanks for you response.

Hi @Anil_G, thanks for sharing, I did check the thread. It was not exactly same but it guided me to get the desired data.

Used this expression in for each loop:
io_ExtractionResults.ResultsDocument.Fields.Where(Function(field) field.FieldName.Equals(“Field Group Name”)).Select(Function(actual_field) actual_field.Values.Select(Function(val) val.Components))(0)(0)(1).Values(0).Components

Used below expression to access field name, value and confidence:
currentResultsDataPoint.FieldName+" : “+currentResultsDataPoint.Values(0).Value +”(“+currentResultsDataPoint.Values(0).Confidence.ToString+”)"

thanks.

5

@Rohit_More

Great that helped and good to know you were thinking ahead in right direction

cheers

6

Hi @Rohit_More,

Nice, your logic is already correct, actually I have used similar logic in one of my previous DU projects, so thought to share the simpler logic here.

Basically the confidence for table fields works like this, in simple words:

  • Table itself = a Field
  • Rows of table = Values
  • Columns in each row = Components
  • Confidence of each column value =Values(0).Confidence

So the structure is like - Field → Values (rows) → Components (columns) → Values(0).Confidence

Instead of writing one big line with LINQ and hardcoded index like ‘1’, in my project I used normal nested For Each loop, much easier to read and debug also:

Code:

For Each tableField In io_ExtractionResults.ResultsDocument.Fields.Where(Function(f) f.FieldName = “Field Group Name”)
For Each rowValue In tableField.Values
For Each columnField In rowValue.Components
Log Message: columnField.FieldName & " : " & _
columnField.Values(0).Value & " (" & _
columnField.Values(0).Confidence.ToString & “)”
Next
Next
Next

Just replace “Field Group Name” with your actual table field name and it should work fine.

It is better to use columnField.FieldName instead of hardcoding column index like ‘1’, because if column order changes in model tomorrow, your code won’t break. This is something I faced issue with in my earlier project itself, so changed to FieldName based approach.

Anyway your solution works fine, this is just same thing written in easier way. Hope this helps someone else too who lands on this thread later.

Thanks.

7

Hi, @Rohit_More

For field groups/tables, the confidence score is usually available at the individual field/Value level, not directly from the field group.

I would first check the extraction output and look inside the Values/field objects for the confidence property. For table fields, you may need to iterate through the rows and columns to get the confidence score for each value.

If you’re not seeing it in the output, check the raw extraction result rather than only the flattened output shown in the picker.