Generative classifier

subject–
Incorrect Confidence Score (1.0) in Generative Classifier for Invalid Prompts

description:-
When using the Generative Classifier activity in UiPath’s Document Understanding framework, the confidence score is returned as 1.0, even when the prompt provided is incorrect or irrelevant to the document. This behavior misrepresents the actual confidence level of the classifier, leading to unreliable results and potentially misclassified documents.

Steps to Reproduce:

  1. Set up a Document Understanding workflow using the Generative Classifier activity.
  2. Provide an incorrect or irrelevant prompt for a document classification.
  3. Observe the confidence score in the output.

Actual Result:
The classifier returns a confidence score of 1.0, regardless of the accuracy or relevance of the prompt.

Expected Result:
The classifier should return a confidence score reflective of the prompt’s accuracy and relevance, with lower confidence for incorrect or irrelevant prompts.

Impact:
This issue undermines the reliability of the Generative Classifier, potentially leading to misclassification and errors in downstream processes relying on accurate document classification.

Environment Details:

This is not a bug, generative classifiers and extractors by their very nature cannot have a confidence score, so it always defaults to 100%.

There is an argument to say remove the confidence score, but that would mean it would require a different data type and wouldnt fit in with the other DU activities neatly as it does.

When using the generative activities you need to use a different method to decide if the result is reliable or not.

1 Like