IXP extraction inconsistent

Help Document Understanding
1

Hi

I’m using IXP gen ai for data extraction.I use gpt model with model settings Temp 0 and rest default settings..i see at times for the same file the extraction outcome differs.Also same pattern file sometimes it extracts all some time it misses data..how can we make it consistent

2

Hi @phoenix2222,

This is expected with LLM-based extraction and cannot be made fully deterministic. To improve consistency, make the prompt very explicit with clear field definitions and examples, lock the same model version, and add validation checks. If fields are missing, re-prompt only for those fields instead of re-running the full extraction.

3

@phoenix2222

Could you elaborate about the inconsistency you have observed like the data is available in file but the model failed to extract/identify or grammatically or values incorrect. Depending on the issue we can fix it by refining the prompts giving specific instructions etc. How many documents have you annotated and what is the model score for your project.

Please not that using LLM we can expect some discrepancies in the results, few we can handle at instructions/prompt level and few at post processing.

4

This is expected behavior from Generative AI. Can we get the same answer for the same question you ask to ChatGPT? no, we can’t because it generate/predict the answer every time and it will vary.

To get the consistent answer or extraction, make sure your prompt is concise, accurate and detailed at the same time so the LLM/transformer will extract the correct answer.

5

@phoenix2222

Few differences would be there due to hallucination

Change the prompting style use few shot or example based prompting to make it little more deterministic. Include group level prompts and file level prompts as well along with field level

Include any identifiers like length or bound values if they are know.

These can improve it . Also its not always advised to use Gen AI for every type of doc..if it is more structured then go with traditional structured document understanding. If the file is highly random and the layouts are highly random then Gen AI makes more sense

@Maheep_Tiwari

May I know how do you re-prompt only for fields where validations fail? also how do you validate if not manually if the extraction is correct?

Answering based on GPT or any other Gen AI tool is fine, but please validate it before posting it here so that it does not confuse the users.

cheers

1 Like
6

Hi @Anil_G ,
You’re right, and I need to correct my earlier answer.

IXP GenAI does not support field-level re-prompting natively. You cannot selectively re-extract failed fields within IXP itself. Validation must be done using rule-based checks in the workflow. If validation fails, the only optioare to rerun the full IXP extraction or call a separate GenAI activity outside IXP for missing fields and then merge the results.

1 Like
7

@Maheep_Tiwari

glad youa re correcting

but again I see Gen AI based answer here which is not verified

for missing you cannot call a separate one or a fall back logic directly as you dont have a dynamic way

We appreciate anyone using Gen AI to the core but bear the limitations of it in mind to help yourself and others as well

Cheers

8

Got it will check and try this

1 Like
9

Change your model setting

  1. Attribution - Keep it Rule-based
  2. Define proper prompt
  3. If you score is coming very low then change your prompt and try it again.
  4. If still not working then change GPT4o to Gemini model and Attribution “Rule-bases”
  5. Define proper taxonomy in IXP.

image
image1153×563 43.9 KB

It will work, thanks and enjoy automation.

10

Hi Anil.

I’m using few short prompt very discriptive one .I m also using field level,group level and overall prompt.The extraction rows values are good ..I just could see missing extraction of rows for few files..when I rerun on the same file I see difference in outcome..i was just wondering if there is a way to control this..

11

Hi…May I know the reason behind ur suggestion for rule based over model based?

12

@phoenix2222

then you are left with playing with the advanced options

Specifically with Seed which works on repeated values

cheers

13
  • Attribution - The method used for attributing predictions to the relevant part or text in the document.
    • Rules-based - Uses an extensive set of rules and heuristics to match the correct spans on a page to the predicted values from the model. This is a low-latency option, but it sacrifices performance in terms of successful attributions compared to the model-based option.
    • Model-based - Uses an additional LLM call to successfully match the predicted values to the correct spans on the page, as these values can often be repeated in different parts of the page. This is the most performant option in terms of successful attributions, but it does add some latency to predictions. This option relies on using Gemini models.