UiPath CV adds an empty column when invoicing in ERP

Help AI Computer Vision
, , ,
1

Hi,

For the past few days, I’ve been struggling with a particular area in the ERP system. Only CV works in this specific section, but even that has proven ineffective. When entering products and invoice costs, I use CV to input data in specific columns and rows. Unfortunately, I noticed that sometimes CV reads as if there are more columns than there actually are, for example, it adds one empty column, which causes the data to be entered into the wrong column. When there is only one line item in the invoice, CV reads everything correctly, but when I add another product and refresh the CV, this non-existent column appears. I have the CV Descriptor set so that the rows are dynamic and the columns are fixed, but it doesn’t help if the columns are sometimes read incorrectly. Any ideas on how to solve this?

2

Hi @pikut,

Can you provide some more details so we can try reproducing the issue? Environment (Windows/Cross-platform project, Studio and UIAutomation package versions, what flavour of CV you’re using (Localserver, on-premise or UiPath Cloud - default one), resolution details), the ERP you’re using in case it’s a publicly available one or some anonymised screenshots.

It looks like an issue with table detection - this one is particularly stronger on UiPath Cloud since Feb '24, when we deployed a Vision Transformer (a much more powerful neural architecture vs. the previous one); this is why I was asking what model you’re using.

cc @Dragos_Bobolea1 @Stefan_Adam

3

Sure, more details:

  • Environment Windows
  • Studio 2024.10.1
  • UIAutomation 23.10.3
  • UiPath Cloud - default
  • UiPathScreenOCR
  • Resolution details - 1440 × 900 (Macbook Air connect to RDP)

Unfortunately, there is no demo version of this application available. The situation is as follows: when I have only one row, CV correctly recognizes the header “Oznaczenie” in column 4. However, if another entry is added, a thin empty line appears between the columns “J.m.” and “Oznaczenie,” causing “Oznaczenie” to be incorrectly positioned in the 5th column.

This image shows a partially filled invoice form with items listed, including quantity, unit price, and total price, in what appears to be an accounting or invoicing software interface. (Captioned by AI)
This image shows a partially filled invoice form with items listed, including quantity, unit price, and total price, in what appears to be an accounting or invoicing software interface. (Captioned by AI)1448×867 120 KB

4

Can you please also attach two clean screenshots of the two states you mentioned? Would help a lot in verifying any potential fix.

5

And now it sees a false column in position 1…

The image shows a user interface screen of a financial or inventory application in Polish, featuring fields for quantity, unit, description, unit price, and total price, along with a control panel for configuration options on the left side. (Captioned by AI)
The image shows a user interface screen of a financial or inventory application in Polish, featuring fields for quantity, unit, description, unit price, and total price, along with a control panel for configuration options on the left side. (Captioned by AI)1438×857 83.9 KB

The image shows a screenshot of an invoice application in Polish, displaying various numeric entries and fields, with a CV Type Into dialog box at the bottom left indicating where data is being inputted. (Captioned by AI)
The image shows a screenshot of an invoice application in Polish, displaying various numeric entries and fields, with a CV Type Into dialog box at the bottom left indicating where data is being inputted. (Captioned by AI)1432×859 78.5 KB

The image shows an invoice entry screen in a software application, detailing items with their quantities, unit prices, and total prices, with additional navigation and input options visible. (Captioned by AI)
The image shows an invoice entry screen in a software application, detailing items with their quantities, unit prices, and total prices, with additional navigation and input options visible. (Captioned by AI)1437×854 60.3 KB

6

Screenshot 2024-10-01 at 15.19.54
Screenshot 2024-10-01 at 15.19.542236×998 227 KB

Hi @pikut, I couldn’t reproduce the issue based on the screenshots you sent; I think it’s mainly because it’s still not a clean screenshot (it still contains your overlays from Studio).

Even so, the cell targeting should still work: if you reference the target cell via its column name (as is the case in my screenshot) vs. column index, ghost columns shouldn’t be an issue. Moreover, you can also specify a row contains (similar to an anchor with a non-table-cell target) to make sure you have the correct row as well.

Also, we’re close to deploying a new AI Model which has been trained a larger tables dataset, so the ghost columns issue might disappear. We could test this if you send a clean screenshot (no overlays).