ModernUI Table Scrape - Occasionally Fails to Find Rows

Help Activities
, ,
1

We have a bot process that automates a website. Working out of a database queue from another one of our services, this bot types an identifier into a field on the website which performs a search, and then scrapes the resulting table on the page.

Between 3% and 7% of the searches we perform, across hundreds per day, the table scrape activity fails to find any rows even when rows are present when manually checked. This results in the bot reporting that the information was not found and passing it to a human reviewer, who then immediately finds the record in question when searching manually.

While the actual number of false positives varies between 10 and 30 records per day, we are interested in identifying ways to reduce the number of these false positives.

I could have it attempt to re-scrape the table when it finds no rows, but this comes with the downside of slowing down the bot when the record is actually not present, a situation that is not an unexpected business scenario and one we are specifically interested in identifying.

Are there any ideas why this activity would occasionally fail and are there any suggestions on how to make this lookup more resilient without slowing down the bot?

2

we keep in mind, that any additional stabilization do need additional time in terms of processing costs. But we can have a chance to come in a balanced approach and go for it.

the situations of the bot is not mandatory the same, when afterwards it is checked by the human. Maybe an issue screenshot / recording log (ootb ORC) can help.

Maybe a better sync on the loading / existence along with a precheck can help, to start with scapping when the table is ready to get extracted.

Kindly note: we can give only more general hints, as we cannot study all details from remote so far