Next button page not always working - Scraping

TransactionalProcess.zip (12.6 KB)
Book1.xlsx (27.2 KB)

Hi guys
I am scraping a webpage. The problem comes when i have to click to the next page button. For an unknown reason, the next page button works until 15 clicks, sometimes 20, sometimes 4… there is not a rule apparently. UiPath does not find anymore the next button and goes in timeout, at least this is my understanding.
maxNumberOfLimit is properly set.

I enclosed the project and an example of Excel generated with the extracted data. The IE browser is automatically opened when you run the application…

Can you help me in debugging why is stopping every time in a random page?

Regards

Hi @Stefano_Skjupyter

Do you want to scrape the whole website for the specifications that you gave?

Hi
well i will not take all data in the page, only 5 field which are repeated 30 times per page, in fact the 30 result per page.
Do you think the problem may be the advertisement is loaded everytime different in a new page?

Regards

Hi @Stefano_Skjupyter

OK so what’s happening is that every time the process stops at a random page, and the most important thing is that it doesn’t give an error it seems like it has finished the scraping gracefully and then load the data to excel like the whole scrapping was done.

What’s really happening is that the scrapping activity waits 300 ms to click on the next button and at some point the page will take longer time to load and the scrape won’t find the next button so it thinks that there is no more data to scrape and it stops.

So the solution is to increase the DelayBetweenPagesMS of the extract Structured Data activty to say 800ms for example.

it will take longer but it will work.

Regards,
Reda

2 Likes

Thanks @reda that solved the problem ideed!

Thanks again
Stefano

Glad it worked @Stefano_Skjupyter

[please close thread by marking the right answer as solution]

Regards,
Reda

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.