Scraping data from slow-loading pages

Help
1

I was creating a UiPath demo that went to the Motley Fool website, and then looked up the top 20 stocks listed on Top Stocks to Buy in September 2022 | The Motley Fool.

The problem is that because of all of the tracking pixels and marketing stuff that needs to load, using INTERACTIVE mode takes 5-10 seconds per page. I tried setting it to wait for NONE, but that didn’t scrape anything.

Any ideas?

2

Not a lot that you can do from the robot side. The issue is that the websites you are trying to scrape data from are slow, not that the robot is slow. Only thing you could do is do an ad-blocker such as pi-hole to prevent those extra items from loading at all, or create/use a browser extension that prevents images from loading

3

That setting doesn’t make scraping data fail.
I suggest before scraping the data, look for Elements Exists which defines the data is loaded and then scrape the data. This will guarantee that data is loaded.

Regards,
Karthik Byggari

4

Thanks Karthik. Is there any way to set it to read the page after 1-2 seconds whether it’s loaded or not?

5

For just delay you can use the following activity -

1 Like