Unable to scrape dynamically loaded website

Hi all, so I am trying to do some web scraping on a store’s product listing but the issue I am running into is that UiPath web scraping is not picking up all the listings. This is likely due to the fact that the website does not load all the products in one go and only shows a partial listing until you scroll down to show you the rest. How would I solve this issue? There are also additional pages of products I wish to scrape from.

This is the product page in question:
https://www.sephora. com/shop/perfume

1 Like

Hi @p3tr1fy

Use Data Scrapping and click what data you want and click yes multiple pages alert before click yes scroll down and check if any next and > this symbol click any one it will scrape multiple pages data.

Regards,
Kommi Jeevan.

Hi @kommijeevan,

That is the current method I am using. I use the data scrapping tool to select the first product on the page and then the last product on the page to create the pattern. I also extract the other correlated data I require in the same manner and then the multiple pages data option. There are a total of 712 products and I am only getting around 130 entries.

Currently the only method I can think of is that I send a hotkey “end” to load the rest of the webpage and do data scraping for that specific page. Next I would go to the next page and repeat these steps which I find very brute forceish :sweat_smile:

please send screen shot of webpage pagenation.

pagenation?
If you are referring to the product webpage I have included link in the question.

Link not working

now it is working here you want 712 product names.

Correct, but I am only able to scrape about 130 across all the pages.

here my internet little bit slow I will send you the sceenshot of workflow.

Here I got total 145 records.

Test.xlsx (8.7 KB)

yeah, that’s the issue I’ve been having with scraping from this webpage. Since the products are only loaded when you scroll down the page, I can’t use the data scraping tool unless I scroll to the end of each product page. Which means multiple web scraping activities

Hi
Welcome to uipath community
While data scrapping with Extract structures datatable activity we got a property named DelayBetweenPagesMS where set the time as 15000 milliseconds
So that before each page is scrapped with data a delay is set to load page which will ensure that all data is obtained
And

Also mention maxnumberofrecords property as 0 so that all data is scrapped

Kindly try this and let know for any queries or clarification
Cheers @kommijeevan

Hi Palaniyappan

I want to get store address for each state of USA from this website https://www.walgreens.com/storelistings/storesbystate.jsp?requestType=locator, to an excel sheet. I am able to get data of one city but the loop is not working for each state and city.

After clicking the state a new page opens up of city and after clicking city a new page opens up and address are mentioned in structured form.

I have just started using Uipath for last couple of days and after much trail and error I am not getting desired results.

I shall be grateful for the help,

Thanks