Data Scrapping - How to scroll down on each page

Hi, I’m scraping data from Linked-In. The data I’m scrapping is made up of 100 pages. Each page contains around 25 records. When I scrape UI path only picks the visible records (3 or 4) then turns to the next page. The website does not load all the products in one go and only shows a partial listing until you scroll down to show you the rest.

The solutions I have are to zoom out to around 25% which then shows around 12 of the 25 records per page. I have tried sending a hotkey to scroll to the bottom which only works for the first page. I there a way or I do I get it to scroll on all the pages?

Thank you.

How are you scraping the data?
Do you have example of what you are trying to scrape?


I’m scrapping the Name, company and position.
I’m suing data scrapping.

The process works but only gets the visible records

is it pulling as extracting structured data?

It depends on how webpage generates the DOM if the page is not visable.

It seems strange isn’t just extracting all the data.

Try play around with UiExplorer to understand if the selectors are valid or not if the page isnt scrolled down.