Web Scrapping - Next page isn't scrapping


#1

Screenshot_1
hi,
i am doing web scrapping, uipath robot is scrapping pages from 1 to 30 but 31 number isn’t scrapping cause there is no next sign, what can i do to solve this problem? i need to scrape the page 31 also.


#2

Hello,

The simplest way to solve this one particular instance of not getting the last page would be to use the scraper, then add a click activity to go to page 31, then scrape again. You’d now have two data tables, but could easily merge them into one and then use that.

However, this will not work if there are not exactly 31 pages. So a better approach would be to do the data scrape, then check how many pages it scraped (either by looking at the current page or by dividing the number of returned results by the number of results per page), and then checking to see if there’s a selector for the next page. So if each page has 100 results, for this you’d do the data scrape, then see there are 3000 (30*100) rows in the datatable and check if there is a 31st page (using Element Exists). If Element Exists returns True, navigate to the 31st page then do a second data scrape and merge the two data tables.

The tricky part is you don’t know what number will be in the selector to get to the last page, so you will need to put a variable for the page number to look for. If you share the website I could help with identifying those selectors.

Ezra


#3

First of all thank you, but i am facing another problem, when i am scrapping one single page, suppose page 1, then i am getting all the links, 10 links per page, but whenever i am doing scrape for all the 30 pages then i am not getting all the links, sometimes i get 123 or 120 links but i should get 300 links in my excel or csv file, what can i do now?


#4

Sorry for the delay in my response. Are you still having this problem? If so, could you share the site you’re trying to scrape and your workflow?