Data Scraping (Web) Spanning pages limit

Hi All,

This might have a simple solution but I cannot find anything about this anywhere, I am trying to web/data scrape within a social media site and the data table is generated my question is when spanning multiple pages is there a way to limit the amount of pages it goes through? I have used data scrape and when asked about spanning pages I usually click yes and in order for it to work I need to click on “Next” but I dont want it to go through all pages as there might be 30+ is there a way to limit this?

I have also tried to use the limit records but it still goes through all the pages anyways…This might take a lot of time as well.

Any help on this would be great.

1 Like

Did you try using this technique…

Capture

thanks…

Hi yes I did try this but it still goes through “X” number of pages. The results granted do save 100 records (tbh i have not tried to check if there are actually 100 results at the end but I am sure it will work) but I’d like it not to go through so many pages. If at all possible?

Thanks

hi @xkarrox, maybe if you just want to take data from 2 or 3 pages, you could make a different extracData for each one of the pages… If they are too much and you cannot specify the exact elements that you want, I would recommend you to extract the whole data into a DataTable and then filter it or clean the fields that you dont need. It should be fast, talking about processing time.

you can do how many pages you want by using dynamic selector for navigating to next page…

you can try this method:

  1. use UIexplorer to identity the selector for page 2.

Capture

  1. then edit the selector search for property name aaname=‘Pagenumber(1,2,3,…etc’) will be present, in this case aaname should be ‘2’ as i am clicking on page number 2.

  2. once you update the selector like this

aaname= ‘“+ Page_Number+”’

  1. use element exists to check next page is present

  2. you loop through it and once your page reaches limit, it will come out ( Use if condition)

thanks
Yogananda

2 Likes

hi @Pablo_Sanchez I would like to use the extractData for each page but I have a situation with that as well. See here UL Tag Data Scraping for Social Site - #7 by whyyouandi

I am not sure what you mean by extract the whole data…is this via the screen scraper by any chance?

Thanks

I mean if you want to extract the data of 20 pages, it gonna be hard making this in different DataScraping, but if you just have to take the data of few pages, like 3 or 4, make 3 or 4 different DataScraping, one for each page, and automate by yourself the click between the pages.

1 Like

@Pablo_Sanchez thanks but this as said before will not work for me as I highlighted with the problem previously:

If I manage to solve that then your solution would be feasible, I think…

@phyogananda thanks for this, I would like to try this but I am actually quite new to UiPath would you be able to share an example? Or try and scrape the data via your method on this page: Sign Up | LinkedIn

Which is what I am trying to do but it spans so many pages, etc. Any help would be really appreciated.

Thanks

Main.xaml (11.5 KB)

1)i have designed to navigate to number of pages based on your input, it works for me.

2)data scrapping i have put just some dummy, you can change it you own, if you scrap one page and put it inside the do- while it will be enough.

  1. only prob is it will start from 2nd page, i.e, page will go to second page and starts data scrapping
    for this you need to have 1st page data scraped in begining ( before the do-while starts) of the do-while loop.

thanks

1 Like

Will try it out today but in all honesty I think I managed to fix some bits last night…was up till 2am trying to figure it out but think i got quite a bit done. I will nonetheless try this method of yours out and let you know the outcome.

Thanks so much again, much appreciated