Trigger NextLinkSelector multiple times?


#1

Hi,

I have access to a SAP BusinessObjects Analysis webpage. I want to use data scraping in order to get the data out as it contains over 300k cells of data and using the export to excel facility only allows a maximum of 100k cells to be exported at one time.

The problem is that the way the page is setup, it only loads 300-400 results at a time and you need to PgDn or scroll 2-3 times in order to load the next lot of results. Due to this when I use the NextLinkSelector to scroll down, it does so once (not triggering any extra data to be loaded) then stops after the timeout period expires.

Is there any way to trigger the NextLinkSelector multiple times? Or have it in a loop where the data is not duplicated?


#2
  • If you expect to have around 300k items you can use PgDn (100-200 times) before extracting the data.

  • If you don’t know how many data you have, create a loop for PgDn and stop the loop when you find an element from footer section (you have to identify this element) or the next button is not displayed (if the button disappear when there are no more results to display). After loop is closed extract the data.


#3

Hi Andrei,

Unfortunately the way the site works, only 200-400 results are loaded at any one time. Scrolling to the bottom doesn’t keep the results from the start.