How to deal with elements on the webpage that are not loaded yet?

I have 160 items on webpage that I need to scrape. When I open that page, there is only 4 items visible, and in order to load all 160 items, I need to scroll down all the way to the bottom of the webpage. Because of this, I am unable to use Find Children method(because in order to use Find Children, elements must be visible on the page.)
Also, I am not able to use For Each activity nor Data Scraping

What could come handy for my case?
Thanks in advance. :pray:

Well, i guess you answered you own question there, if you need to scroll down, scroll down :slight_smile: using send hotkey if possible or using the browser scroll bar…

1 Like

I’ve tried that also and that could be possible solution, but the problem is that there are not always 160 items, sometimes it’s 10, sometimes 270… :confused:

i mean keep scrolling until you finish all…

more details on the page or best the url would be helpfully to adress your scenario more individually.

in general there are two things to analyse:

  • on which info the fully scrolled down page is to bind (eg. 170 of 170 items status bar info)
  • what is the best strategy to scroll down (pg-down, down, click an element, hover an element…)

for bringing some ideas to you have a look here TUI Scrape data until no more search results.

1 Like