How to deal with elements on the webpage that are not loaded yet?

Help Studio
, , ,
1

I have 160 items on webpage that I need to scrape. When I open that page, there is only 4 items visible, and in order to load all 160 items, I need to scroll down all the way to the bottom of the webpage. Because of this, I am unable to use Find Children method(because in order to use Find Children, elements must be visible on the page.)
Also, I am not able to use For Each activity nor Data Scraping

What could come handy for my case?
Thanks in advance. :pray:

2

Well, i guess you answered you own question there, if you need to scroll down, scroll down :slight_smile: using send hotkey if possible or using the browser scroll bar…

1 Like
3

I’ve tried that also and that could be possible solution, but the problem is that there are not always 160 items, sometimes it’s 10, sometimes 270… :confused:

4

i mean keep scrolling until you finish all…

5

@bp777
more details on the page or best the url would be helpfully to adress your scenario more individually.

in general there are two things to analyse:

  • on which info the fully scrolled down page is to bind (eg. 170 of 170 items status bar info)
  • what is the best strategy to scroll down (pg-down, down, click an element, hover an element…)

for bringing some ideas to you have a look here TUI Scrape data until no more search results.

1 Like