Scroll till the end of a dynamic page

Help Studio
, ,
1

Hi, I want to scroll till the end of a dynamic page, but still dont know how to do it cause I have to scrape all the data in the page, but just when I scroll the page, the new data appears so I have to do this to collect the full data.
My website does not include any identifiable element at the end of the page so i cannot use Element Exists.
I have tried Mouse Scroll but dont know how to identify how many scrolls.
I have read some similar topics in this forum but it does no help to me.
The website is: One Way: DAD → TYOA, 13 Sep 2023 (traveloka.com)
Thank you in advance

3

@QUY_T_TAM_8.0_IELTS

Hi Good to See you back

Can you try with Table Extraction ,I have tried and got the rows of the page as 264

I have shared it in the below screenshot

image
image1039Ă—784 55.4 KB

4

But only when you manually scroll the page till end, you are able to scrape the whole data.
If you dont, you can just extract a small number of data.
So i want to automatically scroll by bot after open the browser

5

Is there something at the end of the page that isn’t visible until you scroll that far?

  • Do While NOT endElementExists
    ** Keyboard Shortcut PgDwn
    ** Element Exists (classic) with selector for end page element, output to endElementExists boolean variable
6

Nothing, just a blank :frowning:

7

@QUY_T_TAM_8.0_IELTS - You can try a loop like this:

  1. Capture how many times the word “Choose” appears in the site. Screen Scraping—> Full Text on full web page—> check on Ignore Hidden, and use Regex:
dblCountBeforeScroll = System.Text.RegularExpressions.Regex.Matches(strFullTextWebsite, "Choose").Count

image

  1. Issue the Mouse Scroll command and capture the count of the “Choose” buttons again (dblCountAfterScroll). Compare this with the initial count dblCountBeforeScroll in Step 1.

  2. If more than the initial count (dblCountAfterScroll > dblCountBeforeScroll), repeat Step 1.

  3. Else, exit the loop.

Hope this helps!

8

image
image1920Ă—1080 221 KB

Do you mention this

9

@QUY_T_TAM_8.0_IELTS - Yeah, man. I attached a workflow you can try.

ScrollUntilEnd.zip (4,2 KB)

Let me know if it works for you! :slight_smile:

10

Im so thankful for your help

1 Like
11

Are you sure? Scroll to the bottom, open UI Explorer, indicate and move your cursor around the bottom of the screen. You could also click the last row and then look at the UI model in UI Explorer to see what’s at the end of the page. There have to be tags there, they just might not be visible elements.

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.