Mouse scroll in data scraping

Help Studio
1

My program can only scrap the top 9 of 40 elements in each result page. I suspect that it is likely because the rest of the 31 elements are not loaded until scrolling. I have found this topic on the forum: Scrolling webpage in Data Scraping

The solution is to use the Mouse Scroll, which is a modern activity. While it is possible to find a modern activity, when in the classic mode, one cannot use a modern activity, if the parent activity (e.g., Attach Browser) is a classic activity.

I have also seen the Send Hotkey solution. I tried this by using a loop to send a pgdn key eight times when the page is loaded before scraping. But it does not seem to have any effect.

Can you please help with a workable solution?

2

Hi,

How about using SetFocus activity to some UiElement around the bottom of page before DataScraping?

Regards,

3

Good idea. But it did not work.

4

Hi,

Didn’t SetFocus scroll down? How about Hover activity?

Regards,

5

As another approach InjectJS may work.

Set target selector to HTM tag as the following

<html  title='Robot - Attended Vs Unattended Robots' /> <webctrl tag='HTML' />

Then use the following code

"function(e,v){
    setTimeout(function() {
       window.scrollTo(0,5000);
    },0);
}"

image

Regards,

6

SetFocus did not scroll down. But Hover did scroll down.

Unfortunately, even when it did scroll down, no new data were scrapped.

7

Hi,

if you scroll down manually, does datascraping get all data? If not, it may not be scroll matter.

Regards,

8

I realized that with hover, when it did scroll down, it got the very last of the 40th as well. With this script, it did not seem to add anything beyond that. It got the first nine and the last of the forty.

9

Yes. When I manually scrolled down, I got a few more.

10

So, all these seem to suggest that it requires scrolling down gradually than just jumping to the bottom of the page.

I shall add that the website to be scrapped has left and right part. The left is a map that does not scroll and the right is the result set that can scroll.

When using Hover and SetFocus, I did choose a selector on the bottom of the right. But I don’t know whether the script you wrote can detect the left/right setting.

11

Perhaps, the script can be mortified. Instead of scrolling to 5000 just one time, the script can be called 10 times, the each time the previous ending point (the second parameter in the scrollTo function) becomes the new starting point (the first parameter) and the new ending point increments by a number.

But I don’t know how to write this script.

12

Hi,

Can you try as the following?

image
image1472×802 62.8 KB

"function(e,v){
    setTimeout(function() {
       window.scrollTo(0,v);
    },0);
}"

Please modofy i<10 in While activiy (10 is repeat times) and (i*100).ToString in InputParameter (100 is scroll width)

Sample
Sequence.xaml (7.4 KB)

Regards,

13

Thank you so much for your help.

A quick answer is that it did not work. I put your code at the beginning of processing each result page. I did not see that the result page being scrolled down. I did not see more listings scrapped.

In addition, I have tried a few other things, but none worked, either.

  1. In the loop processing each listing, I added an IF statement. In every 8 listing (1, 9, 17, 25, etc.), it will hover the 8th listing down (9, 17, 25, 33, etc.)
  2. Same as item 1, but replace hover with setfocus.

At this point, it seems nothing works…

14

@SJ_LIN

try to scroll manually, indicate the new data on screen in table extraction and then try to extract and check what is getting extracted

also instead of table extraction you can try to use find children if that could get all the rows and then create the datatable as needed

cheers