Data scraping only works for page one

Help
1

Hi there,

For my data scraping, why is it that only the first page’s data can be exported, but not for the subsequent pages? I did click the next page button during my data scraping recording process and when I play the script, UiPath could only direct me to page 2., and the data is also not exported.

1 Like
2

Hi,
could you please increase maxNumberOfLimit property from 100 to more .
else please provide URL u want to scrape or could you please attach workflow.

3

hi @ddpadil , does the maxNumberOfLimit affect when there is only 10 results showing per page? this means that, so far, I can only extract 10 results out from the data scraping.

I am also sorry to say that it might be a bit hard for me to attach the workflow due to some privacy issues.

4

Nope. I assumed your first page result might contain data more than 100.
While doing data scraping hope your clicking on the yes on the popup and clicking on the button next page of paging.

5

oooh. okay, so I have tried to increase the limit number alr. I also did click on yes to indicate. However, the above problem still persists. any other solutions to recommend?

6

Hi,
I encountered the same issue and my guess is that the UiPath only can read data which is loaded on the screen. For example, some websites load all the query results on a screen at once (but not shows it on the page and some will be shown on different pages). The others only load data which can be shown on a page and then they send query when “next page” is clicked on. At least in my case, the next page selector worked for the former case and not for the latter.
(Again, it’s just my guess. I might need to set up differently for the latter case)

I gave up to use next page selector and used “find element” to find “next page”, then I used “click” if there’s the element and again use data scraping.
Note that datatable doesn’t clear automatically, so the data on the next page will be simply added to the datatable which has the data on the first page.

Hope there’s an actual solution…

7

One last try how about increase your "DelayBetweenPagesMs(ex3000) property of data scraper.
Due to delay in loading next page, UiPath unable to scrape the data from next page.
@a-taniya nice info :slight_smile:

3 Likes
8

@ddpadil it is not working either :frowning: . Think i shall be trying @a-taniya 's method instead. thank you both!

9

can i have url if not internal.

10

sorry! it is an internal website

11

:frowning:

12

hi @a-taniya, may i know what do you put in the foreach blanks?

13

I think it should be ExtractDataTable, because ExtractData activity automatically create that variable and put data into it. Of course you can change datatable name if you want.
Please see property of ExtractData activity.

image.png1012×579 38 KB

14

So do I use for each “item” or is there any other term that I should be using instead?

15

I think you can use anything you want, I prefer “row” but of course “item” is fine.

In my understanding, “X” in “for each X” is just an identifier of rows of the datatable.
You can name it as you like, then you can use it in “Get row item” activity or somewhere.

image.png1000×357 18.7 KB

16

Hi @a-taniya,

I followed your suggestion and the robot check if the next page exists (and open it), but how can I add the extracted data from the 2nd page to datatable generated in the 1st page?

I write the datatable in Excel but only the data from the 1st page appear.

Any suggestion?

best regards,
Francisco

17

Hi, would like to know if you have found any solution for this issue?

1 Like