Data Scraping doesn't work due to new results loading when scrolling to bottom of page

Help
,
1

Hi,

so here is the issue:

I am trying to extract all links related to a certain search. The problem is that only a couple of results are displayed when the results list is loaded, and that additional search results are only loaded when you scroll down to the bottom of the page. So I do not have the option to indicate a next button or something of the sort to tell UIPath where it can find the next batch of links to look for and scrape.

Do you have any idea how I could solve this so that UiPath will scrape all the links the search returned?

Unfortunately I cannot provide a link due to it being an internal website.

Thank you in advance for any help!

2

How about give some delay after you scroll down and scrape.

3

Hi,

I solved it by sending the hotkey end and looping it until the process does not find the image ā€œLoading new resultsā€¦ā€ any more, which is displayed at the bottom of the page if there are more results to be loaded.

Maybe this helps someone who is encountering the same issue.

2 Likes
4

Hi PhilippN,
Thanks for your answer, actually i am also facing the same issue. Can you please let me know in detail how did you solve this ?

Thanks,
Ankit

5

Sure. I simply put the Send hotkey and Image exists activities in a Do while activity.

The hotkey I kept sending was ā€œEndā€ to make sure it scrolls to the bottom of the page. For the image exists I made a screenshot of the message ā€œLoading messagesā€¦ā€ (due to the fact that I think every webpage which uses this way to load more results shows some kind of message while loading them, this should be universally applicable). The condition for the Do while was the result of the check made by the Image exists activity, which outputs a boolean variable. So as long as the variable ā€œMoreResultsAbailableā€ (=variable name of your choice) is true, it will keep sending the hotkey.

Once all results are loaded the Image exists wont find the image any more, ā€œMoreResultsAvailableā€ is false and the process will continue with whatever it is supposed to be afterwards.

Hope this made things more clear :slight_smile:

6

Thanks once again. I am actually new to UIPath and hence may ask things in detail.

in my case its not text i.e. loading Messages but a image i.e. https://content.jdmagicbox.com/icontent/preloader_big.gif

I have these steps

  1. Extract Structured Data ā€˜ULā€™
  2. Send hotkey : end
  3. ?? (should we put do while loop here if yes what will be condition as per my scenario ?)
  4. Write to CSV .

Thanks a lot for your help.

7

Also will be great if you can give some screenshot.

8

Unfortunately I cannot provide any screenshots because it is an internal website.

The steps should be in this order:

  1. Do while
    inside it should be the send hotkey END activity and then an image or element exists activity to check for the loading results message. The result of the exists activity is a boolean variable (name it e.g. MoreResultsAvailable), which is the condition for your Do While (=> while MoreResultsAvailable = True, keep sending hotkey).
  2. Extract data
  3. Write to CSV
9

My criteria is i need to get all the Names in the screenshoot and only scroll bar button is used to display the next set of data.Attached is the screenshoot with the scroll but i am not able to do data scrapping for all the 43 Names.When i am doing Datascrapping i am not able to get total 43 names but i am getting 24 Names. If you have any Solution Please help me out.

image
image1575Ɨ712 75.4 KB

Thanks in Advance.

10

Hi All,

In my case when i use Data Scrapping and write the extracted data into excel using Write Range activity under Excel Application Scope. Sometimes the whole process runs perfectly fine, but most of the times, while Data Scrapping the pages wont traverse or i get Null exception in Write Range.
Can anyone help me on this.
Thanks Inadvance
Shrunga

11

HOw you add the condition? You have an image? Please If you can send me the code to my email will be amazing!!! jrpr_123@hotmail.com

12

Hey @rajithanallagatla I am also facing same issue. Did you got solution for this?