How to Scrap multiple pages data From a Site


#1

How can we Scroll in a Single page and Scrap Web content from a Single page.
Example Linkedin Connection/Names


#2

Hi,
Here a way to do it.
1.Create a counter variable scroll(int)
2.while loop (execute body until it reaches end of page (scroll))
3. inside body of while use “data extract” and page down and write range.


#3

Hi @ddpadil how to find that scroll bar have reached end of the page,since pgdn is Send Hot key activity inside a loop it will be executed infinite times and data extract element scrapes data infinite times over last linkedIn connection/Name page.

And do we need to scroll pages to extract structuted data from single web page?.

Thanks in advance.
Ranjith.


#4

Your putting inside while loop and it will execute certain number (count)of time.
no need use scroll. page down works fine.
you just need to get the count in the variable that how many times u need to hit pagedown
Try to find something unique when it reaches end of page maybe "“Image Exists” or
this might help


#5

I’m getting this error -“No robot available for user domain\user”.
How can i resolve this issue?. Could you help me…
Thanks in advance…


#6

Did you check this.


#7

how to scrape data from multiple pages from internal website where data scraping and web recording cannot be done.
Is there any solution?
Thanks in Advance


#9

@rajithanallagatla
You have to provide next page link(Selector). Please refer this document - https://www.uipath.com/tutorials/web-data-extraction-automation


#10

@skkumar2020 Thank you for the reply

But my criteria is i need to get all the Names in the screenshoot and you can a scroll bar button also in the attached screenshoot.Totally 43 Names are there i want all the names.When i am doing Datascrapping i am not able to get total names only 24 are coming.If you have any Solution Please help me out.
Thanks in Advance.


#11

Can you check the pagination details exists for the above result / last result row in the page


#12

Hi Yes,
Till 21st row i am getting data.Can i know how to get the rest data
Thanks in advance


#13

@skkumar2020
When we are using data scapping it can scrap the first page which contains of 21 details .But to load the second page it takes fraction of seconds to get the data on the page or else it wont scrap .This is what i felt.
Can you please rectify me if me wrong and help me with a solution.
Thanks in Advance


#14

You can use multiple page down using hotkeys/type into and place delays between them, place the extract structured data at the end i.e. when whole page is loaded.

Page

Try the combinations of pgdn that works for you :slight_smile:

Try this if nothing works for you. I’m sure this will work.


#15

Thanks for the reply but pgdn activity is not working:(


#16

In this senario i will get only the last page details whereas i need to get all the page details.But here page numbers are not there so data scrapping is not helping me.Whereas its like scroll button to get the next page details i have tried using scroll button and pagedown activities but those activities are not helping.Can someone help me?
Thanks in Advance.


#17

I had a similar issue when I was working with Data Scraping in Snapdeal.com, and i was able to achieve it by using Delay+Pagedowns+Data Scraping! The WebPage looks absolutely okay and i think data scraping should work. Make sure the selector of attach browser is correct. And you need to perform data scraping at The end i.e. when you have loaded the whole page by using page down either by type into or by send hotkey…
When the whole page gets loaded it will automatically take all the data from the previously loaded page…
Also see if maximum results field is fine!

I hope this might help you!


#18

Hello… i doing how to said but i still can’t stract all data… can you help me ?print%202


#19

I am trying to extract from this page https://www.quintoandar.com.br/alugar/imovel/sao-paulo-sp-brasil/


#20

Try increasing the delay time. So that the webpage is loaded completely…
And place pg downs!
image

Just like this :slight_smile:


#21

I have tried but its not workiing…can you look my robot?