Data scrapping On multiple pages

Data scrapping issue when on multiple pages

@vijay_allena

Welcome to our UiPath community.

May I know what error are you getting here ? Can you please tell more details about the issue for better understanding.

I have a website ,where it consists of 15 pages i need to extract 15 pages data ,it consists only right arrow as next button,when it reaches 15th page,the right arrow gets disabled where the bot needs to stop extracting data and go for the next workflow.But the bot keeps on clicking the same right arrow,even if it is disabled.How to solve this issue.

Try to finetune the selector of the next button. If possible share a screenshot of the selector of the next button

HI Vijay,

Find the selector & look in to the attributes, try to get the position number.

You can able to achieve it

Are You talking about the selector in the data scraping method ? Here is the screenshot of selector

Hi @vijay_allena

Could you explain your query little bit. Any error occur

Can you share the InputExtractmetadata Screenshot

Regards
Gokul

HI Balaji,
Here is the screenshot of the extract meta data ,The issue with the scrapping is ,the bot keeps on clicking the next button(Right arrow) even after it is disabled. I am unable to stop the process after that.kindly please help in this regard

Hi @vijay_allena

Can try to scrap the data again. If possible share the webpage link. I will check in my side

Regards
Gokul

Thank you for asking to share the web link,But i cannot share that ,its a business process.I am trying to finetune the selector but its not recognising the selector even after that.

Hi @vijay_allena

If possible share the screenshot of the scrapping value.

Regards
Gokul

Hi @vijay_allena

You should give → Is Data Spanning Mullite page → Yes

Have look on the document for more info

https://www.analyticsvidhya.com/blog/2021/02/web-scraping-using-rpa-tool-uipath/

Regards
Gokul

1 Like

Actually we don’t need to use any explicit activities to click on that next button
Data scrapping wizard itself provides an option to do for multiple pages by pressing a next button or any arrow button that leads to next page

So when that option to click for multiple pages click yes and have a view on the preview whether you are getting all the records if not change the maxnumberofrecords property as 0 to get all values

Or

If you are not able to select that next button with data scrapping option then use a element exists activity before clicking on the next button

Like use a element exists, get the Boolean output and use a if condition like this bool_exists = True

If true it goes to then block where you use the click activity to click that next button

Cheers @vijay_allena

Yeah,i have given Yes on that,the extraction is working good but its not stopping after extraction of the last page,it keeps on clicking the disabled next button.Thats what the issue us

Hi Vijay,
Try to extract the num of pages (consider variable total_page) and create a current_page variable 0 . use the same in while loop. then you can data scrape the table for 1st page. once it is done click next button also increment the current page value. bot will eventually come out of loop after it reaches the total page value.
Note: you cannot click on next button while data scraping.

you can refer the logic below,

while current_page<total_page:
data scrape the table in the current page.
current_page=current_page+1

usually data scraping will append the data in the same table when you are doing this way.

Thanks!

Hi @vijay_allena

While Data Scrapping do you all the page as an output ?

Try with Element exist activity or Image Exist Activity

Regards
Gokul

Hi@Gokul,
Can u describe briefly about this

Then try with this approach

Or if you would like scrap for each page
Here you go with an example on how to do it

Cheers @vijay_allena

Thanks @Palaniyappan ,
Will try as you said,Thanks.

1 Like

HI @vijay_allena

You can Use Try/Catch Activity

In Try Block → Insert an Data Scraping activity

Need to Remove the Continue on error option in the Extract Structured Data ‘DIV’

Here is sample XAMl file

DaraExtraction.xaml (9.8 KB)

Have try like this method, Hope it will work

Regards
Gokul

Regards
Gokul