Data scrapping On multiple pages

[vijay_allena]

Data scrapping issue when on multiple pages

[lakshman (Ganta lakshman)]

@vijay_allena

Welcome to our UiPath community.

May I know what error are you getting here ? Can you please tell more details about the issue for better understanding.

[vijay_allena]

I have a website ,where it consists of 15 pages i need to extract 15 pages data ,it consists only right arrow as next button,when it reaches 15th page,the right arrow gets disabled where the bot needs to stop extracting data and go for the next workflow.But the bot keeps on clicking the same right arrow,even if it is disabled.How to solve this issue.

[Thanu_Matheesha1 (Thanu Matheesha)]

Try to finetune the selector of the next button. If possible share a screenshot of the selector of the next button

[varunk (Varun Kumar)]

HI Vijay,

Find the selector & look in to the attributes, try to get the position number.

You can able to achieve it

[vijay_allena]

Are You talking about the selector in the data scraping method ? Here is the screenshot of selector

[Gokul001 (Gokul Balaji)]

Hi @vijay_allena

@vijay_allena

Data scrapping issue when on multiple pages

Could you explain your query little bit. Any error occur

Can you share the Input → Extractmetadata Screenshot

Regards
Gokul

[vijay_allena]

HI Balaji,
Here is the screenshot of the extract meta data ,The issue with the scrapping is ,the bot keeps on clicking the next button(Right arrow) even after it is disabled. I am unable to stop the process after that.kindly please help in this regard

[Gokul001 (Gokul Balaji)]

Hi @vijay_allena

Can try to scrap the data again. If possible share the webpage link. I will check in my side

Regards
Gokul

[vijay_allena]

Thank you for asking to share the web link,But i cannot share that ,its a business process.I am trying to finetune the selector but its not recognising the selector even after that.

[Gokul001 (Gokul Balaji)]

Hi @vijay_allena

If possible share the screenshot of the scrapping value.

Regards
Gokul

[Gokul001 (Gokul Balaji)]

Hi @vijay_allena

You should give → Is Data Spanning Mullite page → Yes

Have look on the document for more info

https://www.analyticsvidhya.com/blog/2021/02/web-scraping-using-rpa-tool-uipath/

About Data Scraping

Regards
Gokul

[Palaniyappan (Palaniyappan P)]

@vijay_allena

The issue with the scrapping is ,the bot keeps on clicking the next button(Right arrow) even after it is disabled. I am unable to stop the process after that.kindly please help in this regard

Actually we don’t need to use any explicit activities to click on that next button
Data scrapping wizard itself provides an option to do for multiple pages by pressing a next button or any arrow button that leads to next page

So when that option to click for multiple pages click yes and have a view on the preview whether you are getting all the records if not change the maxnumberofrecords property as 0 to get all values

Or

If you are not able to select that next button with data scrapping option then use a element exists activity before clicking on the next button

Like use a element exists, get the Boolean output and use a if condition like this bool_exists = True

If true it goes to then block where you use the click activity to click that next button

Cheers @vijay_allena

[vijay_allena]

Yeah,i have given Yes on that,the extraction is working good but its not stopping after extraction of the last page,it keeps on clicking the disabled next button.Thats what the issue us

[vaishnavi_velayutham]

Hi Vijay,
Try to extract the num of pages (consider variable total_page) and create a current_page variable 0 . use the same in while loop. then you can data scrape the table for 1st page. once it is done click next button also increment the current page value. bot will eventually come out of loop after it reaches the total page value.
Note: you cannot click on next button while data scraping.

you can refer the logic below,

while current_page<total_page:
data scrape the table in the current page.
current_page=current_page+1

usually data scraping will append the data in the same table when you are doing this way.

Thanks!

[Gokul001 (Gokul Balaji)]

Hi @vijay_allena

While Data Scrapping do you all the page as an output ?

Try with Element exist activity or Image Exist Activity

Regards
Gokul

[vijay_allena]

Hi@Gokul,
Can u describe briefly about this

[Palaniyappan (Palaniyappan P)]

@vijay_allena

Yes on that,the extraction is working good but its not stopping after extraction of the last page,it keeps on clicking the disabled next button.Thats what the issue us

Then try with this approach

@Palaniyappan

If you are not able to select that next button with data scrapping option then use a element exists activity before clicking on the next button

Like use a element exists, get the Boolean output and use a if condition like this bool_exists = True

If true it goes to then block where you use the click activity to click that next button

Or if you would like scrap for each page
Here you go with an example on how to do it

Data scrapping with Next button if exist

Hi Hope theee steps would help you resolve this —first let’s use a ELEMENT EXISTS activity, click on next button as a selector And get the output of boolean type which denotes whether next button exists or not —now use a IF condition and mention as ElementExists = True If true it will go to THEN block where we can have one EXTRACT STRUCTURES DATATABLE which will use next button If the condition is false it will go to ELSE block where we can have another different EXTRACT STRUCTURED DATATAB…

Cheers @vijay_allena

[vijay_allena]

Thanks @Palaniyappan ,
Will try as you said,Thanks.

[Gokul001 (Gokul Balaji)]

HI @vijay_allena

You can Use Try/Catch Activity

In Try Block → Insert an Data Scraping activity

Need to Remove the Continue on error option in the Extract Structured Data ‘DIV’

Here is sample XAMl file

DaraExtraction.xaml (9.8 KB)

Have try like this method, Hope it will work

Regards
Gokul

Regards
Gokul