How to extract data from website using dynamic data scraping?

[PraveenKumar_S (PraveenKS)]

Hi All

I’m new to UiPath. I want to do data scraping in amazon website. That is to search the items from user defined variable and get the product name and price from the result respected to that search.
I have try in many ways such as,
1. creating variable in the selector using UIExplorer,
2. to create the variable in url and use that url instead of using Title tag in selector .
3. try to edit extracted data result in extract wizard window but cannot succeed.
By using these ways .i can search the results but cannot read the data (i.e. product name and price ) from the website.
Anyone could help me?
It’s very helpful to me.

Attached my so for process and the related file for your reference Amazon.zip (6.8 KB)

[Vrushali_Gave]

@PraveenKumar_S Here is your solution. While scraping of web data does it properly

amazon.xaml (12.3 KB) amazon2.xlsx (7.8 KB)

Regards, Thanks

[Vrushali_Gave]

@PraveenKumar_S You can take help from this video : UiPath - How to Data Scrape from a web page and save to Excel - Full Tutorial - YouTube
It will help you to know about data scrapping.

Regards , Thanks

[PraveenKumar_S (PraveenKS)]

Thank you for your reply @Vrushali_Gave , but I am looking to search the product name that what user giving during run time or before start running after created the total work flow.

[PraveenKumar_S (PraveenKS)]

Thank You for your suggestion @Vrushali_Gave , in this video tutorial he just only get the data from what he used to create workflow for data scraping and there after he did not give any product name by himself and getting the data from the product name given given externally by him.So, I am looking for that kind of search. If you know kindly help me do complete the task.

[Vrushali_Gave]

@PraveenKumar_S yes I got it so for this you need to do like this what I have mentioned in the below screenshot.
The output variable (inputDatavalue) of the input dialog activity we need to put into the website to search the product which is given by the user.
Like this : “https://www.amazon.in/sk=“+inputDatavalue+”&page=7&qid=1594911279&ref=sr_pg_6”

I hope this will be helpful to you for completing your task.

Regards, Thanks

[PraveenKumar_S (PraveenKS)]

I tried this work flow but It get the data from what the product i was used to create work flow only. It cannot get the data from the input value given by user.
For example i used the product to create the work flow was “Watches” , and i give the input value as “bags” through URL. It reaches the bags result page but it could not get the data from bags result page instead it takes the data from Watches page. I have attached the screenshots for your reference.

Thank you @Vrushali_Gave

[Vrushali_Gave]

@PraveenKumar_S So for this, you have to edit selector.

Click on selector of Extract Structured Data activity. Change the selector title as I have mentioned below in the screenshot.
Instead of this html : title=‘Amazon.in :Bike parts’
we need to change is like this ,html : title=‘Amazon.in :*’

Regards, Thanks

[PraveenKumar_S (PraveenKS)]

@Vrushali_Gave If i change the selector like this html : title=‘Amazon.in :*’ , then it could not enter the Attach Browser ‘Amazoninw Page’. It throws an error as " Attach Browser ‘Amazoninw Page’: Cannot find the UI element corresponding to this selector: " .

I will try to search some other ways are available or not, kindly let me know if you get an idea.

Thank you @Vrushali_Gave

[Vrushali_Gave]

@PraveenKumar_S Have you change selector of attach browser and Extract Data activity?

[Vrushali_Gave]

@PraveenKumar_S I have attached workflow.

amazon.xaml (13.5 KB)

Regards, Thanks

[PraveenKumar_S (PraveenKS)]

Yes … @Vrushali_Gave I changed both attach browser and Extract Data activity. If i run the workflow after those changes it completed running whole flow but again it could not get the data. I attached my Workflow for your reference

Amazon.zip (2.8 KB)

My doubt is , if we omits the title in Attach browser and search , tag attribute in Extracted structured data activity then how it finds the data from website?

Again this issue is comes to the starting point .

Thank You @Vrushali_Gave

[PraveenKumar_S (PraveenKS)]

Yes i have edited @Vrushali_Gave

[Vrushali_Gave]

@PraveenKumar_S Have you execute my workflow?

[PraveenKumar_S (PraveenKS)]

@Vrushali_Gave it also could not get the data . So i created the new flow but i also did not work.

[Vrushali_Gave]

@PraveenKumar_S It’s working properly in my system. Please try it again and do data scraping carefully. Check all the activities and check the selector too. Take the help of my workflow to compare your workflow with what you did in it.
amazon.xaml (13.8 KB)
amazonData.xlsx (8.8 KB)

Note: You will get only those product data which kind of website view you used while data scrapping.

For example:

1. You will get only horizontal represented data in the website.

2. You will not get any vertical represented data or Data scrapping will not extract this kind of represented data. Because we used data scrapping for horizontal representation.

At your side, when you search for a product than might it represented as vertical therefore your workflow does not extract data.

Please try it carefully and I’m sure you will get your exact result.

Happy Automation
Regards, Thanks

[PraveenKumar_S (PraveenKS)]

Thank You Very Much @Vrushali_Gave . It’s working properly but i have a doubt.
If i used to create workflow using Shirts as a product , it was in horizontal position. Then i give T-Shirts from input it was also in horizontal and could take the result . But if i give Bags as a input even it would be in horizontal position it could not take the result.

kindly you tell me the reason if u know .

Thank You @Vrushali_Gave

[Vrushali_Gave]

@PraveenKumar_S It should not be like that but I don’t have any idea why this happening.
Regards,Thanks

[PraveenKumar_S (PraveenKS)]

Ok . Thank You very much For your Contribution @Vrushali_Gave