Data scraping with multiple pages not working

Help Studio
, ,
1

Dear Forum Members,

I have designed a workflow in which I am extracting Customer Reviews for any product on Amazon website and I am using Data Scraping method for this. After selecting relevant data, when I am clicking on finish then it is asking for “Is data spanning multiple pages” where I am selecting yes and indicate on “Next Page”.

But now the problem is: When I am executing this workflow, after scraping the data from first page, it is clicking on “next page” button and then scrap the data from second page and then instead of clicking on “next page”, it clicks on “previous page”.

And this process keeps going, clicks on next page, then previous page and so on. So, I have to forcefully stop the process. I have tried this with separate “Element Exist” and “click” activity but the output is same.

Any suggestions for this issue.? Thanks in advance.

2

Hi @dimple.khurana

I think there is some issue in Next Page selector

Please check it

Regards
Nived N
Happy Automation

3

Hi @NIVED_NAMBIAR ,

Thanks for you response.

The selector is: " < webctrl parentid=‘cm_cr-pagination_bar’ tag=‘A’ />"

It does not seem that there is any issue. Please tell me if this is fine or not.

Thanks.

4

Hi @dimple.khurana
Can u share the link where u are doing datascarping operation?

5

Yeah Sure.

Link:

This is a sample link but this is not fixed. It can be changed based on user input.

Thanks

6

Hi @dimple.khurana
Can u try with this selector in Next Link selector in property panel

                         "<html app='chrome.exe' idx='*' /><webctrl aaname='Next page→' tag='A' innertext='Next page→' />"

It is working fine

Mark it as solution if it resolves your query

Regards
Nived N

Happy Automation

7

Hi @NIVED_NAMBIAR ,

Awesome, this is working as expected. Thank you so much for your time.

Can you please help me with one more query. Actually I am searching for creating dynamic URL.

As I send above, this URL would be directly goes to customer rating page. Is there any way to create a common URL, so just by changing product name, bot could go to the customer review page for that product.

Thanks in advance.

8

U had to observe the url for each product and then create a dyanmic url based on it depending upon pattern

9

Actually I tried the same but it is giving 404 error.

10

Can u share some example of url of each costumer review of product?

11

Actually this is not fixed. It can be any product like Mobile, Laptop, Ear Buds etc.
But website is Amazon. So, how this could be achieved.

I am writing some sample URLs

one link in the previous message. Please help if this is achievable.

Thanks

12

Hi @NIVED_NAMBIAR ,

I have not found the solution yet. Did you find it.?

13

Hi @NIVED_NAMBIAR ,

Sorry to bother you again. Actually the selector for Next page which you provided yesterday, that time it worked fine but today, it is not.

When I am giving any number to MaxNumberofResults, it should only work for that particular number but it keeps moving to next page which was not happened yesterday.

Do you have any idea about this…?? Thanks in advance.

14

I will check it

15

Hi @dimple.khurana

Since the url changes and I couldn’t find any such pattern for this

So I think u can’t dynamically create url.based on product name

I think there is an API for getting Amazon product review
You can check that

Just Google it

16

Thank you, I will check for the API and also waiting for your response regarding that selector issue. Thanks

1 Like
17

Hi @NIVED_NAMBIAR ,

I am still stuck at that part. I have tried to execute my workflow with that product which has less customer reviews to check that it is extracting the data or not. After processing, when i check the datatable variable then it was blank.

I am attaching a sample workflow, if you could help because I am trying that part since morning and don’t know where I am wrong. Could you please check it once.

Sequence.xaml (18.5 KB)

Thanks

18

Hi @Nithinkrishna , @kantheshm ,

Can you please help me. I am struggling with this from yesterday. I have attached a sample workflow. In this, data scraping is not working properly. The datatable is coming blank. I don’t know where I am wrong.

Sample workflow : Sequence.xaml (18.5 KB)

Please guide me. This is a bit urgent. Thanks in advance.

1 Like
19

Hi @dimple.khurana
Sorry for delayed response
I just got in some works

Check the workflow

Sequence.xaml (22.6 KB)

It is extracting all the reviews correcly

I think u forget to click see all reviews option
image

So i added the click option and then same datascrapping feature is used and i got all reviews

Check the reviews i got in excel
output.xlsx (25.2 KB)

Regards
Nived N
Happy Automation

20

Hi @NIVED_NAMBIAR ,

Actually, I didn’t forgot to add the click activity for “See all reviews”, that was present in the workflow under if condition, after element exist activity.

But I try to execute the sequence which you uploaded. But it is not working because of selectors as it is not finding the uielement. So, I tried to validate the selector but still not working.

Can you please try to execute my workflow and try to resolve the issue, as it is working fine till click on see all reviews. I am having problem in data scraping.

It has 2 issues: one is that data table is blank and second is that it keeps moving on the next page until last page comes but it should work for the number provided in MaxNumberofResults. It just keeps moving to pages but not extracting any data.
Please help with this issue. I don’t know why this is happening as it was working fine 2 days ago and I have not done any changes.

Thanks