Data Scraping - multi layer?

[ghdunn (Gerald)]

Hi
I have got data scraping working and can happily bring back a list of books:

https://www.amazon.co.uk/s?k=rpa&i=stripbooks&ref=nb_sb_noss_1

However, if i wanted to click each individual item on this list and bring back lower level data, could that be done through data scraping. For example, each book has a Product Details field on the individual book web page that has number of pages data on…so i would like to bring page number in as a field.

Alternatively, some web items have abbreviated text where you have to click on the title to ‘read more’ of the description. Example:

https://www.jobsite.co.uk/jobs/rpa/in-london?radius=0

Any way of getting to this?

I have looked at tutorials/other posts but cant see anything that suits.

Thanks

[goodoldme (Venkat)]

Hi @ghdunn,

Welcome to the Community! Data scraping can only extract data that is currently loaded/available. The scenarios you have mentioned require you to navigate to different page(s) and/or load additional information. And that’s exactly what you will need to automate to be able to retrieve them. Hope this helps.

[ghdunn (Gerald)]

Venkat…hi…thanks…Can that be done within a loop using data from the noisy scrape. Or too complicated?

Gerald

[goodoldme (Venkat)]

Hello Gerald,

If it’s a simple use case, you can use a loop construct to drive the workflow based on data from the initial scrape.For complex ones, I suggest you look at REFramework

ReFramework official

Hi everyone! This is the link to the official version of the UiPath ReFramework. It’s purpose is to provide a template which handles the basic needs of any business process automation, like reading and storing a configuration file, retrying Transactions that fail with an Application Exception and logging the status of all processed transactions, failed or successful. The best way to get it is by pulling from our github, located at GitHub - UiPath/ReFrameWork: Robotic Enterprise Framework Templ…

[Indrajit_Banerjee]

Hi @ghdunn,
Sharing one sample workflow based on your scenario and yes you can loop through the extracted URL while doing data scrapping and then do multiple works on the page as in extracting few other text like descriptions etc.

_Test.xaml (9.3 KB)

[ghdunn (Gerald)]

Indrajit,

Many thanks…could you just help me further…how do I change this config?

![image|690x417]

(upload://j0ucyrKmCvNuJppGCTYX9DCybea.png)

Gerald

[Indrajit_Banerjee]

Hi @ghdunn,
Can you please tell me which version of Studio are you working with right now?

The missing activity being shown here is because of package version mismatch for UiPath.System.Activities. Try reinstalling this package again .

As far as the workflow is concerned please see the below snapshot.

[ghdunn (Gerald)]

I am using 2019.5.0 Community edition. Do i reinstall the package using 'Tools>Project Dependencies Mass Update Tool" ? It asks me for a project to put that in…I am not sure how to proceed.

I will try to emulate your screenshot and get it working that way…will let you know how it goes!

Gerald

[ghdunn (Gerald)]

Thank you Venkat…i will work through that.

[ghdunn (Gerald)]

I think i am very close…but not quite there

If i try to scrape an item on each of the underlying pages, the output comes back with the value “Text”

[Indrajit_Banerjee]

HI @ghdunn,

I have tried simulating the steps and i am able to extract the text for the “Paperback” value.
just double check the selector.
.I am using the following selector for get full text →

[ghdunn (Gerald)]

Thanks Indrajit,

I think the Paperback object is a i different kind of object to the Paperback object I selected…I could only capture it as a Generic type. Anyway…got it working so thank you very much for your help.

Gerald