Scraping a Variable Page

Playing about with UI, I’ve gotten how to scrape a single page with the wizard but I’m wondering if there is a way to enter a variable page. What I’m trying to do is: Go to google, search a string variable (For example, it would search for Hat if that was the variable) and from there, scrape the little bit of website preview text they give you for each of the websites (So you’d get a fragment of the start of a wikipedia article, along with the first dozen or so words from a heap of websites).

I’ve got how to open the browser. I’ve got how I can enter the variable into google and hit enter (keyword +[k(enter)]) I’ve got how to scrape a website with the wizard. What I’m not 100% sure on how to do is to put it all together and get ‘Use the website that was just reached as the source of the scraping, using the template I worked out previously’.

I get the feeling I’m missing something really simple to get my hands on that little bit of text.

Hi @Ikiryo

You should definitely check out our Academy to learn more.
A quick guide for Data Scraping can be found here:

There is a wizard that will allow you to achieve just what you want :slight_smile:

Thank you for the help, I’ve been looking over some of the academy guides and I think the issue is me failing to understand something.

Basically, what I’m trying to do is set up a scrape that works with any web page I point it at. So, google search results will all have the same general structure as each other but not the same results. The automation will do the google search for whatever I ask it for and then scrape the results.

I’m failing to quite understand how to get it pointed at ‘whatever web page you want’ however. I think I might be misunderstanding the academy, as it seems like it only works for the web page you’re already at when you start the wizard?

Sorry for causing trouble with my failure to understand.

No issue, it is quite simple and I’ll try to explain :slight_smile:

UiPath automation is based on selectors. Those selectors is how Studio sees the things on your screen. It is possible to make those selectors flexible with wildcards (* and ‘?’, see more info here), and as such it is totally possible to scrap a page of Google searches dynamically (as in making a robot that scraps the results irregardless of the keywords). Use of Data Scraping wizard would be advised here.

The academy has a chapter on Selectors as well as on Data Scraping, so check those out for sure.

However, the selectors can only be flexible so much. In practice, you will need to create a separate Data Scraping activity for each website you want to scrap data from.

Thanks a lot for the help! I’m not 100% sure I get it yet but I’ll play about with it and see if I can get it working. I get the feeling that once I’ve done it once, it will all click into place for me.

To make it a bit easier, see this example I’ve just made :slight_smile:
ScrapGoogleResults.zip (289.0 KB)

It is first asking you what term to search for and then searching for it in Google and saving the results to an Excel file :slight_smile:

Oh, that will help immensely in my understanding! I’ve always been someone who learns best through doing and this will give me a version I can play about and break to learn the ins and outs.

Thank you again.

1 Like