Complex web scraping question

Help
1

Hi all, I am new to UiPath and just playing around… I am trying to do this:

Step 1: Go to

Step 2: Open each article

Step 3: Extract spec data

image
image778×687 9.72 KB

Step 4: Iterate through all articles

Step 5: Write CSV file

Unfortunately, I cannot make it work. Any suggestions?

Would be great

2

Hello @Tom2, could you specify what didn’t work?
And what kind of CSV do you expect?

Not really trying it, but it seems like your specification could be achieved by:

  1. Open Browser and navigate to the website
  2. Use Data Scraping to get list of articles
  3. For Each Particle, open the Particle Detail and do Data Scraping again
  4. Depending on how you want you CSV to be, you could process your DataTable
  5. Write it to CSV
3

My first step is to open the browser and the right url.

Step 2: Create a table with all urls pointing to the article detail pages. I use the “Data Scraping” tool for that. However, I cannot make the “spanning multiple pages” work. I assume that is, because the “next page button” skips always 5 pages (and not 1) when clicking on it.

image

Any suggestions?

The expected CSV file should contain

  • Article name
  • Type
  • Bore Type
  • Cage Type
  • Cage Material
  • Ball Material
  • Precision
4

Hello @Tom2, for the Data Scraping, as you cannot use Next Button, I think it’s easier if you do Data Scaping inside a While so you can set your own “Go to Next Page”, as in:

DO

  1. Get current page
  2. Do data scraping with no spanning, just for current page
  3. Check if the button with the current page + 1 is exists and is clickable
  4. If link to the next page exists, you click the link to next page
    WHILE there is still link to next page.

I noticed that on the 5, 10 and 15 page you have to click the >> button, so you need to add some additional on that.

Here is a sample. It is working at least on my side: Data Scraping With No Next.xaml (20.7 KB)

Note there is always some extra empty row when scraping this web, you need to add remove empty rows later. You can add the next steps after that.

5

Many thx! That was really helpful and I could learn a lot.

I have two more questions

  1. Could you pls help me make the >> button work?
  2. For each product I would need to open the page and get data from there. Could you pls help me out?

Could be great :slight_smile:

6

Hi @Tom2,

  1. I don’t think we can and should make the >> button work, but may be there is other way to modify the Next Page Selector, but that is over my skill.
  2. What have you tried?
7

Hi @whyyouandi

I have tried to locate the >> button using the UIPath UI Explorer. However, it cannot locate the button.

Any suggestions?

8

@Tom2 Oops. You were talking about the >> button (sorry, dunno how I mistook it for the > button). I think the button has no special attributes so you should be able to locate it usually, but you can look at the element tree if you have problem with it.