For Each for data scraping

Hello all,

I am using a For Each loop to data scrape some data from a website. The problem is that I need to scrape the data for each iteration of the For Each loop and so I have added the data scraping piece below the For Each row in variablename section as below:

2019-11-12_135657

The For Each loop doesn’t run when I go into debug or run mode. Is this because the data scrape section needs to be above the used variable? How do I do that and still data scrape for each iteration of the For Each?

Thanks in advance

1 Like

If you’re running the Data Scrape within the scope of the row variable, then it must be connected somehow to the selector of the area you’re scraping. And if the row keeps changing, the selector of the area you’re scraping will likely change.

Did you check to see if there are issues with your data scraping selector? And if something in the row object is a determining factor in it?

1 Like

@AndyMenon - Thank you, I’ve looked at the selector but frankly I don’t know much about them and I’m not sure what to look for. This is what I’m seeing for the section called Attach browser ‘GEMA Page’
2019-11-12_153755

For the section called Extract Structured Data 'TABLE ’ I’m seeing this in the NextLinkSelector:
2019-11-12_153930

What am I missing?

@AndyMenon - Also, Im seeing this in the second section:
2019-11-12_154208

Can you check if the :j_id14920 keeps changing between pages?
Also, please check the Unselected attributes of the Selector in UiPath Explorer. You may find attributes not selected by default, to be useful in crafting reliable selectors

That is the DataTable created by the data scraper to extract results of the scraping.
If data scraping is successful then data will be populated in the datatable

But as you’re working within the scope of each row, the data in that DataTable will change with every row.
Therefore you might want to append it to another DataTable that is global in scope to your sequence.

@AndyMenon - The whole process stops when it gets to the For Each so I’m not sure how to check if :j_id14920 is changing.

And how do I check unselected attributes? MY “Open in UI Explorer” is greyed out

What is GEMATable in the For Each row? Where do you get the data for that datatable from?

@AndyMenon - I understand what you’re saying but I’m not sure how to do what you’re suggesting. It sounds like you want me to save the scraped data to Excel

@AndyMenon - A question I can answer! The variable GEMATable is supposed to scrape this data for each pass of the For Each loop:

Actually it doesnt scrape it. Once the scraping is done the data is held in GEMATable

So what happens the very first time when there is no data in GEMATable?

@AndyMenon - My whole process works up to the point of the for each and then it just shuts down. This for each doesn’t run at all. I think I haven’t linked the variable properly to the for each but I can’t figure out what the issue is. Can you take a look at it? Its the second For each in the second activity

Main.xaml (53.7 KB)

Where is the data that is loaded to the GEMATable coming from? Are you scraping something to get the data in? And are you then using it in the For Each to scrape more data?

@AndyMenon - So I am pulling a number from an Excel table, entering it in the website and then using a For Each loop to scrape the table of data as follows


This data is supposed to be saved as the variable GEMATable.
I am then checking two of the pieces of data in each row of the scraped table. I then use two assigns to get that data. I think this answers all of your questions

Well I don’t understand. If the GEMATable is empty the first time with no rows, how does the For Each even enter the loop when there are zero rows in GEMATable?

@AndyMenon - That is my question. I have to do the data scraping within the For Each loop don’t I? How else can I iterate through each occurrence of the number pulled from the Excel sheet

There is no need for you to loop through multiple times. If I understood right:

  1. when you enter a value into the website, it comes back with a set of records
  2. If the records exceed more than one page , it will have a Next button
  3. You set up the Data Scraping Wizard to scrape the data across multiple pages using the Next button
  4. All the data will be loaded into GEMATable

Steps 1-4 are for one ID from the excel you enter into the website. You take the results from GEMATable and save it to another Global DataTable (let’s say GEMAData).

You repeat these steps for each value you enter into the website. And each time GEMATable is populated with data, you add it to the GEMAData DataTable.

Is my understanding correct?

@AndyMenon - 1 through 4 makes sense to me. So maybe I don’t need the second For Each loop then. I’ll think about it and let you know tomorrow. Thanks for all your help!

Great!

You can check the “Data Table” activities that will help you append data tables.

Thanks