Loop data scrape > add data row to master file

Help Studio
, ,
1

Hi friends and RPA gurus,

I am performing a data scrape of a few websites for client. These websites have many different products on them, which require accessing each product. I have performed initial data scrape to get sub links and created a loop to go to each product page and scrape the product specific info…

Now, I want to add every one of the product info as a row in a master data table… problem is, the data for each product needs to be transposed to a horizontal row + the output of the data scrape is a data table variable, so the “add data ROW” activity will not work… already tried a few different strategies to assign the table to a data row variable.

I would greatly appreciate any thoughts on the structure of code to add each new product data scrape into the master data table. Thank you!

2

@Shelby_Pons The Data that needs to be transformed into a Horizontal row, can you Show how it looks in an Excel. Depending on that I guess you can use this Activity below :

1 Like
3

Scraping data from each product results in this:

image
image305×728 9.26 KB

Which I’ve been able to transpose with the Transpose data activity, resulting in this:

image
image1292×71 4.69 KB

I only want to write to excel at the end of all product data scrapes. In this URL (https://www.satco.com/products/menu/LightBulbs/Technology/Fluorescent) there are 290 products, so I just think this method is best rather than going back and forth between browser and excel to paste.

The issue is, when i scrape the product info, it is a data table output, which I cannot use to do an “add data row” to the master table of products. I think what would fix my issue is to transform that data table output from the scrape to a data row, which I could then feed into the master table. . . would need to be repeatable over all 290 products, which is what is confusing me.

4

Hello,

Sorry if I misunderstood your problem but are you trying to merge datatables (the transposed results)?

1 Like
5

Thank you! I will try this. I think I am a bit confused on the difference between Merge and Join dt activities. I turned away from testing this out because I thought my problem would be solved by some kind of complicated loop + invoke method sequence to turn the dt into a data row variable. Will test soon and get back to you. Thanks!

1 Like
6

You Join ON something (with left, right or inner join): that’s the datatable VLOOKUP :slight_smile:

For the Merge, (sorry for the image…), but think about business merging, squashing all “dealing with” duplicates and normalizing schemas… the difference should stick now…

1 Like