How to manage data table in this web data scraping?

I am performing data scraping of several web pages using For Each Row in Data Tabel activity.
The ExtractDT shape obtained from each page is as follows.
화면 캡처 2021-09-11 214932

  1. How to get temp_DT as below from ExtractDT?
    화면 캡처 2021-09-11 215051

  2. How to add data to temp_DT to finally get final_DT as shown below?
    화면 캡처 2021-09-11 220511

@greenlee

Sorry we didn’t understand anything with above mentioned input and output screenshots. Could you please tell more details about your requirement.

Hi @lakshman

Thanks for asking for more details.

It would be appreciated if you could look at the attached project file.

The purpose of this project is data scraping of 700 web pages. Each page’s Extracted data table has 6 rows with 1 column. I turned this into a data table ( ExtractDataTable_transposed) with 1 row with 6 columns using the Transpose DataTable activity (I tried also your Transpose DataTable activity). And the goal is to get the final file (result.xlsx) with 700 rows.

Please let me know the following:

  1. Why does ExtractDataTable_transposed.Rows.Count.ToString show ‘0’? “tempResult.xlsx” is blank.

  2. How to add each row to excel file (result.xlsx)?

한식메뉴사전_Case2.zip (60.8 KB)

@greenlee

Instead of using Transpose Datatable activity, Refer below and add logic to your workflow

Hope this may help you

Thanks

1 Like

Hi @Srini84

When I put the ‘Transpose DataTable’ code into For each row in datatable activity, I encountered the following message.

Add Data Column: A column named ‘Column0’ already belongs to this DataTable.

How to solve it?

@greenlee

In Read Range remove Add Headers option and check

Hope this may help you

Thanks

1 Like

I got the dt_Sample data table from web data scrapping so I didn’t use Read Range activity.

Hmmm…

I found the way to turn around the problem above. I put the while activity into the if activity which has condition as dt_Final.Rows.Count=0.

image

There is one more thing I want to know.

Each iteration in For Each Row in DataTable activity, old data of dt_Final is removed and updated with new data. Is it possible to append new data to the row below the old data?

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.