Faster way of iterating through scraped data in a datatable

Help
, ,
1

HI Guys,

I need to write the data scraping content to a data table and then iterate through the rows. Im currently using Excel but its very slow. takes about 30-40secs to read a csv file with 2 -3 rows .
Hence thought of doing it on the fly and bypass the Excel write and read. Can I use data scraping to an array or data table instead?

Kind regards,
Ajay

1 Like
2

@Ajay_Gopinath

It won’t take that much time. May I know your RAM size ?

And also may I know what are trying to do here after scraping data ?

3

Hi Lakshman,
Thank you for your response.
RAM size is 8 GB.
Just iterating through the rows … and based on the value of a variable in that row Im send an email.
Kind regards,
Ajay

1 Like
4

@Ajay_Gopinath

Do one thing. Don’t write scraped data into Excel file. Instead of that directly iterate data once scraped from page. It will give output as DataTable right. Use ForEach Row activity to iterate DataTable.

1 Like
5

Hi
Usually data scrapping will give us output in the form datatable only. So I we don’t need to write them again to another datatable
Or
Are we doing screen scrapping Method high is different from data scrapping as screen scrapping gives us output of type string

If it is data scrapping we can directly iterate through the variable by passing that to a For each row Activity and use IF condition to validate them
Or
If it’s a screen scrapping then we can pass that string variable as input to a activity named GENERATE DATATABLE activity where we will be getting an output of type datatable which can be then passed as input to For each row activity

Usually it won’t take much time
If so kindly use READ CSV FILE activity and pass the csv file path as input and get the output variable of type datatable
Then we can use that in for each row activity
That would quite simple and easier

Hope this would help you
Kindly try this and let know for any queries or clarification
Cheers @Ajay_Gopinath

6

Thank you Lakshman and Palaniyappan

Work perfectly.

3 Likes
7

Now it does it in 6-10 seconds. Before it did it in 57secs-to 1.5 mins.

1 Like
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.