How to compare two csv files of different record counts

Help
, , , ,
1

I have to compare two csv files of different record count and write the differences in new csv file.
Say for example csv1 has 225000 records and csv2 has 300000 records.

My approach is as below -
read two csv files and output to two datatables say DT1 and DT2.
Take first record in DT1 and loop through all the records in DT2 and proceed to next record of DT1 and scan all across DT2 and so on.
Issue im facing is , Uipath Robot is not responding!!
I even tried removing matching records in datatable while iterating through the loop but it doesnt help

Please suggest a better way to compare files of different length

2

Hi @Nitin_K,

Uipath is not responding because there is a lot data and when you are looping one to one row it would take around 225000 x 300000 iterations to complete the process.
You better use either select function or use Linq to match data

3

if you have time , could you please share small snippet\example of either select function or Linq?

Thanks!!

4

I Tried using linq as below
DT1.asEnumerable.Except(DT2.asEnumerable().system.data.datarowcomparer.default).copytodatatable

Same issue , uirobot not responding
My system has 8gb ram

image

5

Can you pull your data in chunks? get the first 1000 records > process them> store your rowIndex and then get the next 1000.

Reading the Datatable 225 times should be faster than processing all of them at once.

6

Tried above method, same issue :frowning:

7

If performance is so much of an issue you can try to realize it in a

Invoke Code (C#) Activity. That might be faster.

You dont use debug mode right?. If you use debug its extremely slow.

8

@Nitin_K got any solution for bulk data, if yes can you please share the solution , even i have same requirement.

Thank you