Datatable comparison hanging/taking very long time to process

Help
, ,
1

Hello,

I have 2 large Datatables A and B. Both tables have around 100,000 rows, and I have converted them to a similar column structure for comparison.

I have 3 columns defined for both the tables - Col1, Col2, Col3.

The objective is to find all rows between both tables where Col1 field of Table A matches Col1 field of Table B, Similarly Col2 field of Table A matches Col2 of Table B, However Col3 of Table A should not match Col3 of Table B.

I started of using the dirty/easy method of for each inside a for each to compare by rows, which would take may be a day to finish comparison -
Next i tried if I could sort both tables first and then do a for each and use a break activity inside the nested for each to limit the comparison. Not much happened with this approach, but I will have to research more on this.
Currently I am using the Join Datatable activity to have the filter set as following

image
image812×565 12.9 KB

The Output from this seems to be hanging as well. If i remove the 3rd filter which does not match Col3’s, it gives a fast output with rows matching Col1’s and Col2 's.

Any feedback - since the dataset is huge?
@Palaniyappan ?

2

It would help if you add primary key to your tables. Does this datacolumns have unique values?

3

Thanks for reply @tdhaene

I tried using a query on the data tables and that seems to have worked for my problem.

I would try using your approach of using PK as well.

Good Day!