Large Number of Records in Excel

Hi All,

I have an excel A with 700,000 records(due to the sensitive nature I can’t share the File here). I want to Perform VLookUp using another Excel B of 80,000 records. On reading Excel A to Perform VLookUp and Filter Records, I keep getting Job execution Failed Error. I tried using XL Excel activity to Read the File but I observed it changes the Format of the Duplicate Column (For example 4555272527262 converts to 4.5E12).

I have tried combining the two records to a single excel File i.e combine both A and B, then remove duplicate (with a custom activity) using the column specified, but it keeps running for more than 8hrs, hence not efficient.

Is there a way I can filter for Duplicate and Non Duplicate without reading the two Files? or by using a Custom Activity? or using VB.Net Code?

Thank you.

1 Like

Kindly have a view on this to get duplicate records

And for non duplicates

Cheers @Nafseeidri

for detecting Non Duplicates Set operation Except in combination with Duplicates on the All rows can help much in such scenarios