How to remove duplicates in text file when we have more than 7 lakh records

Help Something Else
1

Hi Team,

Can someone help me in removing the duplicates from text file when we have more than 7 lakh records. We tried converting to excel and tried Remove duplicate rows activity but It didn’t help.

@J0ska

Thanks,
Naresh

2

@Mandava_Naresh

can you share Schema of File with some Extries.

3

Hey!

Use Generate Data Table activity and use the separator as [Comma or Space]

create one output dataTable variable

Use the expression like this to remove the Duplicates

Dt.DefaultView.ToTable(True)

The expression will remove the duplicates in entire table

Regards,
NaNi

4

have a look here and let us know the result

for further questions please share some sample data with us in advance. Thanks

5

@Mandava_Naresh,

Check with this xaml
RemoveDuplicateLines.xaml (6.9 KB)

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.