How to remove duplicates in text file when we have more than 7 lakh records

Hi Team,

Can someone help me in removing the duplicates from text file when we have more than 7 lakh records. We tried converting to excel and tried Remove duplicate rows activity but It didn’t help.




can you share Schema of File with some Extries.


Use Generate Data Table activity and use the separator as [Comma or Space]

create one output dataTable variable

Use the expression like this to remove the Duplicates


The expression will remove the duplicates in entire table


have a look here and let us know the result

for further questions please share some sample data with us in advance. Thanks


Check with this xaml
RemoveDuplicateLines.xaml (6.9 KB)

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.