How to remove duplicates in text file having more than 700000 of records

Help Something Else
1

Hi Team ,

I have a text file containing of more than 700000 of records ,Can some one help me in removing duplicate from file .
We have tried with Uipath Activities but it didn’t help much it is taking more one hour to compete .
Can some one help me to resolve this issue please?
@J0ska

Thanks .

2

Hey! Welcome to community!

Are we trying to remove the duplicates from excel or Text file?

Regards,
NaNi

3

You should provide a sample of the text file with highlighting what you mean by “duplicate”

Cheers

4

Give a try at
Deduplication
arrLines = File.ReadAllLines("YourPath").Distinct().toArray

Getting back the content string
strContent = String.Join(Environment.newLine, arrLines)

5

@Thumu_Suresh,

Check this xaml it may help you
RemoveDuplicateLines.xaml (6.2 KB)

6

can you give some input format , it might help to find easily

8

please find the sample report .
Same text file will contains more 5 Lakhs of records .We have to remove the duplicate.
In txt file data look like this .
ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999
BDCG LISM_RKP 1000005673 19.1 AUD 2215 99999
ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999

9

In text file data look’s like this .
volume will be more than 5 Lakhs .

ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999
BDCG LISM_RKP 1000005673 19.1 AUD 2215 99999
ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999

We need remove the duplicate records in txt .

Thanks.

10

In text file data look’s like this .
volume will be more than 5 Lakhs .

Tried with data table activities but it takes long to remove the duplicates.

ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999
BDCG LISM_RKP 1000005673 19.1 AUD 2215 99999
ABCD ALL_OTHERS 1000000123 15.9 IND 2434 9999

11

trying to remove duplicates in text files.
Thanks.

12

should work with the provided approach from above

13

Hi Peter ,

I have used below test data .

image
image1053×275 4.19 KB

Getting unique count as 3 instead of two .

Can you help me on this.

14

sure. Please share with us the text file. So we can replicate / Analyse with the same data

15

Please find the test files
test.txt (684 Bytes)

16

could maybe be caused by counting also empty lines, when present in the file

we used your text file:
grafik

with the result:

grafik
grafik624×691 125 KB

here also we have a +1 for the empty line

17

as an alternate we implemented are more defensive deduplication with the idea to remove all spaces, tabs…

grafik

(From x In File.ReadAllLines(myFilePath)
Group x By k=System.text.RegularExpressions.Regex.Replace(x, "\s+", String.Empty) Into grp=Group
Where Not String.IsNullOrEmpty(k.Trim())
Select v=grp.First()).toArray

grafik

18

Hi Peter ,

I tried using above solution but facing some below issue .
Can you please help .

image
image1269×349 20.4 KB

image
image1473×511 17.6 KB

Sorry for delay in response .

Thanks,
Suresh.

19

UiPath Project is set to VB.Net as selected Programming Languguage, right?

just isolate the issue by folllowing:

  • create a variable with the path, similar it was done in the sample and use it instead the hard coded filepath
  • isolate the statement into a single assign activity:
arrLines = File.ReadAllLines(myFilePath)

Debug and prototype it within the immediate panel
Understanding the 6 Debugging Panels of UiPath in the easiest way possible! - News / Tutorials - UiPath Community Forum

20

It is working now .
Thanks you so much for your help.

21

Hi Peter,

Last its works fine .while in testing encounter the same issue not sure why it is happening .

image
image1358×519 29.1 KB

image
image1620×160 12.9 KB

Can you please check it Peter.

Thanks,
suresh .T