[LINQ] - Remove Duplicate Rows with Multiple Unique Identifier Columns from Multiple Data Table

[ppr (Peter Preuss)]

@anthonyjr

It is possible to have N+2,3,4 and so on depending on how large the original data source.

not sure if this question was get and answered by you. Let me rephrase:

Your sample had not the case that table1 has a rows which is duplicated in Table3. Could this scenario also happen?

However to preview on solution approaches (when split tables are to kept)

• loop over the tables and track the IDs in a Dictionary as unique Keys - dictLKID
• retrieve IDs from checked table, calculate the duplicated IDs by using the dictLKID and remove the duplicated rows

The reason why we suggest this Dictionary approach is the outcome from another Topic where we reduced the processing time from 24+ hrs to seconds

LINQ query running for long hours

Let us introduce the next changes [grafik] we are creating lists with the concatenated strings from dtold, dtnew and creating a dictionary of only the items common in both lists dtNewSet.asEnumerable.Select(Function (x) String.Join(“#”, arrColSet.Select(Function (k) x(k).toString.ToLower.Trim))).Distinct().toList dtOldSet.asEnumerable.Select(Function (x) String.Join(“#”, arrColSet.Select(Function (k) x(k).toString.ToLower.Trim))).Distinct().toList ListDT1.Intersect(ListDT2).ToDictionary(Fun…