then we simplified the processing by keeping the ja string
and using the containskey result for the column value update.
(it is writing now “True”, “False” instead of yes, no (we are interested in speed currently)
Important to know about the intersect, it is is deduplicating as well. This needs to be respected when we need to keep it as well ( find matches)
Summary of the last optmization actions:
reduce the checklist
shift to dictionary containsKey as we want to check if this will be faster instead of list.contains
Result: Reducing the execution time from 24+ hrs to 10-20 secs
General Analysis Pattern for Performance Optimization
When the execution time is to optimize following general actions can be applied:
remove any unneeded actions e.g. Log messages / Write Line… from the core block, which needs to be optimized on its execution
surround the core block with a Stopwatch and trace the execution time
run the core block and cancel it, when the execution is taking too long time
Data Volume Reduction
Apply techniques like using take(x), reduced Test Data Set to retrieve an initial understanding of the core block execution timings and interpolate it to the full data volume
check and enable the parallel execution of the core block if possible. Maybe the core block has to be rewritten or to be modified
Isolate the time consuming parts within the core block
replace the time consuming parts with alternates and run again test series
Once the optimized format / core block implementation is found run and measure it on the entire data volume
We are using Intersect like you mentioned, post intersect it will only keep the unique elements. I wanted to know as I have duplicate data in both dtNewSet an dtOldSet and want the duplicates to be there. Our this processing will not remove duplicates?
Currently the bot the logic is writing True/False, how can we change to Yes/No in the code please suggest.
We are running a parallel for each which runs asynchronously and not in a sequential
manner, will the results be the same and correct which we might obtain from the LINQ query?
Also, if we need to take some more steps forward on implementation part please guide.