How best to organize a large amount of repetitive data

Help
, ,
1

I have created a data table to store all of the possible commissions that can be paid to an agent based on multiple criteria, but it’s well over 600k rows (and that’s just for one product). I’m wondering if there’s a better way? I can’t upload the document (new user). But here is how it’s laid out.

Col 1 = Insurance Carrier
Col 2 = Issue Age (Age of the client at Issue Date)
Col 3 = Contract Level
Col 4 = Plan Option
Col 5 = State
Col 6 = Commission Percentage

With 8 insurance carriers, 13 issue age rules, 12 Contract Levels, 11 plan options, and 50 states (813121150) we’re well over 600k rows. Any help would be greatly appreciated.

2

Hi Chase,

I love the UiPath activities but I find that the data management/organization methods get bogged down when the data sets are too large like I’m sure you’re also finding out. The best way that I’ve found to manage large datasets is to use Microsoft SQL server and operate it through a series of database activities in UiPath.

If you need additional information, feel free to post your questions here.

3

I’m a little confused as to what you’re trying to achieve?

@Chase_Catalano You can just leave it as a datatable of all possible combinations where each row is one combination (8 * 13 * 12 * 11 * 50 = 686,400 combinations). 686,400 rows is very small for a datatable with only 6 columns. However, what does that actually achieve? What are you trying to do with this information?

To me, these look like the values/properties that would be linked to an individual person and there is no need to combine everything?

4

Hi welcome to the community!
Where do you have this data source? It will depend whether or not you process will need to use all of this data… but for things this big, it is not recommended to have all in memory but use database to filter things and bring only the data you will need to make decisions in your flow…