Bulk Add Queue Items with datatable that has over 80k rows

[itaim]

Hello everyone, I need a little bit of a help since I got a solution in mind but maybe you guys have more elegant way.

I have a final datatable that I want to add to the orch queue, I also want it to be FIFO therefore I know that It’s not possible to get it done instantly with so much rows. so I want to do for each loop that takes 100 rows each times places them in a datatable that will be used on the add queue items and continue the loop till I get all the rows.

Anyone can help?

[Anil_G (Anil Gorthi)]

@itaim

You can do upto 15K items in bulk queue

so to get items as chunks use for loop with list items as dt.AsEnumerable.Chunks(10000) …for safety using 10k

inside loop use add bulk queue item with currentitem.CopyToDatatable as the datatable input

cheers

[itaim]

I don’t have the Chunk method.

Also when it’s a big group like 10k it messes the order it goes to the orchestrator, and I want it FIFO

[Anil_G (Anil Gorthi)]

@itaim

then you can use take and skip…You can do it in 100 as well …

check this

Insert data in queue with Bulk add Queue items error

@Deep_HML To confirm…did you try with only onw row of data? …first try with it…it might be that the columns names are not exactly matching or so…check those… And yes the limit is 15k or so… We can break the table into rows of 10k and add in 3 loops So use for loop with Enumerable.Range(0,Cint(Math.Ceiling(dt.RowCount/10000))).ToArray and change type argument to integer And inside loop to get each chunk use dt.AsEnumerable.Skip(currentitem*10000).Take(10000).CopyToDataTable And use the chun…

cheers

[ppr (Peter Preuss)]

@itaim

I don’t have the Chunk method.

You dont have due the project is set to Legacy?

Then do the following modelling

Assign Activity
SegmentCount | int32 = CInt(Math.Ceiling(yourDT.Row.Count / 1000))

For Each Activity | item in SegmentCount

• Assign Activity: tmpDataTable | DataType: DataTable =

yourDT.AsEnumerable().Skip(item*1000).Take(1000).CopyToDataTable

[ppr (Peter Preuss)]

Find some Visualizations here:

Process repeated for each multiple rows

@christine.tzenghy let me introduce a general approach that can be also adopted to more case specific details. Assumptons: DataTable with 8 Rows Segmentsize: 3 Following Building Blocks are used Calculation of the numbers of segments: [grafik] with the Ceiling method the fractions are uprounded to the next Integer with the Skip() and Take() Method the Rows for a segment can be retrieved. the different segments are bult by following [grafik] and do fetch the different segment rows by: …

For LINQ Introductions have a look here:

[HowTo] LINQ (VB.Net) Learning Catalogue

This topic references the different Learning HowTos related to LINQ. It will be updated on new published HowTos as well LINQ Learning: Topic LINK First Start with LINQ [HowTo] - First Start with LINQ (VB.Net) Exploring the LINQ universe [HowTo] - Exploring the LINQ Universe (VB.Net) Operator [HowTo] Overview Topic LINK Where [HowTo] LINQ - First Start # Where Operator Select work in progress Any [HowTo] LINQ – First Start # Any Operator All [HowTo] LINQ – First Start…

[itaim]

I appreciate all the answers, you guys helped a lot :), I did segments of 100 in order to keep it FIFO in orchestrator if anyone will encounter this.

Thank you!