Bulk Add Queue Items write nonenglish values

I’m writing a robot in which speed matters and found activity Bulk Add Queue Items quite useful because adding around 100 items by Add Queue Item activity takes tens of seconds which is quite long for process.
I noticed that Bulk Add Queue Items adds nonenglish text as unicode entities, though Add Queue Item activity works as expected. It’s not a problem to convert from unicode entities to readable text but it looks quite off and also human can’t understand these queue items.
It would be great if Bulk Add Queue Items works same way as Add Queue Item activity.

Clipboard02

Hi @Yuri,
Thank you for your suggestion. I added it to our internal ideas tracker for our team to consider.

I’m using Bulk Add Queue Items as well and it’s extremely frustrating that one supports it but another don’t, it makes no sense at all why it is build this way and I see this as a bug and it should be fixed asap.

If i use the old for each row method and use the “old” ‘Add Queue Item’ it adds it fine with my localized special character, but it is quite combersome to write the string to add all 17 columns. also the runtime is longer

Bulk should support UTF-8 !

As an enterprise user I’m quite frustrated that i just found this bug now, because i have to go back and revert quite a few robots and use time on writing strings, and this makes me quite mad to be honest. Why would you support it on one queue add handler but not the other?!?

@loginerror please get this prioritized, 6 months since it was considered and this is still an open issue…

Just to clearify some how it kinda works

I can’t show you all fields but there is a few above the green in this image that is the unicode string instead of the actual character, but because it is sensetive information i cannot show it
image

As you can see it works for both key and value some places but not in others

Hi @Rasmus_J

Could you try this solution?

Specific content needs to be annotated with the ODATA type as shown below:

{
	"queueItems": [
		{
			"Priority": "Normal",
			"SpecificContent": {
				"hello_in_russian@odata.type": "#String",
				"hello_in_russian": "Привет",
				"hello_in_chinese@odata.type": "#String",
				"hello_in_chinese": "你好"
			}
		}
	],
	"queueName": "acaEA",
	"commitType": "StopOnFirstFailure"
}

in a code box? else i don’t follow
FYI, I’m not a programmer, i can code some here and there, but for the above i need some more information :slight_smile:

Also i still don’t get it, why not just update the Bulk add to support special characters like russian, chinese, nordic?
DK have 3 special characters ‘æ’ and ‘Æ’, ‘ø’ and ‘Ø’, ‘å’ and ‘Å’

1 Like

It would seem I rushed with the answer, let me dig a bit deeper :slight_smile:

@Rasmus_J

Okay, I seem to have figured it out a bit, but unfortunately it is simply not working properly for the activity, while working correctly within the Orchestrator UI.

If you take this, save it as a CSV file (UTF-8) and upload it in Orchestrator UI, it will work correctly:

Reference;Priority;hello_in_russian@odata.type;hello_in_russian
1;Normal;#String;Привет

image

However, loading this CSV file to a Data Table in Studio and passing it to Orchestrator using the Bulk Add Queue Items returns this for me:
image
(an empty queue element).

I registered this particular issue in our bug tracker.

Lastly, I also feel like we shouldn’t force our users to name their columns like that. We will look into that too.

1 Like

Okay, final update for now -> it also works with Bulk Add Queue Items and I just need some tea :sweat_smile:

I forgot to flip the delimiter for my CSV file in the activity here:
image

With that, it properly added the queue item:
image

@loginerror thx for the effort, but i still don’t see how that solves my issue :thinking:

I get a mail with excel document, this cannot be changed and it will have different columns, with only a few in common in each run and those i use for handeling each row, but i will still need all data for status mail. I do a read range into at DT and use that DT in the bulk add, this way i don’t have to predefine all columns in the DT (i can’t due to the difference between documents). Data is correct in the datatable, the issue is when bulk add sends the info to the queue it converts special character, or rather some of them, to unicode strings.

By using read range combined with bulk add, i get the fastest and easiest way to add a lot of data to the queue system without going through 7 hoops and a little dance before everything is working :wink:

Prioritizing fixing the bug in ‘bulk add’ so it works just like ‘add queue item’ regarding characters seems a bit more important in my opinion than trying to find a way around it, even tho i really appreciate the effort. In any case having two activities that more or less does the same, where one is working perfect using UTF-8 but isn’t great to use with unknown data columns and the other does not support UTF-8 but handles any number of columns without ever defining anything else than the DT, still doesn’t make any sense and it never will.

This feels like swimming across the lake to get water and running a sprint with your shoes tied together.

I might be misunderstanding your posts, let me know :slight_smile: But fixing the UTF-8 issue seems like the proper solution

1 Like

I keep digging and it would seem like the way it works now would be the proper way to feed Orchestrator data (one should explicitly set its type).

Until it is properly documented, the solution to your struggle would be to insert an extra column that explicitly specifies the type of data for that column.

Thus, if your strings are in the column that is called myStrings, then you should add 1 extra column that is called myStrings@odata.type with all values set to #String.

I believe this can be done with a simple Add Data Column and a Default value, like so:

This way you won’t have to loop through anything and it will be ready to be pushed to Orchestrator.

@loginerror I get what you are saying

So since i don’t know what else columns are in the document except for 3 out of everything from 5 to 20 columns i should create a loop that goes through each column, extract the column name, then adds a new column with the [ColumnName]@odata.type with ‘#String’ as default value for each column that it reads?

1 Like

If the goal is to dynamically handle your input file, then that would be the way.

I would say you will need one For Each activity that loops over dt.Columns, maybe one more IF statement in the loop to ignore columns that shouldn’t be handled and then Add Column activity.

After setting the argument type of the For Each loop to a data table column you will be able extract the column name with something like item.name

This was a possible solution, see further down for the real solution. Removed this possible solution to remove any confusion as to what will work
See: Bulk Add Queue Items write nonenglish values

1 Like

@loginerror doesn’t work, after some change compared to the above i got it to add all the odata columns and then when i did the bulk add i just got
Bulk Add Queue Items: Operation returned an invalid status code 'BadRequest’

First off, the change i did was cloning the original DT to a temp after editing the period out of titles, because you can’t loop over the same DT that you are editing when adding and looping over columns, makes sense as it’s an infinite loop.

Here is a snapshot of the DT column names after the last ‘For Each’. everything on the left of the red mark is the column names and then on the right you see the odata columns start in the same DT
image
Here you see the data (again i had to blur out sensitive data)
image

So what is going wrong here, i would say i did what you mentioned, but it’s not allowing me to send it in bulk to orchestrator

EDIT: I tried to replace space with underscore just to see if that was the issue, but no change, still the same error

I think it comes from the columns that shouldn’t be strings. I mentioned above that you might need one more IF statement to only rename the Specific Content columns, and not the ones that are used by the template, such as Reference or Priority.

I tried it and indeed such DT causes me the same error as you got:

[Reference,Priority,hello_in_russian,Reference@odata.type,Priority@odata.type,hello_in_russian@odata.type
1,Normal,Привет,#String,#String,#String
]

Something like this should do the trick:

@loginerror My bad, don’t know why i overlooked it and it makes sense, I came up with this as a temporary fix, devs should be able to implement this in the activity a bit cleaner :slight_smile:

Here is the solution and it seems to work perfectly, considering going through the smallest amount of hoops and dances :rofl:

First do a ‘Read Range’ into a DT created by the activity (in this case named AddToQueue)
Then setup the following

First ‘For Each’ is removing the period from column names (proberly other symbols should be removed? @loginerror )
Assign in first For Each loop:
column.ColumnName = Regex.Replace(column.ColumnName.ToString,".","")

Now copy the datatable as we cannot loop over and modify the same datatable

Then we go through the second ‘For Each’ it loops through columns from the temp datatable, extract the datatype as string, replaces the ‘System.’ in front of every datatype with # and adds that as the new odata column value in the original datatable referring back to the column name

image
Add Data Column
ColumnName:
column.ColumnName.ToString+"@odata.type"

DataTable:
AddToQueue

DefaultValue:
Regex.Replace(TempDT.Rows(0).Item(column.ColumnName).GetType.ToString,“System.”,"#")

Now we have set the proper datatype for each column and we can send the data to an orchestrator queue with full UTF-8 support in Bulk Add

please pleas pleas update the Bulk add to do this by itself :sweat_smile:

1 Like