How is Queue Related Data Represented in Insights?
Understanding How Ingestion of Queue Data Works
Queue Data gets to Insights via the ingestion of QueueItems and QueueEvents. As of writing this documentation (this was written in March 2024) all queue data is derived from the queueItems that are ingested.
Firstly, understand the difference between a QueueItem and a QueueItemEvent.
- A QueueItem is the QueueItem that is created when used to add QueueItems to Orchestrator. It is the transaction.
- The QueueItemEvents, represents all the states or events associated with a particular QueueItem. For example, a QueueItem might be present that went from New, to In Progress, and finally to Successful. For each stage there would be a queueItemEvent.
When a QueueItem is ingested, first ensure that it has reached what is called a 'Terminal State'. This check is done by looking for a terminalQueueItem Event. This is basically a state that is final. Examples could be that the QueueItem is in the Failed (With no more retries), or Successful state. If these conditions are met, ingest the item because it is expected that the state of the QueueItem will never change (Some exceptions would be that the QueueItem was deleted at a later event. This would change the state of the item. But generally try to wait until the QueueItem is no longer in a changing state. This is why a queueItem in new state is never seen in Insights).
When the QueueItem is ingested, take the following actions:
- Ingest the associated queueItemEvents (but these are not used directly in reports)
- Ingest the QueueItem itself.
Implications On Reporting and Other Nuances: To help explain how this all works, consider the following scenario,
- A Folder called ParentFolder is present and within that Folder there is a child folder called ChildFolder
- In the ParentFolder there is QueueA and in the ChildFolder there is QueueB. There is also QueueC which is linked to both folders.
Scenario 1:
QueueA has been created. No queueItems have been processed. In Insights, create a report that list all the Queues in the ParentFolder.
- Results: Since no queueItems have been processed, no queueItem exists and the Insights returns no Queues for the ParentFolder. Once a queueItem has been process it will show up in Insights and it is observed that folder is having a Queue.
- Take Away: Data about a given queue will not show up in Insights until an associated queueItem has been processed and is a Terminal State.
Scenario 2:
We have a process called ProcessQueueB that exists in the ParentFolder. It is processing the QueueItems in QueueB that exists in ChildFolder. Create a query to show all the QueueItems that have been processed in QueueB. Also add the field Folder.
- Results: See that all queueItems processed in QueueB show their folder as the ParentFolder. The QueueItem will always report its Folder based on where the process that processed the QueueItem exists. Considering that Queues can be linked to other folders (at which point it exists in both folders) its not possible to determine where the folder exists. (This is explained more in Scenario 3).
- Take Away: The queueItem Folder, is determined by where the process who processed the QueueItem is located.
Scenario 3:
Consider a process called ProcessQueueC. It exists in both ParentFolder and ChildFolder. A report is required on how many items in QueueC have been processed by ProcessQueueC and the report is filtered by the folder ChildFolder.
Additionally, consider that for the version of ProcessQueueC that exists in ChildFolder there are 20 items processed, but for the version of ProcessQueueC that exists in ParentFolder there are 10 items processed. Finally, in the Orchestrator Queue, a total of 30 items have been processed.
- Results: The report would show that 20 Items were processed for QueueC when filtered by the ChildFolder. Sometimes we get users expecting that even if they filter by ChildFolder, they would still see 30 items processed. This is not the case. For a QueueItem, currently a field defining the Queue Links associated with the QueueItems Queue is not included. In other words, from the ingested QueueItem data alone, it is not possible to infer that the Queue is linked to another folder.
But even if possible, there would be other issues. Consider a dashboard has been created that shows the Queues in each folder and the count of QueueItems that has been processed for each Queue. If full count is shown for each Queue independent of the folder the QueueItem processed in, there would be a dashboard that shows QueueC having had 30 items processed in both the ParentFolder and the ChildFolder. The total would be 60 QueueItems, which would be in accurate. By differentiating where the QueueItem is processed, accuracy is maintained.
- Take Away: To see the total items processed in a linked Queue, account for the different Folders the queue is process in.
Scenario 5:
To create a report of all the number of Queues that exists in a given folder. The items in QueueA that exists in ParentFolder are only processed by processes in ParentFolder. The same holds true for QueueB in ChildFolder. At one time processes in ParentFolder and ChildFolder were processing items in QueueC but the linking is removed so that QueueC only exists in ChildFolder.
- Results: The report would show that in ParentFolder we have QueueA and QueueC and in ChildFolder we have QueueC and QueueB. This is because historically items in QueueC were processed in both the ParentFolder and ChildFolder.
- Take Away: Insights shows historical data and will not update QueueItems based on changes that occurred post ingestion. This would also apply if a queue is moved by linking it to a new folder and then removing the link from the original folder.
Scenario 6:
All Queues are deleted. Do a report on on the number of queues present.
- Results: If any queueItems were ever processed, they will still show up in queries. Also see the Queues that existed in the folders.
- Take Away: Same as the previous Scenario 5. Insights shows historical data and no updates are made to that historical data.