How to find the job that started a queue item processing

[tristaanogre (Robert Martin)]

We’re working on building a project to help monitor and keep a long-running queue process working smoothly. One thing we’re trying to do is look for transactions in a queue that have been stuck in “In Progress” for longer than a configured period of time. If we find one, we want to trace it back to the job that started processing the item and stop or kill that job.

So far, I have no problems finding queue items and manipulating them. However, tracing it back to the initial job… that’s where I have the problem. Does anyone have any ideas, tips, tricks on how to trace back to a job on orchestrator from any given queue item?

Thanks!

[codemonkee (Tim)]

If you know the transactionID of the Queue Item which is passed in the logs as part of rawMessage you can trace it to the jobId.

In the above example, I’m using Splunk (We use NLog to cover our log event payload to JSON and redirect to Splunk), the same can be found in the Orchestrator Database in the dbo.Logs table, however the RawMessage content is in JSON, but that will allow you to trace it back to the JobID / JobKey.

It is also found when viewing the Queue Item in the Orchestrator UI.

Another way to go about it is to look at the Queue Item History where you’ll find when it was placed into “In Progress” and which Robot/User ran the process. You can then filter your Jobs based on Robot and Timeframe of what job was running at the time the item was placed in progress.

[Jon_Smith]

This is an excellent question and one I have been considering as I am working on using the various persistence activities and its also relevant there, especially when using the wait for queue item activity.

I’m not very keen on the idea of diving into the database, especially if you need to use a third party tool to do so and connect up all of this information.

Instead, this is how I am going to tackle it.
Queue items have got two areas we can manipulate during execution that can hold information useful for tracing, the progress status and comments. Personally I am going to use comments.

So what I am going to do is when a job picks up a queue item it will add a comment indicating which job is doing it. I made a custom activity to grab details like the Job ID etc so I’ll have that information at my fingertips. Adding comments can be done with an API call and using the Orchestrator HTTP activity to skip any authentication mess. Basically long term I’ll build a library that works as a wrapper around the ‘get queue item’ activity to add all this information.

With that information in tow I think it will be super easy to trace back from a queue item to a job, and job to queue item is simple as you can just make sure its in the logs.

Now I don’t expect someone to dive in and do some of the tasks I have, it seems getting the Job ID of a running job has never been solved before, at least not shared on the forums, so I’ll be the first if and when I publish that solution.
But I think you can do a lower level solution to this, if comments are too complex with the API call, just use the progress status and have it contain whatever information you can currently grab from the job to identify it. Scale it up if and when you can tackle comments (to have a better history and more information) and the Job ID if and when that becomes easily accessible.

[tristaanogre (Robert Martin)]

So far, excellent ideas. However, it’s not helpful at the point I’m at. I’d like to be able to, using API calls to the orchestrator, make this connection between the transaction being executed and the job or machine doing the execution. I know that if I manually go to the orchestrator, I can see, in the queue, what robot is doing the transaction. What I’m having difficulty finding is any data that comes back in an Orchestrator API call that contains that same information.

[codemonkee (Tim)]

Hi @tristaanogre,

Sorry I should have been more explicit in what I was describing with the UI. The Web UI leverages the same API. So if you are ever not sure which API Endpoint to call, use your browsers inspector to monitor the requests and perform the actions in the UI to determine the endpoints.

For example when you bring up a Queue Item in the UI, three calls are made

• /odata/QueueItemComments/UiPath.Server.Configuration.OData.GetQueueItemCommentsHistory
• /odata/QueueItems({key})/UiPathODataSvc.GetItemProcessingHistory
• /odata/QueueItemEvents/UiPath.Server.Configuration.OData.GetQueueItemEventsHistory

Looking at UiPath.Server.Configuration.OData.GetQueueItemEventsHistory It reveals the Robot which processed the queue Item.

{
  "@odata.context": "https://<orchestrator>/odata/$metadata#QueueItemEvents",
  "@odata.count": 3,
  "value": [
    {
      "QueueItemId": 327194,
      "Timestamp": "2021-09-01T21:13:46.69Z",
      "Action": "Create",
      "Data": null,
      "UserId": 126,
      "UserName": "user@domain",
      "Status": "New",
      "ReviewStatus": "None",
      "ReviewerUserId": null,
      "ReviewerUserName": null,
      "Id": 513168
    },
    {
      "QueueItemId": 327194,
      "Timestamp": "2021-09-01T21:14:04.377Z",
      "Action": "Status",
      "Data": null,
      "UserId": 217,
      "UserName": "K98",
      "Status": "InProgress",
      "ReviewStatus": "None",
      "ReviewerUserId": null,
      "ReviewerUserName": null,
      "Id": 513169
    },
    {
      "QueueItemId": 327194,
      "Timestamp": "2021-09-01T21:14:43.16Z",
      "Action": "Edit",
      "Data": null,
      "UserId": 217,
      "UserName": "K98",
      "Status": "Successful",
      "ReviewStatus": "None",
      "ReviewerUserId": null,
      "ReviewerUserName": null,
      "Id": 513170
    }
  ]
}

From there you can go a few different ways depending on what you expect the state of the job to be in (Completed/Stopped, In Progress)

• Get list of Jobs narrow it based on Robot Name / Timing of when the Queue Item was In Progress
• Use GetRobotDetails will provide you the currentProcess, machineName, etc.
• I’m sure there are other if you explore the https:///swagger

Use the /odata/Jobs as an example you could pass something similar to

/odata/Jobs?
$expand=Robot
&$filter=((CreationTime ge 2021-09-07T15:15:45.940Z) and (State eq '1') and (Robot/Name eq 'K98'))
&$top=10
&$select=id,robot

• Expand is including the related entity [Robot] details
• Filter is looking for
  • CreationTime >= 2021-09-07T15:15:45.940Z
  • State == In Progress
  • “Robot/Name” is looking for a property of a property, the Name of the Robot == “K98”
• Select restricts for the desired properties

{
  "@odata.context": "https://<orchestrator>/odata/$metadata#Jobs(Id,Robot,Robot())",
  "@odata.count": 1,
  "value": [
    {
      "Id": 363125,
      "Robot": {
        "LicenseKey": null,
        "MachineName": "mymachine",
        "MachineId": 98,
        "Name": "K98",
        "Username": "domain\\user",
        "ExternalName": null,
        "Description": null,
        "Version": "19.10.5.0",
        "Type": "NonProduction",
        "HostingType": "Standard",
        "ProvisionType": "Manual",
        "Password": null,
        "CredentialStoreId": null,
        "UserId": 217,
        "Enabled": true,
        "CredentialType": null,
        "RobotEnvironments": "",
        "IsExternalLicensed": false,
        "LimitConcurrentExecution": false,
        "Id": 199,
        "ExecutionSettings": null
      }
    }
  ]
}

Hopefully that gives you some ideas!

[tristaanogre (Robert Martin)]

Perfect! That’s exactly what I needed.