Orchestrator Queues not Displaying Metrics

After many attempts to figure out what is going on, I decided to ask if anyone else is seeing this issue/bug?

Environment

Problem
Recently, the Orchestrator Queues have not been displaying metrics in the dashboard. That is, it used to display “In Progress”, “Remaining”, Average Time, “Successful”, “App Exceptions”, “Biz Exceptions”

No matter what the data is inside the Queues, all these values remain zero (0). Example screenshots below.

Anyone have any idea what’s happening - why are the metrics not being displayed in the Queue dashboard? I’m stumped. No issues I could gleam from Azure side of things…

We had the same issue on one of our Orchestrators, which we just got resolved. They did not disclose the exact solution though. I suspect there’s a disconnect between the SQL database for Queues and Orchestrator, and I’m willing to bet all the analytics are in your database correctly.

Your best bet would be to escalate this to technical support and maybe they will be able to help with this: Contact Technical Support

Regards.

1 Like

Thanks, mate. I suspected this might be a bit beyond what the community might be able to help me with lol

I did pop into the Azure SQL database, ran a few queries in dbo.QueueItems to take a look at the data stored, and the integrity is fine, the most recent queueitems are matching the queue items currently shown in the Orchestrator webapp… So you’re likely right about the disconnect between the SQL Database and the App Service.

I’ll send it over to support and see what they can do!

Curious - did you ever resolve our find out what the root cause is or was? We started to encounter this same issue out of the blue 2 days ago. The Queue Page is stuck, meaning that it display the same metrics no matter what - it is not updating. I queried the SQL Table that holds the metrics and it shows the same metric values that are being displayed on the page. The SQL Table is not being updated. It appears there is an underlying Stored Proc that is suppose to update this Table, but appears that Stored Proc is no longer being executed.
Note, this happened around the same time we were trying to stand up a new instance of Orchestrator (2019.10.x), currently we are on 2018.4.3.

Hello, we are facing same issues and even more in 2019.10.15.
Queue stats are not showing, add queue item and get queue item activities timeouts from time to time.
Some robots schedules are not starting and we even had whole orchestrator application down.

This is thread post Issue in Orchestrator version 2019.10.15 - #4 by Dainius_Mieziunas

Hi @rstaylor62 and @Dainius_Mieziunas,

After a few calls with a couple UiPath Devs, we discovered that the issue was on our Orchestrator Azure SQL Server and Database. We had a dev robot set to log “Trace” information, and this was storing a ton of records and items into Orchestrator’s SQL Database. So we set all robots to only log at the “Information” level, and we manually emptied the tables out with some SQL commands and removed old records and Queues that no longer existed.

We ended up clearing out nearly 700,000+ items worth of logs, queue items, queue definitions, etc… Due to how many items there were to clear, we couldn’t clean it all at once or SQL would time out. Had to run the script for every 50,000 records or something like that until we reached zero.

Example SQL script:

DELETE FROM DBO.QueueItems WHERE ID IN (SELECT qi.Id FROM DBO.QueueItems qi inner join QueueDefinitions qd on qi.QueueDefinitionId=qd.Id WHERE qd.IsDeleted=1)

After doing this, the issue almost immediately fixed and I was able to see the Metrics displayed in the Dashboard again.

We then implemented some of Azure’s SQL Server and Database automatic cleanup features and I wrote a few scripts that automatically run weekly to remove old and unused items and records from the DBO tables used by Orchestrator.

TL;DR, check with your SQL server admins and clean out those databases tables. If this gets too large, SQL Server starts timing out on some of the stored procedures. Then get some SQL server/database management and cleanup best practices in place.

Further Documentation on UiPath SQL Server Maintenance: Orchestrator

Good luck!

2 Likes