Orchestrator-based activity fails with the error: The Operation Has Timed Out

News Knowledge Base
1

When using activities that work with Orchestrator, like Get Asset, Get Folder item, Get Queue item, Get Credential, Get Job, , Upload Storage file, etc, you can get a TimeOut error. This error can happen randomly or as a symptom of an underlying connection issue between the Studio/Robot computer and Orchestrator.

Issue Description

Get Asset: The Operation Has Timed Out.

Get Job: The Operation Has Timed Out.

Upload Storage Files: The Operation Has Timed Out.
etc.

Resolution

  1. Some errors can be transient. If the occurrence rate is very low, consider the following:
    - Increase the Timeout value of the activity
    - Use a Retry activity
    - Upgrade the UiPath.Systems.Activity package to version 25.2.1 or higher - since starting with this version, we've improved the retry mechanism for Orchestrator-related activities.

  2. If the errors correlate to a database connection timeout, then running database maintenance can be a solution, as long as the maintenance is in line with the business retention policies.
    1. Maintenance tools: link
    2. Sample exception that might lead to this recommendation (this is a partial stack trace that simply shows the timeout. We would validate the timeout corresponds to the queue item operation): 
      • System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
  3. If Maintenance is not an option, database performance metrics should be checked to see if this is a hardware limitation.
  4. If the issue has a non-timeout exception, the problem will be addressed by fixing the underlying issue. Most times the database isolation and read committed settings are not enabled.


Diagnosing the Issue

  1. Gather the Robot logs. These can be gathered from the Orchestrator UI (see here) or from the Robot machine using the Diagnostic tool (see here).
    1. It is critical to note the timestamp of the error. It is needed to correlate the exception with Orchestrator.
    2. If using tools like Spunk or Elastic Search to aggregate the logs, pull the logs from there. Ideally, if the error occurred at 7 am, the logs would pull from at least the hour time range around that time (i.e. 6:30-7:30).
  2. For cloud, only the robot logs and the support ID are needed. The rest of the steps are for on-premises only.
  3. Gather the Orchestrator logs. If this is a multi-node instance, the logs from all servers are needed.
    1. The necessary logs are the application logs and the HTTP logs. In Standalone this would be the IIS logs and in Automation Suite this is part of the log bundle. If using a tool like Elastic Search or Splunk for logs and the IIS logs are not included, they need to be manually gathered (only applies to standalone).
    2. It is very important that the logs gathered include the event. If they do not, we cannot make a correlation.
    3. Links for log gathering:
      • Orchestrator Standalone link
      • Automation Suite link
  4. The most common causes of these errors are database issues (When Orchestrator is installed as on-premise). As such there are some things we can check on the database:
    1. Get a count of the database rows, etc: link.
      1. There is no set limit on the size of the database. The size and hardware needed for an Orchestrator database are determined by the use case. Very generic estimates based on some use cases can be found in the docs here: link. However, actual performance may vary based on the use case and it is important to monitor the performance of the database server.
      2. The count servers, for an estimate on the database profile. If, for example, database maintenance is recommended, it will be based on actual performance data (i.e. the IIS logs show a long request time for basic queries, in the Orchestrator logs we see explicit timeouts, etc.).
    2. Hardware specs of the database server.
    3. Check that read committed snapshot and allow snapshot isolation are turned on (Starting with Orchestrator version 21.10 this is mandatory to have enabled). 
      • -- Check if ALLOW_SNAPSHOT_ISOLATION is enabled
SELECT name, snapshot_isolation_state_desc 
FROM sys.databases 
WHERE name = 'MyDatabase';

        – Check if READ_COMMITTED_SNAPSHOT is enabled
        
SELECT name, is_read_committed_snapshot_on
        
FROM sys.databases
        
WHERE name = 'MyDatabase';

    4. If deadlocks or timeouts are observed in the Orchestrator logs, a SQL profiler trace may be requested: link.
      • SQL query store can be helpful as well.


Possible root causes:

  1. Timeout on SQL query
  2. A specific exception related to the query or operation itself
  3. Request not reaching the Orchestrator server (In this case we would usually expect other API calls to suffer the same issue unless it's a firewall-specific setting blocking the calls)

1 Like