Get Queue Items An Error Has Occurred Error Code 0

News Knowledge Base
1

Why do is intermittent Error "An error has occurred. Error code: 0" thrown when trying to Get Queue items or Add Transaction Items

Issue Description:
Get Queue Items Activity within a Process fails intermittently with an error "An error has occurred. Error code: 0"

Root Cause:
This error indicates that a server-side error occurred. It is the same as if the server returned a 500 error. The causes of this could be:

  1. Timeout on SQL query.
  2. A specific exception related to the query or operation itself.
  3. Request not reaching the Orchestrator server (In this case we would usually expect other API calls to suffer the same issue unless it's a firewall-specific setting blocking the calls).
In all cases there should be a corresponding error message in the Orchestrator server logs.


Diagnosing the Issue:

  1. Gather the Robot logs. These can be gathered from the Orchestrator UI (see link) or from the Robot machine using the Diagnostic tool (see link).
    1. It is critical to note the timestamp of the error. It is needed to correlate the exception with Orchestrator.
    2. If using tools like Spunk or Elastic Search to aggregate the logs, pull the logs from there. Ideally, if the error occurred at 7 am, the logs would pull from at least the hour time range around that time (i.e. 6:30-7:30).
  2. For cloud, we only need the robot logs and the support ID. The rest of the steps are for on-premise only.
  3. Gather the Orchestrator logs. If this is a multi-node instance, the logs from all servers are needed.
    1. The necessary logs are the application logs and the HTTP logs. In Standalone this would be the IIS logs and in Automation Suite this is part of the log bundle. If using a tool like ElasticSearch or Splunk for logs and the IIS logs are not included, they need to be manually gathered (only applies to standalone).
    2. It is very important that the logs gathered include the event. If they do not we cannot make a correlation.
    3. Links for log gathering:
      1. Orchestrator Standalone link.
      2. Automation Suite link.
  4. The most common causes of these errors are DB issues. As such there are some things we can check on the DB:
    1. Get a count of the database rows, etc: link.
      1. There is no set limit on the size of the DB. The size and hardware needed for an Orchestrator DB are determined by the use case. Very generic estimates based on some use cases can be found in the docs here: link. However, actual performance may vary based on the use case and it is important to monitor the performance of the DB server.
      2. The count servers to give us an estimate on the DB profile. If, for example, DB maintenance is recommended, it will be based on actual performance data (i.e. the IIS logs show a long request time for basic queries, in the Orchestrator logs we see explicit timeouts, etc.).
    2. Hardware specs of the database server.
    3. Check that read committed snapshot and allow snapshot isolation are turned on (Starting on 21.10 this is mandatory to have enabled). 
      1. -- Check if ALLOW_SNAPSHOT_ISOLATION is enabled
SELECT name, snapshot_isolation_state_desc 
FROM sys.databases 
WHERE name = 'MyDatabase';

        – Check if READ_COMMITTED_SNAPSHOT is enabled
        
SELECT name, is_read_committed_snapshot_on
        
FROM sys.databases
        
WHERE name = ‘MyDatabase’;

    4. If deadlocks or timeouts are observed in the Orchestrator logs, a SQL profiler trace may be requested: link.
      1. SQL query store can also be helpful as well.


Resolution:

  1. If the errors correlate to a database connection timeout then running database maintenance can be a solution, as long as the maintenance is in line with the business retention policies.
    1. Maintenance tools: link
    2. Sample exception that might lead to this recommendation (this is a partial stack trace that simply shows the timeout. We would validate the timeout corresponds to the queue item operation): 
      1. System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
  2. If Maintenance is not an option, database performance metrics should be checked to see if this is a hardware limitation.
  3. If the issue has a none timeout exception, the problem will be addressed by fixing the underlying issue. Most times the database isolation and read committed settings are not enabled.
  4. As a short-term workaround, the timeout on the activity can be increased or it's possible that automations can have their schedules distributed farther apart to prevent the issue.