A job is stuck in Terminating and cannot be removed

News Knowledge Base
1

A job cannot be removed on Orchestrator. It is stuck in Terminating.

Issue Description

A job has been terminated in Orchestrator and is now stuck in the 'Terminating' state.


Root Cause

When a job is terminated it submits a status to the database. If this status update does not reach Orchestrator, the job will be stuck in the terminating state.


Diagnosing/Resolving

  1. First, see the section "Manually Removing Job From Orchestrator" to fix the bad state that Orchestrator is in.
  2. Check the Robot Event viewer logs to see if the Robot gives any indication of what the communication error was. (Event Viewer->Windows Logs->Applications and events from the source UiPath). For example, there might be an error like the following:
    • System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it
  3. Most likely the error that would be seen is a timeout in the connection. If this is the case, check with the Orchestrator admin to see if there was an outage or if Orchestrator is having performance issues. Try searching UiPath KB articles to address the specific issue.
  4. If the following error is encountered, go to the "WAF Configuration" section.
    • submitJobStates: UiPath.Service.Orchestrator.Clients.OrchestratorHttpException: Forbidden
  5. If no KB for the specific issue can be found, open a ticket with UiPath.


WAF Configuration

  1. The error 'submitJobStates: UiPath.Service.Orchestrator.Clients.OrchestratorHttpException: Forbidden' can occur when the Robot is blocked from submitting a job state.
  2. Most likely this means there is a Web Application Firewall (WAF) between Orchestrator and the Robot machine. When a job is terminated multiple messages are sent to Orchestrator informing Orchestrator of the state of the automation. When a job is terminated, the request sent to Orchestrator is getting falsely flagged
  3. Check to see if there is a WAF in the network topology and if it implements standard OWASP 3.1.
    • This implementation is common on Azure WAFs. See: https://docs.microsoft.com/en-us/azure/web-application-firewall/ag/application-gateway-crs-rulegroups-rules?tabs=owasp31#owasp-crs-31
    • Try disabling rule REQUEST-942-APPLICATION-ATTACK-SQLI.
  4. If there is no WAF in the network topology, check the IIS logs to see if a request made it to Orchestrator.
    • Go to the IIS logs on the Orchestrator server. (For default configuration this is typically: C:\inetpub\logs\LogFiles\W3SVC2)
    • Search for '/api/robotsservice/SubmitJobState' with status code 403. The request would look something like POST /api/robotsservice/SubmitJobState - 443 - - - 403 0 0 58
  5. If a message like the one in the above step is found, open a ticket with UiPath.
  6. If no message like the one above is found, let your network admin know. If an error like the above is not found, it means that the request did not reach Orchestrator and was blocked by a WAF or load balancer.


Manually Removing Job From Orchestrator

You can create SQL query to identify the terminating state job, the following is the list of state code used in the dbo.jobs table.

  1. Pending-0
  2. Running-1
  3. Stopping-2
  4. Terminating-3
  5. Faulted-4
  6. Successful-5
  7. Stopped-6


The following query can be handy.

First check are you getting the terminate process in table, execute the below query to get it

Select * from [UiPath].[dbo].[Jobs] where Jobs.State=3

Then update the terminating state to faulted, so that the terminating state is changed to faulted.

Update [UiPath].[dbo].[Jobs] set jobs.state=4 where Jobs.State=3

3 Likes