Intermittently jobs stuck in Pending state after upgrade

The multi-node orchestrator has upgraded and removed one node from the load balancer. Now the single node is running without Redis from the load balancer where few scheduled jobs are going in Pending state.

Cause : As Orchestrator is still running on the node 2 (which is being disconnected from load-balancer) and is accessing same database as node 1 (which is connected), both the nodes send start job command separately. 
Resolution: In scenarios where we want to test functioning of a particular node, we should stop Orchestrator on rest of the nodes instead of just removing it from load-balancer.