Orchestrator Application Restarted Due to Bulk Queue Request Overload IIS Configuration and Performance Issues

News Knowledge Base
1

What are the best practices for configuring IIS memory limits to handle large bulk operations, and how can we prevent similar issues in the future?

Issue Description

An issue with the "Private Memory Limit" setting in application pools is restricting the ability to add large volumes of items at once. This setting controls the maximum amount of memory a worker process (w3wp.exe) can use before it gets recycled, impacting resource management and potentially causing issues with handling large batches of data.


Robot server error

Errors in the robot machine logs:

  • Logged time: 13-09-2024 18:41:38
  • Error: Connection attempt failed. The error occurred while starting the 'WebSockets' transport.
  • Exception Details: System.Net.WebSockets.WebSocketException (0x80004005): The server returned a status code '503' instead of the expected '101'. This indicates that the Orchestrator server did not respond.


Orchestrator Server Error

image.png

Resolution

  1. Open IIS Manager: Go to Start, type "IIS Manager" or "inetmgr" in the search box, and press Enter.
  2. Select Application Pool: In the Connections pane on the left, expand the server node and click on "Application Pools."
  3. Edit Application Pool Settings: Choose the application pool you need to adjust and click on "Advanced Settings" in the Actions pane on the right.
  4. Adjust Private Memory Limit:
    • In the "Advanced Settings" dialog, locate the "Recycling" section.
    • Find the "Private Memory Limit (KB)" field and enter the desired memory limit in kilobytes (KB).
  5. Apply Changes and Restart:
    • Click OK to save the changes.
    • To ensure the new settings take effect, select the application pool and click on "Recycle" in the Actions pane.

Reference image:

Further, check the below settings:

  • Application Pool Settings: In IIS, check the application pool settings for your Orchestrator application. Make sure the recycling settings are appropriate for your workload. Sometimes, frequent recycling can cause disruptions.
  • Memory Leaks: If the application is consuming more memory over time (due to inefficient handling of bulk requests or memory leaks), IIS might recycle the application pool to free up resources. Use tools like memory profilers to identify and fix memory leaks.
  • Configuration Limits: Check if there are any configuration limits in IIS related to request size, queue length, or execution time that might be impacting the application's stability.

Outcome:

After updating the Private Memory Limit, more requests can be successfully sent to the Orchestrator Server.