An error of "Invalid server response. Http Response Code: 500" occurs when calling ML Skills in Automation Suite 24.10.x
Issue Description
This is a bug that will be fixed in a future release.
The workaround is to restart all deployments starting with 'ai-' and to subsequently Stop/Resume all affected ML Skills.
Diagnosing the Issue
The issue would start appearing 3 months after installing Automation Suite 24.10.x, because secret rotation happens every 3 months. So, if the cluster at hand is not within this timeframe, then the cause may be unrelated to this KB.
Run the following command to check for the error: kubectl -n uipath logs {nameOfSkillPod} -c ai-proxy
An error with the following format will be found in the ai-proxy container of the non-working ML Skill pod if it runs into this issue:
ERROR|Microsoft.AspNetCore.Server.Kestrel|Connection id "XXXXXXXXXX", Request id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx": An unhandled exception was thrown by the application.|ConnectionId=XXXXXXXXXX,TraceIdentifier=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx,EventId=13,EventName=ApplicationError|System.Net.Http.HttpRequestException: Could not retrieve token with scope Du.Metering. Status Code: BadRequest. Token is empty: {"error":"invalid_client"}
at UiPath.DocumentUnderstanding.Common.Clients.Identity.IdentityServiceClient.SendTokenRequest(String scope)
at UiPath.DocumentUnderstanding.Common.Clients.Identity.IdentityServiceClient.<>c__DisplayClass11_0.<<GetToken>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at UiPath.DocumentUnderstanding.Common.Caching.AsyncRefreshCache1.Get(String key, Func1 refreshFunction)
at UiPath.DocumentUnderstanding.Common.Clients.Identity.IdentityServiceClient.GetToken(String scope)
at UiPath.AiProxy.Common.Services.License.MeteringValidationService.AuthHeader(String accountId, String tenantId) in /src/UiPath.AiProxy.Common/Services/License/Metering/MeteringValidationService.cs:line 102
at UiPath.AiProxy.Common.Services.License.MeteringValidationService.ValidateApiKey(String apiKey, String tenantId, String accountId, AIUnitsValidationRequest aIUnitsValidationRequest, CancellationToken ct) in /src/UiPath.AiProxy.Common/Services/License/Metering/MeteringValidationService.cs:line 39
at UiPath.AiProxy.Controllers.Auth.ApiKeyAuthHandler.HandleAuthenticateAsync() in /src/UiPath.AiProxy/Controllers/Auth/ApiKeyAuthHandler.cs:line 68
at Microsoft.AspNetCore.Authentication.AuthenticationHandler1.AuthenticateAsync() at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme) at Microsoft.AspNetCore.Authorization.Policy.PolicyEvaluator.AuthenticateAsync(AuthorizationPolicy policy, HttpContext context) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at UiPath.AiProxy.Middlewares.MeteringErrorHandlingMiddleware.InvokeAsync(HttpContext context) in /src/UiPath.AiProxy/Middlewares/MeteringErrorHandlingMiddleware.cs:line 26 at UiPath.AiProxy.Middlewares.InputFormatValidationMiddleware.InvokeAsync(HttpContext context) in /src/UiPath.AiProxy/Middlewares/InputFormatValidationMiddleware.cs:line 32 at UiPath.AiProxy.Middlewares.ApiModeMiddleware.InvokeAsync(HttpContext context) in /src/UiPath.AiProxy/Middlewares/ApiModeMiddleware.cs:line 28 at UiPath.DocumentUnderstanding.Common.Telemetry.Tracking.TrackingMiddleware.InvokeAsync(HttpContext context) at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application)
To Fix the Issue Already Caused by the Rotated Secret
1. Run the following command to restart all affected deployments: kubectl -n uipath rollout restart deploy ai-app-deployment ai-appmanager-deployment ai-deployer-deployment ai-helper-deployment ai-pkgmanager-deployment ai-trainer-deployment
2. Click the ML Skill in the AI Center UI and click the Stop Button, and click the Resume button once it's stopped. The ML Skill will start working again after it's resumed.
Workaround to Prevent Secret Rotation Before the Bug Is Fixed
- Login to ArgoCD > Click the Platform application > Click the DETAILS button > Go to the PARAMETERS tab > Click the EDIT button > add the following parameter under the identity-service section (make sure to do proper indentation):
secretRotation:
enabled: false
The identity-service section should look similar to the one below after editing and clicking the SAVE button:
identity-service:
appConfig: {}
basePath: /identity_
hostAdminUsername: admin
secretRotation:
enabled: false
kerberos:
enableSqlIntegratedAuth: false
ldapAdDomain: ""
krb5KeytabSecret: ""
2. To validate whether the parameter has been applied correctly, run the following command:
kubectl -n uipath get cronjobs | grep identity
If the following 4 cronjobs are not listed, then secret rotation has been disabled successfully: identity-secret-copy, identity-secret-expire, identity-secret-rotate, identity-secret-sync.
Note that it may take a few minutes for the cronjobs to be deleted.