Job logs suddenly don't appear in Orchestrator (when read target is Elasticsearch).
Issue Description:
There will be an error in the Event Viewer Logs in the following format:
ElasticSearch: Bulk item failed: index:tenantName-yyyy.mm result: type:logEvent error:Type: illegal_argument_exception Reason: "Rejecting mapping update to [tenantName-yyyy.mm] as the final mapping would have more than 1 type: [_doc, logEvent]"
This means that the mapping of the index is currently "_doc", but the documentType in the UiPath.Orchestrator.dll.config is currently "logEvent".
- To verify this, open the dll.config file and search for "robotElasticBuffer", to find the nlog settings for Elasticsearch. The settings will be in the following format below. Verify that the documentType is "logEvent".
- Next, go to Kibana > Stack Management > Index Management > Check the checkbox of the "tenantName-yyyy.mm" index > Manage Index > Show index mapping > Mappings.
The mapping will be in the following format. Verify that the mapping is "_doc" (or whichever type is mentioned in the last part of the error).
{
"mappings": {
"_doc": {
"properties": {
Root Cause:
A separate index is created by Orchestrator every month in Elasticsearch using the naming convention of tenantName-yyyy.mm (ie. default-2024-07).
When the index is created, the factor that influences the mapping (for example, _doc, logEvent, etc.) of the index is the documentType that is found in the nlog settings in the dll.config file. By default, the documentType should be "logEvent" (unless the Elasticsearch version is 8.x or above, in which the documentType needs to be "").
This particular error is caused when the index was created with a different mapping (for example "_doc") when the dll.config states that the documentType should be "logEvent".
Usually this gap in the mapping/documentType would be caused by a misconfiguration at some point in the dll.config, since even an upgrade to the most recent Orchestrator version will not change the documentType from "logEvent" to "_doc".
Resolution:
- Change the documentType in the dll.config to match the current documentType in the index mapping, and then stop -> start the UiPath Orchestrator site in IIS. Doing the change the other way around will also work, but there will be data loss, since index mappings cannot be changed once the index is created. The only way to change the mapping is to delete the index and recreating it.
- However, if the preferred method is to delete -> create a new index (and if losing the log data stored in the index to be deleted is fine), then simply deleting the index from the Kibana UI and subsequently running a new job from Orchestrator will force the creation of a new index with the exact same name (matching the mapping of the current documentType in the dll.config file).
Changing the documentType in dll.config to "_doc" will not adversely affect previous Orchestrator logs, meaning that reading logs from old indices that have the "logEvent" mapping will still work without issues. This is because read operations from Elasticsearch do not consider the mapping. Only write operations will throw the exception.