No New Logs Showing On Orchestrator After A Specified Date Event Viewer Showing Elastic Search Error

News Knowledge Base
1

Orchestrator logs not appearing after a specific date. The Orchestrator event logs show the following error: elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, "cluster_block_exception", "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];")

  1. Delete some old indexes to free up space on the Elastic Search server
  2. Run the following curl command to disable read-only mode:
  • $ curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

At this point, logs will show only for new jobs. Unfortunately, because the Elastic Search index was read only, the logs for jobs that have executed while the index was in read-only mode will be lost (unless they are also being written to the SQL database).

In order to prevent this issue from happening in the future:

  1. Regularly maintain the ES server to delete very old indexes. Try not to let memory get <5%
  2. The index gets locked when the configured disk watermark / quota is exceeded. Choose to set the watermark to a very low value by using the following command:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'

{

"transient": {

"cluster.routing.allocation.disk.watermark.low": "30mb",

"cluster.routing.allocation.disk.watermark.high": "20mb",

"cluster.routing.allocation.disk.watermark.flood_stage": "10mb",

"cluster.info.update.interval": "1m"

}

}

'

This will only extend the time that this issue will not occur for a little bit longer. The best approach will be to ensure that regular maintenance is done on the ES server.