Bug ID 1055153: Elasticsearch cluster goes into read-only if disk space runs low

Last Modified: May 20, 2022

Bug Tracker

Affected Product:  See more info
BIG-IQ AppIQ(all modules)

Opened: Oct 14, 2021
Severity: 3-Major

Symptoms

Elasticsearch has a default parameter that changes the cluster to read-only mode when the disk is running low. This is the error log that indicates the read-only mode state: {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403} The mode does not change back to read/write when disk space is available again.

Impact

The elasticsearch cluster is not operational.

Conditions

A data collection device runs low on /var disk space.

Workaround

Run the following command on the Centralized Manager or on any one of the data collection devices: for idx in $(curl -ks https://localhost:9200/_cat/indices | awk '{print $3}'); do if [ $idx != ".opendistro_security" ]; then curl -k -u admin:admin -X PUT "https://localhost:9200/$(echo $(echo $idx | cut -d\+ -f1)*)/_settings?pretty" -H 'Content-Type: application/json' -d'{"index.blocks.read_only_allow_delete": false}'; fi; done

Fix Information

None

Behavior Change