Symptoms

Disk was filled because elastic usage was not checked properly during stats injection and retention causing the DCD to be non-operational.

Impact

The DCD with full disk size will be non-operational. It will not be able to function and receive incoming messages and write them to the elastic cluster. Also the DCD services may be unavailable to the elastic cluster and this will effect the elastic cluster's health in regards to reading/writing activities.

Conditions

Elastic usage is not checked properly because of the following issues in the elastic cluster, such as red index causing the following: - AgentManager failing to check disk size on the DCD's before writing to elastic. - ConfigServer failing to check size on the DCD's when doing the retention by size logic

Workaround

None

Fix Information

Queries to the elastic cluster disk size (of every DCD) are now using an API that returns only the filesystem information https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-nodes-stats.html#fs-info

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips