Symptoms

If you remove then add a data collection device (DCD), BIG-IQ logs messages similar to the following in the /var/log/elasticsearch/eslognode.log: ------------------------------------- [2021-07-10T23:02:55,781][WARN ][o.e.g.DanglingIndicesState] [xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxxx][[.opendistro_security/xxx-xxxxxxxxxxxxx]] can not be imported as a dangling index, as index with same name already exists in cluster metadata

Impact

This might impact data collection and warning messages are logged.

Conditions

This can happen after a DCD is removed and added back to a configuration.

Workaround

Remove unnecessary indexes. 1) Find the indexes by grepping eslognode.log to get the indices that need to be removed : Note: This will return a list of the indices that need to be deleted grep DanglingIndicesState /var/log/elasticsearch/eslognode.log | awk '{print $3}' | cut -d '/' -f2 | tr -d ']' | sort | uniq Example: --------- $ grep DanglingIndicesState /var/log/elasticsearch/eslognode.log | awk '{print $3}' | cut -d '/' -f2 | tr -d ']' | sort | uniq ad5HO156TP6uvi8QiJQjlw n4TExpoYQnqk0Dq3udAq5A oD7vlt9oSTWa5FoMleLCxw 2) Remove those indexes from all BIG-IQs (CMs and DCDs) On *each* CM and DCD run the following commands to remove the stale indexes: * the command will not get any feedback rm -rf /var/config/rest/elasticsearch/data/nodes/0/indices/<index> Using the previous example: rm -rf /var/config/rest/elasticsearch/data/nodes/0/indices/ad5HO156TP6uvi8QiJQjlw rm -rf /var/config/rest/elasticsearch/data/nodes/0/indices/n4TExpoYQnqk0Dq3udAq5A rm -rf /var/config/rest/elasticsearch/data/nodes/0/indices/oD7vlt9oSTWa5FoMleLCxw

Fix Information

None

Behavior Change