Bug ID 721487: ElasticSearch java process stalls when running at high CPU rate and does not recover

Last Modified: May 30, 2018

Bug Tracker

Affected Product:  See more info
BIG-IQ Platform(all modules)

Known Affected Versions:
6.0.0

Fixed In:
6.0.1

Opened: May 23, 2018
Severity: 3-Major

Symptoms

An ElasticSearch java process stalls when running at high CPU and does not recover. Search results in BIG-IQ might become stale and the user interface won't work well, if at all.

Impact

Under some conditions, the ElasticSearch continually fails, even after a restart of the service. The service periodically restarts itself due to insufficient memory to complete the ingest operation.BIG-IQ user interface becomes unusable.

Conditions

A large amount of search data updates are posted by a BIG-IQ process to the ElasticSearch instance in response to a change in data such as a multi-BIG-IP deployment.

Workaround

To work around this issue: run command: bigstart stop restjavad tokumond searchd run command: cd /var/config/rest/searchd run command: pwd Verify the output of the pwd command is /var/config/rest/searchd if in the correct directory according to the above steps run command: rm -rf ./data/ run command: bigstart start searchd tokumond Wait for all data to be re-indexed. The easiest way to determine the re-index is finished is to monitor the command: top and to wait until the ElasticSearch user's java process no longer shows high cpu for at least one minute. run command: bigstart start restjavad Once the BIG-IQ comes back up, verify the CPU utilization of the ElasticSearch user's java process is no longer in the top of the top command's results.

Fix Information

None

Behavior Change