Bug ID 840101: ElasticSearch snapshot view

Last Modified: Mar 26, 2020

Bug Tracker

Affected Product:  See more info
BIG-IQ Config Mgmt Infrastructure(all modules)

Opened: Oct 17, 2019
Severity: 3-Major

Symptoms

Not all existing snapshots appear in UI if an error occurs during snapshot removal.

Impact

ES snapshots take up space in the /var filesystem which can fill up and cause problems with other systems that need available space.

Conditions

When you select multiple ES snapshots for deletion sometimes only one is removed from the filesystem while the others remain but no longer show up in the UI.

Workaround

To work around this in the UI, select only a single ES snapshot at a time for deletion. You can also delete the snapshots from the command line as follows: 1. curl -s localhost:9200/_snapshot/backup/_all | jq > /tmp/es_snapshot_summary 2. From /tmp/es_snapshot_summary get all snapshot names and save in /tmp/snapshot_names 3. Iterate /tmp/snapshot_names to fetch snapshot name one at a time and search for the presence in restcurl -u user:password /mgmt/cm/shared/esmgmt/es-snapshots/ 4. If snapshot name is present in /mgmt/cm/shared/esmgmt/es-snapshots/ then it is not a stale entry and nothing to be done and continue to iterate. 5. If snapshot name is not present in /mgmt/cm/shared/esmgmt/es-snapshots/ then it is a stale entry and trigger { curl -X DELETE "localhost:9200/_snapshot/backup/<snapshot_name>?pretty" } for cleanup. If delete operation failed and reason for failure is 503[another delete request in progress] then wait for sometime [30 seconds] and reattempt[max reattempt count 3] and proceed.

Fix Information

None

Behavior Change