Bug ID 901625: Pre-upgrade data might be missing in the Analytics dashboards following upgrade to v7.1

Last Modified: Jul 29, 2020

Bug Tracker

Affected Product:  See more info
BIG-IQ AppIQ(all modules)

Opened: Apr 21, 2020
Severity: 3-Major

Symptoms

In rare instances, statistics data gathered prior to upgrading to v7.1 might not display in the Analytics UI dashboards, once the upgrade is complete.

Impact

Statistics data gathered before upgrading to v7.1 might not display in the Analytics UI dashboards, once the upgrade process is complete. This is the chart and dimension data that appears in Monitoring > DASHBOARDS (excluding Access reports) and Application Service dashboards found in the Applications tab.

Conditions

1. Upgrade any supported version of BIG-IQ with DCDs to version 7.1. 2. From the UI, query analytics does not include data prior to the upgrade. 3. If the post upgrade itself was successful,according to the BIG-IQ logs And the /var/log/restjavad.0.log (or /var/log/restjavad.1.log if the log was rotated) must contain the following: "Analytics script /var/config/appiq/upgrade/upgrade_analytics_reindex_tmstat_statistics_7_0_to_7_1.py finish successfully" It can take 2 or 4 days after the upgrade for it to appear. 4. You see (3.) but the upgrade scripts are still unable to complete the information processing from the ES, this is because the maximum retries was reached. The log is:/var/log/upgrade_analytics_data.log at this case it must contain ERROR log (not WARN) with the following text: "ES Post failed for url" 5. Run the command at the CM to see if there are some old analytics data that was not converted: curl localhost:9200/_cat/indices/statistics* | grep -v '\-group' If you see this command returns some indexes after the process of post-upgrade finish this is a sign that not all old analytics data was upgraded and hens will not bee seen after an upgrade.

Workaround

After you have verified that all 5 conditions are met the workaround is as follows: 1. Copy the script supplied in the pre-upgrade procedures to a file /tmp/script.sh For pre-upgrade procedures, see https://techdocs.f5.com/en-us/bigiq-7-1-0/upgrading-big-iq-5-4-7-0-dcd-cluster-latest/upgrading-bigiq-version-5-4-7-0_overview.html 2. Run the file with the added script: source /tmp/script.sh 3. Run: python2.7 /var/config/appiq/upgrade/upgrade_analytics_reindex_tmstat_statistics_7_0_to_7_1.py 4. the script will start to process unfinished old analytics data. You can monitor the progress at the /var/log/upgrade_analytics_data.log At the end of the process you should see at the log: REINDEX statistics completed successfully The script might run from couple of hours to couple or days depending on the data size left.

Fix Information

None

Behavior Change