Last Modified: Feb 07, 2024
Affected Product(s):
BIG-IP ASM
Known Affected Versions:
16.1.1, 16.1.0, 16.0.1.2, 15.1.4, 15.1.3.1, 15.1.3, 14.1.4.4, 14.1.4.3, 14.1.4.2, 14.1.4.1
Fixed In:
17.0.0, 16.1.2, 15.1.4.1, 14.1.4.5
Opened: May 09, 2021 Severity: 2-Critical
Asmlogd suddenly deletes all request log protobuf files and records from the database. Additionally, after the deletion happens, newly generated event logs seen in database do not show up in TMUI.
Sudden loss of request logs.
-- ASM provisioned -- Config-sync setup, with frequent sync recovery and/or with a manual-sync device-group with ASM sync enabled.
Delete all empty partitions and restart asmlogd. 1) View all the partitions: # perl -MF5::Db::Partition -MData::Dumper -MF5::DbUtils -e 'print Dumper(F5::Db::Partition::retrieve_partitions_info(dbh => F5::DbUtils::get_dbh(), table_name => "PRX.REQUEST_LOG"))' Take note of the 'PARTITION_NAME' and 'TABLE_ROWS' for each partition. 2) For every partition_X (in PARTITION_NAME) with '0' in TABLE_ROWS, delete it by: # perl -MF5::Db::Partition -MF5::DbUtils -e 'F5::Db::Partition::delete_partition(dbh => F5::DbUtils::get_dbh(), table_name => "PRX.REQUEST_LOG", partition_name => "partition_X")' 3) restart asmlogd if and after done all deletes: # pkill -f asmlogd NOTE: this workaround is temporary. Meaning, that the number of empty partitions will again accumulate and the same issue will happen again. The MAX number of partitions is - 100. Thus, it is advised to monitor the total number of *empty* partitions and repeat the workaround periodically. A cron script should work well. Normally, empty partitions should NOT accumulate. Thus, it is safe to run this script once an hour and remove all empty partitions. Optionally, if you are having TMUI issue with newly generated event logs, perform the additional command below to delete rows from PRX.REQUEST_LOG_PROPERTIES for which no corresponding rows exist in PRX.REQUEST_LOG. # mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'delete from PRX.REQUEST_LOG_PROPERTIES where request_log_id > (select id from PRX.REQUEST_LOG order by id desc limit 1)'
Upgrading to a fixed software version will not delete accumulated partitions and will not clear the symptom away. The fix will prevent ASM from accumulating partitions unnecessarily in the scenarios. If your ASM system is experiencing this and partitions are accumulated, perform the workaround to clear the symptom. Using a fixed software version, you only need to perform the workaround once. You no longer have to perform it periodically.