Bug ID 1017153: Asmlogd suddenly deletes all request log protobuf files and records from the database.

Last Modified: Jun 07, 2022

Bug Tracker

Affected Product:  See more info
BIG-IP ASM(all modules)

Known Affected Versions:
16.1.1, 16.1.0, 16.0.1.2, 15.1.4, 15.1.3.1, 15.1.3, 14.1.4.4, 14.1.4.3, 14.1.4.2, 14.1.4.1

Fixed In:
17.0.0, 16.1.2, 15.1.4.1, 14.1.4.5

Opened: May 09, 2021
Severity: 2-Critical

Symptoms

Asmlogd suddenly deletes all request log protobuf files and records from the database. Additionally, after the deletion happens, newly generated event logs seen in database do not show up in TMUI.

Impact

Sudden loss of request logs.

Conditions

-- ASM provisioned -- Config-sync setup, with frequent sync recovery and/or with a manual-sync device-group with ASM sync enabled.

Workaround

Delete all empty partitions and restart asmlogd. 1) View all the partitions: # perl -MF5::Db::Partition -MData::Dumper -MF5::DbUtils -e 'print Dumper(F5::Db::Partition::retrieve_partitions_info(dbh => F5::DbUtils::get_dbh(), table_name => "PRX.REQUEST_LOG"))' Take note of the 'PARTITION_NAME' and 'TABLE_ROWS' for each partition. 2) For every partition_X (in PARTITION_NAME) with '0' in TABLE_ROWS, delete it by: # perl -MF5::Db::Partition -MF5::DbUtils -e 'F5::Db::Partition::delete_partition(dbh => F5::DbUtils::get_dbh(), table_name => "PRX.REQUEST_LOG", partition_name => "partition_X")' 3) restart asmlogd if and after done all deletes: # pkill -f asmlogd NOTE: this workaround is temporary. Meaning, that the number of empty partitions will again accumulate and the same issue will happen again. The MAX number of partitions is - 100. Thus, it is advised to monitor the total number of *empty* partitions and repeat the workaround periodically. A cron script should work well. Normally, empty partitions should NOT accumulate. Thus, it is safe to run this script once an hour and remove all empty partitions. Optionally, if you are having TMUI issue with newly generated event logs, perform the additional command below to delete rows from PRX.REQUEST_LOG_PROPERTIES for which no corresponding rows exist in PRX.REQUEST_LOG. # mysql -t -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e 'delete from PRX.REQUEST_LOG_PROPERTIES where request_log_id > (select id from PRX.REQUEST_LOG order by id desc limit 1)'

Fix Information

Upgrading to a fixed software version will not delete accumulated partitions and will not clear the symptom away. The fix will prevent ASM from accumulating partitions unnecessarily in the scenarios. If your ASM system is experiencing this and partitions are accumulated, perform the workaround to clear the symptom. Using a fixed software version, you only need to perform the workaround once. You no longer have to perform it periodically.

Behavior Change