Symptoms

PRX.BRUTE_FORCE_* db tables do not have a row_limit, so they can grow to consume all available disk space in /var/lib/mysql.

Impact

/var/lib/mysql can run out of disk space

Conditions

ASM provisioned

Workaround

1. Truncate the two large tables. This clears all the row in those table and should make disk space. Note that existing brute force username and IPs reporting data will be lost. # mysql -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "TRUNCATE TABLE PRX.BRUTE_FORCE_MITIGATED_USERNAMES" # mysql -u root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "TRUNCATE TABLE PRX.BRUTE_FORCE_MITIGATED_IPS" 2. Add row_limit for the two tables to avoid the same issue in the future. Add following lines in the bottom of this file, /etc/ts/tools/clean_db.yaml PRX.BRUTE_FORCE_MITIGATED_USERNAMES: row_limit: 100000 order_by: brute_force_mitigated_username_id PRX.BRUTE_FORCE_MITIGATED_IPS: row_limit: 100000 order_by: brute_force_mitigated_ip_id Restart clean_db process (there is no impact of restarting this process) # pkill -f clean_db Wait 30 sec, and make sure the process came back # ps aux | grep clean_db

Fix Information

None

Behavior Change