Bug ID 947341: MySQL generates multiple error 24 (too many files open) for PRX.REQUEST_LOG DB tables files.

Last Modified: May 29, 2024

Affected Product(s):
BIG-IP ASM(all modules)

Known Affected Versions:
14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.1.0, 16.1.1

Fixed In:
17.0.0, 16.1.2, 16.0.1.2, 15.1.3, 14.1.4.1

Opened: Sep 22, 2020

Severity: 3-Major

Symptoms

1) var/lib/mysql/mysqld.err is filled with log entries that contain: ------------ 200824 11:04:43 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24) 200824 11:18:46 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24) 200824 11:35:58 [ERROR] mysqld: Can't open file: './PRX/REQUEST_LOG.frm' (errno: 24) ------------ 2) There are a lot of PRX.REQUEST_LOG partitions, in some cases up to 1024, many of which are empty.

Impact

MySQL runs out of resources when opening the file PRX.REQUEST_LOG and an error message states the file is corrupt.

Conditions

ASM/AVR provisioned

Workaround

1. If the /appdata partition is filled to 100% and MySQL restarts continuously, refer to the following Knowledge Articles: https://support.f5.com/csp/article/K14956 https://support.f5.com/csp/article/K42497314 2. To identify the empty partitions, look into: mysql -su root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "SELECT * FROM INFORMATION_SCHEMA.PARTITIONS WHERE table_name = 'REQUEST_LOG' AND table_schema = 'PRX'\G" 3. For every partition that is empty, manually (or via shell script) execute this sql: mysql -su root -p$(perl -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw) -e "ALTER TABLE PRX.REQUEST_LOG DROP PARTITION <empty_partition_name>;" Note: <empty_partition_name> must be substituted with the partition name, for example p100001. 4. Increase 'open_files_limit' to '10000'. -------------------------------- In the /etc/my.cnf file: 1. Change the value of the 'open_files_limit' parameter to 10000. 2. Restart MySQL: bigstart restart mysql -------------------------------- 5. pkill asmlogd Note: This workaround does not survive upgrade. It must be reapplied after every upgrade until the upgraded version contains a fix.

Fix Information

This release increases the default 'open_files_limit' to '10000'.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips