Last Modified: Oct 07, 2023
Affected Product(s):
BIG-IP APM
Known Affected Versions:
12.1.0, 12.1.1
Fixed In:
13.0.0, 12.1.2
Opened: Jan 23, 2016 Severity: 2-Critical Related Article:
K11561355
File descriptors used by apmd remain unclosed (TCP and UDP) after loading a complex access policy. After some time, the APM process file descriptor table is exhausted and no more access policies are processed. The following error messages may be observed in the logs: err apmd[16013]: 01490000:3: HTTPParser.cpp func: "readFromSocket()" line: 86 Msg: epoll_create() failed [Too many open files].
The APM process is unable to create new sessions, leading to an inability to process access policy operations.
This can happen at the initial stage after apmd starts, or later when policies are reloaded. Although this is not directly related to log-level, this problem is easier to observe when the access control log-level is Warning or lower (Notice, Info, Debug). File descriptors leak (remain unclosed) after loading complex policies that contain many agents.
This can happen at the initial stage after apmd starts, or later when policies are reloaded. Current preferred workaround is to set log level to ERROR or higher and restart apmd. When a large number of file descriptors has already been observed, the only way to close them other than disabling logging is to raise log levels to ERROR or above, and then issue the following command: bigstart restart apmd Note 1: Do not use sys db variables to change log level for versions 12.0.0 and later. Note 2: Double-check log levels using the following command: tmsh list apm log-setting all-properties Note 3: Opened file descriptors do not close until apmd is restarted. Note 4: When in doubt (about whether file descriptors are leaking), run the following command on the BIG-IP system: lsof -p `pidof apmd` | grep TCP; lsof -p `pidof apmd` | grep UDP. This gives you the number of open files. - Detailed steps to change logging-level to ERROR: Step 1. Modify access control log level using the following command: tmsh modify apm log-setting all access modify { all { log-level { access-control err } } } Step 2. Check the log levels using the following command: tmsh list apm log-setting all-properties Step 3. Manually restart apmd using the following command: bigstart restart apmd
Sockets are now closed properly, so there is no longer file descriptor leakage when loading or reloading complex access policies.