Bug ID 569563: Sockets resource leak after loading complex policy

Last Modified: Oct 06, 2020

Bug Tracker

Affected Product:  See more info
BIG-IP APM(all modules)

Known Affected Versions:
12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2

Fixed In:
12.1.2

Opened: Jan 23, 2016
Severity: 2-Critical
Related AskF5 Article:
K11561355

Symptoms

File descriptors used by apmd remain unclosed (TCP and UDP) after loading a complex access policy. After some time, the APM process file descriptor table is exhausted and no more access policies are processed. The following error messages may be observed in the logs: err apmd[16013]: 01490000:3: HTTPParser.cpp func: "readFromSocket()" line: 86 Msg: epoll_create() failed [Too many open files].

Impact

The APM process is unable to create new sessions, leading to an inability to process access policy operations.

Conditions

This can happen at the initial stage after apmd starts, or later when policies are reloaded. Although this is not directly related to log-level, this problem is easier to observe when the access control log-level is Warning or lower (Notice, Info, Debug). File descriptors leak (remain unclosed) after loading complex policies that contain many agents.

Workaround

This can happen at the initial stage after apmd starts, or later when policies are reloaded. Current preferred workaround is to set log level to ERROR or higher and restart apmd. When a large number of file descriptors has already been observed, the only way to close them other than disabling logging is to raise log levels to ERROR or above, and then issue the following command: bigstart restart apmd Note 1: Do not use sys db variables to change log level for versions 12.0.0 and later. Note 2: Double-check log levels using the following command: tmsh list apm log-setting all-properties Note 3: Opened file descriptors do not close until apmd is restarted. Note 4: When in doubt (about whether file descriptors are leaking), run the following command on the BIG-IP system: lsof -p `pidof apmd` | grep TCP; lsof -p `pidof apmd` | grep UDP. This gives you the number of open files. - Detailed steps to change logging-level to ERROR: Step 1. Modify access control log level using the following command: tmsh modify apm log-setting all access modify { all { log-level { access-control err } } } Step 2. Check the log levels using the following command: tmsh list apm log-setting all-properties Step 3. Manually restart apmd using the following command: bigstart restart apmd

Fix Information

Sockets are now closed properly, so there is no longer file descriptor leakage when loading or reloading complex access policies.

Behavior Change