Bug ID 1040277: Syslog-ng issue may cause logging to stop and possible reboot of a system

Last Modified: Oct 04, 2024

Affected Product(s):
BIG-IP TMOS(all modules)

Known Affected Versions:
13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1, 13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 13.1.3.6, 13.1.4, 13.1.4.1, 13.1.5, 13.1.5.1, 14.0.0, 14.0.0.1, 14.0.0.2, 14.0.0.3, 14.0.0.4, 14.0.0.5, 14.0.1, 14.0.1.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.0.0, 15.0.1, 15.0.1.1, 15.0.1.2, 15.0.1.3, 15.0.1.4, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 15.1.10, 15.1.10.2, 15.1.10.3, 15.1.10.4, 15.1.10.5, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.0.1.2, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 16.1.4.3, 16.1.5, 16.1.5.1, 17.0.0, 17.0.0.1, 17.0.0.2, 17.1.0, 17.1.0.1, 17.1.0.2, 17.1.0.3, 17.1.1, 17.1.1.1, 17.1.1.2, 17.1.1.3, 17.1.1.4

Opened: Aug 12, 2021

Severity: 3-Major

Symptoms

A syslog-ng issue with remote logging to an invalid remote syslog server may lead to logging, even locally, via syslog-ng to stop. CPU use of syslog-ng may increase. For software version 13.1 only it may lead to BIG-IP unexpectedly rebooting due to host watchdog timeout, typically within hours to day or two after syslog-ng gets hung up. The cessation of logging happens at the time of the last 'Syslog connection broken' in /var/log/messages before reboot. That message will appear without a preceding 'Syslog connection established' just before it with same timestamp. At this time syslog-ng typically spins, using near 100% CPU.

Impact

Very rarely syslog-ng hangs in a non-functional state. Sometimes, this may lead to an unexpected reboot of BIG-IP. Loss of logs before restart and traffic disrupted while BIG-IP restarts. Even if it doesn't reboot the loss of logging functionality can cause some daemons to block while logging and thus interrupt service.

Conditions

Invalid syslog-ng server configuration or broken connection from BIG-IP toward configured syslog-ng remote server. A server is configured as a remote syslog destination on the BIG-IP, but it or an intervening system responds to stream of log messages by breaking connection eg by sending ICMP port unreachable to BIG-IP. Syslog-ng will note the connection attempt and that it has broken usually in the same second, and do so every 60s when it retries. There may be many of these log pairs, repeating every minute in /var/log/messages, such as: Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection established; fd='14', server='AF_INET(192.168.1.1:514)', local='AF_INET(0.0.0.0:0)' Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60' The final log will of a broken connection only, usually one minute after the last established/broken pair in the very rare event that syslog-ng hangs. Nov 25 03:15:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60'

Workaround

Ensure syslog-ng server configuration is valid, and that the server is reachable. If a remote server is not reachable remove it from the BIG-IP syslog configuration. If the system has encountered this issue it's important that syslog-ng is restarted if that (or equivalent such as reboot) hasn't already occurred, to resume its normal service and reduce risk of further issues. bigstart restart syslog-ng

Fix Information

None

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips