Last Modified: Sep 13, 2023
Affected Product(s):
BIG-IP TMOS
Known Affected Versions:
13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 15.0.1.4, 15.1.1, 15.1.2, 16.0.0, 16.0.0.1, 16.0.1
Fixed In:
16.1.0, 16.0.1.1, 15.1.2.1, 14.1.4, 13.1.3.6
Opened: Aug 26, 2020 Severity: 2-Critical
A syslog-ng issue with remote logging to an invalid remote syslog server may lead to unexpected reboot. The BIG-IP may unexpectedly reboot after a host watchdog timeout when syslog-ng gets hung up. Logs via syslog-ng are no longer written, though logging not via syslog-ng continues unaffected. This happens at the time of the last 'Syslog connection broken' in /var/log/messages before reboot. That message will appear without a preceding 'Syslog connection established' just before it with same timestamp. At this time syslog-ng typically spins, using near 100% CPU (just one core equivalent, not all CPU capacity on system). Typically things appear fine on rest of system - there will usually be adequate CPU and memory. Hours or days later graphs will have a gap of usually tens of minutes to hours before an unexpected reboot. Post reboot logs (in /var/log/sel for iSeries or ltm log otherwise) show this is a host watchdog reboot. After reboot the system runs correctly, though if the syslog-ng remote server was invalid this remains the case.
Very rarely syslog-ng hangs in a non-functional state. Sometimes, this may lead to an unexpected reboot of BIG-IP. Loss of logs before restart and traffic disrupted while BIG-IP restarts.
Invalid syslog-ng server configuration or broken connection from BIG-IP toward configured syslog-ng remote server. A server is configured as a remote syslog destination on the BIG-IP, but it or an intervening system responds to stream of log messages by breaking connection eg by sending ICMP port unreachable to BIG-IP. Syslog-ng will note the connection attempt and that it has broken usually in the same second, and do so every 60s when it retries. There may be many of these log pairs, repeating every minute in /var/log/messages, such as: Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection established; fd='14', server='AF_INET(192.168.1.1:514)', local='AF_INET(0.0.0.0:0)' Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60' The final log will of a broken connection only, usually one minute after the last established/broken pair. Nov 25 03:15:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60'
Ensure syslog-ng server configuration is valid, and that the server is reachable.
Fixed an issue with syslog-ng hang occasionally causing a system restart. This fix is not a complete fix. You will still need to remove unused syslog-ng servers from the BIG-IP configuration. ID 1040277 ( https://cdn.f5.com/product/bugtracker/ID1040277.html ) tracks the remaining issue.