Bug ID 1040277: Syslog-ng issue may cause logging to stop and possible reboot of a system

Last Modified: Jan 19, 2022

Bug Tracker

Affected Product:  See more info
BIG-IP TMOS(all modules)

Known Affected Versions:
13.1.0, 13.1.0.1, 13.1.0.2, 13.1.0.3, 13.1.0.4, 13.1.0.5, 13.1.0.6, 13.1.0.7, 13.1.0.8, 13.1.1, 13.1.1.2, 13.1.1.3, 13.1.1.4, 13.1.1.5, 13.1.3, 13.1.3.1, 13.1.3.2, 13.1.3.3, 13.1.3.4, 13.1.3.5, 13.1.3.6, 13.1.4, 13.1.4.1, 14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5

Opened: Aug 12, 2021
Severity: 3-Major

Symptoms

A syslog-ng issue with remote logging to an invalid remote syslog server may lead to logging via syslog-ng to stop. For software version 13.1 only it may lead to BIG-IP unexpectedly rebooting due to host watchdog timeout, typically within hours to day or two after syslog-ng gets hung up. The cessation of logging happens at the time of the last 'Syslog connection broken' in /var/log/messages before reboot. That message will appear without a preceding 'Syslog connection established' just before it with same timestamp. At this time syslog-ng typically spins, using near 100% CPU.

Impact

Very rarely syslog-ng hangs in a non-functional state. Sometimes, this may lead to an unexpected reboot of BIG-IP. Loss of logs before restart and traffic disrupted while BIG-IP restarts.

Conditions

Invalid syslog-ng server configuration or broken connection from BIG-IP toward configured syslog-ng remote server. A server is configured as a remote syslog destination on the BIG-IP, but it or an intervening system responds to stream of log messages by breaking connection eg by sending ICMP port unreachable to BIG-IP. Syslog-ng will note the connection attempt and that it has broken usually in the same second, and do so every 60s when it retries. There may be many of these log pairs, repeating every minute in /var/log/messages, such as: Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection established; fd='14', server='AF_INET(192.168.1.1:514)', local='AF_INET(0.0.0.0:0)' Nov 25 03:14:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60' The final log will of a broken connection only, usually one minute after the last established/broken pair in the very rare event that syslog-ng hangs. Nov 25 03:15:01 localhost.localdomain notice syslog-ng[12452]: Syslog connection broken; fd='14', server='AF_INET(192.168.1.1:514)', time_reopen='60'

Workaround

Ensure syslog-ng server configuration is valid, and that the server is reachable. If a remote server is not reachable remove it from the BIG-IP syslog configuration.

Fix Information

None

Behavior Change