Last Modified: Sep 13, 2023
Affected Product(s):
BIG-IP LTM
Known Affected Versions:
11.4.1, 11.5.0, 11.5.1, 11.5.2, 11.5.3, 11.5.4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.5.10, 11.6.0, 11.6.1, 11.6.2, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 11.6.5, 11.6.5.1, 11.6.5.2, 11.6.5.3, 12.1.0 HF1, 12.1.0 HF2, 12.1.1 HF1, 12.1.1 HF2, 12.1.2 HF1, 12.1.2 HF2
Fixed In:
12.0.0
Opened: Oct 24, 2014 Severity: 4-Minor
The ElasticSearch/LogStash/Kibana software stack is a popular mechanism for archiving free format data, such as logs. However the LogStash parser for RFC 5424 compliant logs is buggy and will not accept the Message ID field as generated by the BigIP because it contains colons.
Logs sent to LogStash servers will be lost.
High speed logging to a LogStash server will cause this problem.
Instead of setting the logpublisher.logstash_rfc5424_fix DB Variable to true to fix the problem, a proper RFC5424 parser may be specified for LogStash: SYSLOG5424F5_NILVALUE \- SYSLOG5424F5_NONZERO_DIGIT [1-9] SYSLOG5424F5_DIGIT (?:0|%{SYSLOG5424F5_NONZERO_DIGIT}) SYSLOG5424F5_PRINTUSASCII [\u0021-\u007e] SYSLOG5424F5_SP \u0020 SYSLOG5424F5_BOM \u00ef\u00bb\u00bf SYSLOG5424F5_MSG_UTF8 %{SYSLOG5424F5_BOM}\p{Assigned}* SYSLOG5424F5_MSG_ANY \p{ASCII}* SYSLOG5424F5_MSG (?:%{SYSLOG5424F5_MSG_ANY}|%{SYSLOG5424F5_MSG_UTF8}) SYSLOG5424F5_SD_NAME %{SYSLOG5424F5_PRINTUSASCII}{1,32}? SYSLOG5424F5_PARAM_VALUE \p{Assigned}*? SYSLOG5424F5_PARAM_NAME %{SYSLOG5424F5_SD_NAME} SYSLOG5424F5_SD_ID %{SYSLOG5424F5_SD_NAME} SYSLOG5424F5_SD_PARAM %{SYSLOG5424F5_PARAM_NAME}=\"%{SYSLOG5424F5_PARAM_VALUE}\" SYSLOG5424F5_SD_PARAMS (?:%{SYSLOG5424F5_SP}%{SYSLOG5424F5_SD_PARAM})*? SYSLOG5424F5_SD_ELEMENT \[%{SYSLOG5424F5_SD_ID:syslog5424_sd_id}%{SYSLOG5424F5_SD_PARAMS:syslog5424_sd_params}\] SYSLOG5424F5_STRUCTURED_DATA (%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_SD_ELEMENT}) SYSLOG5424F5_TIME_HOUR (?:[01][0-9]|2[0-3]) SYSLOG5424F5_TIME_MINUTE [0-5][0-9] SYSLOG5424F5_TIME_SECOND [0-5][0-9] SYSLOG5424F5_TIME_SECFRAC (?:\.%{SYSLOG5424F5_DIGIT}{1,6}|) SYSLOG5424F5_TIME_NUMOFFSET (?:\+|\-)%{SYSLOG5424F5_TIME_HOUR:syslog5424_time_numoffset_hour}:%{SYSLOG5424F5_TIME_MINUTE:syslog5424_time_numoffset_minute} SYSLOG5424F5_TIME_OFFSET %{SYSLOG5424F5_TIME_NUMOFFSET:syslog5424_time_numoffset} SYSLOG5424F5_PARTIAL_TIME %{SYSLOG5424F5_TIME_HOUR:syslog5424_time_hour}:%{SYSLOG5424F5_TIME_MINUTE:syslog5424_time_minute}:%{SYSLOG5424F5_TIME_SECOND:syslog5424_time_second}%{SYSLOG5424F5_TIME_SECFRAC:syslog5424_time_secfrac} SYSLOG5424F5_FULL_TIME %{SYSLOG5424F5_PARTIAL_TIME:syslog5424_partial_time}%{SYSLOG5424F5_TIME_OFFSET} SYSLOG5424F5_DATE_MDAY (?:0[1-9]|[1-2][0-9]|3[01]) SYSLOG5424F5_DATE_MONTH (?:0[1-9]|1[0-2]) SYSLOG5424F5_DATE_FULLYEAR [0-9]{4} SYSLOG5424F5_FULL_DATE %{SYSLOG5424F5_DATE_FULLYEAR:syslog5424_full_year}\-%{SYSLOG5424F5_DATE_MONTH:syslog5424_month}\-%{SYSLOG5424F5_DATE_MDAY:syslog5424_mday} SYSLOG5424F5_TIMESTAMP (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_FULL_DATE:syslog5424_full_date}T%{SYSLOG5424F5_FULL_TIME:syslog5424_full_time}) SYSLOG5424F5_MSG_ID (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,32}) SYSLOG5424F5_PROC_ID (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,12}) SYSLOG5424F5_APP_NAME (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,48}) SYSLOG5424F5_HOSTNAME (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,255}) SYSLOG5424F5_VERSION %{SYSLOG5424F5_NONZERO_DIGIT}%{SYSLOG5424F5_DIGIT}{0,2} SYSLOG5424F5_PRIVAL (?:[0-9]|[0-9][0-9]|1[0-8][0-9]|19[0-1]) SYSLOG5424F5_PRI <%{SYSLOG5424F5_PRIVAL:syslog5424_prival}> SYSLOG5424F5_HEADER %{SYSLOG5424F5_PRI}%{SYSLOG5424F5_VERSION:syslog5424_version}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_TIMESTAMP:syslog5424_timestamp}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_HOSTNAME:syslog5424_hostname}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_APP_NAME:syslog5424_app_name}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_PROC_ID:syslog5424_proc_id}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_MSG_ID:syslog5424_msg_id} SYSLOG5424F5_LINE %{SYSLOG5424F5_HEADER}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_STRUCTURED_DATA}(?: |)%{SYSLOG5424F5_MSG:syslog5424_msg}
A DB Variable was introduced to work around this problem by changing the RFC54254 message ID format to drop the last colon and replace the first one with the letter 'p' (priority): tmsh modify /sys db logpublisher.logstash_rfc5424_fix value true To revert this behavior: tmsh modify /sys db logpublisher.logstash_rfc5424_fix value false