Bug ID 486640: LogStash Bug Prevents Parsing valid RFC5424 logs

Last Modified: Mar 21, 2019

Bug Tracker

Affected Product:  See more info
BIG-IP LTM(all modules)

Known Affected Versions:
11.4.1, 11.5.0, 11.5.1, 11.5.1 HF1, 11.5.1 HF10, 11.5.1 HF11, 11.5.1 HF2, 11.5.1 HF3, 11.5.1 HF4, 11.5.1 HF5, 11.5.1 HF6, 11.5.1 HF7, 11.5.1 HF8, 11.5.1 HF9, 11.5.2, 11.5.2 HF1, 11.5.3, 11.5.3 HF1, 11.5.3 HF2, 11.5.4, 11.5.4 HF1, 11.5.4 HF2, 11.5.4 HF3, 11.5.4 HF4, 11.5.5, 11.5.6, 11.5.7, 11.5.8, 11.5.9, 11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 11.6.1 HF2, 11.6.2, 11.6.2 HF1, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4

Fixed In:
12.0.0

Opened: Oct 24, 2014
Severity: 4-Minor

Symptoms

The ElasticSearch/LogStash/Kibana software stack is a popular mechanism for archiving free format data, such as logs. However the LogStash parser for RFC 5424 compliant logs is buggy and will not accept the Message ID field as generated by the BigIP because it contains colons.

Impact

Logs sent to LogStash servers will be lost.

Conditions

High speed logging to a LogStash server will cause this problem.

Workaround

Instead of setting the logpublisher.logstash_rfc5424_fix DB Variable to true to fix the problem, a proper RFC5424 parser may be specified for LogStash: SYSLOG5424F5_NILVALUE \- SYSLOG5424F5_NONZERO_DIGIT [1-9] SYSLOG5424F5_DIGIT (?:0|%{SYSLOG5424F5_NONZERO_DIGIT}) SYSLOG5424F5_PRINTUSASCII [\u0021-\u007e] SYSLOG5424F5_SP \u0020 SYSLOG5424F5_BOM \u00ef\u00bb\u00bf SYSLOG5424F5_MSG_UTF8 %{SYSLOG5424F5_BOM}\p{Assigned}* SYSLOG5424F5_MSG_ANY \p{ASCII}* SYSLOG5424F5_MSG (?:%{SYSLOG5424F5_MSG_ANY}|%{SYSLOG5424F5_MSG_UTF8}) SYSLOG5424F5_SD_NAME %{SYSLOG5424F5_PRINTUSASCII}{1,32}? SYSLOG5424F5_PARAM_VALUE \p{Assigned}*? SYSLOG5424F5_PARAM_NAME %{SYSLOG5424F5_SD_NAME} SYSLOG5424F5_SD_ID %{SYSLOG5424F5_SD_NAME} SYSLOG5424F5_SD_PARAM %{SYSLOG5424F5_PARAM_NAME}=\"%{SYSLOG5424F5_PARAM_VALUE}\" SYSLOG5424F5_SD_PARAMS (?:%{SYSLOG5424F5_SP}%{SYSLOG5424F5_SD_PARAM})*? SYSLOG5424F5_SD_ELEMENT \[%{SYSLOG5424F5_SD_ID:syslog5424_sd_id}%{SYSLOG5424F5_SD_PARAMS:syslog5424_sd_params}\] SYSLOG5424F5_STRUCTURED_DATA (%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_SD_ELEMENT}) SYSLOG5424F5_TIME_HOUR (?:[01][0-9]|2[0-3]) SYSLOG5424F5_TIME_MINUTE [0-5][0-9] SYSLOG5424F5_TIME_SECOND [0-5][0-9] SYSLOG5424F5_TIME_SECFRAC (?:\.%{SYSLOG5424F5_DIGIT}{1,6}|) SYSLOG5424F5_TIME_NUMOFFSET (?:\+|\-)%{SYSLOG5424F5_TIME_HOUR:syslog5424_time_numoffset_hour}:%{SYSLOG5424F5_TIME_MINUTE:syslog5424_time_numoffset_minute} SYSLOG5424F5_TIME_OFFSET %{SYSLOG5424F5_TIME_NUMOFFSET:syslog5424_time_numoffset} SYSLOG5424F5_PARTIAL_TIME %{SYSLOG5424F5_TIME_HOUR:syslog5424_time_hour}:%{SYSLOG5424F5_TIME_MINUTE:syslog5424_time_minute}:%{SYSLOG5424F5_TIME_SECOND:syslog5424_time_second}%{SYSLOG5424F5_TIME_SECFRAC:syslog5424_time_secfrac} SYSLOG5424F5_FULL_TIME %{SYSLOG5424F5_PARTIAL_TIME:syslog5424_partial_time}%{SYSLOG5424F5_TIME_OFFSET} SYSLOG5424F5_DATE_MDAY (?:0[1-9]|[1-2][0-9]|3[01]) SYSLOG5424F5_DATE_MONTH (?:0[1-9]|1[0-2]) SYSLOG5424F5_DATE_FULLYEAR [0-9]{4} SYSLOG5424F5_FULL_DATE %{SYSLOG5424F5_DATE_FULLYEAR:syslog5424_full_year}\-%{SYSLOG5424F5_DATE_MONTH:syslog5424_month}\-%{SYSLOG5424F5_DATE_MDAY:syslog5424_mday} SYSLOG5424F5_TIMESTAMP (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_FULL_DATE:syslog5424_full_date}T%{SYSLOG5424F5_FULL_TIME:syslog5424_full_time}) SYSLOG5424F5_MSG_ID (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,32}) SYSLOG5424F5_PROC_ID (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,12}) SYSLOG5424F5_APP_NAME (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,48}) SYSLOG5424F5_HOSTNAME (?:%{SYSLOG5424F5_NILVALUE}|%{SYSLOG5424F5_PRINTUSASCII}{1,255}) SYSLOG5424F5_VERSION %{SYSLOG5424F5_NONZERO_DIGIT}%{SYSLOG5424F5_DIGIT}{0,2} SYSLOG5424F5_PRIVAL (?:[0-9]|[0-9][0-9]|1[0-8][0-9]|19[0-1]) SYSLOG5424F5_PRI <%{SYSLOG5424F5_PRIVAL:syslog5424_prival}> SYSLOG5424F5_HEADER %{SYSLOG5424F5_PRI}%{SYSLOG5424F5_VERSION:syslog5424_version}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_TIMESTAMP:syslog5424_timestamp}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_HOSTNAME:syslog5424_hostname}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_APP_NAME:syslog5424_app_name}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_PROC_ID:syslog5424_proc_id}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_MSG_ID:syslog5424_msg_id} SYSLOG5424F5_LINE %{SYSLOG5424F5_HEADER}%{SYSLOG5424F5_SP}%{SYSLOG5424F5_STRUCTURED_DATA}(?: |)%{SYSLOG5424F5_MSG:syslog5424_msg}

Fix Information

A DB Variable was introduced to work around this problem by changing the RFC54254 message ID format to drop the last colon and replace the first one with the letter 'p' (priority): tmsh modify /sys db logpublisher.logstash_rfc5424_fix value true To revert this behavior: tmsh modify /sys db logpublisher.logstash_rfc5424_fix value false

Behavior Change