Bug ID 645197: Monitors receiving unique HTTP 'success' response codes may stop monitoring after status change

Last Modified: Nov 07, 2022

Bug Tracker

Affected Product:  See more info
BIG-IP LTM(all modules)

Known Affected Versions:
11.6.0, 11.6.0 HF1, 11.6.0 HF2, 11.6.0 HF3, 11.6.0 HF4, 11.6.0 HF5, 11.6.0 HF6, 11.6.0 HF7, 11.6.0 HF8, 11.6.1, 11.6.1 HF1, 11.6.1 HF2, 11.6.2, 11.6.2 HF1, 12.0.0, 12.0.0 HF1, 12.0.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.0, 12.1.0 HF1, 12.1.0 HF2, 12.1.1, 12.1.1 HF1, 12.1.1 HF2, 12.1.2, 12.1.2 HF1, 12.1.2 HF2, 12.1.3, 12.1.3.1, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1

Fixed In:
13.1.0, 12.1.3.2, 11.6.3

Opened: Feb 15, 2017
Severity: 3-Major

Symptoms

Monitors that return unique HTTP/1.1 200 codes (indicating success) accumulate in the monitor history. Upon monitor status change (such as to 'fail'), this history is sent (from 'bigd' to 'mcpd') to indicate that monitor's new-status, plus historical context. This history may grow too large if no monitor status is detected for an extended time (such as days or weeks) when unique status codes are returned from the web server and accumulated in the history. Upon a monitor status change (such as from 'success' to 'fail'), notification from 'bigd' to 'mcpd' fails due to this too-large history, resulting in the monitor remaining in its previous state (i.e., 'success'). 'bigd' properly records the monitor status and continues to monitor, but 'mcpd' is not notified of that status change (due to message-send failure from the history being too large). This is typically not an issue when the web server returns the same HTTP/1.1 200 code (indicating 'success'), as 'bigd' elides/merges the response-value into the monitor history (so the history does not continue to grow). However, for web servers generating a unique value for each success code (e.g., by appending an always-unique transaction ID to the end of the HTTP/1.1 200 response), the history continues to grow for that monitor until a status-change is detected.

Impact

The monitor remains in the 'success' state, as the status-change is lost' ('bigd' properly recognizes the changed monitor status, but 'mcpd' is not notified of the change). The system may eventually self-correct, such as when 'bigd' detects further monitor status changes, and again forwards status-change notifications for that monitor to 'mcpd'.

Conditions

-- Web server returns unique HTTP/1.1 200 (success) codes, such as an included date/time stamp. -- Success history is accumulated for that monitor without status-change for extended time (typically days-or-weeks); followed by a monitor status change (such as from 'success' to 'fail').

Workaround

Modify the web server configuration to not respond with unique HTTP/1.1 200 codes. (Receiving the same return-code elides/merges content with previously accumulated values in the monitor history.)

Fix Information

HTTP/1.1 200 codes with unique values accumulate for limited history, rather than unbounded history, such that monitor status change notifications are always recorded.

Behavior Change