Bug ID 695117: bigd cores and sends corrupted MCP messages with many FQDN nodes

Last Modified: Sep 13, 2023

Affected Product(s):
BIG-IP LTM(all modules)

Known Affected Versions:
11.6.0, 11.6.1, 11.6.2, 11.6.3, 11.6.3.1, 11.6.3.2, 11.6.3.3, 11.6.3.4, 11.6.4, 11.6.5, 12.0.0, 12.0.0 HF1, 12.1.0 HF1, 12.0.0 HF2, 12.1.0 HF2, 12.0.0 HF3, 12.0.0 HF4, 12.1.1 HF1, 12.1.1 HF2, 12.1.2 HF1, 12.1.2 HF2, 12.1.0, 12.1.1, 12.1.2, 12.1.3, 12.1.3.1, 12.1.3.2, 13.0.0, 13.0.0 HF1, 13.0.0 HF2, 13.0.0 HF3, 13.0.1

Fixed In:
12.1.3.3, 11.6.5.1

Opened: Nov 17, 2017

Severity: 2-Critical

Related Article: K30081842

Symptoms

When configured to monitor large numbers of nodes and/or pool members including FQDN nodes and/or pool members, the following symptoms may occur: - bigd may core (aborted by sod due to missed heartbeat). - bigd may produce corrupted MCP messages. - FQDN nodes and/or pool members may remain in a Checking state indefinitely.

Impact

This issue produces the following impacts: - bigd may core. - nodes and/or pool members may remain in a Checking state indefinitely. - bigd may produce corrupted MCP messages, which generate error messages in the LTM log of the following form: ... err mcpd[####]: 01070712:3: Caught configuration exception (0), Can't parse MCP message, ... Examination of the corrupted MCP message shows objects at the point of corruption that have no hierarchical relationship with the objects referenced at the beginning of the message.

Conditions

These symptoms may occur on affected versions of BIG-IP when a large number of nodes and/or pool members including FQDN nodes and/or pool members are configured. Depending on the capabilities of the platform in use, approximately one thousand (1,000) or more total nodes and/or pool members may be required to produce these symptoms. FQDN nodes and/or pool members generate a more significant workload for the bigd daemon than nodes and/or pool members with statically-configured IP addresses. This additional load contributes to high CPU usage and the other observed symptoms.

Workaround

To work around this issue, use the following approaches singly or in combination: 1. Reduce the number of nodes and/or pool members configured for a given BIG-IP system. 2. Configure nodes and/or pool members with statically-configured IP addresses.

Fix Information

bigd no longer produces corrupted MCP messages, resulting in nodes and/or pool members remaining in a 'checking' state, with up to 2,000 nodes and/or pool members including FQDN nodes and/or pool members configured. This issue is resolved by the FQDNv2 feature re-implementation in this version of the software.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips