Bug ID 1599825: The RabbitMQ connection was broken after the OCP upgrade, and the reconnection is only partial

Last Modified: Oct 19, 2025

Affected Product(s):
BIG_IP_NEXT(SPK) SPK(all modules)

Fixed In:
1.7.9

Opened: Jun 21, 2024

Severity: 2-Critical

Symptoms

During boot up, as part of initialization, CWC creates a connection with RabbitMQ. It also creates a channel and queue to receive requests from F5 controllers. It then starts a routine to send heartbeat messages periodically to all subscribers. According to the logs, the initial connection was successful. It appears that the worker node was upgraded, causing RabbitMQ to move to a different worker node, resulting in CWC losing its connection with RabbitMQ. CWC was notified about the disconnection and started retrying the connection with RabbitMQ. During the retry, if the connection is not successful, CWC waits (sleeps) for 3 seconds before attempting a new connection.

Impact

License status not getting updated in RabbitMQ.

Conditions

Random start up of RabbitMQ pod & multiple restarts of RabbitMQ can cause the issue.

Workaround

Restart CWC pod.

Fix Information

Connection retry shall be performed only by main thread. When CWC receives disconnect notification, it should 1.Stop heartbeat routine/timer, 2.Retry connection and establish connection, create channel and queue 3.Start heartbeat timer This way only one thread retries for connection and above scenario of race-condition doesn’t occur.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips