Bug ID 1040573: REST operation takes a long time when two different users perform tasks in parallel

Last Modified: Apr 24, 2024

Affected Product(s):
BIG-IP LTM(all modules)

Known Affected Versions:
14.1.0, 14.1.0.1, 14.1.0.2, 14.1.0.3, 14.1.0.5, 14.1.0.6, 14.1.2, 14.1.2.1, 14.1.2.2, 14.1.2.3, 14.1.2.4, 14.1.2.5, 14.1.2.6, 14.1.2.7, 14.1.2.8, 14.1.3, 14.1.3.1, 14.1.4, 14.1.4.1, 14.1.4.2, 14.1.4.3, 14.1.4.4, 14.1.4.5, 14.1.4.6, 14.1.5, 14.1.5.1, 14.1.5.2, 14.1.5.3, 14.1.5.4, 14.1.5.6, 15.1.0, 15.1.0.1, 15.1.0.2, 15.1.0.3, 15.1.0.4, 15.1.0.5, 15.1.1, 15.1.2, 15.1.2.1, 15.1.3, 15.1.3.1, 15.1.4, 15.1.4.1, 15.1.5, 15.1.5.1, 15.1.6, 15.1.6.1, 15.1.7, 15.1.8, 15.1.8.1, 15.1.8.2, 15.1.9, 15.1.9.1, 16.0.0, 16.0.0.1, 16.0.1, 16.0.1.1, 16.0.1.2, 16.1.0, 16.1.1, 16.1.2, 16.1.2.1, 16.1.2.2, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 16.1.4.3

Fixed In:
17.1.1, 15.1.10

Opened: Aug 13, 2021

Severity: 3-Major

Symptoms

A considerable delay is observed when different users attempt to execute multiple iControl Rest (iCR) requests in parallel. The below restjavad error log will be observed as async context's state expired before icrd times out during delay in processing requests. This error can be observed when there is considerable delay in request processing irrespective of single user or different users. [WARNING][7777][25 Jan 2024 16:09:47 UTC][RestOperation] Exception in POST http://localhost:8100/mgmt/shared/appsvcs/declare failed. t: java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled

Impact

BIG-IP system performance is impacted.

Conditions

Multiple iControl REST operations are performed by different users in parallel. When attempting multiple requests by single or multiple users with and without bulk config, the following behaviors are observed: 5 ICRD children getting spawned successfully and same are being observed in logs and noticed that these children are serving multiple rest requests fired by multiple users Observed expected results for all below scenarios, except the last scenario which has a caveat: 1. Verify multiple rest requests fired with single user 2. Verify multiple rest requests fired with multiple users(5 users ) 3. Verify single rest request fired with multiple users (5 users) 4. Verify multiple rest requests fired from multiple users with Bulk config(5 users) 5. Verify single rest request fired from multiple users with Bulk Config(5 users) Scenario 5 has a Caveat with the current fix, since this fix limits up to 4 concurrent requests, the connection may be refused for some of the requests if the concurrent requests are more than 4.

Workaround

Use only one user to process the multiple requests. OR Send multiple requests in a single iControl Rest transaction.

Fix Information

Create icrd child per user to avoid context switching. If maxNumChild threshold is reached then allocate users in round robin fashion to all available children to process the requests.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips