Bug ID 1473701: Oauth Discovery task is struck at "SAVE_AND_APPLY" state

Last Modified: Jul 24, 2024

Affected Product(s):
BIG-IP APM(all modules)

Known Affected Versions:
15.1.10.3, 15.1.10.4, 16.1.3, 16.1.3.1, 16.1.3.2, 16.1.3.3, 16.1.3.4, 16.1.3.5, 16.1.4, 16.1.4.1, 16.1.4.2, 16.1.4.3, 17.0.0, 17.0.0.1, 17.0.0.2, 17.1.0, 17.1.0.1, 17.1.0.2, 17.1.0.3, 17.1.1, 17.1.1.1, 17.1.1.2, 17.1.1.3

Fixed In:
16.1.5

Opened: Jan 04, 2024

Severity: 3-Major

Symptoms

Initial symptoms could be one of the following: - Auto JWT discovery task stops or stalls and no reason is provided - OIDC discovery task stops discovering - Auto update of JWK fails - OAuth token does not renew - Oauth Discovery stuck at "SAVE_AND_APPLY" - OAuth Provider Discovery Task does not work anymore Other indications: -> Stale JWK keys will be present in the config and Authentication fails with the following error in /var/log/apm:"OAuth Scope: failed for jwt-provider-list '/Common/VPN_JWT', error: None of the configured JWK keys match the received JWT token, JWT Header: " ->restcurl -X GET tm/access/oidc/discover/ outputs the OIDC discovery task status and status will be in "SAVEANDAPPLY"

Impact

- Config will contain stale JWK keys

Conditions

- jwk keys discovered from the openid well known url should be different from the existing JWK keys in the config - And mcp should fail while applying the config. We can identify that if the /var/log/restjavad does not show the " Applying access policies" log after the "Updating mcp jwt and jwk objects for provide" log

Workaround

- Restart restjavad so that the discovery task starts again

Fix Information

- Moved the apply access policy operation into a child thread so that the parent thread does not block itself until it receives a response from the mcp. - Earlier the OIDC thread would be blocked until it got a successful response from the mcp for "apply access policy" and if it did receive a response, it would be blocked and would stop permanently without rescheduling itself. - Now, even if the apply access policy fails in the current discovery cycle, the OIDC discovery worker will not be blocked and will be rescheduled for the next interval and the apply access policy will be reattempted as part of the next discovery cycle.

Behavior Change

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips