Symptoms

The REST tokens are not deleted from cache /var/run/pamcache when the tokens are expired or deleted.

Impact

The deleted token continue to be available in the cache. Memory is consumed as cache is stored in an in-memory filesystem.

Conditions

- A large number of REST Auth tokens are created in multi-slot VIPRION, multi-slot vCMP Guest, or multi-slot VELOS tenant.

Workaround

First take immediate action to recover memory by removing stale tokens and restarting affected processes. This should be done to free memory, even if planning to update software to prevent reoccurrence. Remove token files from /run/pamcache manually. This may have minor impact to REST API use causing a REST user to need to reauthenticate. Execute the following command by using -print instead of -delete to verify the tokens to be deleted (recommended to not use clsh): # clsh "find /run/pamcache -regextype posix-extended -type f -regex '/run/pamcache/[A-Z0-9]{26}' -delete" httpd processes can be affected - restart them. This has an impact to REST API and GUI for the few seconds until httpd restarts: # clsh bigstart restart httpd Restart csyncd - this is expected to have no adverse impact. # clsh bigstart restart csyncd Alternatively clear any stale content and restart processes simply by rebooting the chassis (ie all blades together). Next, it is possible to prevent the issue reoccurring by the following steps, if not quickly updating software to a fixed version. Execute the following commands in bash to remove the pamcache directory from the set being acted upon by "csyncd": # clsh "cp /etc/csyncd.conf /etc/csyncd.conf.$(date +%Y%m%d_%H%M%S)" # clsh "sed -i '/run\/pamcache/,+2s/^/#/' /etc/csyncd.conf" # clsh "bigstart restart csyncd"

Fix Information

Auth tokens in /run/pamcache are deleted as required.

Guides & references

K10134038: F5 Bug Tracker Filter Names and Tips