High CPU usage in SessionExpirationScheduler thread and/or large amount of ERROR and WARN logs after restarting EAP 8.1.x

Environment

• Red Hat JBoss Enterprise Application Platform 8.1.x

Issue

• After restarting EAP 8.1.x instances that are part of an Infinispan cluster, CPU usage for those instances spikes and logs similar to the following are being continuously output:

    ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-9,...) ISPN000136: Error executing command LockControlCommand on Cache '<deployment>', writing keys []: org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key SessionMetaDataKey(<session id>) and requestor GlobalTransaction{...}. Lock is held by GlobalTransaction{...}
    WARN  [org.infinispan.CLUSTER] (thread-9,...) ISPN000071: Caught exception when handling command LockControlCommand{cache=<deployment>, keys=[SessionMetaDataKey(<session id>)], flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction{...}}: org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key SessionMetaDataKey(<session id>) and requestor GlobalTransaction{...}. Lock is held by GlobalTransaction{...}
    ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-9,...) ISPN000136: Error executing command LockControlCommand on Cache '<deployment>', writing keys []: org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key SessionMetaDataKey(<session id>) and requestor GlobalTransaction{...}. Lock is held by null
    WARN  [org.infinispan.CLUSTER] (thread-9,...) ISPN000071: Caught exception when handling command LockControlCommand{cache=<deployment>, keys=[SessionMetaDataKey(<session id>)], flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_WRITE_LOCK], unlock=false, gtx=GlobalTransaction{...}}: org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key SessionMetaDataKey(<session id>) and requestor GlobalTransaction{...}. Lock is held by null
  

• After restarting EAP 8.1.x instances that are part of an Infinispan cluster, the SessionExpirationScheduler is showing high CPU usage.

Resolution

Currently, there is no known workaround. Please upgrade to EAP 8.1.8 when it becomes available.

Root Cause

Restarting EAP 8.1.x instances that are part of an Infinispan cluster can leave some sessions in the cache past their expiration and this can lead to OutOfMemoryException, high CPU usage in the SessionExpirationScheduler thread, as well as large amount of Infinispan LockControlCommand ERROR and WARN log messages.

The main cause of these issues is being tracked in Content from redhat.atlassian.net is not included.[JBEAP-34005] (8.1.z) WFLY-22009 - Cache listeners not re-registered on resume, which is expected to be incorported in EAP 8.1.8.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.