RESTEasy client calls on EAP 7.4.22+ produce large DirectByteBuffer overhead in Util$BufferCache

Environment

• JBoss Enterprise Application Platform (EAP)
  • 7.4.22+
  • 8.0.0+

Issue

• We have an application making RESTEasy client calls on JBoss EAP. After updating to EAP 7.4.22+, it now sees much larger processes sizes growing from DirectByteBuffers allocated and stored on the JVM's thread local sun.nio.ch.Util$BufferCache. So it seems apparent that something through RESTEasy is now calling sun.nio.ch.Util.getTemporaryDirectBuffer in a way it hadn't before:

Class Name                                                                                                                        | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.jboss.as.ee.concurrent.ManagedThreadFactoryImpl$ManagedThread @ 0x787bb4720  EE-ManagedExecutorService-default-Thread-3 Thread|            1 |          152 |                64 |         9,048
'- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x787d1d440                                                                |            1 |           24 |                64 |         8,448
   '- table java.lang.ThreadLocal$ThreadLocalMap$Entry[64] @ 0x787d1d458                                                          |            1 |          272 |                64 |         8,424
      '- [3] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x787b73c28                                                             |            1 |           32 |                64 |         4,168
         '- value sun.nio.ch.Util$BufferCache @ 0x787b73c48                                                                       |            1 |           24 |                64 |         4,136
            '- buffers java.nio.ByteBuffer[1024] @ 0x787b73c60                                                                    |            1 |        4,112 |                64 |         4,112
               '- [1023] java.nio.DirectByteBuffer @ 0x7885d6b88                                                                  |            1 |           64 |                64 |            64
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Resolution

• To limit the persistent growth that can occur from these temporary DirectByteBuffers stored on the JDK , you can set the following limit to your JVM options so any buffer sized larger than this limit will be released when done with instead of being persistently cached on the JVM's thread local sun.nio.ch.Util$BufferCache:

-Djdk.nio.maxCachedBufferSize=8192

• Update to EAP 7.4.24, 8.0 update 12, or 8.1 update 4 and later

Root Cause

• RESTEasy uses a org.apache.commons.io.output.DeferredFileOutputStream to manage larger request writes. This DeferredFileOutputStream Content from github.com is not included.was changed in apache commons io to use java NIO files in commons io 2.11.0+. EAP 7.4.21 and earlier used commons io 2.10.0 and earlier so did not have this NIO File use and resulting DirectByteBuffer allocation. But EAP 7.4.22+ updates to commons io 2.16.1+ and EAP 8.0.0+ to commons io 2.11.0+ for this change. As a result, the java NIO File implementation now used here by commons io can result in the large sun.nio.ch.Util.getTemporaryDirectBuffer allocations on 7.4.22+/8.0.0+ that never occur in 7.4.21 or earlier.
• RESTEasy 6.2 in EAP 8.0.0+ also switches from using the DeferredFileOutputStream to Content from github.com is not included.its own custom provided outputstream but this uses java NIO Files as well for the same buffer overhead increase as with the commons io change.
• This content is not included.JBEAP-31443
• This content is not included.JBEAP-31444
• This content is not included.JBEAP-31445

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.