How to monitor busy task threads, queue size, and connection count for XNIO/Undertow in JBoss EAP

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.x
    • 8.x

Issue

  • How to monitor the number of active busy threads in XNIO/Undertow task thread pool in JBoss EAP 7?
  • How to monitor the number of active busy threads of XNIO/Undertow web threads in JBoss EAP 7?
  • How to monitor the number of active connection count handled by XNIO/Undertow in JBoss EAP 7?
  • Need to know the number of current request count in ajp/http listener in JBoss EAP 7. Is there a way to know the actual ajp (or http) busy threads?

Resolution

  • Worker runtime statistics are available in EAP 7.0.9 and 7.1+. The runtime statistics can be obtained via JMX MBeans org.xnio:type=Xnio,provider="nio",worker="your-workername" ("your-workername" defaults to default which is defined in standalone-*.xml).

  • Some useful metrics in org.xnio MBean:

    • BusyWorkerThreadCount: This shows the number of busy threads in the task worker thread pool. This is available since EAP 7.1.0 and EAP 7.0.9 or later.
    • WorkerQueueSize: This shows how many counts await for free task worker thread in workQueue. This can be used for monitoring to know if the worker thread pool is full or not. When this is greater than 0, it means the task worker thread pool is full. For example, if it's 1, all task worker threads are busy, and 1 request is awaiting for available task thread.
    • ConnectionCount: This shows the number of active established TCP connections. This has been available since EAP 7.0.5 or later.
      Note: In EAP 7.0.5 and 7.0.6, the ConnectionCount attribute of the org.xnio MBean reports incorrect values. This is tracked in This content is not included.XNIO-297 and This content is not included.JBEAP-11369, and is fixed in EAP 7.0.7 and EAP 7.1.0 or later.

  • Since JBoss EAP 7.2.0+, the same metrics that can be checked via Management CLI (jboss-cli.sh):

    • busy-task-thread-count: The same metrics as BusyWorkerThreadCount in org.xnio MBean

    • queue-size: the same metrics as WorkerQueueSize in org.xnio MBean

    • connection-count: the same metrics as ConnectionCount in org.xnio MBean

    • For example:

      • Standalone mode:
              [standalone@localhost:9990 /] /subsystem=io/worker=default:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "busy-task-thread-count" => 0,
                "core-pool-size" => 2,
                "io-thread-count" => 24,
                "io-threads" => "24",
                "max-pool-size" => 192,
                "queue-size" => 0,
                "shutdown-requested" => false,
                "stack-size" => "0",
                "task-core-threads" => "2",
                "task-keepalive" => "60000",
                "task-max-threads" => "192",
                "outbound-bind-address" => undefined,
                "server" => {
                    "/127.0.0.1:8080" => undefined,
                    "/127.0.0.1:8443" => undefined
                }
            }
        }

        [standalone@localhost:9990 /] /subsystem=io/worker=default/server=\/127.0.0.1\:8080:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "connection-count" => 0,
                "connection-limit-high-water-mark" => 2147483647,
                "connection-limit-low-water-mark" => 2147483647
            }
        }
      • Domain mode:
        For Domain mode worker threads need to be monitored for each server instance separately. The below CLI command is executed for server-one in master host:
              [domain@localhost:9990 /] /host=master/server=server-one/subsystem=io/worker=default:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "busy-task-thread-count" => 0,
                "core-pool-size" => 2,
                "io-thread-count" => 16,
                "io-threads" => "16",
                "max-pool-size" => 128,
                "queue-size" => 0,
                "shutdown-requested" => false,
                "stack-size" => "0",
                "task-core-threads" => "2",
                "task-keepalive" => "60000",
                "task-max-threads" => "128",
                "outbound-bind-address" => undefined,
                "server" => {"/127.0.0.1:8080" => undefined}
            }
        }

        [domain@localhost:9990 /] /host=master/server=server-one/subsystem=io/worker=default/server=\/127.0.0.1\:8080:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "connection-count" => 0,
                "connection-limit-high-water-mark" => 2147483647,
                "connection-limit-low-water-mark" => 2147483647
            }
        }

  • Since JBoss EAP 7.3.0+, the same metrics can be checked through /metrics for standalone mode. (Note: this method is not available for server running in domain mode) The metrics name starts with jboss_io_ like the following:

    $ curl -s http://127.0.0.1:9990/metrics | grep -e "^jboss_io_"
jboss_io_busy_task_thread_count{worker="default"} 0.0
jboss_io_connection_count{worker="default",server="/127.0.0.1:8080"} 0.0
jboss_io_connection_count{worker="default",server="/127.0.0.1:8443"} 0.0
jboss_io_core_pool_size{worker="default"} 2.0
jboss_io_io_thread_count{worker="default"} 48.0
jboss_io_max_pool_size{worker="default"} 384.0
jboss_io_queue_size{worker="default"} 0.0

Caveats: Note that the number of established TCP connections (ConnectionCount) and busy worker threads (BusyWorkerThreadCount) are not directly linked as Undertow/XNIO uses non-blocking I/O.

Root Cause

JBoss Enterprise Application Platform (EAP) 7.0.x releases prior to 7.0.5 do not expose runtime statistics for the XNIO/Undertow worker task thread pool, so the number of busy worker threads and the size of the worker queue cannot be monitored out of the box.

The statistics are exposed incrementally in later releases:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.