Easy JMX discovery & browsing with the open source Agent

What is JMX?

Java Management Extensions (JMX) is a mechanism for managing and monitoring Java applications, system objects, and devices. Most users are familiar with the JMX metrics exposed by applications running in the Java Virtual Machine (JVM) such as Cassandra, Kafka, or ZooKeeper.

JMX represents resources as MBean (Managed Bean) objects. They provide a window through which users can read and interact with the running application.

Collecting JMX metrics: the old way

JConsole, bundled as part of the Java Development Kit (JDK), is the canonical graphical monitoring tool for applications running in the JVM. It provides local or remote access to an application’s MBeans.

The problem

There are two problems with using JConsole to explore JMX metrics:

• it requires X11
• it’s resource-intensive

In production environments, and especially in cloud-based environments, most machines operate as headless servers and probably won’t have X11 installed.

JConsole is also resource-intensive—the JConsole documentation recommends connecting remotely to an MBean server in production environments, since a local connection would reduce performance on the server. (In my own tests, I found JConsole to use on average 320 MiB RAM.)

You could use JMX remotely, but that requires configuring additional security policy (it is recommended to configure SSL authentication for production environments).

Enter the Agent

If you’ve installed the Datadog Agent, you already have two powerful tools for exploring JMX metrics. Both tools:

• don’t use X11
• are lightweight
• require no additional configuration

All versions of the Datadog Agent from 3.5 up to (and including) version 5 come bundled with Jmxterm (in /opt/datadog-agent/agent/checks/libs/ on *NIX) and JMXFetch.

Due to changes made in version 6 of the Agent, you will need to download Jmxterm here instead.

Jmxterm: command line JMX console

Jmxterm is a lightweight, full-featured JMX console, with no external dependencies.

To connect Jmxterm to monitor your Java application, on the local machine run java -jar /opt/datadog-agent/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar --url localhost:<JMX PORT> if you’re using the bundled Jmxterm (version 5.x and lower of the Datadog Agent), or java -jar /path/to/jmxterm-1.0.0-uber.jar --url localhost:<JMX PORT> if you downloaded Jmxterm yourself (necessary for Agent version 6).

Using Jmxterm

Jmxterm has a short list of commands:

about    - Display about page
bean     - Display or set current selected MBean
beans    - List available beans under a domain or all domains
bye      - Terminate console and exit
close    - Close current JMX connection
domain   - Display or set current selected domain.
domains  - List all available domain names
dump     - Display a JSON Formatted dictionary of all the attributes and their values of all MBeans of the specified domain or of all domains if domain is not specified.
exit     - Terminate console and exit
get      - Get value of MBean attribute(s)
help     - Display available commands or usage of a command
info     - Display detail information about an MBean
jvms     - List all running local JVM processes
open     - Open JMX session or display current connection
option   - Set options for command session
quit     - Terminate console and exit
run      - Invoke an MBean operation

Use domains to list all of the MBean domains (similar to what you’d see after initially connecting with JConsole):

Welcome to JMX terminal. Type "help" for available commands.
$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
kafka
kafka.cluster
kafka.common
kafka.consumer
kafka.controller
kafka.log
kafka.network
kafka.server

From there, you can drill down into each domain by setting the domain (domain kafka.server, for example) and listing the beans with the beans command.

$>domain kafka.server
#domain is set to kafka.server
$>beans
#domain = kafka.server:
kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics
kafka.server:name=NumOffsets,type=OffsetManager
kafka.server:name=PurgatorySize,type=ProducerRequestPurgatory
kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool
kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager
...

Once you’ve found a bean you’d like to examine more closely, select it with bean <bean name>.

Beyond Jmxterm with JMXFetch

Also included with the Agent is a JMX check, powered by JMXFetch. This check allows you to see, at a glance:

• all JMX metrics exposed on the system, in a non-hierarchical view

• all JMX metrics currently being collected by the Agent

• all JMX metrics that are not being collected by the Agent

  $ sudo /etc/init.d/datadog-agent jmx

This tool is great for verifying that your JMX checks are properly configured, especially when using more advanced check features, like matching MBeans with regular expressions (see bean_regex here).

The full list of commands appears below:

• list_matching_attributes: List attributes that match at least one of your instances’ configurations
• list_limited_attributes: List attributes that match one of your instances’ configuration but that are not being collected because it would exceed the number of metrics that can be collected
• collect: Start the collection of metrics based on your current configuration and display them in the console
• list_collected_attributes: List attributes that will actually be collected by your current instances’ configurations
• list_not_matching_attributes: List attributes that don’t match any of your instances’ configurations
• list_everything: List every attributes available that has a type supported by JMXFetch

Example:

sudo /etc/init.d/datadog-agent jmx list_matching_attributes tomcat jmx solr

If you have a JMX URI at hand, you can also query the MBeans:

$ java -jar jmxterm-1.0-alpha-4-uber.jar -l <JMX URI> -v silent -n < jmxcommands

# list all of the beans
$> domains
$> beans
or
$> beans -d com.cognos
...
com.cognos:group=System,service=JobQueue,type=MetricHealth
com.cognos:group=System,service=JobQueue,type=Metrics
...

########################
# List all of the attributes; -i * is the key here
########################

$>get -b com.cognos:group=System,service=JobQueue,type=Metrics -i *
#mbean = com.cognos:group=System,service=JobQueue,type=Metrics:
NumberOfGlobalRequests = 0; (Attribute exposed for management)
Health = NotApplicable; (Health)
GlobalLastResetTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
TimeInGlobalQueueLowWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
TimeInGlobalQueue = 0; (Attribute exposed for management)
TimeInGlobalQueueHighWaterMark = 0; (Attribute exposed for management)
GlobalQueueLengthLowWaterMark = 0; (Attribute exposed for management)
TimeInGlobalQueueHighWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
GlobalQueueLength = 0; (Attribute exposed for management)
TimeInGlobalQueueLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
GlobalQueueLengthLowWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
GlobalQueueLengthHighWaterMark = 0; (Attribute exposed for management)
NumberOfGlobalRequestsLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)
TimeInGlobalQueueLowWaterMark = 0; (Attribute exposed for management)
GlobalQueueLengthHighWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management)

# get current values of select
$>get -b com.cognos:group=System,service=JobQueue,type=Metrics GlobalQueueLength GlobalQueueLengthHighWaterMark

#mbean = com.cognos:group=System,service=JobQueue,type=Metrics:
GlobalQueueLength = 0;
GlobalQueueLengthHighWaterMark = 0;

# Live Watch - ENTER to exit live watch
$>domain com.cognos

#domain is set to com.cognos
$>bean group=System,service=JobQueue,type=Metrics

#bean is set to com.cognos:group=System,service=JobQueue,type=Metrics
$>watch GlobalQueueLength

Conclusion

If you’re already using the Agent, the Jmxterm and JMXFetch applications are two more tools already in your monitoring arsenal.

Of course, simply spot-checking metrics can only reveal so much. To implement ongoing, meaningful monitoring requires the ability to store metrics over time to spot trends, as well as the ability to put metrics in context with system changes and other events. With Datadog, you can alert on and track metrics and events, and collaboratively diagnose issues all in one place.

If you’re a Datadog customer, you can start monitoring the metrics collected by these tools with minimal setup.

If you don’t yet have a Datadog account, you can sign up for a free trial and start monitoring your Java applications today.