Java Management Extensions (JMX) is a mechanism for managing and monitoring Java applications, system objects, and devices. Most users are familiar with the JMX metrics exposed by applications running in the Java Virtual Machine (JVM) such as Cassandra, Kafka, or ZooKeeper.
JMX represents resources as MBean (Managed Bean) objects. They provide a window through which users can read and interact with the running application.
JConsole, bundled as part of the Java Development Kit (JDK), is the canonical graphical monitoring tool for applications running in the JVM. It provides local or remote access to an application’s MBeans.
There are two problems with using JConsole to explore JMX metrics:
- it requires X11
- it’s resource-intensive
In production environments, and especially in cloud-based environments, most machines operate as headless servers and probably won’t have X11 installed.
JConsole is also resource-intensive—the JConsole documentation recommends connecting remotely to an MBean server in production environments, since a local connection would reduce performance on the server. (In my own tests, I found JConsole to use on average 320 MiB RAM.)
If you’ve installed the Datadog Agent, you already have two powerful tools for exploring JMX metrics. Both tools:
- don’t use X11
- are lightweight
- require no additional configuration
Jmxterm is a lightweight, full-featured JMX console, with no external dependencies.
To connect Jmxterm to monitor your Java application, on the local machine run
java -jar /opt/datadog-agent/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar --url localhost:<JMX PORT> if you’re using the bundled Jmxterm (version 5.x and lower of the Datadog Agent), or
java -jar /path/to/jmxterm-1.0.0-uber.jar --url localhost:<JMX PORT> if you downloaded Jmxterm yourself (necessary for Agent version 6).
Jmxterm has a short list of commands:
about - Display about page bean - Display or set current selected MBean beans - List available beans under a domain or all domains bye - Terminate console and exit close - Close current JMX connection domain - Display or set current selected domain. domains - List all available domain names dump - Display a JSON Formatted dictionary of all the attributes and their values of all MBeans of the specified domain or of all domains if domain is not specified. exit - Terminate console and exit get - Get value of MBean attribute(s) help - Display available commands or usage of a command info - Display detail information about an MBean jvms - List all running local JVM processes open - Open JMX session or display current connection option - Set options for command session quit - Terminate console and exit run - Invoke an MBean operation
domains to list all of the MBean domains (similar to what you’d see after initially connecting with JConsole):
Welcome to JMX terminal. Type "help" for available commands. $>domains #following domains are available JMImplementation com.sun.management java.lang java.nio java.util.logging kafka kafka.cluster kafka.common kafka.consumer kafka.controller kafka.log kafka.network kafka.server
From there, you can drill down into each domain by setting the domain (
domain kafka.server, for example) and listing the beans with the
$>domain kafka.server #domain is set to kafka.server $>beans #domain = kafka.server: kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics kafka.server:name=NumOffsets,type=OffsetManager kafka.server:name=PurgatorySize,type=ProducerRequestPurgatory kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager ...
Once you’ve found a bean you’d like to examine more closely, select it with
bean <bean name>.
Also included with the Agent is a JMX check, powered by JMXFetch. This check allows you to see, at a glance:
all JMX metrics exposed on the system, in a non-hierarchical view
all JMX metrics currently being collected by the Agent
all JMX metrics that are not being collected by the Agent
$ sudo /etc/init.d/datadog-agent jmx
The full list of commands appears below:
list_matching_attributes: List attributes that match at least one of your instances’ configurations
list_limited_attributes: List attributes that match one of your instances’ configuration but that are not being collected because it would exceed the number of metrics that can be collected
collect: Start the collection of metrics based on your current configuration and display them in the console
list_collected_attributes: List attributes that will actually be collected by your current instances’ configurations
list_not_matching_attributes: List attributes that don’t match any of your instances’ configurations
list_everything: List every attributes available that has a type supported by JMXFetch
sudo /etc/init.d/datadog-agent jmx list_matching_attributes tomcat jmx solr
If you have a JMX URI at hand, you can also query the MBeans:
$ java -jar jmxterm-1.0-alpha-4-uber.jar -l <JMX URI> -v silent -n < jmxcommands # list all of the beans $> domains $> beans or $> beans -d com.cognos ... com.cognos:group=System,service=JobQueue,type=MetricHealth com.cognos:group=System,service=JobQueue,type=Metrics ... ######################## # List all of the attributes; -i * is the key here ######################## $>get -b com.cognos:group=System,service=JobQueue,type=Metrics -i * #mbean = com.cognos:group=System,service=JobQueue,type=Metrics: NumberOfGlobalRequests = 0; (Attribute exposed for management) Health = NotApplicable; (Health) GlobalLastResetTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) TimeInGlobalQueueLowWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) TimeInGlobalQueue = 0; (Attribute exposed for management) TimeInGlobalQueueHighWaterMark = 0; (Attribute exposed for management) GlobalQueueLengthLowWaterMark = 0; (Attribute exposed for management) TimeInGlobalQueueHighWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) GlobalQueueLength = 0; (Attribute exposed for management) TimeInGlobalQueueLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) GlobalQueueLengthLowWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) GlobalQueueLengthHighWaterMark = 0; (Attribute exposed for management) NumberOfGlobalRequestsLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) TimeInGlobalQueueLowWaterMark = 0; (Attribute exposed for management) GlobalQueueLengthHighWaterMarkLastUpdateTime = Thu Sep 28 14:04:15 EDT 2023; (Attribute exposed for management) # get current values of select $>get -b com.cognos:group=System,service=JobQueue,type=Metrics GlobalQueueLength GlobalQueueLengthHighWaterMark #mbean = com.cognos:group=System,service=JobQueue,type=Metrics: GlobalQueueLength = 0; GlobalQueueLengthHighWaterMark = 0; # Live Watch - ENTER to exit live watch $>domain com.cognos #domain is set to com.cognos $>bean group=System,service=JobQueue,type=Metrics #bean is set to com.cognos:group=System,service=JobQueue,type=Metrics $>watch GlobalQueueLength
If you’re already using the Agent, the Jmxterm and JMXFetch applications are two more tools already in your monitoring arsenal.
Of course, simply spot-checking metrics can only reveal so much. To implement ongoing, meaningful monitoring requires the ability to store metrics over time to spot trends, as well as the ability to put metrics in context with system changes and other events. With Datadog, you can alert on and track metrics and events, and collaboratively diagnose issues all in one place.
If you’re a Datadog customer, you can start monitoring the metrics collected by these tools with minimal setup.
If you don’t yet have a Datadog account, you can sign up for a free trial and start monitoring your Java applications today.