Datadog APM is here

Easy JMX discovery & browsing with the open source Agent

/ / / /
Published: June 15, 2016

What is JMX?

Java Management Extensions (JMX) is a mechanism for managing and monitoring Java applications, system objects, and devices. Most users are familiar with the JMX metrics exposed by applications running in the Java Virtual Machine (JVM) such as Cassandra, Kafka, or ZooKeeper.

JMX represents resources as MBean (Managed Bean) objects. They provide a window through which users can read and interact with the running application.

Collecting JMX metrics: the old way

JConsole, bundled as part of the Java Development Kit (JDK), is the canonical graphical monitoring tool for applications running in the JVM. It provides local or remote access to an application’s MBeans.

Example JConsole screen image

The problem

There are two problems with using JConsole to explore JMX metrics:

  • it requires X11
  • it’s resource-intensive

In production environments, and especially in cloud-based environments, most machines operate as headless servers and probably won’t have X11 installed.

JConsole is also resource-intensive—the JConsole documentation recommends connecting remotely to an MBean server in production environments, since a local connection would reduce performance on the server. (In my own tests, I found JConsole to use on average 320 MiB RAM.)

You could use JMX remotely, but that requires configuring additional security policy (it is recommended to configure SSL authentication for production environments).

Enter the Agent

If you’ve installed the Datadog Agent, you already have two powerful tools for exploring JMX metrics. Both tools:

  • don’t use X11
  • are lightweight
  • require no additional configuration

Since version 3.5, the Datadog Agent has come bundled with Jmxterm (in /opt/datadog-agent/agent/checks/libs/ on *NIX, see here for other platforms) and JMXFetch.

Jmxterm: command line JMX console

Jmxterm is a lightweight, full featured JMX console, with no external dependencies.

To connect Jmxterm to monitor your Java application, on the local machine run java -jar /opt/datadog-agent/agent/checks/libs/jmxterm-1.0-DATADOG-uber.jar --url localhost:<JMX PORT>.

Of course, you can always just download Jmxterm, but if you’re already running the Agent, you don’t need to!

Using Jmxterm

Jmxterm has a short list of commands:

about    - Display about page
bean     - Display or set current selected MBean 
beans    - List available beans under a domain or all domains
bye      - Terminate console and exit
close    - Close current JMX connection
domain   - Display or set current selected domain.   
domains  - List all available domain names
dump     - Display a JSON Formatted dictionary of all the attributes and their values of all MBeans of the specified domain or of all domains if domain is not specified.
exit     - Terminate console and exit
get      - Get value of MBean attribute(s)
help     - Display available commands or usage of a command
info     - Display detail information about an MBean
jvms     - List all running local JVM processes
open     - Open JMX session or display current connection
option   - Set options for command session
quit     - Terminate console and exit
run      - Invoke an MBean operation

Use domains to list all of the MBean domains (similar to what you’d see after initially connecting with JConsole):

Welcome to JMX terminal. Type "help" for available commands.
$>domains 
#following domains are available 
JMImplementation 
com.sun.management
java.lang
java.nio  
java.util.logging   
kafka     
kafka.cluster    
kafka.common  
kafka.consumer   
kafka.controller 
kafka.log    
kafka.network 
kafka.server

From there, you can drill down into each domain by setting the domain (domain kafka.server, for example) and listing the beans with the beans command.

$>domain kafka.server     
#domain is set to kafka.server
$>beans
#domain = kafka.server:
kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics
kafka.server:name=NumOffsets,type=OffsetManager
kafka.server:name=PurgatorySize,type=ProducerRequestPurgatory
kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool
kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager
...

Once you’ve found a bean you’d like to examine more closely, select it with bean <bean name>.

Beyond Jmxterm with JMXFetch

Also included with the Agent is a JMX check, powered by JMXFetch. This check allows you to see, at a glance:

  • all JMX metrics exposed on the system, in a non-hierarchical view
  • all JMX metrics currently being collected by the Agent
  • all JMX metrics that are not being collected by the Agent

    $ sudo /etc/init.d/datadog-agent jmx

This tool is great for verifying that your JMX checks are properly configured, especially when using more advanced check features, like matching MBeans with regular expressions (see bean_regex here).

The full list of commands appears below:

  • list_matching_attributes: List attributes that match at least one of your instances’ configurations
  • list_limited_attributes: List attributes that match one of your instances’ configuration but that are not being collected because it would exceed the number of metrics that can be collected
  • collect: Start the collection of metrics based on your current configuration and display them in the console
  • list_collected_attributes: List attributes that will actually be collected by your current instances’ configurations
  • list_not_matching_attributes: List attributes that don’t match any of your instances’ configurations
  • list_everything: List every attributes available that has a type supported by JMXFetch

Example:

sudo /etc/init.d/datadog-agent jmx list_matching_attributes tomcat jmx solr

Conclusion

If you’re already using the Agent, the Jmxterm and JMXFetch applications are two more tools already in your monitoring arsenal.

Of course, simply spot-checking metrics can only reveal so much. To implement ongoing, meaningful monitoring requires the ability to store metrics over time to spot trends, as well as the ability to put metrics in context with system changes and other events. With Datadog, you can alert on and track metrics and events, and collaboratively diagnose issues all in one place.

If you’re a Datadog customer, you can start monitoring the metrics collected by these tools with minimal setup.

If you don’t yet have a Datadog account, you can for a free trial and start monitoring your Java applications today.


Want to write articles like this one? Our team is hiring!
Easy JMX discovery & browsing with the open source Agent