Inspeqtor Process Monitoring With Datadog | Datadog

Inspeqtor Process Monitoring with Datadog

Author Mike Perham

Published: 12月 18, 2014

Mike Perham @mperham is the CEO of Contributed Systems and runs his own blog, mikeperham.com.

A year ago, I had a serious problem. I was the  Director of Engineering  at an eCommerce company running a Ruby on Rails website. Our Rails app was quite stable except for one part: our background worker processes kept increasing in memory over the course of a day from 300MB up to many gigabytes. Eventually this would cause our machines to run out of memory and we’d have to restart the service manually to regain that memory.

Of course we were monitoring everything with Datadog. We could see the memory increasing and we could receive an email alert about the problem, but we couldn’t automate the restart. Inspeqtor to the rescue!

Inspeqtor process monitoring

Inspeqtor is Linux-based software which monitors your critical application infrastructure on the local machine: processes like MySQL or PostgreSQL, Memcached, Redis, Java VM processes, custom daemons, etc. You define simple rules, and Inspeqtor will verify proper behavior and take action if a rule is broken.

To get started with Inspeqtor, first install it on each machine:

    
# For Ubuntu 12.04 and 14.04 LTS
 curl -L https://bit.ly/InspeqtorDEB | sudo bash
 sudo apt-get install inspeqtor
  

and then write rules about your services running on that machine:

    
# /etc/inspeqtor/services.d/background_worker.inq
 check service background_worker
     if memory:rss > 1g then restart, alert
     if cpu:user > 95% for 4 cycles then alert
  

Here, you can see I’m using Inspeqtor to automate the restart of our background worker service. Inspeqtor will scan the worker process every 15 seconds. If it is using more than one GB of memory, Inspeqtor will immediately restart it. I’ve also added a rule to alert me if the process uses an entire CPU for 4 cycles (or 60 seconds).

It seems too simple to work, right? The trick is that Inspeqtor queries any installed init systems – upstart, systemd, runit or init.d – to find a service with that name and any associated processes so it can collect metrics.

Inspeqtor Process Monitoring

Inspeqtor is open source and free; I also offer Inspeqtor Pro with more features, including Statsd integration. With just a single line of code, you can send all metrics collected by Inspeqtor Pro to the Datadog Agent for visualization:

    
# /etc/inspeqtor/inspeqtor.conf
set statsd_location localhost:8125  
  

Within seconds your data should appear in the Datadog system. Here you can see Inspeqtor Pro collecting the total memory used by the Apache 2.x service on my website:

Inspeqtor process monitoring

Using Inspeqtor Pro and Datadog together works great in my experience. Datadog provides a rich, flexible UI for seeing historical data and finding problems while Inspeqtor allows you to take action to solve those problems with rules that are incredibly easy to write. Take a look through Inspeqtor’s documentation or the Getting Started screencast if you want to learn more.