---
title: "Collecting and monitoring Rails logs with Datadog"
description: "See how you can monitor, analyze, and correlate your Rails application logs with Datadog."
author: "Mallory Mooney"
date: 2018-08-30
tags: ["log management", "rails", "ruby"]
blog_type_id: the-monitor
locale: en
---

In a [previous post](https://www.datadoghq.com/blog/managing-rails-application-logs.md), we walked through how you can configure logging for Rails applications, create custom logs, and use Lograge to convert the standard Rails log output into a more digestible JSON format. In this post, we will show how you can forward these application logs to Datadog and keep track of application behavior with faceted log search and analytics, custom processing pipelines, and log-based alerting.

## Set up Datadog to collect application logs

In order to collect logs from your Rails application, you will need to install the [Datadog Agent](https://docs.datadoghq.com/agent.md) on your host. The installation step requires an API key that can be found in your Datadog [account](https://app.datadoghq.com/account/settings/agent/latest?platform=overview). If you don't yet have a Datadog account, you can start a <!-- Sign-up trigger (free trial) omitted -->.

Run the following script to install the Agent:

<!-- Agent install command (platform=linux) omitted -->

Next, create a new directory and YAML file in the Agent's **conf.d** directory to store the configuration necessary to set up Rails log collection:

```text
cd /etc/datadog-agent/conf.d/
mkdir ruby.d
touch ruby.d/conf.yaml
```

In the **ruby.d/conf.yaml** file, add the following snippet, providing the path to the Rails log file you want to tail:

```text
# Log Information
logs:
  - type: file
    path: /path/to/app/log/development.log
    source: ruby
    service: rails-app
```

We recommend setting the *source* parameter to `ruby` for your Rails logs and the *service* parameter to the name of your application. The source parameter instructs Datadog to install the built-in Ruby pipeline and [integration facets](https://docs.datadoghq.com/logs/explore.md#facets) so you can easily search for and customize all of your application logs. The service parameter links your logs to correlated data from the same service in Datadog, such as request traces and application performance metrics in [Datadog APM](https://docs.datadoghq.com/tracing.md).

Restart the Agent and check on its status with the following commands:

```text
sudo service datadog-agent restart
sudo datadog-agent status
```

The status command will show the following output if the Agent is able to tail your application logs successfully:

```text
==========
Logs Agent
==========
ruby
  ----
    Type: file
    Path: /path/to/app/log/development.log
    Status: OK
    Inputs: /path/to/app/log/development.log
```

## Explore your logs 

You should see logs from your Rails application appear in the [Log Explorer](https://app.datadoghq.com/logs) as they are generated. Datadog [reserves some attributes](https://docs.datadoghq.com/logs/log_collection.md#reserved-attributes), like **date** and **status**, and turns them into *facets*. With facets, you can easily search and explore logs in the Log Explorer. You can even create new facets from other attributes in your logs. This enables you to search and filter by any attribute you care about, drilling down to the logs you need in seconds. To create a new facet, click on an attribute from an individual log and choose **Create facet**. The example below shows creating a facet for an application controller:

![Creating a new facet based on Controller](https://web-assets.dd-static.net/42588/1776356769-managing-rails-logs-with-datadog-create_a_facet.png)

To perform a search based on that new facet, simply use `@controller: controller_name` in the search bar and Datadog will retrieve all logs with the controller name you specify.

![Search the Log Explorer for the new facet](https://web-assets.dd-static.net/42588/1776356774-managing-rails-logs-with-datadog-controller_search.png)

Facets enable you to search and filter your logs using structured data, making it easier to pinpoint issues from a specific area of your application. You can also create [measures](https://docs.datadoghq.com/logs/explore.md#measures) from numerical data points in your logs. Measures are log metrics you can graph and aggregate with [Log Analytics](https://docs.datadoghq.com/logs/analytics.md).

## Log processing pipelines

When you configure the Agent to collect logs, Datadog automatically installs and enables an integration pipeline based on your specified log source. The Ruby pipeline is designed to capture and parse standard Ruby logs as well as to enhance logs in JSON format.

![Out-of-the-box Ruby pipeline](https://web-assets.dd-static.net/42588/1776356778-managing-rails-logs-with-datadog-ruby_pipeline.png)

This pipeline captures the *status* attribute in your JSON logs and remaps it to `http.status_code`. It also includes a *Log Status Remapper* that sets the status of a log (e.g., Info, Warn, Error) based on a *level* attribute. If you customized your Lograge logs to include that attribute, as detailed in the [previous post](https://www.datadoghq.com/blog/managing-rails-application-logs.md), Datadog will automatically set the appropriate level for the log, making it easier to determine the severity of events being logged in your application.

![Header for error log](https://web-assets.dd-static.net/42588/1776356782-managing-rails-logs-with-datadog-error_header_new.png)

With integration pipelines, you can begin parsing your logs with minimal setup. Check out our list of [integrations](https://docs.datadoghq.com/integrations.md#cat-log-collection) to see all available log sources. If an integration is not yet available, or you need to modify a standard pipeline, you can create new processing pipelines to customize how Datadog parses your logs.

## Enhance logs with a custom pipeline

Custom processing pipelines allow you to create your own processing rules or edit those created for existing integrations. For example, the out-of-the-box Ruby pipeline includes an [**Attribute Remapper**](https://docs.datadoghq.com/logs/processing.md#remapper) for capturing application errors and stack traces. This creates an **Error Stack** section in your processed logs that includes stack traces or exceptions generated by your application. By default, the pipeline is configured to look for an *exception* or *stack_trace* attribute. However, a Rails application may not use those attributes for capturing errors if one of its libraries creates its own attribute for logging. Lograge, for example, uses the *error* attribute by default:

```text
{
   "method":"GET",
   "path":"/articles/11",
   "format":"html",
   "controller":"ArticlesController",
   "action":"show",
   "status":500,
   "error":"NameError: uninitialized constant ArticlesController::Article",
   "duration":34.72,
   "view":0.0,
   "db":0.0,
   "params":{
      "controller":"articles",
      "action":"show",
      "id":"11"
   },
   "level":"ERROR"
}
```

You can modify the Ruby log pipeline in Datadog to include the value of that **error** attribute with the appropriate attribute remapper. In order to edit a processor, you will need to clone the Ruby pipeline, since pipelines installed by the Agent are read-only. Navigate to your [pipeline list](https://app.datadoghq.com/logs/pipelines) and click on the "Clone" button for the Ruby pipeline.

![Clone an existing pipeline](https://web-assets.dd-static.net/42588/1776356786-managing-rails-logs-with-datadog-clone_pipeline.png)

This will disable the out-of-the-box Ruby pipeline automatically. Enable your new pipeline and edit the processor that remaps exceptions and stack traces to include the *error* attribute.

![Attribute Remapper for custom pipeline](https://web-assets.dd-static.net/42588/1776356791-managing-rails-logs-with-datadog-attribute_remapper.png)

After you save the processor, Datadog will automatically apply this rule to new logs. Now when the application generates an error log, you will see the log's error message in the *Error Stack* message column.

![Error log](https://web-assets.dd-static.net/42588/1776356795-managing-rails-logs-with-datadog-error_full_new.png)

This is a simple example of modifying a pipeline processor, but pipelines give you more control over log customization by including several parsing options (e.g., Grok Parser and URL Parser) and remappers for log dates, severity, and attributes. You can learn more about modifying processing pipelines in [our documentation](https://docs.datadoghq.com/logs/processing.md#pipelines).

## Set up alerting for your logs

If you need to know when specific events occur within your application, such as when the number of error logs reach a certain threshold, you can create a new [alert](https://docs.datadoghq.com/monitors/monitor_types/log.md) based on attributes from your processed logs. For example, you can get alerted when your application generates and logs a certain number of 500 errors, or when a server's response time surpasses a certain threshold.

To create a new alert, navigate to the [monitor page](https://app.datadoghq.com/monitors#create/log). The first section includes a search bar where you can query the type of log you want to monitor. For error logs generated by your Rails application, you can search by your host (e.g.,`host:railsbox`) and `status:error`. You will see the search results update as you build your query so you can confirm you are selecting the logs you want to monitor.

![Search query for alerts](https://web-assets.dd-static.net/42588/1776356800-managing-rails-logs-with-datadog-search_query.png)

In the following two sections, you can set conditions for triggering the alert and create a message for notifying team members. The condition section includes fields for setting both alert and warning thresholds, so you can create appropriate notifications depending on the severity of the issue.

![Setting up conditions for alert](https://web-assets.dd-static.net/42588/1776356804-managing-rails-logs-with-datadog-alert_conditions.png)

The notification you create should include any relevant information needed to resolve the issue. This could be troubleshooting steps or specific commands to run on a server. Read [our docs](https://docs.datadoghq.com/monitors/notifications.md) for more information on customizing your alert notifications. The last section of the creation process lets you specify any individual, team, or communication channel that should receive alert notifications.

## Get started monitoring your Rails application

You can collect, manage, and monitor your Rails application logs in one place with [Datadog's log management](https://www.datadoghq.com/product/log-management/) tools and built-in Ruby integration. Datadog also supports 1,000+ other integrations so you can correlate data from your Rails apps with logs and metrics from every part of your infrastructure. Moreover, for a more complete picture of your Rails environment, you can monitor infrastructure metrics from your hosts, databases, and web servers, along with the logs and request traces from the Rails application itself. Read [our guide to Rails monitoring](https://www.datadoghq.com/blog/monitoring-rails-with-datadog.md) with Datadog to learn more.

To start collecting, monitoring, and analyzing your Rails application logs, sign up for a <!-- Sign-up trigger (free Datadog trial) omitted -->. If you're already using Datadog, see our [documentation](https://docs.datadoghq.com/logs.md) for more information about Datadog log management.