Export RUM and Product Analytics data with Export Pipelines

Datadog Real User Monitoring (RUM) and Product Analytics help engineering and data teams understand application availability and performance, as well as user behavior from user session data. Teams can ingest the full scope of this data into Datadog for monitoring and analysis. However, they might also need the event data in infrastructure they control to support custom analytics and reporting, machine learning (ML) workflows, or long-term data access. 

Export Pipelines provide a native way to continuously send RUM and Product Analytics data from Datadog to your cloud storage. You can then load the files into data warehouses like Snowflake, BigQuery, or Databricks, or process the output in your custom pipelines.

In this post, we’ll show how you can:

• Continuously send user session data from Datadog to your cloud storage

• Use exported data for analytics, ML models, and compliance-driven retention

Continuously send user session data from Datadog to your cloud storage

Export Pipelines move RUM and Product Analytics data into your own cloud infrastructure. You configure the pipeline from the Datadog UI, select the performance or behavioral data that you want to export, and choose a destination in Amazon S3, Azure Blob Storage, or Google Cloud Storage. After you configure the pipeline, Datadog continuously sends matching events to that destination without requiring your team to maintain custom export scripts or scheduled jobs.

Export Pipelines support Parquet and JSON file formats so that you can choose a format based on how downstream systems will process the data. Parquet loads directly into Snowflake, BigQuery, and Databricks without preprocessing. 

JSON suits teams that have their own processing pipelines or need a more flexible representation of each event. 

Datadog organizes files by date and hour in the storage location, so downstream processes can target files from the relevant time period instead of scanning every exported file. You can enable, disable, or update an export configuration as your requirements change. The Export Pipelines page displays each pipeline’s current state—Active, Error, Pending, or Paused—so you can see whether your pipelines are running as expected.

Use exported data for analytics, ML models, and compliance-driven retention

Sending RUM and Product Analytics events to your cloud storage makes that same technical and behavioral dataset available to workflows beyond Datadog. Teams can load the files into a data warehouse for analysis, feed them into ML pipelines, or retain them long-term to help meet compliance requirements.

For example, an analytics team might want to understand how application behavior relates to business outcomes that are already modeled in its warehouse. The team can export Product Analytics events to a Google Cloud Storage bucket and load the Parquet files into BigQuery. There, analysts can join application actions and user behavior with datasets such as subscriptions, account attributes, or financial data without introducing a second set of application instrumentation.

Data science teams can also send RUM and Product Analytics events to cloud storage for ML workflows. They can then use their own data transformation and modeling pipelines to train models for personalization, churn prediction, recommendations, or anomaly detection.

Export Pipelines can also help organizations meet compliance requirements that call for long-term access to event data in infrastructure they control. Datadog RUM provides standard 30-day retention and offers 6-month storage with RUM Managed Archive, and Product Analytics provides 15-month retention. Organizations that need access to event data beyond these periods can export the relevant events to cloud storage and apply their own storage, access, and life-cycle policies.

Get started with Export Pipelines

Export Pipelines give you a direct path to move RUM and Product Analytics data from Datadog into infrastructure you control without writing or maintaining custom export code. Whether you’re feeding a data warehouse, training ML models, or addressing long-term compliance retention, your data stays available in the format and destination that works best for your workflows. To learn more, read the Export Pipelines documentation. 

If you’re new to Datadog, you can sign up for a 14-day free trial to start exporting RUM and Product Analytics data to your cloud storage.