Kyra Abbu
Product Manager
Cody Murray-Bruce
Product Marketing Manager
Teddy Gesbert
Product Manager
Eddie Cai
Technical Content Writer
As engineering organizations deploy AI coding agents such as OpenAI Codex across their teams, access alone provides an incomplete view of adoption. Engineering leaders need visibility into how developers use Codex across models and clients, how much Codex-generated code they accept, and whether they find its code review feedback useful. Connecting these signals to software delivery outcomes can help leaders understand how AI-assisted development relates to engineering performance.
The Datadog OpenAI Codex integration collects per-user Codex activity from the Codex Analytics API and reports it as Datadog metrics. The out-of-the-box OpenAI Codex Overview dashboard brings together usage, token consumption, client activity, code attribution, and code review data. Teams can also send Codex engagement data to AI Impact, where they can compare software delivery performance between AI-assisted and non-AI work.
In this post, we’ll show you how to:
Track Codex adoption across users, models, and clients
Organization-wide usage totals can tell you that Codex adoption is growing, but they don’t tell you how that adoption is distributed. For example, a small group of engineers could account for a large share of activity, or developers might rely heavily on one Codex client while barely using another. Understanding these differences helps engineering leaders identify where Codex has become part of everyday development workflows and where adoption remains limited.
The OpenAI Codex Overview dashboard gives you a central view of Codex activity across your enterprise workspaces. Datadog collects per-user activity once per day and tracks threads, turns, consumed Codex credits, and tokens. You can filter the dashboard by workspace, account, user, model, or Codex client to investigate how usage differs across your organization.
Threads represent Codex conversations that an engineer starts, while turns represent messages sent across those threads. Metrics such as openai.codex.usage.threads and openai.codex.usage.turns therefore provide complementary measures of engagement. The dashboard plots these metrics over time and ranks users by activity, helping you determine whether changes in aggregate usage reflect broad adoption or activity concentrated among a smaller group of engineers.
The dashboard also tracks consumed Codex credits and token usage. Credits reflect Codex credit consumption rather than dollar amounts. The integration does not collect spend or currency data. You can break token consumption down into cached input, uncached input, and output tokens to understand how Codex processes context and generates responses over time.
Model-level views show how engineers use Codex across different models. The dashboard breaks down input and output token usage by model and speed tier, and ranks models by total token consumption. Engineering leaders can use these views to identify which models account for the largest share of usage and how those patterns change over time.
You can also compare adoption across Codex clients. The dashboard tracks threads and turns for clients such as the CLI, desktop app, GitHub, web, and IDE integrations. Because the Codex Analytics API does not provide a token breakdown by client, these views focus on threads and turns. Together, these dimensions help answer whether engineers primarily use Codex from the command line, a desktop application, or directly within development workflows.
Measure Codex contributions to your codebase
Usage metrics show how often engineers interact with Codex, but they do not indicate how much Codex contributes to the codebase. An engineer could have many conversations with Codex without using the resulting code, while another could use it to generate substantial changes with relatively few interactions.
The OpenAI Codex integration collects the lines of code added and removed that are attributable to Codex. The OpenAI Codex Overview dashboard plots these additions and removals over time, giving you another measure of adoption beyond conversations and token consumption. For example, rising attributed lines alongside rising thread and turn counts can indicate that increased engagement is also translating into a larger Codex footprint in the codebase.
Code attribution is useful when considered alongside the usage dimensions described above. If one team generates many turns but relatively few attributed code changes, while another has fewer turns and substantially more attributed code, the teams may be using Codex for different kinds of work. Engineering leaders can use these signals to understand how Codex fits into different development workflows rather than relying on a single organization-wide adoption metric.
Evaluate how developers engage with Codex code reviews
Code generation is only one part of Codex usage. Codex can also review pull requests and reason about changes in the context of a larger codebase. As engineering organizations expand AI-assisted review, measuring the number of generated comments alone does not reveal whether developers find that feedback useful.
The OpenAI Codex integration tracks completed reviews and review comments by priority: P0, P1, and P2. It also tracks how developers respond to those comments through upvotes, downvotes, other reactions, and replies. The OpenAI Codex Overview dashboard combines these signals so you can compare review volume with developer engagement over time.
These metrics can help you evaluate whether developers are engaging with Codex feedback. For example, replies and upvotes can indicate which comments prompt developer interaction, while downvotes can point to feedback that teams may want to examine more closely. Tracking these responses over time gives engineering leaders quantitative context to pair with direct developer feedback about AI-assisted code review.
At Datadog, code review is one area where Codex has demonstrated value beyond the immediate diff. In OpenAI’s case study about Datadog’s use of Codex, engineers reported that Codex identified interactions with modules outside a diff, missing test coverage involving cross-service coupling, and API contract changes with downstream risk.
Datadog’s AI Development Experience team also tested Codex against historical incidents by reconstructing pull requests that had contributed to those incidents. Codex found more than 10 cases, or roughly 22% of the incidents examined, where the engineers who owned the incidents said that its feedback would have made a difference. Following this evaluation, Datadog deployed Codex more broadly, and more than 1,000 Datadog engineers use it regularly.
Review metrics from the integration give other engineering organizations a way to examine similar questions within their own workflows. Rather than treating the number of generated comments as a measure of success, teams can examine comment priority and developer responses to understand how Codex review activity is received.
Connect Codex adoption to software delivery performance
Usage, code attribution, and review metrics show how engineers are adopting Codex, but engineering leaders also need to understand how that adoption relates to software delivery performance. Increased Codex activity alone does not indicate whether AI-assisted development is associated with faster delivery or changes in reliability.
Codex usage data feeds Datadog AI Impact, which measures how AI coding assistants affect software delivery performance. AI Impact classifies pull requests as AI-assisted or non-AI and compares their performance, helping teams measure AI adoption and correlate it with changes in delivery outcomes.
For Codex, AI Impact can infer AI assistance from user activity. When Codex reports that an engineer created lines of code with the tool on a given day, Datadog can classify commits that the engineer created that day as AI-assisted. A pull request is then considered AI-assisted when it contains at least one AI-assisted commit. This attribution gives teams a consistent way to connect Codex activity with the code that moves through their delivery process.
With AI Impact, you can track the percentage of pull requests that are AI-assisted and compare PR throughput, PR cycle time, change failure rate, and recovery time between AI-assisted and non-assisted development. For example, you can examine whether higher Codex adoption corresponds with changes in how many pull requests developers merge or how long those pull requests take to move from their first commit to merge. You can then consider those velocity signals alongside change failure rate and recovery time to understand how AI-assisted development relates to reliability.
Start monitoring OpenAI Codex with Datadog
The Datadog OpenAI Codex integration helps engineering leaders measure Codex adoption and connect AI-assisted development with code contributions, review engagement, and software delivery performance.
To configure the integration, an organization needs a ChatGPT Enterprise workspace with Codex enabled. A workspace owner or admin creates a workspace-scoped Admin key in the OpenAI Admin Console, grants it Codex analytics API read access, and provides the key and Workspace ID in the Datadog OpenAI Codex integration tile.
For setup requirements, supported metrics, dashboards, and configuration instructions, see the OpenAI Codex integration documentation. You can also read about how Datadog uses Codex for system-level code review.
If you don’t have a Datadog account, sign up for a 14-day free trial to start monitoring Codex adoption and its relationship to software delivery performance.
