Building AI systems for real enterprise operations

LayerX is building AI systems designed to automate complex operational work inside large enterprises. Guided by its mission to digitize economic activity, the company develops AI products that help organizations reduce manual processes tied to contracts, financial records, proposals, and internal documentation.

The company’s Ai Workforce platform supports workflows such as proposal search, credit-assessment reporting, lease-asset registration, document classification, and enterprise knowledge search. Customers include financial institutions and major enterprises managing large volumes of operational and compliance-heavy documents.

At the center of the platform is an agentic AI architecture that dynamically combines searches, tool calls, LLM interactions, and workflow execution based on the business objective and customer-specific context. Rather than following a fixed process every time, workflows adapt in real time depending on the request and the information available. “We focus on the space where general-purpose AI cannot easily operate,” says Kenta Ishida, Site Reliability Engineer on the AI Platform team at LayerX. “Enterprise workflows involve company-specific knowledge, documents, and operational rules that AI needs to reason through dynamically.”

As Ai Workforce expanded across production enterprise environments, LayerX recognized that conventional monitoring approaches were no longer enough. The company already used Datadog Logs, Metrics, Application Performance Monitoring (APM), and Real User Monitoring (RUM) to monitor infrastructure and applications, but AI workflows introduced new operational challenges that traditional telemetry alone could not fully explain.

The platform runs primarily on Python with infrastructure managed through Terraform on Microsoft Azure. LayerX built its observability stack on OpenTelemetry and added Datadog Agent Observability to instrument workflows, agents, tasks, tools, and LLM calls alongside conventional application telemetry.

Going beyond traditional APM to understand AI agent behavior

As LayerX evolved from deterministic workflows into increasingly agentic systems, troubleshooting became far more complex. In fixed workflows, engineers could usually identify which step produced an unexpected result. But with agentic workflows, execution paths change dynamically depending on the request and the model’s reasoning process.

Which tools are selected, how many searches occur, which documents are referenced, and how intermediate decisions are made are all determined during runtime. Even when workflows technically completed successfully, the team often lacked visibility into why output quality declined or why the workflow behaved unexpectedly.

“The hardest part was understanding what the AI agent was actually doing inside a request,” Ishida explains. “Even when a request succeeded, it was difficult to determine where quality degraded or why the workflow behaved a certain way.”

Before implementing Datadog Agent Observability, LayerX relied on conventional APM, application logs, and manual review of execution results to investigate workflow behavior. That process made it difficult to isolate whether problems originated in retrieval quality, workflow orchestration, tool execution, intermediate reasoning, or external LLM services.

The challenge became especially important when evaluating prompt and model changes. A single update could influence latency, cost, tool selection, search behavior, and output quality simultaneously, but those signals were spread across multiple systems. Because LayerX supports enterprise customers including financial institutions and leasing companies, maintaining reliability and operational trust was essential.

Connecting workflows, infrastructure, and LLM telemetry to speed up investigations

Datadog Agent Observability gave LayerX a unified view across workflow execution, application telemetry, and infrastructure monitoring. By connecting workflow, agent, task, tool, and LLM-call spans into a single trace, the team gained visibility into the full execution path behind every request. Now engineers can see which workflow executed, which tools were selected, where latency increased, how searches behaved, and which LLM calls generated errors—all alongside conventional APM telemetry in the same platform.

“Datadog lets us follow the workflow step by step,” Ishida says. “We can quickly determine whether an issue originates in the workflow logic, a tool call, the LLM itself, or an external provider.”

This visibility became especially valuable during incidents involving external LLM providers. In one workflow, errors associated with a specific LLM call began increasing. Previously, identifying the source would have required lengthy investigation across prompts, logs, and application behavior. Using Datadog Agent Observability, the team isolated the failing span within the trace and quickly determined whether the issue originated internally or with the provider.

By unifying workflow, agent, task, tool, and LLM calls into a single trace, LayerX estimates bug investigations that previously required one to two days can often be completed in tens of minutes and typically within an hour, resulting in a 95%+ reduction in investigation time.

Provider-side triage also improved dramatically. Issues that previously required extensive back-and-forth with external LLM providers can now often be resolved in minutes by sharing a single Datadog trace URL, resulting in an estimated 95% reduction in provider triage time.

The integration between Datadog APM and Agent Observability also gave the team a connected operational view across frontend systems, backend APIs, workflow orchestration, and AI execution layers. Engineers can determine whether latency or failures originate in the Web/API layer, orchestration logic, tool execution, or external providers without moving between separate systems.

Scaling AI operations beyond the engineering team

As observability matured, Datadog became more than a troubleshooting platform for LayerX. Workflow traces evolved into a shared operational language used across SRE, QA, and Forward Deployed Engineering teams to investigate incidents, validate workflows, and support customer operations collaboratively.

Customer validations, bug investigations, and incident response discussions now happen around the same workflow traces, allowing teams outside the original implementation group to independently understand how a workflow behaved step by step.

“Agent Observability became a shared language for understanding AI workflows,” Ishida says. “Everyone can see what happened inside a workflow using the same trace.”

The visibility also changed how LayerX ships AI updates into production. Every prompt and model rollout is now reviewed through trace inspection, allowing teams to evaluate search behavior, tool selection, intermediate reasoning steps, latency, cost, and error rates before broader deployment. In a representative case, the team reduced an improvement cycle from approximately five days to one day, an 80% reduction in iteration time driven by faster investigation and validation workflows.

Looking ahead, LayerX plans to expand its use of Datadog Agent Observability’s pre-production capabilities, including datasets, experiments, playgrounds, and automated evaluations. By connecting production insights with pre-production testing, the team aims to validate AI changes more confidently before deployment.

LayerX is also interested in Datadog Patterns to help automatically identify recurring behaviors and emerging issues across production AI workflows at scale.

“The value of observability is that the entire team can understand what is happening inside production AI workflows with the same lens,” Ishida says. “That visibility is what allows us to scale enterprise AI responsibly.”