Michael Bevilacqua-Linn
Senior Staff Engineer
Tanguy Renaudie
Applied Scientist
Collecting high-quality user feedback on agents, like from thumbs-up or thumbs-down buttons, is an important part of agent development. User feedback is needed for everything from basic gut checks on whether your agents are behaving well to planning and creating robust eval sets. It’s a critical part of Datadog’s Agent Observability, which provides explicit end-user feedback features for collecting and analyzing it. But while these features can be wired up to UI elements like thumbs-up buttons, you can’t force your users to actually click on them. In practice they rarely do, as we noticed while working on Bits Chat.
From a data science point of view, thumbs up, thumbs down, and similar user feedback are just another type of label. This led us to wonder whether we could derive good-enough user feedback from existing Agent Observability traces and other Datadog telemetry by using a technique called weak labeling. We used this technique to create a public session classification skill, which reads traces and other telemetry data to approximate the feedback you’d get from a manual button.
In this post, we’ll explore the weak labeling technique and show you how we tested and validated a classification skill that approximates user feedback from Datadog telemetry.
How weak labeling works
Weak labeling is commonly used in traditional ML projects to generate labels when collecting real ground truth is too expensive or too difficult, as it often is when trying to get comprehensive user feedback. It’s best for generating large amounts of good-enough training data.
The basic idea behind weak labeling is to start with the data you have and use it to derive the label you want via heuristics, trained models, or any automatable means. For many problems, it’s possible to combine several data sources to get a proxy for what you want. For example, a user explicitly clicking thumbs up on a social media post is the ultimate signal of whether they liked it. But combining data on how long they looked at the post along with data on whether they posted a positive comment can get you fairly close.
These derived labels are rarely as accurate as ground truth, so they need to be compared to a smaller golden dataset to understand how close they are and what they can be used for. In our case, the signal we’re after is whether a user had a good interaction with an agent. Did the agent answer their questions, or did the user walk away unhappy?
Choosing our three telemetry types
Well-instrumented applications already collect quite a bit of telemetry data that can help answer this question: Agent Observability collects detailed agent traces; Real User Monitoring (RUM) gives you visibility into what your users are actually doing, where they click, and how long they hover; and Audit Trail surfaces changes made across the platform. Our plan was to generate weak labels approximating a thumbs-up or thumbs-down response from these three telemetry types, then check them against a golden dataset of hand-labeled sessions.
Agent Observability traces contain details about an entire chat session, including transcripts from which we can derive user sentiment. RUM captures how the user interacted with the chat session: Did they bounce right away, or did they accept the agent’s suggestions? Finally, Audit Trail provides details around whether a given Datadog artifact, such as a dashboard or metric, changed. Many Bits Chat conversations involve exactly these changes, so we suspected this would be a useful signal. Together, these three sources give us the arc of a session: what the agent said, what the user did about it, and whether anything in Datadog changed as a result.
What our session classification skill does
You can find our session classification skill in our Datadog Labs repo if you’d like to try it yourself. It’s designed to produce useful results from as little data as possible, and quality improves as you add more. You can run it just on Agent Observability traces, but it improves with RUM and Audit Trail data.
The skill accepts three kinds of modes: an entire application (in which case it samples traces), or an individual trace or session, which it labels each directly. That makes it easy to label a sampled set of traces, where each label stands in for a thumbs up or down from an end user. That’s a useful signal when you’re troubleshooting an agent’s behavior over the last day in Agent Observability. The skill can also be chained into longer pipelines to label individual samples.
How we validated the skill
We wanted to answer two questions: whether we could extract a useful signal about user satisfaction from Datadog telemetry, and whether that signal would improve as we added more telemetry types. So, we built a simple ablation stack in Agent Observability Experiments that starts with traces alone, then traces and RUM, and finally traces, RUM, and Audit Trail. We then ran each version against a golden dataset of hundreds of Bits Chat sessions carrying hand-applied thumbs-up and thumbs-down labels. We cared most about concurrence with the binary thumbs-up and thumbs-down labels from Bits Chat, and because we had a reasonably balanced dataset, we chose accuracy as the primary optimization metric. The notebook below outlines our basic experiment setup and implementation:
A true positive here means that our label matched the hand label, while a false positive means it didn’t. We then measured accuracy across the three different versions of the classifier.
Experiment results
We expected that Agent Observability traces would provide the strongest single signal since they contain the agent conversation itself. We got decent results with just Agent Observability traces, which reached 78% accuracy compared to our ground truth dataset. By adding RUM and then Audit Trail on top of that, we reached 80% and 82% accuracy, respectively. The notebook below shows these results:
While 82% isn’t perfect, it’s a useful first pass to identify sets of traces that are worth inspecting manually. This cuts down the search space to 18% of your overall trace population, and it’s especially useful as a backstop if you lack a true customer-generated thumbs up or thumbs down.
We also validated the approach. The more data sources we added, the more our accuracy improved on the internal validation dataset. Achieving 100% accuracy was not expected, as that generally means you’re overfitting the dataset rather than succeeding. Still, our results suggest we can continue to improve accuracy with additional telemetry types, such as APM and log data.
Collect better feedback from your own agents
This approach works across a wide range of agents, so we’ve published the skill as part of our Datadog Labs agent skills repo. Agent Observability customers and general users can install the skill today from our repo. You can also use the techniques in it as a starting point for building your own skill.
To learn more about monitoring your agent’s performance, costs, traces, token usage, and errors, check out our Agent Observability documentation.
If you’re new to Datadog, get started with a free trial.
