Gaëtan Piquenot
Account takeovers (ATOs) are one of the most common threats facing online platforms. Attackers buy leaked usernames and passwords on underground markets then test them at scale across websites, hoping that password reuse will give them easy access. Today, ATOs have grown so sophisticated and fast-moving that manual incident response often can’t keep pace, requiring intelligent defense systems for detecting compromised credentials and preventing misuse at scale. This post will discuss how we developed an automated system to proactively defend against these attacks in a way that scales with our growing user base.
Credential stuffing and the challenge of preventing ATO across platforms
One recent ATO attempt we observed targeted Brazil, where attackers used automated tools to try compromised credentials in bulk—commonly known as credential stuffing—from primarily Brazilian accounts.
In this case, our investigation revealed several clear patterns in the attacker’s behavior. The malicious actor used a different IP address for nearly every login attempt, distributing them globally to evade detection. They randomized user agents to resemble common browsers and operating systems. Interestingly, they operated in batches of roughly 20,000 login attempts at a time, pausing for several days between each wave. Despite these precautions, they also managed to bypass our CAPTCHA mechanism in some cases.
We detected successful logins by the malicious actor and reset the affected accounts to prevent further unauthorized access. The actor was able to validate a subset of used credentials, which means customers who reused passwords could be at risk of ATOs on other platforms. Remediation required manual intervention at the time; when we detected a potential ATO, a security engineer reset the affected session and, if applicable, the password.
This incident highlighted the need for automated, real-time defenses to protect customers from distributed credential stuffing campaigns.
Two approaches for designing an automated defense system
Datadog customers can sign in using two main methods: standard password authentication, which supports the use of authenticator applications for multi-factor authentication, or SAML authentication through their organization’s identity provider. We decided to focus exclusively on password-based authentication for now. In the future, our approach for other methods such as SAML or OAuth will follow the same logic, but instead of handling direct input, we will monitor and act on the session cookies associated with those flows.
When we began designing this system, we evaluated two potential approaches. The first was to perform a daily fetch of breach data that regularly pulled newly discovered leaks and cross-checked them against customer emails. This method would allow us to remediate faster by resetting sessions and passwords as soon as credentials were confirmed to be compromised, while also covering all types of login methods. However, this approach came with significant drawbacks: retrieving the full list of users each day would place a heavy load on our infrastructure, and exporting millions of user records is not efficient or sustainable. In addition, API licensing limits meant we couldn’t effectively protect every account at Datadog’s scale without prioritizing higher-tier customers.
That’s why we chose the second approach, which involves conducting real-time checks during login events. In this model, each time a user logs in, signs up, or resets a password, we perform a lookup to determine if their credentials have appeared in a known breach, using k-anonymity to do so securely. This approach allows us to cover all customers using password authentication and doesn’t add ongoing load to our systems. The trade-off, however, is a potential impact on login performance. Third-party API’s introduce latency (~250 ms) that will greatly slow down the login process, and we wanted to minimize any disruption to the user experience.
Minimizing login delay and proactively remediating compromised credentials
To prevent login delay, we decided to take a fire-and-forget approach, where we send a verification in the background without being in the critical path of the login flow, so it does not delay user login. Focusing on the password login flow, our authentication system uses a CLI to build a payload which contains the user handle and a hash using the k-anonymity technique, which ensures the server never sees the full password or hash. In our case, we followed our provider implementation, which is the first eight characters of the SHA-512 of the username and password concatenation: SHA512(username+password)[:8].
When designing the remediation workflow, we debated how much responsibility should fall on Datadog versus on the customer. One option was to simply notify users of compromised credentials and let them decide when to reset. Ultimately, we chose to proactively reset passwords when a confirmed leak was detected to make sure accounts were protected before attackers could exploit them.
To balance this with transparency, every action is accompanied by an audit log event that records what was done and why, along with an email notification that explains the situation and guides the customer through the reset process. We decided against storing a persistent mapping of “user ↔ compromised status” in our systems, as it would create unnecessary risk. This way, we help ensure accounts are safe while still respecting customer privacy.
For Datadog customers, this initiative brings stronger and more integrated protection to every login. If a password associated with an account appears in a known breach, our system can identify the exposure and take action before attackers have the chance to use it. Because the process is automated, remediation happens quickly and consistently, reducing the window of opportunity for abuse without requiring manual intervention. All of this runs quietly in the background, so customers can benefit from increased security without added friction or disruption to their normal login experience.
What’s next
We plan to expand protection beyond local passwords, with a particular focus on detecting threats like cookie theft—a technique increasingly used by attackers to bypass traditional authentication controls. We’ll continue to encourage customers to enable MFA and follow account security best practices, adding another layer of defense against account takeover attempts.
As breach intelligence evolves, we’ll keep iterating based on customer feedback to ensure our defenses remain effective and adaptive. We also aim to deepen integration with the Safety Center, giving administrators clearer visibility into credential exposure and remediation actions. Together, these next steps build on the stronger protections and faster remediations already in place, making Datadog logins even more resilient without adding friction for customers. Check out our documentation for Datadog Cloud Security to learn about all our latest capabilities, and see how you can further mitigate ATOs with Datadog App and API Protection. If you’re new to Datadog, sign up for a 14-day free trial to get started.
