Manual workflows and fragmented tooling slow incident response
Krafton PUBG Studio is the core development and operations studio behind PUBG: Battlegrounds, one of the world’s most widely played online games. With millions of active players across every time zone, even brief service disruptions carry real consequences—for player experience, trust, and the studio’s reputation. Keeping the game stable and responsive around the clock falls to the studio’s site reliability engineering team.
As PUBG: Battlegrounds scaled, the SRE team set out to bring real structure to how it handled incidents. At the time, response relied on a patchwork of manual steps and disconnected tools. When something went wrong, on-call engineers had to manually create a Slack war room and invite stakeholders one by one. Important updates got buried in thread replies. In some cases, separate teams—such as the network and middleware groups—would respond to the same incident independently, generating duplicate work and slowing resolution. An internal effort to build a custom incident and case management tool highlighted just how complex the problem had become, and made clear that a purpose-built platform was the better path forward.
Frequent false-positive alerts compounded the problem, eroding trust in critical notifications and adding to the cognitive burden on engineers. And because detection and resolution were tracked manually, Krafton PUBG Studio had no reliable way to measure MTTD and MTTR—making it nearly impossible to identify trends, set benchmarks, or drive meaningful improvement over time.
Designing a structured, five-stage incident response process
Rather than patching the existing workflow, Krafton PUBG Studio’s SRE team took a deliberate approach: they designed a formal five-stage incident response process from the ground up—Awareness, Declaration, Communication, Action, and Postmortem—and built it entirely on Datadog.
The team was already using Datadog for metric collection and monitoring, but recognized they needed to go further. “We didn’t have a defined response process,” explains Park Dal, Part leader of SRE at Krafton. “Handling was manual and fragmented across multiple tools.”
The goal was to centralize every stage of the process in one place, replacing the ad hoc coordination that had been slowing them down.
In the Awareness stage, incidents are detected using telemetry data—API latency metrics, error rates, and logs—that trigger the response process. Declaration happens when an on-call engineer formally declares the incident in Datadog Incident Management, which automatically creates a dedicated Slack war room and invites the relevant team members. All messages in that channel are logged to the incident timeline, giving responders a consolidated, auditable record as events unfold. In the Communication stage, community managers, game service operators, and other stakeholders are briefed in non-technical language. The Action stage focuses on mitigation—service restarts, gateway throttling, forced scale-outs, failover—executed automatically or semi-automatically through Workflow Automation. Finally, the Postmortem stage drives continuous improvement: the team reviews the incident timeline, identifies root causes, and establishes preventive measures.
Keeping alert quality high and engineer fatigue low
One of the team’s most deliberate design decisions was around alert discipline—and it started with a simple rule: only incidents that directly impact players qualify as critical. Those alerts route to a dedicated Slack channel where on-call engineers are required to keep notifications on at all times, ensuring that when something real happens, it gets immediate attention. To keep that channel trustworthy, monitors are periodically reviewed and pruned during postmortem meetings. “Excessive alerts erode confidence,” says Dal. “Minimizing false positives is a priority.”
The team also introduced a shared vocabulary for tracking incident progress: five severity levels and three statuses—Active, Stable, and Resolved—give all responders a consistent framework, while MTTD and MTTR are tracked automatically and used as benchmarks for continuous improvement.
“Excessive alerts erode confidence. Minimizing false positives is a priority.”
Extending the process with AI-assisted postmortems
Rather than treating postmortems as a manual burden, the team engineered them as a structured, largely automated output. When an incident is resolved, Datadog Incident Management surfaces a postmortem draft built from the incident timeline. The team went further by integrating Workflow Automation with the OpenAI API, enabling AI to generate an initial draft that engineers then review and refine—a human-in-the-loop model that reduces documentation overhead without removing accountability.
MTTD drops to 3.89 minutes
The results speak to what’s possible when structure, automation, and alert discipline work together. By consolidating every stage of incident response into a single, well-defined workflow, Krafton PUBG Studio eliminated the redundant coordination and duplicate work that had previously slowed them down—and finally had the data to prove it. “We integrated and unified our entire incident response process into the Datadog platform,” says Dal. “As a result, we significantly reduced false positives, decreased engineer fatigue, and improved alert reliability—ultimately achieving an MTTD of 3.89 minutes.”
For a studio operating a live game at global scale, that’s the difference between a disruption players barely notice and one they don’t forget.
“We significantly reduced false positives, decreased engineer fatigue, and improved alert reliability.”
Looking ahead, the team is evaluating Datadog On-Call and Bits AI postmortem generation as the next step in maturing their process, with a focus on multi-person on-call rotation support, AI-generated postmortems in Korean, and custom prompt support—capabilities that would deepen the automation they’ve already built.
