The Need: Monitoring for a Decentralizing and Growing Development Department
Bazaarvoice had grown its product line from a single customer ratings and reviews offering to a suite of social depth platforms that allowed customers to turn user generated content into actionable insights. As a result, the development team had ballooned from 30 to over 200 in just a few years. Maintaining this aggressive growth required that Bazaarvoice find ways to accelerate product innovation and sustainably scale their development team. “It all became a question of scale. Both in terms of scaling our ability to innovate and deliver services, but also scale how we can work effectively with a much bigger development team,” says Verleye. To address these challenges, Bazaarvoice chose to decentralize engineering and create smaller, product-focused teams. As Verleye says, “We needed to make it such that a bunch of different teams could move independently and make their own decisions rather than centralizing a lot of decision making and operations through a single team.”
Although decentralization allowed Bazaarvoice to more quickly respond to market trends, it made monitoring their complex and growing infrastructure more difficult. Bazaarvoice was already using a number of different monitoring solutions on their legacy systems. Scaling the existing monitoring systems would have required each team to not only run their own monitoring server, but also, to learn specific domain expertise in infrastructure monitoring. For Bazaarvoice, this would have involved more than 50 different monitoring servers, each with their own custom setup, metrics, and naming conventions. Maintaining easy and effective operations on such a disjointed system would have been nearly impossible for a team that was doubling in size every 6 to 9 months. Bazaarvoice required a monitoring solution that could support their evolving and decentralized approach to development.
“ What we liked about Datadog as opposed to the competition was the number of integrations you have… We don’t have to define the metrics we care about, Datadog just knows.”
Manager, Infrastructure Engineering, Bazaarvoice
As Verleye describes, “The point of the decentralization is that each service team has different needs so you make different decisions for their architecture. We wanted to find a solution that allowed us to have a lot of people contributing or basically having a point and click easy to use monitoring solution.”
Seamless Integration, Unified Monitoring, and Better Business Insights
To deliver its services, Bazaarvoice uses on-premise servers as well as public and private cloud services provided by Amazon Web Services (AWS). Bazaarvoice’s development department also utilizes a wide variety of tools to support engineering efforts including Java, Tomcat, Solr, MySQL, Jetty, and Cassandra. With so many different tools, it was important for Verleye to find a monitoring service that could easily connect metrics from their large assortment of current tools, and make it easy to add new tools as they evolved in the future. In addition to providing an easy way to connect monitoring from a variety of tools, Datadog’s integrations incorporated out-of-the-box domain expertise about the specific metrics that were the most important to monitor for each tool. “Integrations were important to us because we
were using such a huge set of technologies,” says Verleye. “What we liked about Datadog as opposed to the competition was the number of integrations they had. So if, say, we wanted to go monitor Cassandra, we don’t have to define the metrics we care about, Datadog just knows.”
Flexible monitoring that serves different and unique needs per team
As Bazaarvoice evolved to a more decentralized IT department, they quickly realized how important it was to give each team the flexibility to monitor what was important for each specific environment. Some teams wanted dashboards that could be used as a heads up displays to watch key performance and business metrics. Other teams only wanted to log in and evaluate metrics when they ran into an issue. As Verleye describes, the goal is to “allow everyone to do what they need to for their particular use case and enable more consistency and more visibility into business intelligence metrics.” With Datadog, Bazaarvoice has empowered each of its product teams to create custom metrics and dashboards for their individual systems in an automated fashion.
Custom Metrics for Evolving Products
Bazaarvoice relies heavily on custom metrics as product teams add new tools to their decentralized environments. Despite the rapid changes that Bazaarvoice makes on a monthly basis which affect the volume of custom metrics created, Datadog has been able to keep pace. As Verleye describes, “One of the reasons we like being a Datadog customer is the sheer speed at which they deliver new functionality and fix problems. That’s been great.” Going forward, Verleye and his team see Datadog becoming an even more important part of their continued success. “Our intent is to get as much as possible into Datadog. It gives us a really good vision as to what’s going on in a larger complicated system.” say Verleye.
Consistent and Consumable Metrics for Senior Business Leaders
In addition to the breadth of integrations and flexibility within Datadog, the system’s ability to offer a unified monitoring solution that can be easily customized for various needs was critically important for product teams in communicating with leadership. “We’ve started to realize value, both in terms of responding to incidents and outages, but also in getting more business metrics back to our leadership,” says Verleye. As manager of infrastructure engineering, Verleye’s ultimate goal is to have a single place where his team can consistently track performance data for all the various product teams and begin to do comparisons in a way that is consumable for senior management and R&D. Datadog is allowing him to achieve that goal.
“ One of the reasons we like being a Datadog customer is the sheer speed at which they deliver new functionality and fix problems. That’s been great.”
Manager, Infrastructure Engineering, Bazaarvoice