Updated April 2017. Earlier editions of this article were published in October 2015 and June 2016.
With thousands of companies using Datadog to track their infrastructure, we can see software trends emerging in real time. Today we're excited to share our latest research into true Docker adoption—no hype, just the facts.
Docker is probably the most talked-about infrastructure technology of the past few years. We started this project to investigate how much Docker is used in production, and how fast real adoption is growing. We found the answers to these questions—and more that we discovered along the way—to be fascinating.
The research that follows was based on a sample of 10,000 companies, and 185 million containers in real-world use. As far as we know, this is the largest and most accurate review of Docker adoption that has ever been published.
Throughout this article we refer to companies' adoption status: "adopted", "dabbling", or "abandoned". Our method for determining adoption status is described in the Methodology section below.
Real Docker Adoption Has
Increased 40% in One Year
At the beginning of March 2016, 13.6 percent of Datadog's customers had adopted Docker. One year later that number has grown to 18.8 percent. That's almost 40 percent market-share growth in 12 months.
Docker Now Runs on 15%
of the Hosts We Monitor
This is an impressive fact: Two years ago, Docker had about 3 percent market share, and now it's running on 15 percent of the hosts we monitor.
The graph below illustrates that the Docker growth rate was somewhat variable early on, but began to stabilize around the fall of 2015. Since that time, Docker usage has climbed steadily and nearly linearly, and it now runs on roughly one in six hosts that Datadog monitors.
Are Still Leading Adoption
The knock on larger companies is that they tend to be slower to move. But in the case of Docker, we've seen larger companies leading the way since the first edition of this report in 2015. The more hosts a company uses, the more likely it is to have tried Docker. Nearly 60 percent of organizations running 500 or more hosts are classified as Docker dabblers or adopters.
Whereas previous editions of this report showed organizations with large numbers of hosts clearly driving Docker adoption, the latest data shows that organizations with midsize host counts (100–499 hosts) have made significant gains. Adoption rates for companies with medium and large host counts are now nearly identical.
Editorial conclusion: Docker first gained a foothold by solving the unique needs of large organizations, but is now finding use as a general-purpose platform at companies of all sizes.
Orchestrators Are Taking Off
As Docker increasingly becomes an integral part of production environments, organizations are seeking out tools to help them effectively manage and orchestrate their containers. As of March 2017, roughly 40 percent of Datadog customers running Docker were also running Kubernetes, Mesos, Amazon ECS, Google Container Engine, or another orchestrator. Additional organizations may be using Docker's built-in orchestration capabilities, but that functionality did not generate uniquely identifiable metrics that would allow us to reliably measure its use at the time of this report.
Among organizations running Docker and using AWS, Amazon ECS is a popular choice for orchestration, as would be expected: more than 35 percent of those companies use ECS. But we also see significant usage of other orchestrators (especially Kubernetes) at companies running AWS infrastructure.
Adopters Quintuple Their
Container Count within 9 Months
Docker adopters nearly quintuple the average number of running containers they have in production between their first and tenth month of usage. This internal-usage growth rate is quite linear, and shows no signs of tapering off after the tenth month. In another indication of the robustness of this finding, this pattern of growth among adopters has remained steady since our previous report last year.
The Most Widely Used Images Are
NGINX, Redis, and Elasticsearch
The most common technologies running in Docker are:
- NGINX: Docker is being used to contain a lot of HTTP servers, it seems. NGINX has been a perennial contender on this list since we began tracking image use in 2015.
- Redis: This popular key-value data store is often used as an in-memory database, message queue, or cache.
- Elasticsearch: Full-text search continues to increase in popularity, cracking the top 3 for the first time.
- Registry: 18% of companies running Docker are using Registry, an application for storing and distributing other Docker images. Registry has been near the top of the list in each edition of this report.
- Postgres: The increasingly popular open source relational database edges out MySQL for the first time in this ranking.
- MySQL: The most widely used open source database in the world continues to find use in Docker infrastructure. Adding the MySQL and Postgres numbers, it appears that using Docker to run relational databases is surprisingly common.
- etcd: The distributed key-value store is used to provide consistent configuration across a Docker cluster.
- Fluentd: This open source "unified logging layer" is designed to decouple data sources from backend data stores. This is the first time Fluentd has appeared on the list, displacing Logspout from the top 10.
- MongoDB: The widely-used NoSQL datastore.
- RabbitMQ: The open source message broker finds plenty of use in Docker environments.
Docker Hosts Often Run
Seven Containers at a Time
The median company that adopts Docker runs seven containers simultaneously on each host, up from five containers nine months ago. This finding seems to indicate that Docker is in fact commonly used as a lightweight way to share compute resources; it is not solely valued for providing a knowable, versioned runtime environment. Bolstering this observation, 25% of companies run an average of 14+ containers simultaneously.
Containers Churn 9x
Faster Than VMs
At companies that adopt Docker, containers have an average lifespan of 2.5 days, while across all companies, traditional and cloud-based VMs have an average lifespan of 23 days.
Container orchestration (see fact 4) appears to have a strong effect on container lifetimes, as the automated starting and stopping of containers leads to a higher churn rate. In organizations running Docker with an orchestrator, the typical lifetime of a container is less than one day. At organizations that run Docker without orchestration, the average container exists for 5.5 days.
Containers' short lifetimes and increased density have significant implications for infrastructure monitoring. They represent an order-of-magnitude increase in the number of things that need to be individually monitored. Monitoring solutions that are host-centric, rather than role-centric, quickly become unusable. We thus expect Docker to continue to drive the sea change in monitoring practices that the cloud began several years ago.