Logging is here!

This article was originally published October 2015. This research was last updated June 2016.

With thousands of companies using Datadog to track their infrastructure, we can see software trends emerging in real time. Today we're excited to share what we can see about true Docker adoption—no hype, just the facts.

Docker is probably the most talked-about infrastructure technology of 2016. We started this project to investigate how much Docker is used in production, and how fast real adoption is growing. We found the answers to these questions—and more that we discovered along the way—to be fascinating.

The research that follows was based on a sample of 10,000 companies and tracks real usage, not just anecdotally reported usage. As far as we know, this is the largest and most accurate review of Docker adoption that has ever been published.

Throughout this article we refer to companies' adoption status: "adopted", "dabbling", or "abandoned". Our method for determining adoption status is described in the Methodology section below.

ONE

Real Docker Adoption
Is Up 30% in One Year

At the beginning of May 2015, 8.2 percent of Datadog's customers had adopted Docker. One year later that number has grown to 10.7 percent. That's almost 30 percent market-share growth in 12 months.

TWO

Docker Now Runs on 10%
of the Hosts We Monitor

This is an impressive fact: 18 months ago Docker had about 2 percent market share, and now it's running on 10 percent of the hosts we monitor.

However, the graph below illustrates two other interesting findings. First there was a distinct local maximum of percent hosts running Docker around August 2015. This could be caused by Datadog reaching new customers who did not use Docker yet, or by organizations cooling on Docker while the technology matured, or by both. Second, growth as a percentage of hosts seems to have transitioned from "extremely rapid growth" to "rapid growth" after that time.

THREE

Larger Companies
Are Leading Adoption

This one bucks the stereotype that larger companies are slower to move. The more hosts a company uses, the more likely it is to have tried Docker, and the more likely it is to have adopted Docker. This fact is particularly surprising since the more hosts a company uses, the higher the number of Docker containers the company must use to be considered an "adopter". Notably, in the eight months since this article was originally published, this finding has only strengthened.

Editorial conclusion: Docker is solving problems felt most acutely by companies with larger numbers of hosts.

FOUR

2/3 of Companies
That Try Docker Adopt It

Good news for Docker just keeps coming. We were surprised to find how many companies who try Docker actually adopt it, and fast. Most companies who will adopt have already done so within 30 days of initial production usage, and almost all the remaining adopters convert within 60 days.

FIVE

Adopters 5x Their Container
Count within 9 Months

Docker adopters approximately quintuple the average number of running containers they have in production between their first and tenth month of usage. This phenomenal internal-usage growth rate is quite linear, and shows no signs of tapering off after the tenth month.

SIX

The Most Widely Used Images Are
Still Registry, NGINX, and Redis

The most common technologies running in Docker are:

  1. Registry: 21% of companies running Docker are using Registry. While Registry is still in the #1 position, its share has declined from 25% eight months ago.
  2. NGINX: Docker is being used to contain a lot of HTTP servers, it seems. It is interesting that Apache (httpd) doesn't make the top 10.
  3. Redis: This popular in-memory key/value data store is often used as an in-memory database, message queue, or cache.
  4. Elasticsearch: Full-text search moved up from the #6 position in the last revision of this article.
  5. MySQL: The most widely used open source database in the world moved up from its previous position at #9.
  6. Logspout: For collecting logs from all containers on a host, and routing them to wherever they need to go.
  7. Quay: With Registry losing a bit of ground, Quay charts on the leaderboard for the first time.
  8. etcd: A new addition to the leaderboard, etcd is the only distributed configuration store to chart.
  9. Postgres: The second-most widely used open source database in the world. Adding the MySQL and Postgres numbers, it appears that using Docker to run relational databases is surprisingly common.
  10. CAdvisor: Used by Kubernetes to collect metrics from containers.

Knocked off the leaderboard:

  • Ubuntu: Ubuntu would have retained its ranking, but was removed because a variety of very different images were built on this default base image.
  • MongoDB: The widely used NoSQL datastore.
SEVEN

Docker Hosts Often Run
Five Containers at a Time

The median company that adopts Docker runs five containers simultaneously on each host, up from four containers eight months ago. This finding seems to indicate that Docker is in fact commonly used as a lightweight way to share compute resources; it is not solely valued for providing a knowable, versioned runtime environment. Bolstering this observation, 25% of companies run an average of 10+ containers simultaneously.

EIGHT

VMs Live 6x Longer
Than Containers

At companies that adopt Docker, containers have an average lifespan of 2.5 days, while across all companies, traditional and cloud-based VMs have an average lifespan of almost 15 days. These numbers have changed from 3 and 12, respectively, just 8 months ago. One possible explanation is that as more companies move dynamic workloads to Docker, the average container lifespan shrinks, leaving traditional hosts to run for longer periods of time.

As discussed in fact #7, it is common to run five containers per host simultaneously. So you might expect that the median VM would run 30 containers total in its lifetime (6 generations of 5 simultaneous containers). But due to uneven distributions, the median VM actually runs 14 containers in its life.

Containers' short lifetimes and increased density have significant implications for infrastructure monitoring. They represent an order-of-magnitude increase in the number of things that need to be individually monitored. Monitoring solutions that are host-centric, rather than role-centric, quickly become unusable. We thus expect Docker to continue to drive the sea change in monitoring practices that the cloud began several years ago.