4 min read

Risk vs. Reward: A Guide to Understanding Software Containers

View all articles

Those of us who are old enough can remember a day when software was delivered primarily by physical media. The spread of broadband internet and smartphones has led us to the age of the web service—software hosted in the cloud accessed by user clients such as browsers and apps.

Not too long ago, web applications were run directly on physical machines in private data centers. For ease of management, these applications were usually monolithic—a single large server would contain all of the back-end code and database. Now, web hosting services like Amazon and the spread of hypervisor technology have changed all of that. Thanks to Amazon Web Services (AWS) and tools like VirtualBox, it has become easy to package an entire OS in a single file.

Using services like EC2, it has become easy to package machine images and string together sets of virtual servers. Along came the microservices paradigm—an approach to software architecture wherein large monolithic apps are broken up into smaller focused services that do one thing well. In general, this approach allows for easier scaling and feature development as bottlenecks are quicker to find and system changes easier to isolate.

Pets to Livestock

I became an infrastructure engineer right at the height of this trend. I recall building my first production environment in Amazon using a series of bash scripts. The servers were like pets to me. I gave each of them cute names. I monitored them carefully. I responded to alerts quickly and kept them healthy. I treated those instances with love and affection because it was painful to try to replace them—much like a beloved pet.

Along came Chef, a configuration management tool, and almost immediately my life got easier. With tools like Chef and Puppet, you can take away most of the manual pain associated with managing a cloud system. You can use its “environments” construct to separate development, staging, and production servers. You can use its “data bags” and “roles” to define configuration parameters and push sets of changes. Now, all of my “pet” servers had graduated from obedience school.

A graphic representation of a crane managing shipping containers

Then in 2013, along came Docker, and a new era began: the age of software as livestock (apologies to any vegans in the audience). The container paradigm is one of orchestration, not configuration management. Tools like Kubernetes, Docker Compose, and Marathon focus on moving around predefined images instead of adjusting config values on running instances. Infrastructure is immutable; when a container goes bad, we don’t try to fix it—we shoot it in the head and replace it. We care more about the health of the herd than individual animals. We don’t give our servers cute names anymore.

The Rewards

Containers make a lot of things easier. They let businesses focus more on their own special sauce. Tech teams can worry less about infrastructure and configuration management and instead worry mostly about app code. Companies can go a step further and use managed services for things like MySQL, Cassandra, Kafka, or Redis so as not to have to deal with the data layer at all. There are several startups offering “plug and play” machine learning services as well to allow companies to do sophisticated analytics without worrying about the infrastructure. These trends have culminated in the serverless model—a software architecture approach that allows teams to release software without managing a single VM or container. AWS services like S3, Lambda, Kinesis, and Dynamo make this possible. So to extend the analogy, we have gone from pets to livestock to some sort of on-demand animal service.

All of this is very cool. It is crazy that we live in a time where a twelve-year-old kid can spin up a sophisticated software system with a few clicks. We should remember that, not very long ago, this was impossible. Just a few US presidents ago, physical media was the standard and only big companies had the means to manufacture and distribute software. Bug fixes were a luxury. Now, that twelve-year-old kid can create an AWS account and make his software available to the entire world. If there’s a bug, someone will bug him on Slack and, in a few minutes, a fix is out for all users.

The Risks

Very, very cool, but not without its price—reliance on cloud providers like Amazon means reliance on big corporations and proprietary technologies. If Richard Stallman and Edward Snowden haven’t made you worry about such things, the recent debacle with Facebook certainly should have.

Greater abstraction away from hardware also brings with it the risk of less transparency and control. When something breaks in a system running hundreds of containers, we have to hope that the failure bubbles up somewhere we can detect. If the problem is with the host operating system or underlying hardware, it might be hard to determine. An outage that could have been resolved in 20 minutes using VMs may take hours or days to resolve with containers if you do not have the right instrumentation.

It isn’t just failures either that we need to worry about when it comes to things like Docker. There is also the problem of security. Whatever container platform we use, we have to trust that there are no backdoors or undisclosed security vulnerabilities. Using open-source platforms is no guarantee of safety either. If we rely on third-party container images for parts of our system, we may be vulnerable.

Wrap Up

The livestock paradigm is attractive for a number of reasons, but it is not without its downsides. Before rushing to containerize the entire stack, tech teams need to think about whether or not it is the right choice and ensure they can mitigate the negative effects.

Personally, I love working with containers. I’m excited to see where things go in the next ten years as new platforms and paradigms arise. However, as a former security consultant, I am cautious enough to know that everything comes with a price. It is up to engineers to remain vigilant to ensure that we don’t give up our autonomy as users and developers. Even the easiest CD/CI workflow in the world would not be worth the cost.

Understanding the Basics

What is a container in Docker?

A container in Docker is an isolated environment where processes are restricted to their own set of operating system resources without having to abstract the hardware layer.

About the author

Jonathan Bethune, Japan
member since February 13, 2018
Jonathan is an experienced DevOps engineer based in Tokyo and New York. He's worked for a number of startups building the infrastructure and improving security. He's extremely knowledgeable about machine learning, data analysis, infrastructure development, and data layer optimization. One of his best qualities is his knack for quickly learning new things which is why he landed in DevOps because it is the best field for trying new technologies. [click to continue...]
Hiring? Meet the Top 10 Freelance DevOps Engineers for Hire in December 2018

Comments

Enrique Conci
Great Article Jonathan! I share your point of view about containers and new paradigms in general. This industry is full of people sitting in the peak of the hype wave, but you've pointed the risks.
Michael Gruben-Trejo
I’d love to hear more from this author on this subject
davenicoll
"An outage that could have been resolved in 20 minutes using VMs may take hours or days to resolve with containers if you do not have the right instrumentation." What is the right instrumentation? I'm interested in what you've used, and your experiences. Interesting post, thanks.
Jonathan Bethune
Thanks for the compliment!
Jonathan Bethune
I appreciate the compliment! I wrote a more philosophical version of this article on my personal blog. Check it out if you're interested. https://medium.com/@jbethune777/software-delivery-and-society-d32ac6ed41b2
Jonathan Bethune
Great question. I could write a whole article just about instrumentation. In my experience people tend to be more thorough with metrics when they use VM's instead of containers. Many container solutions such as ECS or Google Cloud do not provide a lot of the underlying system metrics by default. So for example, you may get basic app logs and system metrics in something like Cloudwatch, but if you have a more subtle failure - a bug in a library, an OS config failure, a noisy neighbor datacenter problem, etc - you may not get any indicators of the real problem when you are as abstracted away from the hardware as containers tend to force you to be. Containers let you treat infrastructure as disposable. If something breaks, just destroy it and replace it. Unfortunately some people think this means that they do not need to know as much about what is going on under the hood. They think they can get away with collecting fewer metrics. Sometimes they are right. When they are wrong, it is painful. My approach has been to use solutions like Prometheus and Datadog to ensure that I am getting just as much data about the system whether I am using containers or not.
Michael Gruben-Trejo
Interesting and thought-provoking, thanks!
davenicoll
Thanks for answering, Jonathan. One of the many reasons in favour of containers :) I've been using datadog for a while, but I'm relatively new to Prometheus, so I'll have to try it out.
@TaDaDaDeK_
Just reading your stuff like a child some magic stories. Sounds like Silmarilion to me.
Mark Patrov
Nice content!Bitcoin Mining Explained http://cryptodetail.com/best-practices-mining-cryptocurrencies
comments powered by Disqus
Subscribe
Free email updates
Get the latest content first.
No spam. Just great articles & insights.
Free email updates
Get the latest content first.
Thank you for subscribing!
Check your inbox to confirm subscription. You'll start receiving posts after you confirm.
Trending articles
Relevant Technologies
About the author
Jonathan Bethune
Java Developer
Jonathan is an experienced DevOps engineer based in Tokyo and New York. He's worked for a number of startups building the infrastructure and improving security. He's extremely knowledgeable about machine learning, data analysis, infrastructure development, and data layer optimization. One of his best qualities is his knack for quickly learning new things which is why he landed in DevOps because it is the best field for trying new technologies.