A Brief History of Time
In order to understand DevOps, we have to travel back in time to the old age when there was nothing but programmers.
As the Tao teaches us:
The programmers of old were mysterious and profound. We cannot fathom their thoughts, so all we do is describe their appearance:
- Aware, like a fox crossing the water
- Alert, like a general on the battlefield
- Kind, like a hostess greeting her guests
- Simple, like uncarved blocks of wood
- Opaque, like black pools in darkened caves
The Programmer gave birth to machine language. Machine language gave birth to the assembler. The assembler gave birth to the compiler. Now there are thousands of languages. Each language has its purpose, however humble. Each language expresses the Yin and Yang of software. Each language has its place within the software.
At the time, software development offices were mostly called labs, and programmers were scientists. In order to create a good application, programmers had to fully understand the problem the application was solving. They had to know where the application will be used and what kind of system will be running it. In essence, programmers knew everything about their application from specification, through development, to deployment and support.
And then human nature kicked in, and we started to ask for more. More speed, more features, more users, more everything.
Being modest, humble, and peaceful creatures, programmers had very little chance to survive such burst of needy users always asking for more. The best chance to prevail was to start evolving in different directions, and create castes. Soon, programmers were extinct as forefathers of new breeds called developers, software engineers, network administrators, database developers, web developers, system architects, QA engineers, and many more. Evolving fast and adapting to challenges from the outside world became part of their DNA. New caste could evolve in a matter of weeks. Web developers became back-end developers, front-end developers, PHP developers, Ruby developers, Angular developers … it was all going to hell.
Soon they all forgot that they came from the same father, a Programmer. A simple and peaceful scientist that just wanted to make the world a better place. They started fighting each other, claiming that each of them is true descendant of “The Programmer” and that their blood is more pure than others.
As time passed by, they reduced their interaction to a minimum and spoke to each other only when they really had to. They stopped celebrating success of their distant family members, and eventually even stopped sending a postcard every once in awhile.
If they only searched their feelings, they would discover that the spark of the Programmer was in their hearts, waiting to shine and bring peace to the galaxy.
In their selfish and egocentric race to conquer the world, programmer’s descendants forgot the very purpose of their job - solving problems for their clients. Clients started to feel the pain of such behavior as projects got delayed, too expensive, or even failed.
Every once in awhile, a bright star would shine and someone would get an inspiration to try and make peace among The Descendants. They came up with Waterfall. This was a brilliant idea, as it utilized the fact that different groups of developers communicated only when necessary. When one group finished their part of the job, it communicated with the next group to send work down through the process and never looked back at it.
This worked for a while, but as usual, humans (Clients) wanted more again. They wanted to take more active part in the whole process of creating software, give their opinion more often, and ask for changes even when it was too late.
Software projects became so prone to failure that it was accepted as an industry standard. Statistics showed that over 50% of projects were failing, and it seems like there was nothing to be done about it (ZDNet, CNet)
Every generation had a few open-minded individuals who knew that all these groups of developers have to find a way to work together, communicate, and be flexible to assure their clients will receive the best possible solution. There are traces of these attempts even as early as 1957, by colleagues of the great John Von Neumann. However we had to wait until early 2001 to start the revolution, when a dirty dozen created an Agile Manifesto.
The Agile Manifesto is based on twelve principles:
- Customer satisfaction by early and continuous delivery of valuable software
- Welcome changing requirements, even in late development
- Working software is delivered frequently (weeks rather than months)
- Close, daily cooperation between business people and developers
- Projects are built around motivated individuals, who should be trusted
- Face-to-face conversation is the best form of communication (co-location)
- Working software is the principal measure of progress
- Sustainable development, able to maintain a constant pace
- Continuous attention to technical excellence and good design
- Simplicity—the art of maximizing the amount of work not done—is essential
- Self-organizing teams
- Regular adaptation to changing circumstance
Agile manifesto was the first big step in bringing peace to the Galaxy and restoring balance to the Force. For the first time in a very long time, connecting all stakeholders in software development process was based on the cultural and “human” way, as much as on procedural and mechanized manner. People started to talk to each other, meet on a regular basis, and exchange ideas and comments all the time. They realized that they have much more in common that they thought, and clients became part of the team, not just some external factor that was expected to send the money and ask no questions.
There were still a few hurdles to overcome, but the future seemed brighter than ever. Being agile means being open and willing to adapt to constant changes. However, with all too many changes, it is difficult to stay focused on the ultimate goal and delivery. And that is how Lean Software Development came to life.
Hooked on LSD (pun intended) and risking to be exiled and outcast, some of the descendants looked for solutions outside of their caste and the software industry. They found salvation in the works of a major car manufacturer. Toyota Production System was amazing in its simplicity and it was so obvious that its lean manufacturing can be easily applied to software development.
There are 7 principles of lean:
- Eliminate Waste
- Build Quality In
- Create Knowledge (Amplify learning)
- Defer Commitment (Decide as late as possible)
- Deliver as fast as possible
- Respect People (Empower the team)
- Optimize the Whole
Added on top of Agile, Lean principles supported the mentality and the focus on doing the right things, while being flexible during the whole process.
Once Agile and Lean were adopted by software development teams, it took just one more step to apply the whole set of principles to IT as a Whole - which finally brought us to DevOps!
Enter DevOps - Three lane highway
The old-school view of software development teams included business analysts, system architects, front-end developers, back-end developers, testers, and so on. Optimizing software development process, including agile and lean principles, was mostly focused on these. Once the application was “production ready”, it was sent over to “Operations” including systems engineers, release engineers, DBAs, network engineers, security professionals, etc. Removing the great wall between Dev and Ops is the main driving force of DevOps.
DevOps is the result of implementing lean principles to the whole IT value stream. IT value stream extends development into production, combining all ‘distant relatives’ that descended from the original Programmer.
The best explanation of DevOps is given by Gene Kim, and If you haven’t read The Phoenix Project I suggest you take some time and do it.
You should not hire a DevOps engineer and DevOps should not be a new department in your IT. DevOps is a culture, mindset, and it is part of IT as the Whole. There are no tools that will make your IT a DevOps organization, and no level of automation will empower your teams to deliver maximum value to your clients.
DevOps is usually referred as method of three ways, but I see them as three lanes of the same highway. You start in lane one, then you speed up and switch to lane two, and eventually you are speeding by in the third lane.
Lane one - Performance of the system as whole is main focal point and it is emphasized over performance of any individual element of the system
Lane two - Make sure that there is a constant feedback loop sent upstream, and not ignored
Lane three - Nurture experiments, constant improvement, and fail fast
Lane one - Getting up to speed
Understanding the importance of the whole system, and prioritizing work items properly is the first thing an IT organization has to learn when adopting DevOps principles. No one in the IT value stream is allowed to create a bottleneck and reduce the flow of work.
Assuring uninterrupted workflow is the ultimate goal for everyone included in the process. Regardless of the role a person or a team has, it is imperative that they seek to achieve profound understanding of the system. Adopting this way of thinking has a direct impact on quality, as defects are never sent down the stream because they would cause bottlenecks.
Making sure that work is not stalling is not enough. A productive organization should always seek to increase the flow. There are numerous methodologies to increase the flow. You can find a solution in Theory Of Constraints, Six Sigma, Lean, or Toyota Production System. Feel free to pick any of these, come up with your own, or combine them.
DevOps principles do not care what team you belong to, if you are a System Architect, DBA, QA, or network administrator. The same rules apply to everyone, and everyone is expected to follow two simple principles:
- Keep the system flow uninterrupted
- Increase and optimize workflow at all times
Lane two - Gear up
Uninterrupted system flow is one-directional, and it is expected to go from development to operations. In an ideal world this means that software is created quickly with high quality, deployed to production, and delivering value to clients.
However, DevOps is not Utopia, and if one-directional delivery was possible the waterfall method would be sufficient. Evaluating deliverables and communicating up the flow is very important to assure quality. First “upstream” communication channel that has to be implemented is Ops to Dev.
Giving a high-five to yourself is easy, but asking for feedback and giving feedback is the way to see the real picture. It is imperative that each small step downstream is followed by an instant upstream confirmation.
It does not matter how you establish the feedback loop. You can invite developers to join support team meetings, or bring in the network admin to weekly sprint planning. As long as your feedback is in place, and comments are picked-up and handled, your organization is speeding down the DevOps highway.
Lane three - Warp 10
DevOps fast lane is not for weak minded. To get on the DevOps fast lane your organization must be mature enough. It is all about taking risks and learning from failure, continually experimenting, and accepting that repetition and practice is the prerequisite to mastery. Quite often you will hear the term Kata, and this is for a reason. Continuous training and repetition leads to mastery because it makes complex moves become a routine.
But before you start combining moves, it is imperative that you master each individual step first.
“What is appropriate for the Master is not appropriate for the novice. You must understand Tao before transcending structure.”
DevOps Third Way/Fast Lane include allocating time for continuous experimenting on a daily basis, constantly rewarding the team for taking risks, and introducing faults into the system to increase resilience.
In order to assure that your organization will not implode when implementing these kinds of measures, you must create frequent feedback loops between all teams, while making sure that all bottlenecks are clear and system flow is uninterrupted.
Implementing these measures makes your organization alert at all times and capable of responding to challenges quickly and effectively.
Summary - DevOps organization checklist
Here is a checklist that you can use to validate how DevOps enabled your IT organization is. Please, feel free to comment on the article and add your own points.
- There is no wall between development team and operations team. Both are part of the same process
- Work that is sent over from one team to another is always verified and top quality
- There is no “piling” of work, and all bottlenecks are handled
- Development team is not using Operations time for its activities, because deployment and maintenance are part of the same time box
- Development team does not deliver the code for deployment at 5PM on Fridays, leaving operations to clean up their work over the weekend
- Development environments are standardized, and operations can easily reproduce and scale them into production
- Development team can deliver new versions as they find appropriate and operations can easily deploy them into production
- There are clear communication lines between all teams
- All team members have time to experiment and work on improvement of the system
- Defects are introduced (or simulated) and handled in the system on a regular basis. Lessons learned from each experiment are documented and shared with all relevant people. Incident handling is a routine and mostly handled in a known way
Using modern DevOps Tools like Chef, Docker, Ansible, Packer, Troposphere, Consul, Jenkins, SonarQube, AWS, etc. does not mean that you are applying DevOps principles. DevOps is a way of thinking. We are all part of the same process, we share the same time and deliver value together. Every person that takes part in the software delivery process can speed up or slow down the entire system. A bug created during development can bring down the system, as well as the wrong firewall configuration.
All of the people are part of DevOps, and once your organization understands that, tools and technology stack that will help you manage it will magically show up.