United StatesToptal Member Since November 21, 2019
Dmitry is a cloud architect and site reliability engineer with over a decade of intense professional experience strictly adhering to the DevOps methodology. He has architected and built multiple platform-agnostic infrastructures from scratch for modern cloud systems. Dmitry has a proven track record of hands-on operations in high-scale environments. He is also proficient with IaC, automation, scripting, as well as monitoring and observability.
Sagi is a top-performing, Microsoft Certified Senior Azure DevOps engineer with ten years of solid hands-on experience in DevOps, programming, scripting, and business intelligence. Sagi specializes in architecting and implementing DevOps processes using Azure DevOps and Azure Cloud platforms. By utilizing his gained experience in multiple application development areas, Sagi has become one of the most prominent experts in the market.
In 2012, Arthur earned a master's degree in computer engineering but he soon learned his true north was in system administration. His programming background has helped him automate most of his tasks along the way and he eventually ended up in cloud computing as it gave him even more possibilities. Arthur is a full-stack DevOps who has particularly strong development skills with all things AWS—which his numerous certifications can attest to.
Simon is a full-stack engineer with eight years of experience in mobile development and four years of Python development. He is an innovative and highly dedicated software engineer with excellent communication skills. He is known for his great instincts, entrepreneurial mindset, and his ability to balance best practices and productivity while staying on budget. If you want high-performing applications, Simon is your guy.
Keidrych reconciles technology to humanity by helping organizations with Cyvive Foggy Computing by making planet-scale, ubiquitous computing as natural as breathing. He pushes the limits of cyber-adaptivity by leveraging scorched earth capabilities to advance technological maturity, enhance security, and gravitate toward zero impact on production systems.
Ryan is an experienced software engineer of reliable and scaleable production Cloud systems. He specializes in DevOps, microservices and architecting applications. He has a solid background in Cloud and back-end jobs and is skilled in Node.js and Python. He has good soft skills and has worked in teams of all sizes. Ryan has an excellent ability to understand the business need behind requirements.
United StatesToptal Member Since September 17, 2013
Mike is a senior data engineer and freelance architect experienced across the development stack. He has extensive cloud and infrastructure experience, with multiple certifications from Microsoft, ISC2, Powersoft, and more. He currently builds data back ends for web apps at scale on RDBMS or NoSQL platforms.
Fernando is a telecommunications engineer working in his chosen field since 2013. He's gained hands-on experience working in challenging environments as an individual contributor and in leadership positions. Fernando joined Toptal to work on projects where he can make a meaningful contribution and deliver real value to his clients.
Guilherme is a computer engineer who has a passion for solving challenges and building beautiful software. He is a flexible and dynamic developer who has been able to learn new technologies and start building idiomatic code very quickly. He is enthusiastic about elegant solutions and sharing ideas with other people.
Fernando is a technology enthusiast with more than ten years of experience in IT, seven of which are in AWS-built environments. Always driven by measurable metrics and big value efforts, he is a generalist with a great capacity and focus on aggregating context and can take deep dives into specific topics when problems arise and solutions are needed. Fernando believes leading by example and attentively coaching is the only way to multiply the impact on an organization.
Fredrik is a developer with over fifteen years of contracting and entrepreneurial experience. He specializes in back-end product development, lifecycle maintenance, and secure data handling and operations, in everything from cluster implementations in charging systems to full-stack product development for one-person startups.
DevOps is like full-stack development: Covering both development and system operations, it cuts across a very wide spectrum of tech. DevOps engineers can be expected to know everything from ad-hoc SQL data mining to automating Docker via Bash scripts.
Recruit and interview DevOps engineers effectively with this handy hiring guide.
... allows corporations to quickly assemble teams that have the right skills for specific projects.
Despite accelerating demand for coders, Toptal prides itself on almost Ivy League-level vetting.
Building a cross-platform app to be used worldwide
Tripcents wouldn't exist without Toptal. Toptal Projects enabled us to rapidly develop our foundation with a product manager, lead developer, and senior designer. In just over 60 days we went from concept to Alpha. The speed, knowledge, expertise, and flexibility is second to none. The Toptal team were as part of tripcents as any in-house team member of tripcents. They contributed and took ownership of the development just like everyone else. We will continue to use Toptal. As a start up, they are our secret weapon.
Brantley Pace, CEO & Co-Founder
I am more than pleased with our experience with Toptal. The professional I got to work with was on the phone with me within a couple of hours. I knew after discussing my project with him that he was the candidate I wanted. I hired him immediately and he wasted no time in getting to my project, even going the extra mile by adding some great design elements that enhanced our overall look.
Paul Fenley, Director
K Dunn & Associates
The developers I was paired with were incredible -- smart, driven, and responsive. It used to be hard to find quality engineers and consultants. Now it isn't.
Ryan Rockefeller, CEO
Toptal understood our project needs immediately. We were matched with an exceptional freelancer from Argentina who, from Day 1, immersed himself in our industry, blended seamlessly with our team, understood our vision, and produced top-notch results. Toptal makes connecting with superior developers and programmers very easy.
Jason Kulik, Co-Founder
As a small company with limited resources we can't afford to make expensive mistakes. Toptal provided us with an experienced programmer who was able to hit the ground running and begin contributing immediately. It has been a great experience and one we'd repeat again in a heartbeat.
Stuart Pocknee , Principal
Site Specific Software Solutions
We used Toptal to hire a developer with extensive Amazon Web Services experience. We interviewed four candidates, one of which turned out to be a great fit for our requirements. The process was quick and effective.
Abner Guzmán Rivera, CTO and Chief Scientist
Sergio was an awesome developer to work with. Top notch, responsive, and got the work done efficiently.
Dennis Baldwin, Chief Technologist and Co-Founder
Working with Marcin is a joy. He is competent, professional, flexible, and extremely quick to understand what is required and how to implement it.
André Fischer, CTO
We needed a expert engineer who could start on our project immediately. Simanas exceeded our expectations with his work. Not having to interview and chase down an expert developer was an excellent time-saver and made everyone feel more comfortable with our choice to switch platforms to utilize a more robust language. Toptal made the process easy and convenient. Toptal is now the first place we look for expert-level help.
Derek Minor, Senior VP of Web Development
Networld Media Group
Toptal's developers and architects have been both very professional and easy to work with. The solution they produced was fairly priced and top quality, reducing our time to launch. Thanks again, Toptal.
Jeremy Wessels, CEO
We had a great experience with Toptal. They paired us with the perfect developer for our application and made the process very easy. It was also easy to extend beyond the initial time frame, and we were able to keep the same contractor throughout our project. We definitely recommend Toptal for finding high quality talent quickly and seamlessly.
Ryan Morrissey, CTO
Applied Business Technologies, LLC
I'm incredibly impressed with Toptal. Our developer communicates with me every day, and is a very powerful coder. He's a true professional and his work is just excellent. 5 stars for Toptal.
Pietro Casoar, CEO
Ronin Play Pty Ltd
Working with Toptal has been a great experience. Prior to using them, I had spent quite some time interviewing other freelancers and wasn't finding what I needed. After engaging with Toptal, they matched me up with the perfect developer in a matter of days. The developer I'm working with not only delivers quality code, but he also makes suggestions on things that I hadn't thought of. It's clear to me that Amaury knows what he is doing. Highly recommended!
George Cheng, CEO
As a Toptal qualified front-end developer, I also run my own consulting practice. When clients come to me for help filling key roles on their team, Toptal is the only place I feel comfortable recommending. Toptal's entire candidate pool is the best of the best. Toptal is the best value for money I've found in nearly half a decade of professional online work.
Ethan Brooks, CTO
Langlotz Patent & Trademark Works, Inc.
In Higgle's early days, we needed the best-in-class developers, at affordable rates, in a timely fashion. Toptal delivered!
Lara Aldag, CEO
Toptal makes finding a candidate extremely easy and gives you peace-of-mind that they have the skills to deliver. I would definitely recommend their services to anyone looking for highly-skilled developers.
Michael Gluckman, Data Manager
Toptal’s ability to rapidly match our project with the best developers was just superb. The developers have become part of our team, and I’m amazed at the level of professional commitment each of them has demonstrated. For those looking to work remotely with the best engineers, look no further than Toptal.
Laurent Alis, Founder
Toptal makes finding qualified engineers a breeze. We needed an experienced ASP.NET MVC architect to guide the development of our start-up app, and Toptal had three great candidates for us in less than a week. After making our selection, the engineer was online immediately and hit the ground running. It was so much faster and easier than having to discover and vet candidates ourselves.
Jeff Kelly, Co-Founder
We needed some short-term work in Scala, and Toptal found us a great developer within 24 hours. This simply would not have been possible via any other platform.
Franco Arda, Co-Founder
Toptal offers a no-compromise solution to businesses undergoing rapid development and scale. Every engineer we've contracted through Toptal has quickly integrated into our team and held their work to the highest standard of quality while maintaining blazing development speed.
A Toptal director of engineering will work with you to understand your goals, technical needs, and team dynamics.
Work With Hand-Selected Talent
Within days, we'll introduce you to the right DevOps engineer for your project. Average time to match is under 24 hours.
The Right Fit, Guaranteed
Work with your new DevOps engineer for a trial period (pay only if satisfied), ensuring they're the right fit before starting the engagement.
How are Toptal DevOps engineers different?
At Toptal, we thoroughly screen our DevOps engineers to ensure we only match you with talent of the highest caliber. Of the more than 200,000 people who apply to join the Toptal network each year, fewer than 3% make the cut. You'll work with engineering experts (never generalized recruiters or HR reps) to understand your goals, technical needs, and team dynamics. The end result: expert vetted talent from our network, custom matched to fit your business needs. Start now.
Can I hire DevOps engineers in less than 48 hours through Toptal?
Depending on availability and how fast you can progress, you could start working with a DevOps engineer within 48 hours of signing up. Start now.
What is the no-risk trial period for Toptal DevOps engineers?
We make sure that each engagement between you and your DevOps engineer begins with a trial period of up to two weeks. This means that you have time to confirm the engagement will be successful. If you're completely satisfied with the results, we'll bill you for the time and continue the engagement for as long as you'd like. If you're not completely satisfied, you won't be billed. From there, we can either part ways, or we can provide you with another expert who may be a better fit and with whom we will begin a second, no-risk trial. Start now.
How to Hire a Great DevOps Engineer
What is a DevOps engineer? Clearly, there’s no one-size-fits-all answer to the question: Skimming DevOps engineer vacancies from just a few companies is enough to show how diverse the role’s requirements can be.
A DevOps specialist should be able to mold their skills into the unique mechanics of a company’s IT processes. A candidate is usually expected to be fluent in deployment automation and networks, understand software licensing, be wise in the ways of hardware and software, work with multiple operating systems, and write scripts in various languages to build reports and automate tasks. They will have to continuously communicate with teammates in order to identify and prioritize tasks and excel at documenting and demonstrating the tools they create.
Assessing so many areas of technical knowledge and communication skills in a single person is tricky. That is where this guide comes in: We’ll tell you what you need to know in order to successfully hire a DevOps specialist, give you example DevOps interview questions, and highlight the skills a candidate should be expected to demonstrate.
Let’s start with a fundamental concept that every DevOps specialist should be fluent in nowadays.
Virtualization and Sandboxing
Understanding virtualization is essential for all DevOps candidates. Many contemporary ideas and strategies in the DevOps profession were built on top of it, and are at work in many companies no matter their structure or purpose.
The idea behind virtualization is to abstract away the underlying hardware or software and make things less dependent and more isolated. It’s actually a broad concept and applies to servers, networks, storage, desktops, and even applications.
For example, a server operating system (OS) with all services and tools installed can be configured the same way as on a company’s real, bare-metal server, but instead be put in a virtual machine, or VM. Then VMs can be launched on developer laptops to mimic the same real structure and relationship between services and tools as in the server environment, even if the developers use different laptops for their work. This has become especially easy today with virtualization tools like Vagrant and lightweight container management systems like Docker.
Another example is how a complex network with multiple resources can be hidden behind a few virtual layers, then partitioned into manageable parts.
The same goes for storage virtualization: Multiple physical storage devices are virtually “joined” and appear as a single storage unit, and then they can be partitioned further as the company needs.
Finally, application virtualization will let the same applications run encapsulated on multiple operating systems.
Make sure [you ask] the candidate about how they actually saw these benefits in an implementation they've done themselves.
It is important that the candidate can explain the key benefits of using virtualization:
Greater control of IT resources
Faster setup and configuration of IT infrastructure
Faster release and recovery times
Better isolation, thus higher security
Lower risks brought by changes
Better testability of software and services
While it might not be required for the candidate to list all of the points, understanding why to use virtualization and where it can be implemented is essential. Without it, a DevOps engineer will be very limited in their possibilities and won’t be able to propose efficient solutions.
Besides the why, the how component is what will allow you to distill whether they’re a truly skilled person—make sure some of the points are demonstrable by asking the candidate about how they actually saw these benefits in an implementation they’ve done themselves.
With virtualization comes cheap and efficient sandboxing. To sandbox means to put a piece of software, like a running program, in an isolated box, and control what resources it can access and when.
Sandboxing is very useful for both security and testing. When it comes to security, this process can stop the spread of software vulnerabilities or system failures. In testing, it is a great way to test software before shipping it to live servers. In a sandbox, you can emulate various corner cases and test various user input, see how a program performs when overloaded with multiple simultaneous requests, automate Quality Assurance testing, and so on.
At the end of the day, virtualization and sandboxing allow you to develop faster, roll out new features more often and more seamlessly, and adapt to the constantly changing company course. Eventually they save money by reducing many unavoidable risks, and by automating plenty of routine processes.
Useful terminology includes virtual machine, host and guest operating system, container, and hypervisor. The candidate should be able to explain these.
Example interview questions:
What tools exist to implement virtualization? What are key players in the market today?
What would be impossible without using virtualization? What are the consequences?
What is the difference between VMWare and Docker?
How can you standardize the environment for developers so that they work with software as if it were on your live servers?
How can you release more often and be able to roll back in minutes, not in hours?
How do version control systems (VCSes, e.g. Git) work together with virtualization?
The last question from the list above is especially important. The idea here is that the VM configuration should be tracked in a VCS together with the project’s code. This way the correct configuration for a virtual machine can be reproduced for any given state of code as committed at that time. Tracking the application code together with the configuration of the system it’s running on makes it possible to reproduce bugs related to VM configuration and versions of OSes, packages, and libraries.
Now that we’ve checked some essential basics, let’s see what a DevOps specialist should understand about today’s hardware, including cloud computing.
Cloud Technologies and Hardware
Over are the days when owning hardware in order to run an IT business was in many cases a necessity. Nowadays, companies like Amazon and Google maintain hardware and rent it out, taking monthly payments that are a mere fraction of the original hardware cost.
Renting hardware instead of buying it is a popular way to go for many businesses today. It eliminates lots of hardware setup and maintenance tasks by outsourcing them to dedicated teams. This also speeds up launching a business, or building a proof-of-concept product, or providing clients with instant dedicated trial setups. It also allows you to focus on the business logic straight away instead of spending time on setting up your own hardware.
A company can even build their own, internal cloud, by joining their hardware into a single network and abstracting over it. Hybrid colocation-cloud solutions are also popular. One way or the other, cloud computing will be present in the day-to-day job of a DevOps specialist. That is why assessing their relevant knowledge and skills will help you make sure your company won’t miss on the speed, flexibility, and scale that cloud computing offers.
To start with, ask the candidate about the benefits that cloud computing brings, like:
Reduced setup and maintenance cost
Ability to pay for what you use only, and add more resources as you go
Automatic software updates by suppliers
Robust disaster recovery
Even though the question might be less practical and more philosophical, it is always good when people have a general understanding of the technology and its goals. If this is the case, they can learn faster when needed, and, having a broader understanding, will have more intelligent applications of existing technologies to your business.
When it comes to introducing cloud computing to the company’s IT infrastructure, different service models are proposed by providers. Have your DevOps candidate say a few words about the differences between the standard models, and give examples:
Infrastructure as a service (IaaS)
Platform as a service (PaaS)
Software as a service (SaaS)
It’s also worth asking about their experience using any of these models.
Next, discuss your business processes in the context of those models. For example, if your company sends emails heavily, can it use a SaaS solution like the Amazon Simple Email Service, or should it set up its own service hosted using third-party IaaS? What would be the difference? What are the limitations of both? How easy will it be to transition from one model to another? Will there be any lock-in?
Another example is implementing an internal project management or a public user support system. In both cases, there are many solutions available on the market—you’ll rarely want to build your own from scratch. Some of them are hosted and maintained by providers, and DevOps will take part in integrating it with the other software used by the team. Others should be downloaded, installed, configured, and updated regularly, which is also up to the competence of the DevOps team.
Therefore, you should expect the candidate to effortlessly answer questions like these:
What is a cloud service API, which pricing models exist, and what is an API usage quota?
Which of IaaS, PaaS, and SaaS are applicable to our company? (This implies the candidate read about your company and its goals before submitting their application.)
What are the risks of renting hardware instead of owning it?
What are the risks of using your own hardware, and when is it the only available choice? (Hint: Unreliable internet service is a prime example.)
In which cases will buying your own hardware benefit the company?
Which compliance issues are related to using cloud services, and how can you check whether a given service is compliant with company policies, standards, and existing certifications?
Because DevOps might alter the company infrastructure with time, it’s important that they understand standards compliance and certification. If your products or services are regulated in the company’s country of residence, DevOps will need to cooperate with your Information Security Engineer to maintain existing compliance or apply for new compliance. If this is the case, let the candidate talk about their experience in the field: How they accomplished compliance and what their role was in the process.
Here is an idea for a quick practical task that will help you assess the DevOps candidate’s knowledge in compliance and certification. If you previously applied for any, you will have lots of application forms filled out. Take an excerpt from one of them and remove all the answers, and give it to the candidate. Let them:
Explain the meaning of the questions, and
Tell you the questions they will ask the infrastructure, security, and developer departments in order to get the information needed to fill out the questions in the form.
Even if you have a dedicated compliance specialist, chances are that DevOps will be heavily engaged in both compliance application and validation. If the DevOps candidate lacks understanding in those, you are risking your compliance status.
Back to the hardware topic, knowledge about it pays off. One of the ways to unlock full software potential is by using the right hardware and combining it in the right way. For example, look at how Stack Overflow served 150 million HTTP requests in 2013 and 210 million in 2016. In the hardware list, you’ll notice that four Microsoft SQL Servers were used in 2013, and there are still only four of them in 2016. The number of the database servers did not increase, and only two of the servers got upgraded. This means that there was an intelligent plan behind the scenes. Not only does it save money on hardware directly, but also on hardware installation, colocation, and maintenance.
So, depending on the business processes in your company, you may want to ask the DevOps candidate some of the following questions:
Which types of software require more of the following?
Processing power (CPU speed)
CPUs or cores (for parallelization)
Give an example of redundancy in hardware setup. (You can use the Stack Overflow article as a guide to familiarize yourself with one possible use case.)
What is load balancing and when it is needed?
Your company may own zero hardware and use cloud services for its whole IT infrastructure, yet to make the best choices, it is important that a DevOps developer can measure hardware performance and understand how to tune software to work best with the given hardware. This skill is especially important if you don’t hire a dedicated server administrator.
Next, we’ll look at something you’ll see in every DevOps job: Eliminating regular repetitive manual work when testing and deploying software.
Build, Test, and Deployment Automation
The process of developing software has transformed considerably over the past decade thanks to the creation of new developer tools and the introduction of automation to many processes—especially when it comes to building, testing, and deployment. This is often referred to as “Continuous Delivery” (CD), but it’s worth learning about the relationship between DevOps and CD before you start interviewing a DevOps specialist.
With CD, teams tend to ship new features more often, implement big tasks in stages by dividing them into smaller subtasks, test new ideas by rolling them out to a limited number of clients, and easily roll back when a problem gets identified in a new release. All this becomes possible with the help of a DevOps developer and allows businesses to be more flexible and adapt to always-changing markets faster and with less stress.
DevOps engineers have a key role to play in setting up, adjusting, and maintaining CD. Here’s how it works.
Step 1. The DevOps team consults other teams in order to gather information about how the development, testing, and deployment processes are set up now.
Step 2. The DevOps team learns the planned features, assesses the company’s growth speed, estimates the range of current expenses on infrastructure, finds out the most time-consuming tasks in the company—all these being factors that will affect what exactly should be automated and improved, and in which order. Even though this section is focused on automating test and deployment, the two are usually an integral part of all other processes and affect them. So the more tech stack research, the better decisions can be made about test and deployment automation.
Step 3. DevOps engineers plan and prioritize their automation implementation tasks in such a way that the most critical and/or expensive processes are tackled first. At this stage, it is usually heavily discussed with the leads of other teams in order to make most out of the test/deployment automation for everyone.
Step 4. Automation is implemented in stages.
For companies with continuous delivery already in place, there might simply be a todo list ready for a new DevOps employee to take over. Yet it is still beneficial to go through the steps above—it never hurts when the DevOps engineer learns the overall setup and can propose improvements to automation while working on the tasks planned.
Many tools and concepts are at work when it comes to automating builds, tests, and deployments. It’s worth considering which are the most relevant ones to discuss with the candidate as we explore the categories below.
A unit tests is a specially developed script that tests the smallest testable parts (units) of the application for proper operation.
More often than not, unit testing is automated. The job of the DevOps team is to integrate unit test runners into the virtual machines used in development as well as CI servers so that the tests can be hooked in to run automatically for every new version before it is released, or triggered manually by a developer or tester. Reporting should also be configured so that a developer finds out instantly if their new code breaks existing functionality.
Furthermore, for large applications with lots of unit tests to be run on a regular basis, DevOps developers usually set up dedicated continuous integration servers where unit tests are run in parallel while developers continue their coding in their dev environments. Without such CI servers, when run on a single developer laptop, unit test automation can sometimes take hours, if not longer.
There are various types of tests:
Functionality tests: Do all the key app features work?
Compatibility tests: Does the app maintain its functionality in various environments? For instance, a web application should work in all supported browsers, an app that supports multiple databases should consistently work with all of them, a driver should function correctly with all supported hardware models, and so on.
Performance tests: How does traffic or other types of loads affect the app’s performance?
Security tests: Can attacks reveal flaws in the app’s security mechanisms, so that protected data is accessed without authorization?
Usability tests: Can the app’s users intuitively understand how to use it without reading its documentation? Does it work as expected?
Some of the tests from the above groups can be joined further into a “smoke test” (also acceptance or sanity test)—a test executed before a release to reveal whether the app meets a certain standard.
DevOps Tools for Build Automation
A DevOps developer will typically use build automation software—and the list of such pieces of software is huge. A good question to a DevOps candidate would be to let them talk about which of these tools they’ve used to solve which tasks. The tools can be categorized in two ways:
Tools specific to a programming language. For example, Rake is a Ruby-based build tool while Apache Ant is popular for Java.
Tools specific to an OS: FinalBuilder is for Windows software developers, Make-based tools (like GNU make and mk) is for Linux/Unix and Mac OS.
It is important that the DevOps engineer who is going to work in your team is familiar with build tools specific to the OS and programming languages used in your projects.
Continuous Integration, Delivery, and Deployment
Often people conflate and confuse these three terms, so let’s go through them quickly first:
Continuous integration helps to keep the mainline (the base branch of the project’s code tracked in a version control system like Git) up to date. It’s achieved by merging all developer working copies into the mainline often, i.e. a few times per week (or more) vs once per release.
Continuous delivery makes sure a deployable working state of an app is produced often, i.e. by small increments. This way you have the confidence that an application can be deployed to production when the business is ready for it.
Continuous deployment is the process of putting the app into production. The continuity of integration and delivery is a prerequisite for the deployment stage.
All three practices are possible nowadays thanks to automation and are implemented by a DevOps engineer. All three require scripting skills and familiarity with corresponding tools. That’s why you should include this topic in your interview.
Before you talk about it with the candidate, it’s important to find out from your developers, system administrators, and other technical teams about the state of the subject in your company. If any of the integration, delivery, or deployment schemes are already in place, make notes about the tools and scripting languages at work to discuss with your candidate. If none are implemented, ask the candidate if they have implemented any for projects using the same stack of languages and frameworks as yours.
It’s not critical that a candidate has experience with every last technology on your list. That’s because there are many different tools for various cases, and the main point is the understanding of their purpose, as well as being able to learn them and start using them quickly. A consultant with enough experience is usually capable of learning about new tools in a matter of days.
Because so many servers, services, microservices, databases, etc. form the building blocks of an application, it would be risky not to monitor the operability and availability of them all continuously and rigorously.
A DevOps engineer must be able to set up monitoring themselves, of course. But it’s no less important that they can:
Analyze failures and understand their causes
Fix failures and restore functionality
Bring in measures to prevent such failures in the future
Outline all the details in internal reports and improve internal documentation with notes, tips, and procedures related to a given issue
Advise the product team on improving the code that caused the issues
Help the support team explain the issues in simple words in blog posts and in support tickets
Ask the DevOps candidate to talk about how they set up monitoring and then handled some failure. From their story, you will be able to identify whether they are familiar with the points outlined above.
You should also ask the candidate about:
What software they used for monitoring
Whether they used third-party services (like Pingdom) or open source solutions (like Zabbix and Nagios), and
Finally, when it comes to performance and failure analysis, being able to configure logging and read log output is the key to effective handling of failures.
Here is a vocabulary of terms related to monitoring for you to get familiar with before interviewing a DevOps candidate:
Server uptime monitoring. This means testing the availability of the company’s website, applications, servers, and services.
Uptime test locations. It makes sense to monitor website availability from different locations, especially from those where most of the website users reside.
Status pages. These display both current and historical data of website or service uptime. There might be public and private status pages—the latter usually face the product team, the server administrators team, and other engineers, and include comprehensive monitoring details and statistics used to manage the whole infrastructure. It’s usually the responsibility of a DevOps specialist to set up such pages.
Alerting. I.e. informing responsible parties through various channels when events occur. Well-thought-out alerting decreases your time to resolution.
Log rotation. This simply means archiving dated log files on a schedule.
Log server. This is a dedicated server used to collect and store all logs. Having one simplifies the process of searching log data, and allows you to filter it to quickly audit the system.
One good question to ask about page speed optimization is: What measures can a DevOps specialist take to optimize page speed? It’s true that many improvements are done by developers inside the code—but there are some a DevOps engineer can do as well:
Enabling compression at the web server level
Improving server response time by fine-tuning web servers (like Apache and Nginx) and load balancers
Enabling the usage of a content delivery network (CDN)
What we just discussed above is usually called performance and availability monitoring. But there is another aspect of monitoring: Security monitoring. The idea is that all operations and activities of users and programs must be tracked.
No matter how big or small your clients are, your company is responsible for the confidentiality and security of the personal data you store on your servers. The same goes for:
Intellectual assets, like your application’s source code. You want to store it securely and not leak to your competitors.
Corporate email and other communication channels, including user support. You don’t want everyone in your company to access those.
Internal documents, like competitor analysis, development strategy, financial reports, etc. It’s strategically important to limit and control access to them.
Therefore, it is critical to observe and record all operations and activities at all levels of the company infrastructure, from networks to applications to employees.
You don’t want a DevOps engineer to run an ill-designed SQL query over a database with 500 million records and make it freeze.
Solving the tasks discussed above is only possible using programming. In fact, you should expect the DevOps candidate to have coding skills in multiple programming languages.
Let’s elaborate: For one thing, scripting in bash (a Unix/Linux shell and a command language) is widely used for automating server software setup and configuration, backups, builds, and deployment; running monitoring; building reports; and consuming cloud APIs. Indeed, any kind of automation will need scripting.
For another, to smoothly integrate various system services with the application, the DevOps engineer you hire will sometimes need to know the programming language used in the application.
A good example is a multitenant PHP, Ruby, or Python web application solution running in a cloud virtual server provided by Amazon Web Services (AWS). Here are some typical tasks that will require a DevOps specialist to employ his or her coding skills:
Create a bash script to instantiate a new VPS server at AWS. The script will need to consume the AWS API to create a new virtual server under the company’s account, deploy the application to it, and create a database schema and configure application accounts in the database. It may also be a requirement to connect to a CRM (like SalesForce) and update a user record with the server ID.
Code an internal method in the application that reports the application state to an external monitoring script.
Implement an internal web page that reports on the status of all servers, and shows some usage statistics.
Make changes to an existing production Dockerfile (if Docker is used for containerization).
Create a bash script that makes scheduled backups, as well as restores from backups by request.
Create a bash script to roll out a new application version to some (or all) servers, as well as roll it back in case a critical bug is discovered in the new version.
Actually, there are many more such tasks in the day-to-day work found in a DevOps job. That is why there is no clear boundary between developers and DevOps—they both are able to write code. The former are just better at algorithms and programming business logic in a given domain, while the latter have wider knowledge in system administration. But many tasks can be done by both.
So, it is not a bad idea to include programming questions and problems in a DevOps interview. At a minimum, you should ask the candidate which programming/scripting languages (besides bash) they’ve used.
Another option would be to use online coding tests. There are plenty services with pre-defined tests (of varying complexity) for different programming languages and domains. Some of the services would also allow you to build your own tests, thus making it possible to select candidates with coding skills specific to your company. Whether you rely on pre-built tests or compose your own, you will need to consult your product/developer team to make the test questions relevant.
Sometimes, DevOps developers are asked to extract or examine data from a database that only they have been granted access to. For these cases, being capable of writing SQL queries will make a difference. So, if your project uses a database, you may also want to discuss the candidate’s experience working with your particular database engine. You don’t want a DevOps engineer to run an ill-designed SQL query over a database with 500 million records and make it freeze.
Configuring SQL table partitioning is a complex field—c.f. the documentation on PostgreSQL table partitioning. If your DevOps hire will need to get into this, knowing SQL syntax is not enough. They’ll need a deep understanding of indexing and factors that affect database performance, and you’ll need to hear about their prior experience in it.
Version Control and Branching Strategies
It’s difficult nowadays to find an IT project that is not managed by Git, Subversion, Mercurial, or another piece of version control software (VCS). From documentation to configuration to code to database schemas—as well as revisions made to documents of any kind—such things are usually stored in and tracked using a VCS.
Thus understanding how to use VCS systems is essential for any candidate.
As per the 2018 Stack Overflow Developer Survey, the most popular VCSes among developers are by far Git (87.2%) and Subversion (16.1%). It’s usually enough for the candidate to have experience with at least one of them. While Git has become the de-facto standard for the IT industry, SVN is still used in some recognized projects—WordPress among them, for example, as of 2018.
The idea behind all VCS systems is the same—to allow for the management of changes and branching.
The management of changes, or revision management, allows you to track every single change a given document or code file, and restore it later at will. Every change tracked in VCS makes it possible to restore the original source tracked at that time, i.e. it acts as taking a snapshot of the whole project, documentation, database schema etc. Such a snapshot is usually called a commit in VCS terminology.
Branching is a technique that allows for multiple people to simultaneously collaborate on the same source effectively—it reduces the number of conflicts between revisions and automates the merging of work made by multiple people (or by the same person who worked on multiple features simultaneously.)
Because there are different approaches to branching (also called branching strategies), like Git Flow and GitHub Flow, it won’t make much sense to require the candidate to know more than one: If they understand how one of them works, they are likely to easily learn any others.
Now, let’s look at why it is important for a DevOps engineer to be able to use VCS. In fact, there are many things done by them that require VCS skills:
Deploying a particular version of an application, or reverting back to one of its older versions
Deploying a particular branch containing a particular feature to a Continuous Integration server, so that tests can be run and reports generated
Comparing two versions of the software to identify any changes that may affect compliance (when the person you want to hire for DevOps will also play the role of a security engineer)
Tracking internal scripts and corresponding documentation in VCS
Git is a sophisticated tool that offers many possibilities. Of course, it offers basic revision tracking, but it also has advanced features that help solve complex tasks. For example, the git bisect command allows you to efficiently pinpoint which commit introduced a bug. Like with programming languages, one can improve one’s skills in Git over time by solving various tasks and reading documentation and tutorials. Therefore, you should not expect a DevOps candidate to have comprehensive expertise in VCS. It should be enough to ask the candidate about just two things:
Whether they have used VCSes in the past, like Git, Svn, or Mercurial
To find out if they are familiar with any branching strategy
Even though it is easy to come up with short tasks to let the DevOps candidate demonstrate their skills in using VCS, it’s probably not necessary if you intend to have them do a trial project—just integrate the requirement there instead of testing separately.
To help you evaluate the quality of commits and branches in the trial project’s VCS repository, you should ask those who use your company’s VCS the most—your developers (the product team). Still, here is a list of things to turn your attention to when checking the repository:
Commit messages (i.e. the messages accompanying commits/snapshots) should be descriptive and concise and follow the same style
Every single feature, fix, or change in documentation should be committed separately (the “commit often” principle)
There should be no commits with half work done (or it should be identified in the commit messages)
No “log” files or other files unrelated to the project should be tracked
Best practices for using VCS have been developed over time and described in articles like this—at least skim such a resource if you will be the one reviewing the trial project.
IT Security Best Practices for DevOps
DevOps drives so many aspects of a company’s development process. So it’s naturally the perfect place to inform the wider implementation of security practices and cultivate security awareness in all teams. With that in mind, it becomes obvious why security is another area you’ll want your DevOps candidate to excel in, especially if you are not backed by a dedicated security team.
As IT solutions are employed more and more by industries, the number of penetration possibilities is growing as well: From hardware to software to firmware to drivers to business solutions, each one is a separate item that needs protection and security monitoring. As a result, the demand for security specialists is rising like never before. That is why a DevOps developer who is also good at security is a very valuable asset.
The first recommendation would be to put the requirements related to IT security up front, right in your job advertising. If you are ready to accept a candidate who has no concrete experience in implementing security best practices when solving real-world problems, it would still make sense to mention that it is required to learn the topic, and/or to make it a requirement in the personal yearly education plan.
Next, when interviewing the candidate, closer to the end of the interview and after familiarizing them with the technology stack your company uses, ask the candidate to list a few measures they would take to improve your overall security implementation. Here are a few good examples you might expect:
Send weekly security awareness letters—as all employees are human beings susceptible to various types of attacks, and as new vulnerabilities are discovered regularly, a weekly letter that sums up the findings and gives advice may make the company less prone to such attacks.
Launch a dedicated security channel—setting up a dedicated flow in Slack, Flowdock, or any other corporate chat is a good way to publish timely announcements and discuss them.
Develop a security education plan—it never hurts to study security formally; this might be just short online courses taken twice per year. Taking courses has also a good side-effect of team building.
Build the list of tools, software, and cloud services used in the company—and watch them closely for new vulnerabilities.
Use VPN, enforce password rotation—it is a common practice to hide internal resources, like project management software and source code hosting, behind a VPN.
Set up security monitoring—as discussed in the Monitoring section above.
Finally, if you plan a trial project as part of the interviewing process, make sure to say in the project description that security best practices will be assessed in the deliverables, if that applies to the project. (Chances are that it does, whatever project you come up with.) Then ask your security/product team to review the project and have their say.
As with any job involving collaboration, communication skills are one of the keys for the team to progress quickly as a whole. DevOps specialists are no exception to this requirement.
There are many areas where DevOps professionals will apply their communication skills in their day-to-day work:
Collecting requirements from different teams (Product, Customer Success, Sales, etc.)
Documenting and demonstrating the tools and solutions they create
Brainstorming to find solutions in architecting software and hardware infrastructure
Reporting to stakeholders
Your Human Resources department can help you filter out candidates who have obvious trouble with communication. It is not uncommon for highly qualified and experienced professionals to leave companies because of communication issues. So it’s important to identify them at an early stage before you sign a contract.
At the very minimum, both during the interview and when evaluating the trial project, make sure the DevOps candidate:
Can express their thoughts and ideas clearly, both verbally and in writing
Shares the value of moving fast as a team vs moving fast as an individual
Understands the culture of sharing
Is willing to work towards a joint target
Collaborativeness is a very important personal trait. While not being testable per se, it will affect how fast the candidate will get involved in the teamwork. Thus, if a few candidates have approximately the same level of technical expertise and skills, prefer the one with the better communication and collaboration traits.
Here are a few ideas for topics to discuss during an interview:
Have you worked remotely (or in an open space, depending on your company’s environment) before?
Describe how you presented a tool and/or process you developed as a DevOps engineer.
What tools did you use for collaboration and communication with your teammates?
Screening DevOps developers is one thing. What about adapting candidate requirements to your company, its type, and its structure?
Finding a Perfect Fit for Your Company
There is a conceptual difference between hiring a DevOps manager who will lead major transformation in the company and recruiting an engineer to join an existing team in a DevOps-established company.
To establish DevOps practices in a company from scratch, you will want a candidate who:
Previously worked as a Lead DevOps Engineer, or
Spent a few years doing DevOps in a medium or big company as part of an existing DevOps team
The idea here is that setting up DevOps infrastructure in a company that does not have it requires, besides theory, a lot of skills and empirical knowledge. Once the infrastructure is established and relied on by the other teams in the project, it will be too costly to radically change it. This means the cost of errors in design and structure is very high in DevOps. Think of it as designing software from scratch vs hiring someone to just add more features to existing software. Same goes with the DevOps profession—you should not risk betting on a person who has limited real-world experience in the field.
If, on the other hand, the new hire is to be guided and checked by an existing DevOps team, the requirements on experience may be loosened, and more focus put on attitude and communication skills.
In any case, it is important that the candidate will be able to plug into the company’s product framework, so they must be familiar with some or all of the company’s technology stack. Those include:
Programming languages (like PHP, Ruby, Java, or Scala)
Application frameworks (like Laravel for PHP or Play! for Scala)
Operating systems (OSes) used on servers and other company hardware (like Ubuntu, Red Hat Linux, or Windows Server)
Databases (like PostgreSQL, MariaDB, Oracle, or Microsoft SQL Server)
Tools (like backup tools and asset compilation tools)
Many times a DevOps engineer will need to develop a plugin for an existing framework using the corresponding programming language. As this will happen more often than not, their familiarity with the company’s technology stack is truly crucial.
A simple way of keeping track of it is consulting your teams and building a detailed list of the technologies used at your company, then including it in the job advertisement. If that list is considered confidential, present it to the candidate straightforwardly at the beginning of an interview instead. Such a list with items checked per candidate will come in very handy when comparing candidates—sometimes that list will make your choice obvious when comparing other traits does not help.
When it comes to deciding on interviewing strategy, the following rules of thumb apply:
Split interviewing into a few stages, and filter out by the traits critical for your project sooner than later, as well as eliminate candidates who are troublesome in terms of communication.
When interviewing, always involve people from various departments, not just the product team (software developers). Let them formulate and ask questions that are important to them, then collect their feedback.
As the last stage, assign a trial project.
Trial projects are a great way for candidates to show off. For some people, especially introverts, going through interviews means a lot of stress, so you might not be able to see their full potential until they deliver something amazing through a trial project.
Finally, because DevOps and software development have many things in common, we can learn a lot from the latter.
Fact 22. “Eighty percent of software work is intellectual. A fair amount of it is creative. Little of it is clerical.” The right DevOps engineer, just like a software developer, will create a lot of valuable solutions that will be part of your company’s intellectual assets.
Fact 23. “One of the two most common causes of runaway projects is unstable requirements.” That is why your DevOps candidate’s ability to continuously and effectively communicate with the project teams is as important as having technical skills.
Fact 27. “There is seldom one best design solution to a software problem.” Here, also being about software problems, DevOps requires some intuition and empirical knowledge. Again, it’s vital to hire an experienced DevOps engineer when the global task is to setup DevOps infrastructure in the company.
So there you have it: Build a hiring plan. Focus on your company needs. Listen to your team feedback. Make a skills and quality comparison table. Identify weaknesses and plan a continuous self-education schedule including follow-ups. Use an online service for hiring management. In the end, investing in the DevOps hiring process means benefitting afterward:
“Easy choices, hard life.
“Hard choices, easy life.”
- Jerzy Gregorek