Senior Infrastructure Engineer
Toptal is a global network of top freelance talent in business, design, and technology that enables companies to scale their teams, on-demand. With $200+ million in annual revenue and over 40% year-over-year growth, Toptal is the world’s largest fully remote company.
We take the best elements of virtual teams and combine them with a support structure that encourages innovation, social interaction, and fun. We see no borders, move at a fast pace, and are never afraid to break the mold.
We are looking for an experienced engineer to build and scale services in a cloud environment within our Infrastructure team. Our Infrastructure Engineers work with a high-energy, fast-paced team responsible for supporting initiatives and operations across Toptal. This is a remote position that can be done from anywhere. Due to the remote nature of this role, we are unable to provide visa sponsorship. Resumes and communication must be submitted in English.
Toptal services are deployed across hundreds of servers. You will be responsible for designing, building, deploying, and maintaining highly available production systems, with shared ownership with the development teams.
- We are embracing DevOps practices, where the Infrastructure team develops systems, automation, tooling, and workflows and has a consulting/mentoring role in enabling developer teams to own the whole lifecycle of the software they are making.
- Implement monitoring for automated system health checks, develop procedures, and maintain system troubleshooting and maintenance documentation.
- Collaborate regularly with engineering teams to improve the company’s engineering tools, systems, procedures, and data security, not just administer clusters and cloud services.
- Join daily scrum standups (GMT-3 to GMT+5). Expect pair programming, engaging in peer code reviews, and using collaboration tools like Slack and Zoom.
In the first week, expect to:
- Join our team and get introduced to Toptal’s practices and processes.
- Learn about our team’s processes and get familiar with the code that maintains our infrastructure resources.
In the first month, expect to:
- Gain insight into our systems by learning why they are built the way they are and how to improve them.
- Monitor systems security, performance, and availability.
- Begin to learn a variety of roles in a wide range of Infrastructure projects.
In the first three months, expect to:
- Start working on support tasks to get familiar with the core tools and setup and everyday challenges.
- Provide excellent customer service by understanding and addressing the teams’ needs and expectations through effective communication and collaboration while learning about our infrastructure.
- Deliver internal Infrastructure and services such as monitoring, logging, automation, and data services targeted at our internal users.
In the first six months, expect to:
- Support Infrastructure design, architecture, and implementation support.
- Have opportunities to be involved in systems design, identify new technologies to support the business, and resolve infrastructure compatibility and performance problems as they arise.
- Participate in the on-call rotation schedule (during business and after hours) to support all infrastructure-related systems.
- Report any downtime or performance issues faced by the system, drill down to find out what caused it, and coordinate with other teams to resolve them.
- Handle incident resolution if a developer is not needed.
- Participate in our Disaster Recovery, change control, and security standards initiatives.
In the first year, expect to:
- Communicate with key partners on project engagements.
- Partner closely with our teams in the engineering area to develop infrastructure automation and management solutions with a strong focus on scalability, observability, automation, reliability, security, and quality in Google Cloud Platform.
- Plan and coordinate testing of changes, upgrades, patches, new releases, and new services.
- Participate in technology initiatives that enable developers to deliver their services to our customers with minimal friction and a high degree of quality.
- Solid understanding of Linux debugging, networking routing, IP addressing, Load Balancing, and VPNs. We don’t run our own AS and maintain thousands of BGP Routes but please do know how traceroute works or when to choose L4/L7 balancers.
- Previous experience managing infrastructure configuration and provisioning through code for large, distributed systems on public cloud platforms (preferably AWS and GCP but if you have had experience with Azure, Oracle Cloud, DigitalOcean, Linode, and others, that’s fine too).
- Knowledge of Infrastructure-as-Code principles and hands-on experience with automation tools. We prefer Ansible and Terraform but if you know Puppet or Chef very well, that’s great too. Basic understanding of version control is needed because our code is kept in Git.
- Hands-on experience with system and application metric collection and alerting services like Graphite, Grafana, Prometheus, InfluxDB, Sensu, or others. A keen focus on what makes a system observable.
- Experience running RDBMS, especially PostgreSQL experience is an added advantage. You generally won’t need to manually optimize SQL queries but it’s important to have a solid understanding when things break or aren’t performing as well as needed. Transferable knowledge from MySQL, SQLite and others is great too.
- Outstanding troubleshooting skills. Experience in resolving complex problems through various troubleshooting protocols and processes.
- Be eager to help teammates, share knowledge with them, and learn from them.
- A strong understanding of modern systems and service oriented architecture.
- Proficient in scripting languages such as Python, Bash, Ruby, etc.
- Experience with Docker, Docker Compose, and building optimized Dockerfiles is an advantage.
- Experience with Kubernetes environments: production operations, troubleshooting, debugging, cluster provisioning, and management is an advantage.