
Bob Clarke
Verified Expert in Engineering
DevOps Engineer and Developer
Bob is a highly experienced DevOps lead with a track record of designing and building large-scale, highly available systems within the financial sector, both cloud-based and on-premise. Bob has done container orchestration (Kubernetes), infrastructure as code, CI/CD, service mesh (Istio), TCP/IP networking, performance engineering, and system telemetry. He has worked with large programs and small greenfield projects alike. Bob is passionate about building robust solutions to deliver value.
Portfolio
Experience
Availability
Preferred Environment
Linux, Kubernetes, Azure, Helm, Docker, Bash, Terraform, Istio, Prometheus, Distributed Tracing
The most amazing...
...thing I've achieved is the design and implementation of a large-scale production Kubernetes infrastructure to host my clients' global microservices platform.
Work Experience
DevOps Lead
Dentsu International
- Designed and built a large-scale production Kubernetes platform in Azure Kubernetes Service (AKS) to host my client's global microservices platform. Currently in the process of implementing Istio service mesh on this platform.
- Established and led a Platform Engineering and DevOps team of 16 engineers responsible for infrastructure design, build automation, operations, and DevOps tooling/release management.
- Built a continuous delivery and release management system to support over 25 development teams running over 120 pipelines. The tech stack used is Brigade.
- Established standard infrastructure as code (IaC) patterns and developed parameterized Terraform modules for all components used on the platform.
- Implemented monitoring and centralized logging for my client's Kubernetes estate using Prometheus, Grafana, and Loki. Currently in the process of adding distributed tracing by leveraging the native support for this that is built into Istio.
- Built and automated a large-scale Kafka platform on Kubernetes to support my client's workflow platform. Developed a new pattern for exposing Kafka broker endpoints via Kubernetes Ingress.
- Defined architecture patterns and engineering standards for the migration to the Istio service mesh.
Senior DevOps Engineer
Lloyds Banking Group
- Developed automated build scripts for WebSphere Liberty.
- Developed Splunk dashboards for platform telemetry.
- Provided support and training for the IBM UrbanCode platform.
- Built Jenkins pipeline libraries to provide a common set of functions for development teams.
DevOps Lead
HSBC
- Managed the infrastructure design, DevOps tooling, automated environment builds, and configuration management for HSBC's DTC program.
- Developed a fully automated, on-demand runtime to provide dedicated CI/CD machinery, including dedicated pipeline tooling and a set of VMs for test environments, for each development team.
- Developed a library of Terraform modules to facilitate the automation of each system component, such as Postgres, Redis, Nginx, and more.
Engineering Lead
Lloyds Banking Group
- Acted as engineering lead for LBG's UrbanCode automation program. The program objective was to deliver an automation platform to provide deployment and configuration management for my client's retail online banking platform.
- Developed automated builds for WebSphere Application Server, DataPower, and F5 BigIP.
- Defined engineering standards for the production of the WebSphere infrastructure. This included automation, network security, high availability, monitoring, and capacity management.
Engineering Consultant
Deutsche Bank
- Designed and built Deutsche's strategic business process management (BPM) platform for the DB Palace project.
- Developed automated builds for BPM platform and rolled out across route to live and production.
- Created automation to test and release BPM workflows.
- Built a custom web-based monitoring and inventory system.
- Worked closely with development teams post-implementation to iron out operational issues.
- Produced system documentation and run-book, carried out handovers and system sign-offs, and provided technical training for Deutsche Bank support teams.
Lead Technical Architect
PwC UK
- Designed and implemented a business process management infrastructure based on IBM's business process management (BPM) system.
- Developed automated build scripts and rolled out test and production environments.
- Provided training for in-house teams and produced system run-books for support teams.
Senior Technical Consultant
Fidelity International
- Led the migration for the Fidelity FundsNetwork platform from WebSphere version 4.0 to 6.1.
- Designed and developed a toolset for WebSphere's automated delivery and configuration management.
- Designed and developed a browser-based inventory tool that provided a dynamic view of server utilization, configuration, and software inventory across the server estate.
- Carried out the platform's day-to-day systems administration, support, and monitoring.
Market Data Specialist
Tullett Prebon
- Designed and built a new Reuters Market Data System (RMDS) platform and migrated 400 trading floor positions from the legacy system.
- Developed automated build and configuration management scripts for the platform.
- Created a browser-based tool for monitoring key systems metrics and usage data.
Market Data Specialist
Dresdner Kleinwort Wasserstein
- Provided operational support of the production RMDS infrastructure.
- Provided system design and engineering services for in-house development teams.
- Developed a browser-based market data usage and cost reporting tool.
Market Data Specialist
NatWest Markets
- Migrated existing RMDS infrastructure from a standalone network to my client's corporate LAN.
- Co-designed and developed a browser-based monitoring and management tool using Perl/CGI.
- Provided day-to-day support for trading floor users.
Experience
Large-scale Production Kubernetes Platform
As part of this project, I designed and automated the build of a new internet landing zone based on Azure Application Gateway and migrated to this from my client's legacy F5-based system.
I automated TLS certificate creation and renewals using Let's Encrypt and implemented system-wide monitoring and logging using Prometheus, Grafana, and Loki.
I also set up a CI/CD toolchain to support a global team of approximately 200 developers, and I automated all builds using Terraform. During the project, I established and documented a set of IaC standards for adoption by other engineering teams.
Production Kafka Platform
Automating Kafka and its related components was fairly straightforward. We chose Confluent Kafka and this came with a set of Helm charts. I needed to modify these Helm charts as they did not ship with Ingress resources (which I needed in order to comply with our existing Kubernetes Ingress standards). This being the case, I forked the chart code and hosted the packaged charts on our internal Chart Museum service.
The biggest and most interesting challenge was exposure to the internet via Kubernetes Ingress. Although the patterns were available in the community for exposing Kafka endpoints to the internet via Kubernetes services, there did not appear to be any existing patterns for doing so via Kubernetes Ingress. With this in mind, I set about forming a new pattern that leveraged Nginx TCP services. This worked very well, so I formalized the pattern with our architecture team and automated the build with Terraform.
Istio Rollout to Production
• Granular, policy-based network controls (east-west and egress)
• JWT validation at Ingress Gateway
• Transport encryption between pods
• Telemetry
I was the technical lead for the project; the key challenges I had to overcome were as follows:
• Migration of Ingress resources to Virtual Services in 200+ existing Helm charts
• Cutover from Ingress controllers to Istio Ingress Gateway
• Auditing services that would be MESH_EXTERNAL and configuring necessary Istio ServiceEntry's
• Developing a Terraform module for Istio deployment
• Updating Helm charts for Kubernetes Jobs to ensure Istio sidecar exit
• Integration of Kiali with existing Prometheus estate
Skills
Tools
Terraform, Helm, Istio, GitHub, Ansible, Grafana, Splunk, Azure Application Gateway
Paradigms
DevOps
Platforms
Kubernetes, Docker, Linux, Unix, Azure, Apache Kafka
Other
Infrastructure as Code (IaC), TCP/IP, CI/CD Pipelines, Shell Scripting, Container Orchestration, Prometheus, HashiCorp Vault, Loki, IT Infrastructure, Transport Layer Security (TLS), PKI, Content Delivery Networks (CDN), Distributed Tracing, Env0, Containers
Languages
Go