Systems Developer in Dublin, Ireland

A bi-located, fault-tolerant audio recording, storage, and serving system operating 24/7/365 providing mission-critical service to thousands of users.

I architected the solution within the confines of the corporate environment. This included clustered VMware origin servers with SCSI attached shared storage presented as RDMs to the origin VMs and consumed as LVM PVs.

Designed the API and the front end (a web-based audio player), outsourced the front end build, and developed the back end myself.

The back end consisted of a PHP codebase for the API, which leveraged server-side-binaries to slice up the audio and stream it to the client. I also wrote a Python program to synchronize the central storage with the satellite recorders every 750 milliseconds using HTTP range requests. This turned out to be extremely efficient and was rolled out further to a separate radio production system to use excellent reliability.

Instrumented the system so that operators could monitor various parameters, including disk usage and sync latency.

The API was designed to be used by other applications to automate tasks (cut audio for publication online), provide long-term archive access, and facilitate the production of future radio programs.

Developed a continuous delivery (CD) pipeline and infrastructure-as-code (IaC) to provision a three-tier, cloud-based web app capable of serving thousands of back-end requests per second. The stack was operated on by three distinct roles — DevOps engineers, operations teams, and app developers.

Stack included monitoring (Prometheus) and log collection (Grafana Loki) as well as automated backups (AWS Backup) of these metrics, logs, and the application's underlying database (Postgres).

I used Github actions to provide continuous delivery of code triggered by pushes and pull requests that include modifications to the application codebase. The actions include building a Docker image for the front-end or back-end service as necessary, testing the new image, tagging it with the appropriate version number (Docker Hub), and initiating a production rollout of the upgraded version on Kubernetes (Amazon EKS).

We automatically secured all public endpoints with TLS (cert-manager and Let's Encrypt).

Scaling was handled semi-automatically at the client's request by providing a simple management script that an in-house Ops team could call to scale up and down as they saw fit. The cluster then managed to scale AWS resources as necessary.