Brandon Peskin

Led (as the technical lead) a team of SREs responsible for the maintenance of the hardware to support and the release of the company's SaaS product. Established direct connectivity to the cloud provider via Terraform/network infrastructure as code. Established all of the resources needed from the cloud provider via Terraform. Worked with engineering to set up a Spinnaker pipeline to deploy the code via Kubernetes pods which replaced our home-brew Bash script we used for years on bare metal.

As a site reliability engineer tasked with creating SLA/SLI/SLT, I was tasked with building out an observability platform.

• Completed vendor evaluation and developed selection criteria for a new observability platform. Evaluated all major vendors as well as the DIY model with an ELK stack. Datadog was the selected vendor.
• Over 6 months, scaled out an observability platform using Datadog. Also provided for the InfoSec team to use Datadog as a SIEM.
• The project was not as simple as installing the Datadog agent and key. I worked with engineers to reduce log volume and identify the signal-to-noise ratio. This was required because the current logging volume would have made Datadog very expensive. Reduced logging volume equivalent to $40,000 a month, Datadog spent before full onboarding.
• Identified key metrics and components for which visibility was required. Was able to leverage some of Datadog's built-in functionality, but the rest required a DogStatsD implementation so we could roll our own metrics.

The end result was that we were receiving far less frequent but more meaningful information about how the application stack was performing. Instituted a "if action cannot be taken, we should not be paged" policy around alerts.