Adrian Sandulescu

While working on one Kappa architecture analytics pipeline I ran into issues with the database nodes being very expensive.
The only solution to significantly lower the costs was to use the AWS cloud spare capacity (SPOT), made available at a much lower cost,
but with an extremely high risk of losing the virtual machines. Losing the VMs could sometimes happen even multiple times per day.

The lost VMs could be replaced but this still meant that any new VMs had to reload data from cloud storage (S3) over the course of several hours in order to become fully operational.
During this time, the database would lose replication and would be vulnerable to downtime, a risk we couldn't take.

In order to solve this, I used a script that would allow newly launched VMs to reuse the virtual hard disks left behind by any lost VMs, thus allowing them to become available as soon as they were launched.

While working with high volume Tomcat application deployed on hundreds of nodes I ran into issues with the deployment procedure being both too slow and too time-consuming (not completely automate due to being too complex).

The deployment procedure involved both a manual canary step (where only one of the servers would be updated and monitored for errors pending approval) and a rolling update step (where the entire fleet would be updated a few VMs at a time).

Due to the number of VMs that needed to be updated the entire deployment could take up to 30 minutes but what's worse is that a rollback could also take up to 30 minutes.

In order to improve this, I implemented Spinnaker, a tool that allows automating all deployment steps (including canary and manual approval by stakeholders) as a deployment pipeline and also allows running red/black deployments.

In a red/black deployment, a completely new set of VMs is provisioned and traffic is routed to them. In order to rollback traffic simply needs to be routed back to the VMs deployed with the previous application version meaning that rollbacks could now be performed in seconds instead of tens of minutes minimizing the impact of failed deployments.

While working with applications that required access to AWS cloud services while being deployed in Kubernetes I ran into an issue with using dynamically generated credentials (to ensure proper rotation) while also making sure that applications could not use each other's credentials.

While AWS does facilitate using dynamic credentials by assigning a role to each VM, this would mean that all pods running on a VM would have the same access policy.

In order to solve this issue, I adopted an application called Kube2Iam that proxies access to AWS' credential server and allows configuring separate access policies to each pod running on a VM.

While working with Kafka in AWS I ran into an issue with using throughput optimized virtual hard disks on the Kafka VMs.

The maximum throughput of the disks was several times lower than what was advertised, meaning both a performance penalty and a cost penalty if we were to switch to using SSDs.

Digging through the documentation I found that the advertised throughput is only guaranteed if write size is at least 1MB.

Kafka is optimized specifically for efficient, large sequential writes so this was most likely a kernel configuration issue.

Searching for more information online I found that the maximum write size for the OS we were using was limited to 256KB and needed a boot parameter change in order to allow being increased after boot. A new VM image was created with the required changes thus solving the issue.

Working extensively with Puppet for automating server provisioning I ran into the issue of new servers taking a very long time to provision since every new server would need to install and configure all required packages.

In order to tackle this limitation, I pioneered and encouraged the shift to use immutable infrastructure, using Packer to create VM images.

Since new servers would now be provisioned from pre-configured images launching them would be several times faster.

As an added benefit, configuration drift would no longer be possible.