Roman Gershkovich

Due to the new networking design in modern DCs without IP v4 available for the "fastbone" network, we had to migrate to IP v6 for internal data transfers at Yandex.Market. It was a challenging task because of multiple factors:

01. We were inexperienced with running dual-stacked systems (IP v4 was needed for the "backbone" network to serve real user traffic).
02. At the time, IP v6 implementation in a Linux kernel was pretty buggy.
03. We used BitTorrent for internal data transfers, so we had to find a tracker and a client that properly supported IP v6; it was not straightforward back then (ended up with Opentracker and Aria2c),
04. Even the basic infrastructure needed to be updated to work with IP v6, e.g., automation to generate reverse DNS for AAAA records.
05. A rollback plan was a must since this process of updating data was business-critical.

I was not the only one working at this project, but I was responsible from the infrastructure side, i.e., choosing correct open-source solutions, setting up and testing IP v6 everywhere, and cooperating with networking operations for debugging purposes. Eventually, we managed to solve the issues above and fully migrate to IP v6—genuinely satisfying!

Managing Kubernetes is hard. While bringing up GKE or EKS clusters and deploying sample "Hello world" apps is made super-easy, running Kubernetes on your own hardware and actually managing configurations without relying on kubeadm to fully control what's going on in your cluster is a totally different story.

Choosing the right CNI plugin, thinking about how to organize load balancing, certificate management, version upgrades, compatibility between different components, optimal hardware usage with full (but not critical) saturation, and so on—all this needs planning and careful execution of changes.

At OZON, I was the point person responsible for these operations and bootstrapped several clusters with Calico, iBGP peering for L3 routable container network, Puppet-managed configurations, and dynamic node provisioning. The architecture evolved significantly from KVM nodes with limited resources to full hardware nodes running up to 250 containers per-machine simultaneously. I also took part in common pipeline development for deployments, organized fair requests/limits for namespaces, and integrated Vault for secrets and more.