Gabor Hermann

I was working within a team of data scientists and data engineers, leading the set up of the infrastructure and tooling for migrating a recommendation system to the Google Cloud Platform. We had Hadoop/Pig jobs and a Java service using Cassandra.

We then migrated it to BigQuery and PySpark jobs running on Dataproc, scheduled with Airflow, serving on Kubernetes backed by Bigtable. We also set up a new CI/CD pipeline in GitLab CI/CD. I also introduced SRE practices for service maintenance.

I was mostly in charge of establishing the infrastructure and tooling, which included GitLab CI/CD, Docker images, deployment with Kubernetes, creating clusters to run PySpark on, setting up permissions, monitoring services, scheduling with Airflow, and so on.

I led a small team that implemented distributed machine learning algorithms as part of a research project. Using Scala, we implemented matrix factorization (iALS, DSGD) on top of Apache Flink and Spark. I also implemented the Parameter Server architecture on top of Apache Flink, which can be used to implement many machine learning algorithms more easily (matrix factorization and passive-aggressive classifier).

I established the analytics engineering practices for an analytics team.

This included setting up data testing and coaching analysts on how to verify their assumptions. Implementing CI/CD made the development cycle faster; this included SQL code-style checks, query validation, fast Docker image builds, and automatic deployment.

I also made the local development environment easier to set up and use with Docker. I then set up the scheduling of pipelines with Airflow and implemented pipeline monitoring by loading DBT metadata to the data warehouse.