Igor Gorbenko

I designed and implemented a lakehouse system based on Apache Iceberg and S3, capable of processing 2PB of data. The architecture utilized Apache Spark as the primary ETL tool and featured two data ingestion paths: batch and real-time layers.

Every day, the system processes up to 10TB of new data, efficiently handling both historical and streaming data. The batch layer manages large-scale data processing tasks, enabling efficient transformation and loading of massive datasets. The real-time layer ingests streaming data, allowing for immediate analytics and up-to-the-minute insights. This dual ingestion approach significantly enhanced data processing efficiency and expanded analytics capabilities.

By leveraging Apache Iceberg for table formats and S3 for scalable storage, the system provides robust data storage solutions with support for ACID transactions and schema evolution. Apache Spark's powerful processing engine facilitates complex data transformations and computations across large clusters.

A GCP-based system for the purpose of recommendation. This system allows users to receive the most relevant content based on their interests.

I was the engineering team leader and owned the entire development process on the data and cloud sides.

A GCP-based recommendation system. The system of recommendations for the interior design of apartments offers the most optimal arrangement of furniture based on a given apartment plan.

I was a project architect, as well as a data engineer and back-end developer. I designed the architecture of the system and the interaction of all components.

The main task of this project was to convert raw data into a standardized format. The original datasets could be of various types and stored in different storages, such as AWS S3, GCP GCS, Hadoop HDFS, PostgreSQL, Amazon Redshift, and more. The project needed a tool to prepare a conversion in automatic mode and minimize the issues during the ETL process with Spark SQL.

I was a tech lead on this project. My responsibilities were developing the core part of the framework’s components using Python, which allowed us to automate scheduled ETL steps and run other tasks after conversion, such as unit tests, stats reports, and so on. I also performed code reviewing and ran the ETL pipelines.