Andrei Biketov

In this project, I was involved in designing and implementing a data mesh solution centered around Reactive Streams and enterprise integration patterns. This solution enabled various General Educational Development (GED) teams to generate real-time and near-real-time analytical and risk assessment reports. The goal was to equip traders and business users with comprehensive and current market information.

Contracted by HBC to migrate existing legacy applications to the AWS platform by building microservices and data processing pipelines based on the Reactive Streams architecture.

ROLE AND RESPONSIBILITIES
As a technical team lead, I was in charge of designing, implementing, and deploying distributed data integration solutions using the Lightbend Cloudflow stream processing SDK. I utilized Scala, Akka Streams, Spark Structural Streaming, Kafka, Amazon Elastic Kubernetes Service (Amazon EKS), Amazon EMR (formerly Elastic MapReduce), and API Gateway as the core technologies for building this system.

Contracted by Equifax Canada to assist in constructing a new credit data management system using the Cloudera Data Platform (CDP) and subsequently transitioning it to the Google Cloud Platform (GCP).

I designed and executed the extract, transform, and load (ETL) process to import historical credit data from the mainframe, transform it into Avro format, and store it in the HBase database and Solr search engine within a Cloudera Hadoop cluster. I also developed batch Spark jobs and real-time data pipelines utilizing Kafka with Spark Streaming to produce credit information reports and compute credit scores.