Aleksei Burenin

An Airflow migration project.

WORK DONE
• Conducted analysis of the current EC2 standalone solution, evaluating its strengths and weaknesses.
• Set up a new AWS EKS cluster to support the updated architecture.
• Configured Helm Charts to deploy the Airflow application within the Kubernetes environment.
• Implemented FluxCD to embrace the GitOps paradigm for managing Kubernetes infrastructure and application.
• Provided fully hybrid environments:
a) Two cloud-based environments for production and staging.
b) One local environment for DAG development and debugging within an IDE.
c) Docker Compose for automated unit testing and integration tests.
• Developed unit tests for Airflow DAGs, enabling execution both locally and in CI/CD pipelines.
• Introduced a CI/CD pipeline within the GitLab ecosystem to ensure a proper software development lifecycle (SDLC) with mandatory unit tests during merge requests.
• Secured external access through dynamic DNS provisioning for Kubernetes with Cloudflare for enhanced security.
• Configured a hybrid logging S3 and local storage.
• Implemented database change management using Liquibase.

The final solution had a GitOps approach, scalability, flexible configuration, and proper SDCL support.

A travel agency startup that required a data analytics solution.

What was done:
• Analyzed data sources for structure and built S2T mapping.
• Selected target data storage as BigQuery; integrated with customer GCP environment.
• Set up data collectors to extract real-time changes from Firebase to BigQuery RAW schema.
• Created an ETL framework to transform data from RAW schema to the ODS layer (GCP Cloud Scheduled queries).
• Added metadata-based transformation for Firebase documents JSON fields to BQ schema.
• Created reporting layer in DWH with recent snapshots and historical data.
• Engineered a cloud set up and BI tool Metabase.
• Migrated manual exports from Retool to five BigQuery-based Metabase dashboards with more than 30 reports.
• Designed data quality policy and processes:
- Freshness checks for each fact table
- Consistency checks for source Firebase collections; targeted raw tables, specifically Python Cloud Functions.
- Calculated uptime KPI metric.
• Enabled alerting for DQ checks via Metabase alert feature for the dev team and business users.
• Documented all solutions in Notion.
• Implemented BigQuery cost monitoring Cloud Logging Sink to BigQuery and alerting via Metabase question subscription.

I completed a data warehouse solution for a real estate company.

Project milestones:

1. Made the ETL process using Airflow for loading various sources—Amazon RDS MS SQL sources, S3 Flat files (Excel, CSV), and API sources—into the ingesting layer of the data warehouse.
2. Built the data model and ETL DAGs for data layer storage—preprocessing and detailed data storage (DDS)—on PostgreSQL Amazon RDS.
3. Created DQ processes for controlling freshness and consistency between data sources and ingestion layer tables, ingestion layer and preprocessing, and preprocessing and mart layer. This included the data structure for customizing checks, saving aggregated and detailed results, and Airflow DAGs for running these checks.
4. Created the preprocessing layer's adjustment framework, which allowed the following:
• a user can define datasets for the target table;
• validation of changesets before applying changes (Alembic style but for data);
• reverse changes if required;
• log of changes and tracking for audit purposes.
5. Implemented SCD 4 for DDS.
6. Built the mart layer for views and materialized views for reporting purposes.
7. Prepared guidelines and built the POC for the migration to Snowflake for future extensions.

I created an ETL framework for loading different data sources to the Vertica MPP system. The scheduling was done on Airflow.

The created framework could generate a DAG-based on YAML configuration files:
• Typical pipeline for file storage: grab a file from the remote file storage.
• Push to a staging table in Vertica with minimum transformation.
• Merge or overwrite to a destination table in Vertica.
• Apply business-specific transformation in Vertica using provided SQL (for the reporting layer).

Supporting features:
• Push date linage and statistical information to data metric storage.
• Push data freshness values.
• Data source data validation and alerting it not correct.

An additional type of tasks:
• Run DQ checks to check integrity between data layers and accuracy.
• CI/CD was done via TeamCity.

The created solution became scalable, monitorable, and more robust. CI/CD enabled tracking on source changes and deployment process.

I integrated one of the source systems into an enterprise data lake. It was a customer service automation software built on MongoDB and a reporting REST API.

Tasks:
• Interacted with vendor representatives to understand how to extract data from the API.
• Analyzed data available in the REST API.
• Designed transactional (ODS) and data mart (DM) data models for storage API data in Hadoop.
• Collaborated with the dev team responsible for system implementation about multi-datacenter HA deployment to consider future ETL and DQ tools.
• Built an ETL Apache Spark-based solution for incremental and full data loads.
• Integrated ETL app metrics to the enterprise monitoring system based on Cassandra and Grafana.
• Built the ETL for transformation data from the transactional layer to business-specific data marts.
• Created Metabase reports for business owners, which helped analyze the flow's effectiveness built by the flow interaction dev team.
• Established data quality processes to check consistency, accuracy, and freshness for each data pipeline stage.

This project created tremendous value for the analytical department, increased agent interaction performance transparency, and enabled real-time analyses.