Eduardo Bartolomeu

A data lake to store data from several company systems in Snowflake. We leveraged ELT methods to keep all the data in Snowflake and then be able to perform all necessary transformations. Both the raw zone of data and the trusted and refined zone were stored in Snowflake.

An education company acquired smaller ones, prompting the need to establish a centralized data lake to manage data from various company systems. We built that required data lake, as well as data pipelines and data warehouses from scratch, leveraging the AWS tech stack. We used document management systems (DMS) to migrate data from relational database services (RDS), MySQL, and PostgreSQL. We also employed AWS Glue jobs, workflows, and crawlers for ETL processes across different data lake layers. To automate CI/CD, we implemented Lambda Functions through API Gateways triggered by Git actions. Terraform was used as infrastructure as code (IaC).

A major healthcare company in Brazil possessed historical data of beneficiaries but lacked access to exam results. Collaborating with data engineers and a business team, including doctors and nurses, we developed an algorithm that assesses whether a beneficiary has a low, medium, or high probability of having cancer. This project enabled the company to undertake proactive measures for these beneficiaries, enhancing their health journey.

Initially, we migrated data sheets to BigQuery, focusing on the most common procedures for beneficiaries with cancer. Subsequently, we developed a Python algorithm to export the results for storage as both CSV files and tables. Everything was orchestrated using Composer and Apache Airflow.

I worked as a data engineer on a project to integrate and analyze accident-related data across Los Angeles County. I unified information from LAPD APIs, XML files, and existing Snowflake tables into a single analytics environment. I built ingestion and transformation pipelines using Azure Data Factory, SnowPark, and SnowSQL, applying business rules and data normalization. All development was version-controlled in Git, and deployments were automated through Azure DevOps CI/CD, including ADF environment management via ARM templates.

I organized data into analytical domains, implemented Slowly Changing Dimensions to preserve history, and enriched records with Service Planning Area (SPA) geographic boundaries for regional insights. I also delivered semantic models for Power BI, enabling structured and governed dashboard consumption. Additionally, I documented data lineage and technical details in Confluence, ensuring clarity, maintainability, and continuity.