Eduardo Bartolomeu

A data lake to store data from several company systems in Snowflake. We leveraged ELT methods to keep all the data in Snowflake and then be able to perform all necessary transformations. Both the raw zone of data and the trusted and refined zone were stored in Snowflake.

An education company acquired smaller ones, prompting the need to establish a centralized data lake to manage data from various company systems. We built that required data lake, as well as data pipelines and data warehouses from scratch, leveraging the AWS tech stack. We used document management systems (DMS) to migrate data from relational database services (RDS), MySQL, and PostgreSQL. We also employed AWS Glue jobs, workflows, and crawlers for ETL processes across different data lake layers. To automate CI/CD, we implemented Lambda Functions through API Gateways triggered by Git actions. Terraform was used as infrastructure as code (IaC).

A major healthcare company in Brazil possessed historical data of beneficiaries but lacked access to exam results. Collaborating with data engineers and a business team, including doctors and nurses, we developed an algorithm that assesses whether a beneficiary has a low, medium, or high probability of having cancer. This project enabled the company to undertake proactive measures for these beneficiaries, enhancing their health journey.

Initially, we migrated data sheets to BigQuery, focusing on the most common procedures for beneficiaries with cancer. Subsequently, we developed a Python algorithm to export the results for storage as both CSV files and tables. Everything was orchestrated using Composer and Apache Airflow.