Brenda Alexsandra Januário

A pipeline of agricultural data to be consumed by machine learning models and dashboards. The historical and daily data were extracted from APIs and spreadsheets. The data lake in the AWS S3 bucket stored the extracted raw data to create this pipeline. Data lake data was ingested as raw as possible into the bronze layer of the data lakehouse in Databricks. Then, the bronze layer data was manipulated to adjust the type, format, and interpolation of missing values and ​​outliers, to finally insert them into the silver layer, clean and consistently ready to be consumed by business analysts, data scientists, and MLOps.

The data was tested at every stage using the data quality open-source Great Expectations, and Sentry was enabled to notify in case of failure.

A data lakehouse and visualizations for B2B eCommerce. A public personal data engineering study project on my GitHub portfolio. In this data lakehouse, I created random data for a transactional base B2B eCommerce system and an analytical base with weblog data.

I developed ETL pipelines and created data tests and the infrastructure as code (IaC) using Terraform to build the data lakehouse. I generated more than 700 million rows, and the final structured data were used to create visualizations in tabular and dashboards format, to work with BigData.

Contributed to the development of Traive's Debt Enrichment API that consolidates debt data from multiple sources like SCR, CERC, B3, BNDES, and income tax, providing a comprehensive view of farmers' debt profiles.

Credit risk assessments often rely on outdated and incomplete data, primarily from income tax reports. By integrating various API endpoints, our solution normalizes, cleans, deduplicates, and classifies short- and long-term obligations in real time. This enables credit analysts to make more accurate risk assessments, reducing manual effort and misclassification risks.

Delivered via Kafka, this enriched data allows our customers to make faster, data-driven decisions while improving the performance of Traive's models. The solution also helps lower costs by eliminating the need for 3rd-party data providers, streamlining the credit decision process and enhancing overall user trust in our platform.