Matheus Gaignoux Raiol

In this project, I designed and developed the steps of a refactored fraud analysis data pipeline for purchase orders made in a web marketplace. The primary purpose of this pipeline was to analyze these orders based on key factors and either approve or cancel them. The decision is formulated by superpositioning an ML model score threshold and applying a set of dynamic and flexible rules to each order. After that, all the information is returned to the front-end application, establishing a link between OLAP and OLTP systems. The changes and new features added to this pipeline significantly decreased processing time by 75%—from one hour down to just 15 minutes at most.

I designed and implemented a data pipeline capable of extracting files from several sources and then processing and sending them to downstream applications handling critical information on chargebacks. The following stages of data storage were created:

• Staging layer (raw files)
• Bronze layer (same information as the previous stage but unifying the file schemas)
• Silver layer (last status of a set of keys)
• Gold layer (combined information)

The medallion architecture provided the needed flexibility to the extent that changes in some business rules could only be applied to the appropriate stage without compromising the rest. Furthermore, this architecture ensured a unified source of truth for any application after the gold layer. Rather than having each consumer process raw files in a potentially inconsistent way, building this ready-to-use refined business enabled a high level of reusability. Lastly, implementing programming best practices guaranteed quick completion of jobs and a lower level of cluster memory and cloud resources.

The key idea was to design and implement data workflows following a custom paradigm—each workflow related to only one subject—to replace some processes related to feeding tables in a fraud analysis team data mart. This brought about improvements and increased reliability for all applications relying on the pipelines. Since the data environment was not built on top of a lakehouse architecture, our custom paradigm assumed the existence of two layers for data storage: the raw and business layers. The raw layer should have all the historical loads. Each load is a daily snap of a set of transformations applied to the data concerning the business rules that lead to it. In this layer, it is optional for the data to have uniqueness. On the business layer, the data is deduplicated and has a small volume, considering only the time interval needed for business analysis. Unlike the other layer based on a daily append process, the tables are overwritten every time the workflows run.