Renvaldas Mackevicius

Demanding users need to know about data loading delays, errors, and other problems. Therefore, I planned, designed, and implemented an integrated monitoring solution for a DW maintenance team.

This solution helped to see a clear picture of the situation in the short run and saved a lot of time in the longer run, avoiding debugging and helping to solve any quality problems.

For many years, my client struggled in loading external XLSX files from various locations due to data quality problems and even some "file opened and lock" problems. That resulted in failed pipelines, unrefreshed reports, and unhappy business users.

Developers tried to build their own solutions but kept getting the same problems while loading those files. So, I proposed and developed a universal XLSX file loader via configurations with error handling that allows developers to use the same package with everything they need for configuring metadata. After that, we even forgot that problem existed.

For this project, I planned the migration updating code, which included redesigning pipelines, packages, and the data structure. During the migration, I developed pipelines using Azure Data Factory and new data formats—like Parquet—available there. I continued working on data quality control and performance during and after the migration.

Wood is IKEA’s most important raw material, and they are one of the biggest buyers of wood in the world. To ensure that all the suppliers meet the requirements of the many rules that IKEA has on forestry, the suppliers need to register every future delivery of wood in WTS. After that, IKEA can analyze risks—such as illegal logging, forest legislation, protected species, and so on—for the area from where the wood is originated before they buy it. Geography is, of course, critical in those analyses.

For this project, I designed the system architecture and the database architecture. I also dealt with installation, data migration, DB structure development, SP, views, models in Python, map features, geoprocessing services, and data export and import services.

I designed the architecture and developed integrated front and back-end applications for a black call list subsystem and an address data control and load subsystem, building a persistent layer (database schema).

I implemented an ASP.NET API for managing OData in the SQL Server database. I also created the database design and table, stored procedures for managing attribute data from JSON and binary files, and data load test procedures in PowerShell. For automation, I used Apache Airflow.

I built Azure ADF pipelines to load data from Salesforce to Azure SQL. I also created stored procedures to upsert incoming data, test cases for data and performance, and documentation for the application's maintenance team.