Yuri Franzoni da Silva

This project aimed to implement a data streaming pipeline from multiple Progress OpenEdge databases, which would be polled by a Kafka cluster to automatically apply changes to several monitored tables into a Redshift cluster at AWS.

To make this happen, I had to learn how to build a successful change data capture (CDC)-enabled OpenEdge environment and set up and configure a Confluent Kafka platform from scratch, including using Kafka connectors, KSQL streams, and other components and deploy and configure an Amazon Redshift instance.

I conducted technical research and coordinated a small team with their corresponding tasks and milestones.

After deploying the solution we created, the company was able to start migrating several legacy reports and dashboards using modern data analytics tools. Also, the company's clients could log into the data warehouse environment and connect their software for querying data.

The company had around 400 separate Progress OpenEdge databases, one per client, created using pre-existing templates. Over time, these databases grew and required frequent maintenance work such as re-indexing, dumping, and loading. The source templates were created long ago using type I storage architecture, which is now obsolete.

This project aimed to migrate these 400 databases to the type II structure, which had sizes varying from 1 gigabyte to 45 gigabytes, using knowledge and best practices from Progress for this task. We had to analyze the usage statistics for every client to decide how to split the new database structure into multiple database areas across several hard drives.

I helped develop a PowerShell script to execute a full database backup and binary dump, deleting the existing database, recreating a new one, and finally running a binary load and re-indexing.

As an outcome, we had a performance increase ranging from 15% to 40%, dramatically reducing client downtime for database maintenance.