Data Developer in Macungie, PA, United States

I created data pipelines to ingest multiple daily input files from different sources. This process decrypts and loads the daily input files, takes data elements from these files, compares them against historical data, calculates other custom column values, and loads the data into the online transaction processing (OLTP) database. The data pipelines were created using Azure Data Factory, which integrates Databricks notebooks written in Python and PySpark.

I was heavily involved in extensive T-SQL coding for a terminal seaport management system that tracks shipments from overseas to the US addresses. It is a complex system that integrates electronic data interchange (EDI) messages from oversea shipping lines. The system can locate a small package in a ship that carries thousands of containers and transfer those containers to trains and trucks for onshore distribution.

An SSIS-based, fully configurable data correction back-end system built on top of an SQL Server database for proper customer billing. The system corrects data inconsistency caused by human errors. It examines the incoming data feed and compares the configured data mapping and data from multiple systems to assign customers to proper products or programs. It also aligns underlying customer data with their contracts so customers are billed correctly, reducing disputes and unnecessary customer call volumes. This project has been highly successful as it addressed a complex data correction requirement with many variables and inputs.

The SSIS processed large raw-impression data files from ads on top media websites. Raw data is loaded, transformed, and loaded again into an SQL server database. Analysts and data scientists then use that data to run their R programs for predicting future demands and pricing.