Sidney Park

With the global expansion of the business, the company needed to build a business intelligence system in order to create a single source of truth data repository for enterprise reporting and enable user self-service reporting ability.

I developed the data pipeline to ingest data from different sources and placed the data into the star-schema modeled tables in the data warehouse, following the Kimball dimension modeling methodology.

I created the PowerBI workspace environments and developed data sources with all the tables required for user self-service reporting by defining table relationships and dashboards for monitoring sales pipelines.

The technologies used were Snowflake, PowerBI, SQL, Python, Matillion, Azure DB, and AWS.

A metadata-driven data pipeline practice using Luigi in Python. Python makes it much easier to dynamically populate update scripts for source data to be updated to the data store tables. For this practice, data sources are defined as a MySQL database and a weblog file loaded into Redshift. Scripts to demo the data warehouse presentation layer in dimensional modeling are included, but it has not been incorporated into the Luigi data pipeline due to a time constraint.

A small personal work to demonstrate how to stream Twitter tweets using Kafka in Python. The producer sends a Twitter API request to retrieve tweets by searching for a particular keyword. When a response is received, it remembers the last tweet ID and applies a data transformation so that it is in a format that is easier to read. The producer sends data to the consumer, which waits and consumes data in real-time. The consumer uploads the data received to S3 as files, including the last tweet ID, to be used for further processing. The last tweet ID will work as a watermark for a producer to retrieve incremental tweets the next time.

I created a simple Amazon S3 data lake with CSV files in this project.

The CSV files are created by extracting data from the PostgreSQL DVD rental sample database publicly available.

Then Python script is used to streamline the process of extracting data from the PostgreSQL database, saving outputs to CSV files, and then uploading them to the Amazon S3 bucket in a specific folder structure.

Glue Crawler is used to scan and create table metadata on Glue Data Catalog. Once the tables are formed in the Glue Data Catalog, you can use EMR or Athena to query tables—in this case, Athena is used.

Loyalty NZ had different streams for ingesting transaction data and it caused transactions to be scattered over different places. I managed the project as the single resource by gathering requirements, understanding the business logic, analyzing data, identifying improvement gaps, developing the data pipelines, and training the users. I delivered the single source of truth for all the transactions for the consumer transactions related to the company's loyalty program. The principal data scientist called the project the biggest improvement ever implemented to the company's transaction data. Throughout the project, I also participated as a scrum master.

Developed and supported business intelligence ETL pipelines and reports or dashboards and performed data analysis for clients in Australia, New Zealand, and the UK. I focused on Microsoft and Azure products but also covered other tools such as Tableau.

Pacific National is one of the largest freight logistics companies in Australia. I developed and supported their business intelligence system end-to-end for over four years. My job included developing, monitoring, debugging, and supporting ETL pipelines and Tableau servers and reports, data analysis, consulting with business users, and providing recommendations.

Designed data modeling for the company's finance-related source data, developed ETL to bring into a data warehouse, and then created business reports following complex reporting requirements. I made a range of improvements to the ETL pipeline, which enhanced data modeling design and significantly reduced ETL pipeline duration to meet SLA. Also, I implemented SQL Server upgrades from SQL Server 2008 R2 to SQL Server 2016 in a high-availability environment.

Enhanced and expanded the client's business intelligence system built for their claims data model. Also, I monitored the business system to meet the company's SLA for data availability and improved performance.

Clients requested to create customized business reports using their finance and operations data from their operation tools, QuickBooks Online and Service Ledger. My job included source data analysis, developing an extract process using QuickBooks Online API in Python and T-SQL to extract data from Microsoft SQL Server, building datasets in appropriate data models in Power BI, and creating customized finance and operations business reports using complex calculations.