Pavlo M Kurochka

This proof of concept project demonstrates the power and flexibility of the methodology and toolset that I use for data migrations.

Skills: ETL, Microsoft SQL Server, T-SQL, Python, Hubspot, Zoho-CRM, data migration

Please follow the link for a demo video.

I developed and configured data conversion logic for 15 conversion objects into SAP CRM, SD, and FI.

I executed complex customer data conversion that involved integration with Satori Address Validation software and de-duplication in spreadsheets.

I'm excited to share that I've completed the final project for the Data Engineering Zoomcamp by DataTalksClub! During this course, I had the opportunity to dive deep into various aspects of data engineering, including:

• Containerization and Infrastructure as Code: Learned about Docker, docker-compose, and setting up infrastructure on GCP with Terraform.

• Workflow Orchestration: Explored Data Lake and Workflow orchestration with Mage. Got inspired to research this field more extensively. Discovered, learned, and used in my project Kestra for orchestration.

• Data Warehouse: Gained insights into BigQuery, partitioning, clustering, and BigQuery best practices.

• Analytics Engineering: Delved into the basics of analytics engineering, dbt (data build tool), BigQuery and dbt, Postgres and dbt, dbt models, testing, documenting, deployment to the cloud and locally, and visualizing the data with Google Looker Studio and Metabase. Again, researched similar popular new technologies by myself and learned and started using DuckDB and SQLMesh.

• Batch Processing: Learned about batch processing and Spark Dataframes.
By applying these technologies, I was able to tackle my Capstone Project.

I am thrilled to share my latest project: a streamlined approach to building data warehouses.
This open-source project, inspired by the Data Engineering Zoomcamp, showcases the power of a modern data engineering stack. All you need to install locally on your computer is Visual Studio Code. The rest, including cloud storage and computing resources, is free.
Simply create accounts on MotherDuck and GitHub and follow my detailed instructions. Within minutes, you'll have a robust and responsive data warehouse ready to handle your queries.
The example data pipeline transforms some online gaming data, but the described approach could be applied to any data source or subject matter.
Big thanks to Alexey Grigorev and the DataTalksClub team for putting together such an innovative and inspiring course.

Technologies used: Docker, Kestra, MotherDuck, SQLMesh, dlt, Python, SQL, Streamlit, DataEngineering, DEZoomcamp, dltHub, Tobiko Data

A single Python script that works as a command line application (CLI). This script can achieve the same results that some companies' platforms are offering, for which they charge hundreds of thousands of dollars in software licenses just to enable the data migration process. This proof of concept could apply to any business system (CRM, ERP, PLM, etc.). It takes a publicly available data set of US companies maintained by a US government agency, transforms it, and loads it as Accounts into Salesforce.com. I wanted to showcase my ability to get up to speed with the load process and methods in modern business systems that are new to me.

It covers all the steps of a proper data migration project:
• Enabling source and target system discovery and establishing connectivity.
• Making source and target table snapshots locally.
• Data profiling and column mapping.
• Generating the code for data transformation rules that create a staging table
• Producing pre-load reports.
• Loading data into the target system while marking loaded records and capturing any load errors.
• Producing post-load reports with column-by-column comparisons between staged and loaded values.