Sidney is available for hire
Hire SidneySidney Park
Verified Expert in Engineering
BI and Data Engineer and Developer
Auckland, New Zealand
Toptal member since February 12, 2022
Bio
Sidney is an experienced business intelligence and data engineer. His expertise includes BI dimensional modeling, data pipelines, dashboards, reports, and handling API-related data. He has served as team scrum master, led migration and upgrade projects, mentored team members, and was recognized as the top employee for his ability to build excellent client relationships.
Portfolio
Freelance Clients
Python, SQL, Azure, Snowflake, Matillion ETL for Redshift, Microsoft SQL Server...
Cognition360
SQL, Microsoft SQL Server, Data Engineering, Transact-SQL (T-SQL)...
Analytical Technologies Group
Microsoft Power BI, Microsoft 365, ServiceLedger, Microsoft Power Apps, SQL...
Experience
- Data Pipelines - 9 years
- Data Visualization - 9 years
- Business Intelligence (BI) - 9 years
- SQL - 9 years
- Data Engineering - 9 years
- Data Modeling - 7 years
- Python - 3 years
- Microsoft Power BI - 3 years
Work Experience
Data Engineer | Data Analyst
2022 - PRESENT
Freelance Clients
- Developed data pipelines for complex data transformations.
- Conducted data analysis and developed dashboards to gain insights into organizations' data.
- Performed root cause analysis to handle BAU issues.
Technologies: Python, SQL, Azure, Snowflake, Matillion ETL for Redshift, Microsoft SQL Server, SQL Server Integration Services (SSIS), SSAS Tabular, SQL Server Analysis Services (SSAS), Microsoft Power BI, SQL Server Reporting Services (SSRS), Data Engineering, Data Architecture, Data Migration, Data Analysis, Git, Azure DevOps, Dimensional Modeling, Azure Databricks, Databricks, Spark, WhereScape RED, Azure Data Factory (ADF), Delta Live Tables (DLT), Dashboards, Data Visualization, Tableau, Data Reporting, Power BI REST APIs, APIs, ETL Pipelines, Data Cleaning, Data Warehouse Design, Business Analysis, Database Migration, Power BI Desktop, Apache Airflow, Performance Optimization, Data Orchestration, Database Schema Design, Amazon EventBridge, Data Marts, Fivetran, Microsoft Dynamics, Stored Procedure, Power BI Report Server, Analytics, Customer Data, Data Governance, AI Data Classification, Data Transformation, Claude, OpenAI, Document Parsing, REST, Copy Activity, Metadata
Data Engineer
2023 - 2023
Cognition360
- Developed stored procedures with CDC ETL processes from the legacy data curation process.
- Analyzed the legacy process and identified gaps and improvements to simplify and optimize the complicated stored procedures with critical thinking.
- Developed a template to generate scripts to save development time dynamically.
Technologies: SQL, Microsoft SQL Server, Data Engineering, Transact-SQL (T-SQL), Data Warehousing, Azure SQL, Azure Data Factory (ADF), Azure, Change Data Capture, ETL Pipelines, Data Cleaning, Database Schema Design, Data Marts, Stored Procedure, Analytics, Data Transformation
Data Engineer | Report Developer
2022 - 2023
Analytical Technologies Group
- Analyzed finance and operations data and developed data models optimized for Power BI reporting.
- Developed a data pipeline that extracts finance data via QuickBooks Online API and processes it using Python.
- Developed finance and operations reports using Power BI for complex reporting requirements.
Technologies: Microsoft Power BI, Microsoft 365, ServiceLedger, Microsoft Power Apps, SQL, Salesforce, Intuit QuickBooks, Expensify, QuickBooks API, QuickBooks Desktop, QuickBooks Online, Data Modeling, Finance Reports, Dimensional Modeling, Data Structures, ETL Pipelines, Data Cleaning, Data Warehouse Design, Power BI Desktop, Data Orchestration, Database Schema Design, Analytics, Board Reporting, Data Transformation, REST
Senior Data Engineer
2022 - 2023
Greater Western Water
- Developed a data pipeline and modeling for automating critical business processes using CRM and ERP data to enable integrated reporting for merged businesses.
- Developed Power BI modeling and visualization optimally with the best practices for business reporting that meet requirements.
- Developed data quality check processes that enable business users to self-monitor and self-resolve data quality issues.
Technologies: Azure, Azure SQL, Azure Data Lake, Databricks, Azure Databricks, Spark, Python, SQL, Data Engineering, Microsoft Power BI, Data Modeling, Data Visualization, Data Quality, Git, Azure DevOps, Dimensional Modeling, Change Data Capture, CI/CD Pipelines, Power BI REST APIs, APIs, ETL Pipelines, Data Cleaning, Data Warehouse Design, Power BI Desktop, Data Orchestration, Database Schema Design, Data Marts, Analytics, Customer Data, Data Governance, Data Architecture, Data Transformation, REST, Copy Activity, Metadata
Data Engineer
2022 - 2022
BFB Pty Limited
- Developed a Python script and SQL Server stored procedures to scan and pull JSON blobs from Azure Blob Storage, then process JSON files into the SQL Server tables, including designing the table schemas and analyzing JSON data.
- Designed the data flow for frequently ingesting webhook JSON payloads, leveraging existing technologies and the Azure Python SDK.
- Documented the solution design and instructions for further changes for the business users.
Technologies: SQL, Azure Blob Storage API, JSON, Python, APIs, Git, Data Structures, Databases, ETL Pipelines, Data Cleaning, Data Orchestration, Database Schema Design, Webhooks, Stored Procedure, Data Transformation, Metadata
Data Engineer
2021 - 2022
Loyalty NZ
- Developed a data pipeline to handle complex consumer transactions and loyalty data from SFTP and API data feeds in a structured and semi-structured format using PySpark, Amazon Athena, SQL, Python, Airflow, Amazon S3, and AWS Glue.
- Built a data pipeline to ingest real-estate market data, apply transformations, and address clean-up, then present data to be used for marketing campaigns using Amazon Athena, SQL, Python, Airflow, Amazon S3, and AWS Glue.
- Facilitated team Agile ceremonies as the team's scrum master, collected feedback constantly and initiated a few changes to Agile practices to work better for the team.
- Assisted business users with source API changes by testing, identifying, and assessing the impact on the organization's data platform.
Technologies: Amazon S3 (AWS S3), AWS Glue, Apache Hive, PySpark, Apache Airflow, SQL, Python, JSON, GitHub, Scrum Master, Data Engineering, Data Analysis, Amazon Athena, Spark, Amazon Web Services (AWS), Business Analysis, Data Lakes, Data Warehousing, Data Warehouse Design, Apache Spark, Data Architecture, Data Pipelines, Agile, ETL Tools, Data Modeling, Git, Data Structures, ETL Pipelines, Data Cleaning, Performance Optimization, Data Orchestration, Database Schema Design, Amazon EventBridge, Data Marts, Analytics, Customer Data, Data Transformation, REST, Metadata
Business Intelligence and Data Engineer
2021 - 2021
EROAD
- Developed a metadata-driven ETL framework and data pipeline to extract data from D365 BYOD, Salesforce, Amazon S3, and other databases, then transform and load into EDW in Kimball dimensional modeling methodology using Matillion and Snowflake.
- Designed Power BI workspace architecture by considering access and security by business departments and different data use cases for reporting and developed Power BI datasets and reports as per agreed reporting requirements.
- Confirmed data and reporting requirements with different stakeholders to be extracted from correct sources with the right access. Arranged, transformed, and presented them to meet end users' expectations.
Technologies: Snowflake, ETL, Microsoft Dynamics 365, Salesforce, Azure SQL Databases, GitHub, SQL, Data Engineering, Data Analysis, Microsoft Power BI, Python, Amazon Web Services (AWS), Amazon S3 (AWS S3), Dashboards, DAX, Business Analysis, Data Warehousing, Data Warehouse Design, Geospatial Data, Reports, Microsoft Power Automate, API Integration, Power Query, Power Query M, Data Pipelines, Technical Architecture, Data Architecture, Monitoring, ETL Tools, Data Modeling, Analytical Dashboards, Git, Dimensional Modeling, ETL Pipelines, Data Cleaning, Performance Optimization, Data Orchestration, Database Schema Design, Amazon EventBridge, Data Marts, Microsoft Dynamics, Analytics, Board Reporting, Data Transformation, Metadata
BI Consultant of Managed Services
2014 - 2021
Altis Consulting
- Awarded and recognized as the best employee for contributions to excellent client relationships in 2017.
- Led upgrades and migration projects with clients and contributed to the success by resolving unexpected issues in a given timeframe, like SQL Server migration to always-on availability, cloud to on-premises, and on-premises to cloud migrations.
- Automated numerous processes, including manual monitoring and reporting tasks, e.g., spent 1.5 days' effort to develop a script that eliminated steps equivalent to two days a month by a finance user.
- Enhanced data pipelines and significantly improved batch ETL performances for the clients' BI environments. For example, the overnight ETL run duration was reduced from 10+ hours to 5-6 hours to meet one client's data availability SLA.
- Onboarded numerous new clients with various technologies and tools, and supported their environments by self-training in new technologies in a given timeframe.
- Mentored other team members as senior-level consultants to suggest guidelines for issues they could not resolve easily.
- Developed data pipelines to onboard new data required, apply transformations, and present data for reporting in Kimball dimensional modeling in clients' BI environments.
- Created and enhanced complex finance and HR reports for clients using different reporting tools, such as Power BI, SSRS, and Tableau.
Technologies: SQL, Microsoft BI Stack, Microsoft Power BI, Tableau, Azure, Azure Data Factory (ADF), Azure Data Lake, SQL Server BI, Business Intelligence (BI), Star Schema, IBM Cognos, MicroStrategy, Oracle, SAP, SAP Business Intelligence (BI), Data Engineering, Dashboards, DAX, SQL Server DBA, SQL Server Integration Services (SSIS), SQL Server Analysis Services (SSAS), SQL Server Reporting Services (SSRS), Microsoft Excel, Microsoft SQL Server, Business Analysis, Data Warehousing, Data Warehouse Design, Excel 365, Reports, Windows Server 2016, Communication, ODBC, Power Query, Power Query M, Monitoring, Data Auditing, ETL Tools, Data Modeling, Azure SQL Data Warehouse, Dedicated SQL Pool (formerly SQL DW), Analytical Dashboards, Subversion (SVN), Git, Dimensional Modeling, Production Support, ETL Pipelines, Data Cleaning, PostgreSQL, Performance Optimization, Data Orchestration, Database Schema Design, Data Marts, SSIS Custom Components, Stored Procedure, Power BI Report Server, Analytics, Customer Data, Data Transformation, Metadata
Project History
Luigi Data Pipeline
A metadata-driven data pipeline practice using Luigi in Python. Python makes it much easier to dynamically populate update scripts for source data to be updated to the data store tables. For this practice, data sources are defined as a MySQL database and a weblog file loaded into Redshift. Scripts to demo the data warehouse presentation layer in dimensional modeling are included, but it has not been incorporated into the Luigi data pipeline due to a time constraint.
EROAD Business Intelligence
With the global expansion of the business, the company needed to build a business intelligence system in order to create a single source of truth data repository for enterprise reporting and enable user self-service reporting ability.
I developed the data pipeline to ingest data from different sources and placed the data into the star-schema modeled tables in the data warehouse, following the Kimball dimension modeling methodology.
I created the PowerBI workspace environments and developed data sources with all the tables required for user self-service reporting by defining table relationships and dashboards for monitoring sales pipelines.
The technologies used were Snowflake, PowerBI, SQL, Python, Matillion, Azure DB, and AWS.
I developed the data pipeline to ingest data from different sources and placed the data into the star-schema modeled tables in the data warehouse, following the Kimball dimension modeling methodology.
I created the PowerBI workspace environments and developed data sources with all the tables required for user self-service reporting by defining table relationships and dashboards for monitoring sales pipelines.
The technologies used were Snowflake, PowerBI, SQL, Python, Matillion, Azure DB, and AWS.
Twitter Tweet Streaming Using Kafka in Python
A small personal work to demonstrate how to stream Twitter tweets using Kafka in Python. The producer sends a Twitter API request to retrieve tweets by searching for a particular keyword. When a response is received, it remembers the last tweet ID and applies a data transformation so that it is in a format that is easier to read. The producer sends data to the consumer, which waits and consumes data in real-time. The consumer uploads the data received to S3 as files, including the last tweet ID, to be used for further processing. The last tweet ID will work as a watermark for a producer to retrieve incremental tweets the next time.
Creating a Simple Amazon S3 Data Lake Using Glue Crawler From PostgreSQL Sample Database
I created a simple Amazon S3 data lake with CSV files in this project.
The CSV files are created by extracting data from the PostgreSQL DVD rental sample database publicly available.
Then Python script is used to streamline the process of extracting data from the PostgreSQL database, saving outputs to CSV files, and then uploading them to the Amazon S3 bucket in a specific folder structure.
Glue Crawler is used to scan and create table metadata on Glue Data Catalog. Once the tables are formed in the Glue Data Catalog, you can use EMR or Athena to query tables—in this case, Athena is used.
The CSV files are created by extracting data from the PostgreSQL DVD rental sample database publicly available.
Then Python script is used to streamline the process of extracting data from the PostgreSQL database, saving outputs to CSV files, and then uploading them to the Amazon S3 bucket in a specific folder structure.
Glue Crawler is used to scan and create table metadata on Glue Data Catalog. Once the tables are formed in the Glue Data Catalog, you can use EMR or Athena to query tables—in this case, Athena is used.
Loyalty NZ Single Source of Truth for Transactions
Loyalty NZ had different streams for ingesting transaction data, which caused transactions to be scattered over different places. I managed the project as a single resource by gathering requirements, understanding the business logic, analyzing structured data as well as semi-structured API payloads, identifying improvement gaps, developing the data pipelines, and training the users. I delivered the single source of truth for all the consumer transactions related to the company's loyalty program. The principal data scientist called the project the biggest improvement ever implemented to the company's transaction data. Throughout the project, I also participated as a scrum master.
Altis Consulting Business Intelligence
Developed and supported business intelligence ETL pipelines and reports or dashboards and performed data analysis for clients in Australia, New Zealand, and the UK. I focused on Microsoft and Azure products but also covered other tools such as Tableau.
Pacific National Business Intelligence
Pacific National is one of the largest freight logistics companies in Australia. The company's main operation data was from its TMS platform. I analyzed the TMS source data carefully by communicating with business users and developed and supported their business intelligence system end-to-end for over four years. My job included developing, monitoring, debugging, and supporting ETL pipelines and Tableau servers and reports, data analysis, consulting with business users, and providing recommendations.
TransGrid Business Intelligence
Designed data modeling for the company's finance-related source data, developed ETL to bring into a data warehouse, and then created business reports following complex reporting requirements. I made a range of improvements to the ETL pipeline, which enhanced data modeling design and significantly reduced ETL pipeline duration to meet SLA. Also, I implemented SQL Server upgrades from SQL Server 2008 R2 to SQL Server 2016 in a high-availability environment.
Partners Life Claim Business Intelligence
Enhanced and expanded the client's business intelligence system built for their claims data model. Also, I monitored the business system to meet the company's SLA for data availability and improved performance.
Analytical Technologies Group Finance and Operations Reporting
Clients requested to create customized business reports using their finance and operations data from their operation tools, QuickBooks Online and Service Ledger. My job included source data analysis, developing an extract process using QuickBooks Online API in Python and T-SQL (Transact-SQL) to extract data from Microsoft SQL Server, building datasets in appropriate data models in Power BI, and creating customized finance and operations business reports using complex calculations.
JTI KPI Scorecard Dashboard
A new KPI scorecard dashboard was requested to be developed for complex calculations. I performed data analysis to understand data for the requirements and investigated existing reports for templates. I took the existing report as a template and developed the new dashboards, but I also remodeled datasets and relationships so that they could be optimized for Power BI reporting. As a result, the newly developed dashboards came with fewer table objects, fewer DAX calculated measures, and calculated fields, but they served all the complicated KPI calculations and improved the performance.
TVNZ Customer Data Platform
AWS services were developed to execute and orchestrate the services required to perform the customer data matching process, ensuring customer data security through required encryption. Results from the matching process were integrated into the Azure Databricks data platform using APIs. A Power BI dashboard was created to provide analysis and insights into how customers are matched from clients, and the company can be analyzed in different segments to support marketing analysis.
Health New Zealand Data Migration
Health New Zealand rolled out a new source system, replacing a legacy source system for the hospital episode, operational, patient, and other related data. I worked on an on-premises to cloud migration, developing an Azure Databricks data platform by migrating the existing ETL pipeline using SQL scripts and modeling the new source system data into the existing reporting model, analyzing the data model from the new system. Achieved a mass migration within a tight deadline without impacting the existing business reports.
MTF Finance Databricks Onboarding New Data Source
Onboarded a new data source. Responsibilities include source data mapping, developing a framework for automated Azure Data Factory pipelines for data ingestion, creating a framework for automated data pipelines to store and transform data in the lakehouse, and applying data quality checks using the DQX library.
Report Tool Evaluations for Migration
Analysed the current state of the existing reports and found the patterns defined by user requirements. Evaluated Power BI, Tableau, and Looker Studio to identify pros and cons and make a decision on which reporting tool would be most suitable to migrate to. Investigated the existing reports and made recommendations to make the reports more accessible as per WCAG guidelines.
New Plymouth District Council Data/Reporting Platform Migration
Data platform migration project from SQL Server to Snowflake by migrating ELT processes from SSIS to Matillion. Reporting platform migration for the business reports from SSRS to Power BI for the local district council.
Farmlands AWS, Snowflake, and Airflow Data Platform
I set up an AWS MWAA Airflow environment using the Snowflake data platform and dbt for data transformations. I configured DAGs based on business requirements and developed integrations with external services such as APIs and Power BI. I also created a CI/CD pipeline to support deployments across multiple environments.
Education
2006 - 2014
Bachelor's Degree in Accounting and Information Systems
University of Auckland - Auckland, New Zealand
Certifications
JUNE 2024 - PRESENT
Fabric Analytics Engineer Associate
Microsoft
JUNE 2023 - PRESENT
The Databricks Certified Data Engineer Professional
Databricks
MARCH 2023 - PRESENT
Microsoft Certified: Azure Solutions Architect Expert
Microsoft
MARCH 2023 - PRESENT
Microsoft Certified: Azure Administrator Associate
Microsoft
JUNE 2022 - PRESENT
AWS Solutions Architect – Associate
Amazon Web Services (AWS)
JUNE 2022 - PRESENT
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks
MARCH 2022 - PRESENT
AWS Certified Data Analytics - Specialty
Amazon Web Services (AWS)
NOVEMBER 2020 - PRESENT
Microsoft Certified: Power BI Data Analyst Associate
Microsoft
AUGUST 2020 - PRESENT
Microsoft Certified: Azure Data Engineer Associate
Microsoft
APRIL 2020 - PRESENT
Azure Fundamentals
Microsoft
APRIL 2016 - PRESENT
Microsoft Certified Professional: MCSA SQL Server 2012/2014
Microsoft
NOVEMBER 2015 - PRESENT
Stephen Few: Visual Business Intelligence Workshop
Perceptual Edge
MARCH 2015 - PRESENT
Dimensional Modelling: The Kimball Method Workshop
Kimball Group
Skills
Libraries/APIs
PySpark, Luigi, Azure Blob Storage API, QuickBooks API, Pandas, ODBC, SendGrid API, Power BI REST APIs
Tools
SQL Server BI, Microsoft Power BI, Power BI Desktop, Power BI Report Server, Tableau, AWS Glue, Apache Airflow, Amazon Athena, Microsoft Excel, Git, IBM Cognos, GitHub, Amazon Elastic MapReduce (EMR), SSAS, WhereScape RED, Power Query, Microsoft Power Apps, Subversion (SVN), Matillion ETL for Redshift, AWS Step Functions, n8n, Looker, Microsoft Dynamics, Claude
Languages
SQL, Python, Stored Procedure, Snowflake, Transact-SQL (T-SQL), Power Query M, Scala, Excel VBA
Paradigms
Business Intelligence (BI), Dimensional Modeling, ETL, ETL Implementation & Design, API Architecture, REST, Agile, Azure DevOps, Kimball Methodology, DevOps
Platforms
Microsoft BI Stack, Amazon Web Services (AWS), Azure, Databricks, AWS Lambda, Oracle, Microsoft Dynamics 365, Salesforce, Apache Kafka, Windows Server 2016, Microsoft Power Automate, Azure SQL Data Warehouse, Dedicated SQL Pool (formerly SQL DW), Azure Synapse, Azure Service Fabric, Azure Data Lake Storage, Docker
Storage
SQL Server 2012, Data Pipelines, Microsoft SQL Server, SQL Server Integration Services (SSIS), Data Lakes, Databases, Azure SQL Databases, Amazon S3 (AWS S3), JSON, PostgreSQL, SQL Server 2008, Apache Hive, Redshift, SQL Stored Procedures, SQL Server DBA, SQL Server Analysis Services (SSAS), SQL Server Reporting Services (SSRS), Azure SQL, Azure Blobs, Oracle 11g, SQL Server 2016, SSAS Tabular, MySQL, Azure Storage, Database Architecture, Database Migration
Frameworks
Spark, Flask, Apache Spark, Delta Live Tables (DLT), Streamlit
Industry Expertise
Accounting, Marketing
Other
Star Schema, Data Analysis, Data Warehouse Design, Data Visualization, ELT, Data Engineering, Reporting, Data Analytics, ETL Tools, Data Modeling, ETL Development, Microsoft Data Transformation Services (now SSIS), Data, Analytical Dashboards, ETL Pipelines, Data Cleaning, Performance Optimization, Data Orchestration, Database Schema Design, Data Marts, Analytics, Data Transformation, Copy Activity, Metadata, Azure Data Factory (ADF), Azure Data Lake, APIs, DAX, CI/CD Pipelines, Excel 365, CSV Import, CSV Export, Logistics, Business Analysis, AWS Certified Solution Architect, Reports, Communication, Data Architecture, Data Migration, Amazon EventBridge, SSIS Custom Components, Customer Data, Data Governance, Board Reporting, AI Data Classification, Document Parsing, Business Processes, Financial Accounting, Management Accounting, Finance, Enterprise Systems, MicroStrategy, Scrum Master, SAP, SAP Business Intelligence (BI), Data Warehousing, BI Reporting, Amazon RDS, Dashboards, ServiceLedger, Intuit QuickBooks, QuickBooks Online, Azure Databricks, Delta Lake, Data Quality Management, Data Quality Analysis, Data Quality, Tableau Server, Web Scraping, SSRS Reports, WhereScape, Data Build Tool (dbt), Geospatial Data, Microsoft 365, API Integration, Technical Architecture, Monitoring, Data Auditing, Cloud, Solution Architecture, Expensify, QuickBooks Desktop, Finance Reports, Matillion, Azure Blob Storage, Unity Catalog, Data Structures, TMS, Data Reporting, Financial Analysis, Pivot Tables, Change Data Capture, Production Support, Data Strategy, Looker Studio, Accessible Design, Web Content Accessibility Guidelines (WCAG), Amazon Managed Workflows for Apache Airflow (MWAA), Language Models, Large Language Models (LLMs), AI Model Integration, GitHub Actions, Retrieval-augmented Generation (RAG), Vector Databases, Artificial Intelligence (AI), Webhooks, Fivetran, OpenAI
Collaboration That Works
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
1
Share your needs
Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2
Choose your talent
Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3
Start your risk-free talent trial
Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.
Top talent is in high demand.
Start hiring