Senior Data Engineer2021 - PRESENTWILBUR
Technologies: Databricks, Python, SQL, Unix, AWS, Azure, Datafactory, Terraform, Windows PowerShell, Pandas, SQL Server 2016, Azure SQL Data Warehouse (SQL DW)
- Designed and built a scalable data vault in Delta Lakehouse for multiple source systems and their tenants, using AWS, Databricks, a database, Python, Powershell, and Unix. Provisioned environments in AWS using Terraform.
- Developed notebooks in Databricks (dynamic) to load an enterprise Delta Lakehouse. Configured the Databricks platform from scratch on AWS and orchestrated the design of a Delta Lakehouse (data vault model).
- Made high performance dashboards available to insurance clients with architecture, a total win-win for Wilbur as a third party.
Senior Data Engineer2020 - 2021Transport for NSW
Technologies: AWS, Talend ETL, Python, Amazon Aurora, Datalake, Data Warehouse Design, Redshift, AWS Athena
- Developed dashboards and data pipelines to project near real-time traffic travel times from point A to Point B. The Government used this data to make the lives of citizens easier.
- Built real-time dashboards in Tableau and dynamic data pipelines in Talend. Managed the data feed from different sources and integrated them in a data warehouse in Aurora and Redshift. Made Data Lakes in Amazon Athena that housed data for ML models.
- Showcased traffic patterns and helped government stakeholders to make decisions for building or maintaining infrastructure to help the citizens of the region.
Senior Data Analyst2017 - 2019Jones Lang LaSalle
Technologies: Databricks, Azure, Python, Informatica ETL
- Architected and developed data workflow for multiple source systems. Data was shared across the globe for organizational groups providing them with daily insights to make business decisions.
- Developed data pipelines in Databricks on Azure Cloud, dynamic notebooks were built that could handle any data loading strategy for a data warehouse and data vault. Developers and business users were made aware of data status on a day-to-day basis.
- Created dynamic and scalable architecture so that build time was significantly under control. 360-degree views of businesses have helped real estate agents to ace business deals.
- Developed data feeds on Informatica PowerCenter and Informatica Cloud to ingest into EDW.
Senior Consultant2012 - 2017Deloitte
Technologies: AWS, Informatica ETL, Talend ETL, Python, Unix, Redshift
- Migrated client's data warehouse and data pipelines to the cloud from on-premise.
- Architected and developed data pipelines to ingest data feeds into a data warehouse, using Informatica ETL and Talend ETL. Created a library of Unix functions to reduce the build time in projects and to promote function reusability.
- Transitioned to the cloud, saving up to two million dollars in licensing, administration, and maintenance. Scalable architecture was reused in multiple projects that resulted in bringing down the build time and overriding all manual tasks.