Huajun is available for hire

Huajun Gao

Verified Expert in Engineering

DB and Data Lake Developer

Location

Glen Huntly, Victoria, Australia

Toptal Member Since

March 19, 2021

Huajun is an experienced senior developer who excels in end-to-end cloud data solutions, optimizing efficiency and accessibility. Their expertise in Azure, Snowflake, and Hadoop enables seamless migration and pipeline enhancement. With strong skills in CI/CD, infrastructure management, and Python programming, they ensure smooth development and efficient data exports. Clients can trust Huajun's reliable support and troubleshooting, delivering exceptional results.

Databricks ETL Hadoop Azure Data Lake Data Engineering Spark SQL Bash Apache Hive Jenkins Java Data Pipelines Python Python 3 Scala Azure Data Factory

Portfolio

Coles Supermarkets

Azure Databricks, Azure Data Factory, Azure Data Lake

Servian

Azure, Hadoop, Spark, Bash, Scala, Python

Experience

Spark - 4 years Databricks - 4 years ETL - 4 years Azure - 4 years Data Pipelines - 3 years Hadoop - 3 years Data Engineering - 3 years Azure Data Factory - 2 years

Availability

Part-time

Preferred Environment

Azure Databricks, Azure Data Factory, Python

The most amazing...

...cloud migration project I led improved data processing efficiency and accessibility, resulting in cost reduction and enhanced performance.

Work Experience

Senior Developer

2021 - PRESENT

Coles Supermarkets

Spearheaded the design and implementation of an end-to-end data solution, resulting in a 40% increase in data processing efficiency and enhanced accessibility.
Developed and managed robust event-driven data processing pipelines, handling large volumes of data on a daily basis and achieving a 60% reduction in processing time.
Orchestrated and optimized ETL pipelines, resulting in a 30% faster data landing onto the cloud data lake and substantial cost savings.

Technologies: Azure Databricks, Azure Data Factory, Azure Data Lake

Consultant

2018 - 2021

Servian

Developed comprehensive ETL pipelines, improving data accessibility and processing speed by 80% on an on-premises Hadoop system.
Streamlined routine tasks through automation tools, improving productivity and operational efficiency.
Documented and implemented CI/CD best practices, reducing code deployment time and enhancing development process efficiency.

Technologies: Azure, Hadoop, Spark, Bash, Scala, Python

Experience

Architect Azure Solution for Cloud Migration

I designed and implemented a full, end-to-end solution for a cloud migration project from on-premises Hadoop to Azure. The technical stack on Azure included Azure Data Lake Storage Gen2, Azure Data Factory, and Databricks, with Databricks Delta as the sink format.

Skills

Frameworks

Hadoop, Spark

Paradigms

ETL, Database Design

Platforms

Azure, Databricks

Other

Azure Data Factory, Azure Data Lake, Networking, Data Engineering, Software Development, Azure Databricks

Languages

Bash, SQL, Java, Python, Scala, Python 3

Tools

Jenkins

Storage

Apache Hive, Data Pipelines, Google Cloud

Education

2016 - 2017

Master's Degree in Information Technology

Monash University - Australia

Certifications

NOVEMBER 2019 - PRESENT

Microsoft Certified Azure Fundamentals

Microsoft

JULY 2019 - JULY 2021

Associate Cloud Engineer

Google Cloud

AUGUST 2014 - AUGUST 2016

CCIE Routing & Switching

Cisco

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring