Wei Ding, Developer in Sydney, New South Wales, Australia
Wei is available for hire
Hire Wei

Wei Ding

Verified Expert  in Engineering

Data Engineer and Developer

Location
Sydney, New South Wales, Australia
Toptal Member Since
September 27, 2022

Wei is a senior system engineer with over a decade of experience in system development in the Australian finance and media fields. He has a bachelor's degree in engineering and two master's degrees in engineering management and IT. Wei specializes in data modeling and system solution architecture and has delivered Python, Java, and Spark applications.

Portfolio

ANZ Group
Java, Python, Linux, OLAP, Docker, Spark, SQL, Apache Airflow, Tableau, Jenkins...
Network TEN
Project Leadership, Amazon Web Services (AWS), Python, Linux, Databases, PL/SQL...
FreightExchange Pty Ltd
JavaScript, CSS, HTML, Git, Programming

Experience

Availability

Part-time

Preferred Environment

Linux, Docker, Java, Spark, Amazon Web Services (AWS), Python, Databases, SQL

The most amazing...

...work I've delivered was a re-engineered solution for a dynamic booking system, which improved the process performance from half an hour to seconds.

Work Experience

System Engineer

2020 - PRESENT
ANZ Group
  • Developed big data cluster and Spark-based data pipelines.
  • Carried out extensive performance tuning in the Spark application.
  • Delivered components in Java online analytic processing applications for banking regulatory compliance.
  • Refined continuous integration and delivery (CI/CD) pipelines and test strategies for projects.
  • Conducted interview screening and mentored new members.
Technologies: Java, Python, Linux, OLAP, Docker, Spark, SQL, Apache Airflow, Tableau, Jenkins, Parquet, Spring, Data Engineering, Programming, Agile, Data Modeling, Business Systems Analysis, DevOps, Microsoft Excel, Apache Spark, Apache Maven

Developer Analyst

2015 - 2020
Network TEN
  • Led a team of three developers and one business analyst and successfully delivered the project.
  • Developed back-end services for enterprise systems.
  • Implemented extract, transform, and load (ETL) data pipelines and required infrastructure services.
  • Designed data warehouse modeling and technical solutions.
  • Handled performance tuning in the relational database.
Technologies: Project Leadership, Amazon Web Services (AWS), Python, Linux, Databases, PL/SQL, Docker, PostgreSQL, Apache Airflow, Data Engineering, Programming, Agile, Data Modeling, SQL, Business Systems Analysis, DevOps, Microsoft Excel

Front-end Developer

2015 - 2015
FreightExchange Pty Ltd
  • Focused on building the front-end components of the company website.
  • Developed a web form that allows users to submit their shipping price quotes.
  • Created CSS consistent across browsers and devices.
Technologies: JavaScript, CSS, HTML, Git, Programming

IT Intern

2011 - 2012
The Walt Disney Company
  • Assisted with data migration for the contract management system migration project.
  • Conducted work using Excel and DocuSign Selectica.
  • Organized daily progress reports and reported to the project lead.
Technologies: Databases, Microsoft Excel, Programming

Market Risk Viewing Solution

Developed big data infrastructure, pipelines, and reporting for banking regulatory compliance.

I implemented multi-threaded tasks and improved source data extraction performance by four times. I also delivered an Alluxio in-memory file system, which improved Hive query performance. Additionally, I optimized the Parquet partition and improved the Spark application performance.

The stacks I used in this project include Spark, Airflow, S3, Alluxio, Hive, Presto, Docker, and Tableau. The project also involved a daily data size of over 50 gigabytes of compressed data and over 500 gigabytes of uncompressed data.

Broadcast System Integration Solution

Managed a team of three developers and one business analyst, successfully delivering the project as the project lead.

I designed a data modeling solution and set up a project solution architecture, including application services and CI/CD. I also managed the development scrum, business requirements, user acceptance testing, and product release.

The stacks I used in this project include Python, Flask, Oracle, PostgreSQL, AWS, Docker, and Airflow.

Dynamic Trading Platform

Delivered a dynamic trading platform which was a new sales/revenue source.

I developed API services and a batch ETL pipeline. I also improved the platform's process performance, reducing the time needed to deliver the end product from half an hour to mere seconds.

Languages

Python, SQL, Java, Python 3, C, JavaScript, CSS, HTML

Frameworks

Apache Spark, Spark, Flask, Spring, Presto DB

Tools

Apache Airflow, Git, Apache Maven, Tableau, Jenkins, Microsoft Excel, MATLAB, Spark SQL

Paradigms

Testing, Business Intelligence (BI), Agile, ETL, DevOps, OLAP, Agile Project Management

Platforms

Docker, Amazon Web Services (AWS), Linux, Oracle, Spark Core

Storage

Databases, PL/SQL, Data Pipelines, PostgreSQL, Amazon S3 (AWS S3), Apache Hive, Alluxio

Other

Programming, System Design, Systems Engineering, Data Modeling, Data Engineering, IT Project Management, Business Systems Analysis, Big Data Architecture, Project Leadership, Excel 365, Parquet, Cisco, Leadership, Analysis, Solution Architecture, APIs, AWS Database Migration Service

Libraries/APIs

Spark ML

2013 - 2015

Master's Degree in Information Technology

University of Technology Sydney - Sydney, New South Wales, Australia

2012 - 2013

Master's Degree in Engineering Management

University of Technology Sydney - Sydney, New South Wales, Australia

2007 - 2011

Bachelor's Degree in Engineering

North China Electric Power University - Beijing, China

NOVEMBER 2022 - PRESENT

Databricks Certified Associate Developer for Apache Spark 3.0

Databricks

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring