Wei Ding
Verified Expert in Engineering
Data Engineer and Developer
Sydney, New South Wales, Australia
Toptal member since September 27, 2022
Wei is a senior system engineer with over a decade of experience in system development in the Australian finance and media fields. He has a bachelor's degree in engineering and two master's degrees in engineering management and IT. Wei specializes in data modeling and system solution architecture and has delivered Python, Java, and Spark applications.
Portfolio
Experience
- SQL - 7 years
- Databases - 7 years
- Docker - 5 years
- Python - 5 years
- Data Modeling - 5 years
- Amazon Web Services (AWS) - 3 years
- Java - 3 years
- Spark - 2 years
Work Experience
Senior Engineer
ANZ Group
- Developed big data cluster and Spark-based data pipelines.
- Carried out extensive performance tuning in the Spark application.
- Delivered components in Java online analytic processing applications for banking regulatory compliance.
- Refined continuous integration and delivery (CI/CD) pipelines and test strategies for projects.
- Conducted interview screening and mentored new members.
Analyst Developer
Network TEN
- Led a team of three developers and one business analyst and successfully delivered the project.
- Developed back-end services for enterprise systems.
- Implemented extract, transform, and load (ETL) data pipelines and required infrastructure services.
- Designed data warehouse modeling and technical solutions.
- Handled performance tuning in the relational database.
Front-end Developer
FreightExchange Pty Ltd
- Focused on building the front-end components of the company website.
- Developed a web form that allows users to submit their shipping price quotes.
- Created CSS consistent across browsers and devices.
IT Intern
The Walt Disney Company
- Assisted with data migration for the contract management system migration project.
- Conducted work using Excel and DocuSign Selectica.
- Organized daily progress reports and reported to the project lead.
Project History
Market Risk Viewing Solution
I implemented multi-threaded tasks and improved source data extraction performance by four times. I also delivered an Alluxio in-memory file system, which improved Hive query performance. Additionally, I optimized the Parquet partition and improved the Spark application performance.
The stacks I used in this project include Spark, Airflow, S3, Alluxio, Hive, Presto, Docker, and Tableau. The project also involved a daily data size of over 50 gigabytes of compressed data and over 500 gigabytes of uncompressed data.
Broadcast System Integration Solution
I designed a data modeling solution and set up a project solution architecture, including application services and CI/CD. I also managed the development scrum, business requirements, user acceptance testing, and product release.
The stacks I used in this project include Python, Flask, Oracle, PostgreSQL, AWS, Docker, and Airflow.
Dynamic Trading Platform
I developed API services and a batch ETL pipeline. I also improved the platform's process performance, reducing the time needed to deliver the end product from half an hour to mere seconds.
Education
Master's Degree in Information Technology
University of Technology Sydney - Sydney, New South Wales, Australia
Master's Degree in Engineering Management
University of Technology Sydney - Sydney, New South Wales, Australia
Bachelor's Degree in Engineering
North China Electric Power University - Beijing, China
Certifications
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks
Skills
Libraries/APIs
Spark ML
Tools
Apache Airflow, Git, Apache Maven, Tableau, Jenkins, Microsoft Excel, Amazon EKS, MATLAB, Spark SQL
Languages
Python, SQL, Java, Snowflake, Python 3, C, JavaScript, CSS, HTML
Frameworks
Apache Spark, Spark, Flask, Spring, Presto
Paradigms
Testing, Business Intelligence (BI), Agile, ETL, DevOps, OLAP, Agile Project Management
Platforms
Docker, Kubernetes, Amazon Web Services (AWS), Linux, Oracle, AWS IoT, Spark Core
Storage
Databases, PL/SQL, Data Pipelines, PostgreSQL, Amazon S3 (AWS S3), Apache Hive, Alluxio
Other
Programming, System Design, Systems Engineering, Data Modeling, Data Engineering, IT Project Management, Business Systems Analysis, Big Data Architecture, Project Leadership, Excel 365, Parquet, Cisco, Leadership, Analysis, Solution Architecture, APIs, AWS Database Migration Service (DMS)
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring