Hao Luo
Verified Expert in Engineering
Data Engineer and Developer
London, United Kingdom
Toptal member since January 14, 2021
Hao is a data engineering manager who's led complex and innovative projects in the fields of banking/investment. When a new challenge arises, Hao can always identify paths to solutions while dealing with uncertainties. Although Hao's main focus is in data engineering, he has broad knowledge from full-stack development to DevOps engineering.
Portfolio
Experience
- Python - 8 years
- SQL - 8 years
- Data Engineering - 6 years
- Azure - 4 years
- Hadoop - 4 years
- Spark - 4 years
- Docker - 4 years
- Google Cloud Platform (GCP) - 2 years
Preferred Environment
Python, Azure, Google Cloud Platform (GCP), React, Go, Hadoop, Spark, Python 3, Kubernetes, Docker
The most amazing...
...thing I've done was to redesign a large data system to fit with the project's modern architecture.
Work Experience
Data Engineering Manager
Top UK Financial Institution
- Delivered six financial data ELT (extract, load, transform) data projects with reporting visualization. The data comes from multiple sources such as APIs, folders, and emails.
- Designed and developed a high-performance time-series aggregation engine to support highly complex equation aggregation rules with custom functions. It supports different data layers such as files, RDBMS, Parquet, Hive, BigQuery, etc.
- Reduced the data point modeling (XBRL) schema update effort from three months every year to only one week by delivering a low-maintenance solution.
- Reduced the cost of training new developers by developing a chatbot with BERT.
- Designed the organization's first cloud big data solution with an options paper (Google Cloud and Azure).
- Supported two high-performance data engineers who are now technical leads.
Data Engineer
Top UK Financial Institution
- Optimized a data processing pipeline that is now five times faster.
- Delivered a critical functional change to an end-user application within one day of the request from production.
- Built an automation functionality in a time series data system to save three days of business effort every quarter.
- Developed a continuous integration and deployment pipeline that saved five hours of manual build and deployment every month.
- Led the delivery of a data analysis reporting sharing platform by using React and .NET Core.
Software Developer
University of Greenwich
- Developed an inventory management system by using C# and WPF to replace Excel.
- Reduced the asset support daily effort from three hours to ten minutes by establishing a SharePoint workflow to automate the asset support process.
- Built a motivation system to encourage individuals to join events by using C# and MVC.
Software Developer
Qin Dynasty Ltd.
- Delivered an automatic postage tracking system by consuming APIs.
- Automated the dispatch process by integrating the APIs from eBay, Amazon, and other custom shopping portals.
- Developed a web scraping tool to obtain competitor information.
Experience
AI-driven Social Distancing Tool
It uses:
1. Python and Flask as the web API to serve data.
2. Python and Image AI as the image process engine.
3. React as the front end.
4. MongoDB as the data storage.
Big Data Cluster Docker Images
http://github.com/sporule/big-data-clusterIt supports key big data components:
• Airflow
• Hadoop
• Hive
• Spark
• Nifi
• Livy
• Kafka
Data Basket
http://github.com/sporule/data-basketSporule
Key Features:
1. RSS output
2. Schedule article publishing
3. Theme integration
4. Offline search
5. Progressive web app
Werewolf Chatbot
Grater | Distributed Web Scraping Tool
https://github.com/sporule/graterKey Features:
1. It supports unlimited scale-out worker nodes.
2. It supports transformation and validation in the process of scraping.
3. It supports complex custom scraping rules.
4. All events are configuration based.
5. It supports automatic proxy obtaining through scraping and APIs.
Education
Master's Degree in Central Banking and Financial Regulation
Warwick Business School - London, United Kingdom
Bachelor's Degree in Computer Science
University of Greenwich - London, United Kingdom
Certifications
Azure Solutions Architect Expert
Microsoft
Professional Cloud Architect
Google Cloud
Professional Data Engineer
Google Cloud
ITIL Foundation Certification
British Computer Society
Skills
Libraries/APIs
REST APIs, React, Windows Forms (WinForms), Telegram Bot API
Tools
Tableau, Microsoft Power BI, Apache Airflow, Webpack
Languages
C#, Python, SQL, Snowflake, HTML, CSS, JavaScript, R, Go, Java, C++, UML
Frameworks
Spark, Hadoop, Flask, ASP.NET, Windows Presentation Foundation (WPF), MVC Razor, Gin-Gonic
Paradigms
ETL, Web Architecture, DevOps
Platforms
Docker, Google Cloud Platform (GCP), Azure, SharePoint, Firebase, Kubernetes
Storage
Data Pipelines, Database Modeling, Database Migration, NoSQL, PostgreSQL, Apache Hive, Data Lakes, Microsoft SQL Server, MongoDB
Other
Data Engineering, ELT, Data Modeling, Big Data, Data Cleaning, Data Cleansing, Team Leadership, Architecture, Data Analytics, ETL Tools, Data Warehousing, Data Architecture, Data Analysis, Google BigQuery, Complex Data Analysis, Quality Assurance (QA), Data Marts, Dashboards, Data Visualization, Azure Data Factory (ADF), Data Aggregation, Performance Tuning, Finance, Credit Risk, Leadership, Risk, Economics, Projects, ITIL V3 Foundation Certified, Machine Learning, Chatbots, Scraping, APIs, Machine Learning Operations (MLOps)
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring