Data Engineer ||
2022 - PRESENTAmazon.com- Supported and developed a big data pipeline for machine learning models. Managed ETL services for a large-scale data warehouse for consumer transaction data.
- Provided automation solutions for existing manual steps to maintain metadata for big data pipeline, saving client 99% times of processing.
- Provided automation solutions to workflows maintained by data scientists and data analysts.
Technologies: Python, SQL, PySpark, AWS EMR, Redshift, Amazon S3 (AWS S3), AWS Glue, AWS LambdaSenior Database and Data Warehouse Engineer
2019 - 2022Honda- Maintained and supported 150+ OLTP database ETL processes and 20+ OLAP data warehouse ETL processes. Resolved 99.9% of production issues, including complaints from the app and analytics teams regarding process failure or data questions.
- Completed a sensitive data migration project, including PII and non-PII data. The project included sending a data feed to the cloud by decrypting sensitive info and processing the reverse feed sent from the cloud into an on-premise database.
- Developed an ad-hoc process to check and load the missing data back and passed the validation; the data warehouse had missed some data, which resulted in data inaccuracies that the business team and analytics team questioned.
- Created a program that can quickly search programs/scripts by entering keywords to solve the issue that many legacy database ETL processes that didn't document well; before I took over, many processes were only recalled by human memory alone.
- Found the root cause of performance issues with a large back-end database store process returning hierarchy JSON data for API usage (which caused the API to time out) and resolved the timeout issue.
- Migrated a database and data warehouse to the cloud and tested and ensured that all the database and ETL programs worked fine after the migration.
- Guided and coached junior developers on the overall workflow and tasks; also reviewed essential programs developed and/or modified by junior developers.
- Enhanced the weblog ETL process by using Spark to read JSON logs and create partition files in AWS S3.
- Built a high-quality payment estimator data workflow, including an ETL process and store procedures returning data for web API usage.
- Created helper programs to reduce the repeatable human work, including but not limited to Python and C# class to work with the database server, FTP, SFTP, file system, file security, cloud, and Spark.
Technologies: SQL, Python, SQL Server Integration Services (SSIS), JSON, XML, APIs, C#Database Engineer
2019 - 2019Honda- Completed an ETL project in two weeks and deployed it to production; it loads large text files to a relational database and it had been left unfinished by the previous contractor due to its complexity.
- Provided business intelligence solutions by creating and maintaining business intelligence reports and data visualizations.
- Worked with the project team on database design, database programs, and development.
Technologies: SQL, Python, C#, SQL Server Integration Services (SSIS), Data Modeling, Dimensional ModelingDatabase Engineer
2018 - 2019Helm360- Assisted a senior database engineer with database development.
- Designed a relational data model to meet project needs.
- Contributed to the development of ETL pipelines and business intelligence reports.
- Wrote SQL functions and store procedures for use by the API and ETL data pipelines.
Technologies: SQL, SQL Server Integration Services (SSIS), C#, Python, Data Modeling, Dimensional Modeling, Azure, Microsoft Power BI
Hang Guo
Database Engineer and Developer in Irvine, CA, United States
Member since May 20, 2021
Hang is a data and database engineer who excels at writing programs and obtaining data insights. He's worked with a variety of data sources and destinations, including on-premise databases (such as SQL Server and PostgreSQL), cloud databases (such as AWS Redshift), and big data technologies (such as Spark). Hang also has experience working with application teams for back-end database development, ETL, and data pipelines and in creating business intelligence solutions with the analytics team.
Hang is now available for hire
Portfolio
- Amazon.comPython, SQL, PySpark, AWS EMR, Redshift, Amazon S3 (AWS S3), AWS Glue...
- HondaSQL, Python, SQL Server Integration Services (SSIS), JSON, XML, APIs, C#
- HondaSQL, Python, C#, SQL Server Integration Services (SSIS), Data Modeling...
Experience
- Data Modeling 3 years
- SQL Server Integration Services (SSIS) 3 years
- ETL 3 years
- SQL 3 years
- Python 3 years
- JSON 2 years
- PySpark 1 year
- Amazon S3 (AWS S3) 1 year
Availability
Part-time
Preferred Environment
PyCharm, SQL Server Management Studio, pgAdmin, Jupyter Notebook, Amazon Web Services (AWS)
The most amazing...
...project was supporting and developing the back-end databases, data warehouse, and ETL pipeline for the app team and analytic teams at a Fortune 500 company.
Employment
Experience
- Support and Development of a Powersports Website Database and ETL
Tasks Accomplished:
• Supported and enhanced a back-end database and ETL process for the main web application API.
• Supported cross-environment data publishing and data validation.
• Troubleshot database performance issues caused by large database programs. - Inventory Data Flow and Database
Tasks Accomplished:
• Built and supported an auto inventory and data management database (OLTP).
• Created an ETL program to manage inventory and offer data flow across different environments.
• Developed database programs for the use of the Web API. - Cloud Sensitive Data Migration
The project involved migrating sensitive PII information from an on-premise database to the cloud, and I was the main developer and created a custom ETL package.
Custom ETL Package Process:
• Extracted and decrypted the PII information.
• Transformed and applied the cloud business rule to the data and sent it to the destination.
• Retrieved data from the cloud and built an automation process to populate cloud data back to the on-premise database. - Weblog Data ETL Pipeline
Tasks Accomplished:
• Created ETL data pipeline loads for JSON weblogs from AWS S3 using PySpark; it then writes the Parquet files back to AWS S3.
• Developed a program that quickly reads and analyzes these Parquet files using PySpark.
The company's consumer application team uses the weblog ETL/analyze process. - Jira System's Data ETL and Generation of a Campaign Calendar
Project Process:
• Extracted the data from the old ticketing system and loaded it into the Jira ticketing system.
• Extracted the data from the Jira ticketing system daily and loaded it into a data warehouse.
• Transformed the data into a format required by the business.
• Generated a marketing campaign calendar based on the data set and then the calendar was emailed. - Enhancement of a Data Warehouse Lead System
This is the process that loads the feeds sent by third-party vendors to a data warehouse. The old process used fuzzy logic to map the lead information between the OLTP and OLAP and sent the result to the analytics team, which cause data inaccuracies in rare cases.
We decide to enhance the logic by introduce a unique identifier between the two systems. The process involves a redesign of a few tables belonging to the data warehouse and modifying the ETL process.
Skills
Languages
Python, SQL, C#, C#.NETParadigms
ETL, Dimensional ModelingStorage
SQL Server Integration Services (SSIS), JSON, Amazon S3 (AWS S3), Redshift, NoSQL, Cassandra, SQL Server Management Studio, PostgreSQLOther
Data Modeling, Big Data, APIs, Business Administration, Data Analysis, Machine LearningFrameworks
Flask, AWS EMRLibraries/APIs
PySpark, SQLAlchemy, Flask-RESTfulTools
Apache Airflow, Microsoft Power BI, PyCharm, pgAdmin, AWS GluePlatforms
Amazon EC2, Jupyter Notebook, Linux, AWS Lambda, Amazon Web Services (AWS)
Education
- Master's Degree in Information Technology2013 - 2015University of La Verne - La Verne, CA, United States
- Bachelor's Degree in Business Administration2006 - 2010Guang Dong University of Technology - Guang Zhou, China
Certifications
- Machine Learning EngineerAUGUST 2021 - PRESENTUdacity
- Full-stack Web DeveloperMAY 2021 - PRESENTUdacity
- Data EngineeringMARCH 2021 - PRESENTUdacity
