Wenjie Xu

This was a PySpark-based ETL project developed using Spark-SQL and Python script to transform CSV/XML data into a relational database. Te project integrated data from various formats and sources into the data warehouse to ensure that target data adhered to data quality and accessibility standards. On top of the transformed data, I developed a dashboard using Power BI to provide actionable insights.

I used the Scrapy framework to capture construction projects' bidding info on agency websites (MTA, Port Authority) to deliver clean and reliable bidding data to the marketing department. This data was used for further decision-making purposes. This project captured over 500+ records per day from various websites, which replaced manual searching and significantly improved work efficiency for the marketing team.

I built a Python-Flask-based data entry system independently to enable data quality check-up functionality and eliminate data noise. I automated the data acquisition process to reduce human errors significantly. A dynamic dashboard based on Power BI was also deployed on this system and tracked data trends and insights as new data fed in. This system was a configured server environment on AWS EC2 and RDS with reliable security groups.