Verified Expert in Engineering
Data Warehouse Design Developer
Nigel is a senior software and data engineer on Cloud, Linux, AWS, GCP, Snowflake, Hadoop, and almost all computer and database platforms. He's led and contributed to eCommerce and self-driving startups as well as the world's largest brokerage, retail, semiconductor, communication, network, and storage enterprises on the data analytics, ETL data pipeline, transaction processing, self-driving, and data science teams.
Amazon Web Services (AWS), Python, Redshift, Hadoop, Snowflake, Linux
The most amazing...
...thing that I've worked on as the only available data engineer was a $200 million eCommerce startup to run 25+ data pipelines and support all business teams.
Senior Data Engineer
- Performed as a member of the Workforce Staffing data engineering team. Developed ETL pipelines, data mapping, modeling, data lake, and data flow to fill labor order.
- Developed Airflow DAG for tasks, operator, and connection with Python and SQL.
- Worked with business intelligence engineers and a data analyst to create dashboards.
Senior Data Engineer
- Took part in the eCommerce ROI data engineering team. Developed ETL pipelines, data mapping, modeling, and data flow for 20+ advertising media sources, including Nielsen, Google, Amazon, Facebook, Twitter, OMD, and more.
- Developed an Airflow DAG for tasks, operator, and a connection variable that brings data from AWS S3 into Snowflake.
- Developed a data vault schema and table in Snowflake. Supported the Snowflake database, role, warehouse, schema, and table.
- Created a self-driving car fleet management system analytics, data pipelines, and ETL.
- Used AWS Redshift, EC2, S3, Python, Database Migration Service, and MongoDB.
- Developed data pipelines and data flow of vehicle heartbeats, and weather API data. Fed Tableau analytics dashboards.
- Developed a B2B customer contact hub dataset. Supported machine learning, AI, software renewal, NPS survey, and sales campaign automation.
- Built a data pipeline framework, guidelines, production procedures, data architecture, and code review process. Led and educated junior Python developers.
- Developed an internal Salesforce contact dataset and sync it with an external Salesforce object.
- Migrated data foundation from Hadoop to Snowflake, GCP BigQuery, GCE, GCS, Airflow, and Cloud Gateway Server.
Big Data Engineer
- Developed and supported enterprise data management big data engineering for worldwide head and drive wafer fab production image and data ETL pipelines.
- Rebuilt, managed, and tuned large production enterprise data management AWS Redshift clusters to allow large volume pipelines and user queries.
- Supported AWS Redshift, Redshift Spectrum, ElasticSearch, Kinesis, S3, EC2, RDS, MySQL, PostgreSQL, Aurora, and CloudWatch. Managed Control-M, Spotfire, and SnapLogic ETL.
- Supported the wafer images defect model machine learning platform.
- Worked with Slack, Hadoop, Hive, Impala, Python, NumPy, SciPy, SVM, SVD, GitHub, Bitbucket, Jenkins, Tidal, Java, Jira, Wiki, and Confluence.
Lead Data Engineer
- Developed and maintained an online shopping eCommerce data engineering, data analytics, 25 ETL pipelines, and a data warehouse as the only available data engineer.
- Constructed and managed Salesforce eCommerce Cloud (aka Demandware), Square POS, eCommerce replication Percona FelexCDC, Adobe Omniture Marketing Cloud, Oracle Responsys, ScientiaMobile WURFL, Qualtrics, Zodiac, ShopKeep, Acuity, and RetailNext.
- Developed data pipelines with various vendors using GitHub, Python, C/C++, Java, REST API, JSON, XML, CSV, TSV, Jira, and Slack.
- Designed Azure migration of Azure SQL Data Warehouse, Blob Storage, and Linux VM.
Software System Engineer
- Built a new portfolio accounting system on Linux as the very first engineer.
- Led the sparks team and built a cost basis accounting system and a reporting repository data warehouse.
- Built and supported Eagle Investment Systems STAR and PACE products.
- Supported and migrated the mainframe-based system to RedHat Linux/Solaris VMware server and 100TB+ scale Oracle 9/10/11/12 RAC/TAF/EMC/HDS based DataGuard/Golden Gate environments.
- Developed and supported partitioning, parallel processing, ESP scheduling, high availability/failover, disaster recovery, Tivoli monitoring, Splunk, and Zenoss.
- Implemented and supported both the development and production of OLTP, OLAP, ETL, distributed Messaging (MQ), iPlanet/Apache, Application Server, Oracle 9/10/11/12 RAC databases, and DataGuard.
- Built and supported multiple TB scale development and performance/volume/stress testing environments.
- Developed systems and applications with Java, Perl, Shell, Python, SQL, PL/SQL, and XML languages.
- Educated the team with SQL and RDBMS, MySQL/SQL Server, and a data-driven documents library.
Enterprise Sales AI Data Engineering
The pipeline includes eight tasks: data extraction and ingestion, data deduplication, data transformation, data incremental load, data filtering, offer data generation, offer motion data generation, and data enrichment. GitHub, JSON, XML, JIRA, Wiki, and Confluence. Fully and solely completed Hadoop to Snowflake migration. Incubated junior engineers.
eCommerce Data Pipeline Migration
Brokerage Portfolio Accounting System
Enterprise Data Management
Python, C, Bash, SQL, Snowflake, Java, Perl, Python 3, C++
PySpark, REST APIs, NumPy, SciPy, Spark Streaming
MongoDB Shell, Jira, GitHub, Google Compute Engine (GCE), Amazon Elastic MapReduce (EMR), Cisco Tidal Enterprise Scheduler, Spark SQL, BigQuery, Docker Compose, Tableau, Google Cloud Dataproc, Google Cloud Composer, Apache Airflow, Kafka Streams
Amazon EC2, Google Cloud Platform (GCP), Linux, Oracle, Red Hat Linux, Amazon Web Services (AWS), Salesforce
MongoDB, Amazon S3 (AWS S3), MySQL, PostgreSQL, Apache Hive, Elasticsearch, JSON, Redshift, Google Cloud Storage, Google Cloud SQL, Datastage
Machine Learning, Data Warehouse Design, Software Development, Google BigQuery, eCommerce, Data Warehousing, Tableau Server, Document Management Systems (DMS), Kubernetes Operations (kOps), Data Build Tool (dbt), Argo CD, CI/CD Pipelines
Master of Science Degree in Computer Science
Indiana University - Bloomington, Indiana
Bachelor of Science Degree in Engineering
National Taiwan University - Taipei, Taiwan