Nigel Chang, Data Warehousing Developer in San Francisco, CA, United States
Nigel Chang

Data Warehousing Developer in San Francisco, CA, United States

Member since March 13, 2019
Nigel is a senior software and data engineer on Cloud, Linux, AWS, GCP, Snowflake, Hadoop, and almost all computer and database platforms. He's led and contributed to eCommerce and self-driving startups as well as the world's largest brokerage, retail, semiconductor, communication, network, and storage enterprises on the data analytics, ETL data pipeline, transaction processing, self-driving, and data science teams.
Nigel is now available for hire


  • Cyngn
    Amazon Web Services (AWS), REST API, Jira, Tableau, Python, MySQL, MongoDB...
  • Cisco
    Jira, GitHub, Datastage, BigQuery, Google Cloud Storage...
  • Western Digital
    Amazon Web Services (AWS), Elasticsearch, Bash, PostgreSQL, AWS EC2, AWS S3...



San Francisco, CA, United States



Preferred Environment

Amazon Web Services (AWS), Python, Redshift, Hadoop, Snowflake, AWS, Linux

The most amazing...

...thing that I've worked on as the only available data engineer was a $200 million eCommerce startup to run 25+ data pipelines and support all business teams.


  • Data Engineer

    2019 - PRESENT
    • Created self-driving car fleet management system analytics, data pipelines, and ETL.
    Technologies: Amazon Web Services (AWS), REST API, Jira, Tableau, Python, MySQL, MongoDB, Document Management Systems (DMS), AWS S3, AWS EC2, Redshift, AWS
  • Data Engineer

    2019 - PRESENT
    • Supported machine learning, AI, and sales campaign automation.
    • Built data pipeline framework, guidelines, production procedures, data architecture, and code review process.
    • Led junior Python and PySpark developers.
    • Migrated data foundation from Hadoop to Snowflake.
    Technologies: Jira, GitHub, Datastage, BigQuery, Google Cloud Storage, Google Compute Engine (GCE), Google Cloud Platform (GCP), Snowflake, JSON, Apache Hive, Hadoop, Spark SQL, PySpark, Python
  • Big Data Engineer

    2017 - 2018
    Western Digital
    • Develop and support Enterprise Data Management Big Data Engineering, world-wide head and drive wafer fab production image and data ETL pipelines.
    • Rebuild, manage and tune large production Enterprise Data Management AWS Redshift clusters to allow large volume pipelines and user queries.
    • Support AWS Redshift, Redshift Spectrum, ElasticSearch, Kinesis, S3, EC2, RDS, MySQL, PostgreSQL, Aurora, CloudWatch. Manage Control-M, Spotfire, SnapLogic ETL.
    • Support wafer images defect model Machine Learning platform.
    • Tooled Slack, Hadoop, Hive, Impala, Python, numpy, scipy, SVM, SVD, GitHib, BitBucket, Jenkins, Tidal, Java, JIRA, wiki, Confluence.
    Technologies: Amazon Web Services (AWS), Elasticsearch, Bash, PostgreSQL, AWS EC2, AWS S3, Python, Redshift, AWS
  • Lead Data Engineer

    2015 - 2017
    • Developed and maintained an online shopping eCommerce data engineering, data analytics, 25 ETL pipelines, and data warehouse as the only available data engineer.
    • Constructed and managed Salesforce E-commerce Cloud (aka DemandWare), Square POS, E-commerce replication Percona FelexCDC, Adobe Omniture Marketing Cloud, Oracle Responsys, ScientiaMobile WURFL, Qualtrics, Zodiac, ShopKeep, Acuity, and RetailNext.
    • Developed data pipelines with various vendors using GitHub, Python, C/C++, JAVA, REST API, JSON, XML, CSV, TSV, JIRA, Slack.
    • Designed Azure migration of Azure SQL Data Warehouse, Blob Storage, and Linux VM.
    Technologies: Amazon Web Services (AWS), PostgreSQL, MySQL, Python, Bash, AWS EC2, AWS S3, Redshift, AWS
  • Software System Engineer

    2002 - 2015
    Charles Schwab
    • Built new portfolio accounting system on Linux as the very first engineer.
    • Led SPARKS team and built Cost Basis Accounting System, Reporting Repository Data Warehouse.
    • Built and supported Eagle Investment Systems STAR and PACE products.
    • Supported and migrated mainframe based system to RedHat Linux/Solaris VMware server and 100TB+ scale Oracle 9/10/11/12 RAC/TAF/EMC/HDS based DataGuard/Golden Gate environments.
    • Developed and supported partitioning, parallel processing, ESP scheduling, high availability/failover, disaster recovery, Tivoli monitoring, Splunk, and Zenoss.
    • Implemented and supported both development and production OLTP, OLAP, ETL, distributed Messaging (MQ), iPlanet/Apache, Application Server, Oracle 9/10/11/12 RAC databases, and DataGuard.
    • Built and supported multiple TB scale development and performance/Volume/Stress testing environments.
    • Developed systems and applications with JAVA, Perl, Shell, Python, SQL, PL/SQL, and XML languages.
    • Educated team with SQL and RDBMS, MySQL/SQL Server, and Data Driven Documents library.
    Technologies: SQL, Bash, Perl, Red Hat Linux, Oracle, Linux


  • Enterprise Sales AI Data Engineering (Development)

    Sales AI campaign data science required data pipeline and machine learning models. Major sources from external REST API, Webhook, and S3. Snowflake, GCP, Hadoop, and Hive. Primarily email and phone contact. Hadoop, Spark, PySpark, Hive, Google Could Platform, and Snowflake.

    The pipeline includes eight tasks: data extraction and ingestion, data deduplication, data transformation, data incremental load, data filtering, offer data generation, offer motion data generation, and data enrichment. GitHub, JSON, XML, JIRA, Wiki, and Confluence. Fully and solely completed Hadoop to Snowflake migration. Incubated junior engineers.

  • eCommerce Data Pipeline Migration (Development)

    Developed and supported 25+ eCommerce product, transaction, KPI, marketing, merchandising, planning, finance, fraud detection, LTV, Square, REST API, and fulfillment data pipelines single-handed. Migrated in-house built transaction and ETL pipelines AWS Redshift, MySQL, MongoDB, Postgres, EC2, S3, Data Migration Services to Salesforce Commerce Cloud and Azure. Tableau Server and desktop dashboards. Pipelined 20+ marketing partners including Google Analytics, Adobe Omniture, Oracle Responsys, and Salesforce Marketing Cloud. Python, Bash, JSON, XML, JIRA, Slack.

  • Brokerage Portfolio Accounting System (Development)

    Build a new and first Linux and Oracle-based portfolio accounting system for the largest brokerage firm in the West Coast with 16 million customers.

  • Enterprise Data Management (Development)

    Built wafer data ETL pipelines from wafer factories all over the world for the world's largest storage company. AWS Redshift, EC2, S3, Elastic Search, JSON, Python, Teradata, SQL Server, Oracle, and Control M. Re-created all the tables in Redshift to make it perform.

  • Self-Friving Analytics (Development)

    Built data pipelines for self-driving car company fleet management system with real-time heartbeats, analytics dashboards, and products. AWS Redshift, EC2, MySQL, MongoDB. AWS DMS, Python, and JSON Tableau.


  • Languages

    Python, C, Bash, SQL, Snowflake, Java, Perl, C++
  • Frameworks

    Hadoop, AWS EMR
  • Libraries/APIs

    PySpark, REST APIs, NumPy, SciPy, REST API, Spark Streaming
  • Tools

    MongoDB Shell, Jira, GitHub, Google Compute Engine (GCE), Cisco Tidal Enterprise Scheduler, Spark SQL, BigQuery, Tableau, Google Cloud Dataproc, Google Cloud Composer, Apache Airflow, Kafka Streams
  • Paradigms

  • Platforms

    AWS EC2, Google Cloud Platform (GCP), Linux, Oracle, Red Hat Linux, Amazon Web Services (AWS)
  • Storage

    MongoDB, AWS S3, MySQL, PostgreSQL, Apache Hive, Elasticsearch, AWS RDS, JSON, Redshift, Google Cloud Storage, Google Cloud SQL, Datastage
  • Industry Expertise

  • Other

    Machine Learning, Software Development, Google BigQuery, Data Warehousing, Tableau Server, AWS, Document Management Systems (DMS)


  • Master of Science degree in Computer Science
    1986 - 1987
    Indiana University - Bloomington, Indiana
  • Bachelor of Science degree in Engineering
    1978 - 1982
    National Taiwan University - Taipei, Taiwan

To view more profiles

Join Toptal
Share it with others