Senior Manager Data Engineering, ETL2020 - PRESENTPluto TV, the ViacomCBS company
Technologies: Spark, PySpark, Scala, Databricks, Snowflake, Apache Airflow, SQL, AWS, AWS Athena
- Built a Revenue Data Mart and added a server-side subject area to the Data Lake.
- Managed a team and oversaw ETL monitoring, optimization, and performance tuning.
- Represented the Data Engineering team in the company's Architecture Guild activities.
Data Engineer2020 - PRESENTMaisonette (via Toptal)
Technologies: Amazon Web Services (AWS), Fivetran, Looker, Python, Apache Airflow, Snowflake, PostgreSQL, AWS
- Built a data platform and data lake.
Consultant | Co-founder | CEO2016 - PRESENTCrowd Consulting
Technologies: Amazon Web Services (AWS), Data Warehouse Design, Data Warehousing, AWS Athena, Tableau, Luigi, Scala, Python, AWS S3, AWS DynamoDB, MySQL, PostgreSQL, Redshift, AWS, AWS Lambda, Apache Hive, Databricks, Spark, Hadoop, AWS EMR
- Worked on full data warehouse implementations for multiple clients.
- Provided big data training and support.
- Engineered and built an ETL pipeline for AWS S3 data warehouse using AWS Kinesis, Lambda, Hive, Presto, and Spark. The pipeline was written in Python.
Data Engineering Architect2020 - 2020CVS Health (via Toptal)
Technologies: RAPIDS, Scala, Python, Spark, Databricks, Azure
- ETL and feature engineering - personalization engine.
Data Engineer2019 - 2020Teespring (via Toptal)
Technologies: Amazon Web Services (AWS), APIs, Redshift, Apache Airflow, Python, Spark, Snowflake, Databricks, Fivetran, AWS
- Migrated a data warehouse ETL pipeline from Airflow/Redshift to Fivetran, Databricks, and Snowflake.
Data Engineer2018 - 2019BCG GAMMA (via Toptal, Three Contracts)
Technologies: Ansible, Boto 3, Apache Airflow, PostgreSQL, Relational Database Services (RDS), AWS Glue, AWS Athena, Presto DB, Apache Hive, Spark, Python
- Provided engineering support for data scientists.
- Designed and built a featured engineering data mart and customer 360° data lake in AWS S3.
- Designed and developed a dynamic S3-to-S3 ETL system in Spark and Hive.
- Completed various DevOps tasks included an Airflow installation, development of Ansible playbooks, and history backloads.
- Worked on a feature engineering project which involved Hortonworks, Spark, Python, Hive, and Airflow.
- Built a one-on-one marketing feature engineering pipeline in PySpark on Microsoft Azure and Databricks (used ADF, ADL, Databricks Delta Lake, and ADW as a source).
Vice President, Data2017 - 2018Enervee
Technologies: Amazon Web Services (AWS), Redash, Apache Airflow, Python, AWS S3, Amazon Aurora, MySQL, PostgreSQL, Redshift, Apache Hive, Presto DB, Spark, AWS EMR, Hadoop, AWS
- Managed the data engineering, BI reporting, and data science teams.
- Worked as a hands-on data engineer.
- Built a data lake on AWS.
- Developed a reporting system with Redash/Presto.
Big Data Architect2016 - 2017ITG
Technologies: Q, Kdb+, Informatica, Sybase, Python, Spark, Apache Hive, Hadoop
- Worked in a full-time position, as a data architect for a transaction cost analysis system.
- Installed a four-node Apache Hadoop/Spark cluster on ITG's private cloud.
- Conducted platform POC embedding Apache Spark technology into ITG's data platform.
- Supported the development of a platform POC for Kx Kdb+; also converted Sybase IQ queries to Kdb+ Q language.
Data Engineer2016 - 2017American Taekwondo Association (via Toptal)
Technologies: Pentaho, Oracle, SQL
- Converted data from a legacy Oracle database to a newly designed SQL Server database.
- Wrote SQL scripts, stored procedures, kettle transformations.
- Administered two databases.
- Performed extensive data cleansing and validation.
Director, Data Warehouse2015 - 2016Connexity
Technologies: Amazon Web Services (AWS), Linux, Python, Perl, Tableau, Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Cognos 10, Impala, Hadoop, Redshift, AWS, PL/SQL, Oracle
- Managed two data warehouses and BI teams for both PriceGrabber and Shopzilla. Connexity is also known as PriceGrabber, Shopzilla, and BizRate.
- Handled operational support for the PriceGrabber data warehouse. Recovered data warehouse after the data center migration.
- Merged one data warehouse into another and retired one of them. Hands-on designed business and data integration architecture; developed data validation scripts and ETL integration code. Managed the transfer of a BI reporting system from Cognos to OBIEE and Tableau.
- Defined the technology platform change strategy for the combined data warehouse.
- Created SQL: PL SQL stored procedures, packages, and anonymous scripts for ETL and data validation.
- Completed an Amazon Redshift project.
- Worked on and completed a Cloudera Impala project.
Director, Data Warehouse2008 - 2015PriceGrabber
Technologies: Pentaho, Linux, Python, Perl, MySQL, PostgreSQL, Apache Hive, Apache Pig, Hadoop, Oracle
- Oversaw the company's data services, defined the overall and technical strategy for data warehousing, business intelligence, and big data environments.
- Hired and managed a mixed on-shore (US)/off-shore (India) engineering team.
- Replatformed a data warehouse to Oracle Exadata X3/Oracle ZFS combination, added big data and machine learning components to the data warehousing environment.
- Supported 24x7x365 operations in compliance with the company's top-level production SLA.
- Wrote thousands of lines of PL/SQL, PL/pgSQL, MySQL, and HiveQL code.
- Worked with big data on multiple types of projects (Hadoop, Pig, Hive, and Mahaut).
- Developed a tool-based ETL for a Pentaho (Kettle) CE ETL redesign project.
- Worked on machine learning for various types of projects (Python, SciPy, NumPy, and Pandas).
Director, Data Warehouse2007 - 2008Edmunds
Technologies: Linux, Perl, Informatica, Oracle
- Managed a data warehouse team and project pipeline; supported operations.
- Created PL/SQL stored procedures, packages, and anonymous scripts for ETL and data validation.
- Worked on a tool-based ETL for multiple Informatica projects.
Manager, Data Warehouse2003 - 2007Universal Music Group
Technologies: Linux, Perl, C#, Cognos 10, MySQL, Microsoft SQL Server, Oracle
- Managed, developed, and operated a CRM data warehouse.
- Wrote PL/SQL, MySQL, and Perl code.
- Administered to a Cognos reporting system.
- Worked on C# for multiple supporting projects for the OLAP reporting system.
- Designed and developed a MSAS OLAP cube system.
Director, Decision Support and Financial Systems2001 - 2003MediaLive International
Technologies: Unix, VB, Microsoft SQL Server, Oracle EBS, Oracle
- Managed a data warehouse, BI, and CRM systems.
- Assumed responsibilities over an Oracle EBS application team.
- Developed the PL/SQL coding for a data warehouse ETL and Oracle Application integration.
- Worked with SQL server for multiple Transact-SQL and analysis service projects.
- Worked on a tool-based ETL for multiple epiphany EPI*Channel projects.
Senior Principal Consultant (Professional Services, Essbase Practice)1999 - 2001Hyperion (Currently: Oracle)
Technologies: Essbase, Hyperion, Informatica, Visual Basic for Applications (VBA), Microsoft SQL Server, Oracle
- Led a practice for a consulting company covering for multiple clients.
- Developed Essbase satellite systems: relational data warehouses and data marts, reporting systems, ETL systems, CRM's, EPP's, ETL in and out of Essbase and with Essbase itself.
- Worked on multiple PL/SQL projects, by providing full support of the team's Oracle project pipeline.
- Helped to develop SQL servers for multiple Transact-SQL and analysis services projects.
- Developed a tool-based ETL for an Informatica project.
- Worked with Hyperion, Essbase, Enterprise, Pillar, planning, financial analyzers, and VBA projects.