Freelance Data Engineer2018 - PRESENTBCG GAMMA (via Toptal, Three Contracts)
Technologies: Python, Spark, Hive, Presto, Athena, Glue, RDS, PostgreSQL, Airflow, Boto 3 API, Ansible
- Provided engineering support for data scientists.
- Designed and built a featured engineering data mart and customer 360-degree data lake in AWS S3.
- Designed and developed a dynamic S3-to-S3 ETL system in Spark and Hive.
- Completed various DevOps tasks included an Airflow installation, development of Ansible playbooks, and history backloads.
- Worked on a feature engineering project which involved Hortonworks, Spark, Python, Hive, and Airflow.
- Build a one-on-one marketing feature engineering pipeline in PySpark on Microsoft Azure and databricks (used ADF, ADL, Databricks Delta Lake, and ADW as a source).
Consultant | Co-founder | CEO2016 - PRESENTCrowd Consulting
Technologies: AWS EMR, Hadoop, Spark, Presto, Hive, AWS Lambda, AWS Redshift, AWS RDS: Postgres, MySQL, DynamoDB, AWS Lambda, AWS S3, Python, Scala, Luigi, Tableau
- Worked on full data warehouse implementations for multiple clients.
- Provided big data training and support.
- Engineered and built an ETL pipeline for AWS S3 data warehouse using AWS Kinesis, Lambda, Hive, Presto, and Spark. The pipeline was written in Python.
Vice President of Data2017 - 2018Enervee
Technologies: AWS EMR: Hadoop, Spark, Presto, Hive, AWS Redshift, AWS RDS: PostgreSQL, MySQL, AuroraDB, AWS S3, Python, Airflow, Redash
- Managed the data engineering, BI reporting, and data science teams.
- Worked as a hands-on data engineer.
- Built a data lake on AWS.
- Developed a reporting system with Redash/Presto.
Big Data Architect2016 - 2017ITG
Technologies: Apache Hadoop, Hive, Spark, Python, Sybase ASE, Sybase IQ, Informatica, Kdb+, Q
- Worked in a full-time position, as a data architect for a transaction cost analysis system.
- Installed a four-node Apache Hadoop/Spark cluster on ITG's private cloud.
- Conducted platform POC embedding Apache Spark technology into ITG's data platform.
- Supported the development of a platform POC for Kx Kdb+; also converted Sybase IQ queries to Kdb+ Q language.
Freelance Data Engineer and DBA2016 - 2017American Taekwondo Association (via Toptal)
Technologies: MS SQL Server, Oracle, Pentaho (Kettle)
- Converted data from a legacy Oracle database to a newly designed SQL Server database.
- Wrote SQL scripts, stored procedures, kettle transformations.
- Administered two databases.
- Performed extensive data cleansing and validation.
Director, Data Warehouse2015 - 2016Connexity
Technologies: Oracle, PL/SQL, AWS Redshift, Hadoop, Impala, Cognos, OBIEE, Tableau, Perl, Python, Linux
- Managed two data warehouses and BI teams for both PriceGrabber and Shopzilla. Connexity is also known as PriceGrabber, Shopzilla, and BizRate.
- Handled operational support for the PriceGrabber data warehouse. Recovered data warehouse after the data center migration.
- Merged one data warehouse into another and retired one of them. Hands-on designed business and data integration architecture; developed data validation scripts and ETL integration code. Managed the transfer of a BI reporting system from Cognos to OBIEE and Tableau.
- Defined the technology platform change strategy for the combined data warehouse.
- Created SQL: PL SQL stored procedures, packages, and anonymous scripts for ETL and data validation.
- Completed an Amazon Redshift project.
- Worked on and completed a Cloudera Impala project.
Director, Data Warehouse2008 - 2015PriceGrabber
Technologies: Oracle, Hadoop, Pig, Hive, PostgreSQL, MySQL, Perl, Python, Pentaho (Kettle), Linux
- Oversaw the company's data services, defined the overall and technical strategy for data warehousing, business intelligence, and big data environments.
- Hired and managed a mixed on-shore (US)/off-shore (India) engineering team.
- Replatformed a data warehouse to Oracle Exadata X3/Oracle ZFS combination, added big data and machine learning components to the data warehousing environment.
- Supported 24x7x365 operations in compliance with the company's top-level production SLA.
- Wrote thousands of lines of PL/SQL, PL/pgSQL, MySQL, and HiveQL code.
- Worked with big data on multiple types of projects (Hadoop, Pig, Hive, and Mahaut).
- Developed a tool-based ETL for a Pentaho (Kettle) CE ETL redesign project.
- Worked on machine learning for various types of projects (Python, SciPy, NumPy, and Pandas).
Director, Data Warehouse2007 - 2008Edmunds
Technologies: Oracle, Informatica, Perl, Linux
- Managed a data warehouse team and project pipeline; supported operations.
- Created PL/SQL stored procedures, packages, and anonymous scripts for ETL and data validation.
- Worked on a tool-based ETL for multiple Informatica projects.
Manager, Data Warehouse2003 - 2007Universal Music Group
Technologies: Oracle, SQL Server, MySQL, Cognos, C#, Perl, Lynux
- Managed, developed, and operated a CRM data warehouse.
- Wrote PL/SQL, MySQL, and Perl code.
- Administered to a Cognos reporting system.
- Worked on C# for multiple supporting projects for the OLAP reporting system.
- Designed and developed a MSAS OLAP cube system.
Director, Decision Support and Financial Systems2001 - 2003MediaLive International
Technologies: Oracle, Oracle EBS, SQL Server,VB, Epiphany, Unix
- Managed a data warehouse, BI, and CRM systems.
- Assumed responsibilities over an Oracle EBS application team.
- Did the PL/SQL coding for a data warehouse ETL and Oracle Application integration.
- Worked with SQL server for multiple Transact-SQL and analysis service projects.
- Worked on a tool-based ETL for multiple epiphany EPI*Channel projects.
Senior Principal Consultant (Professional Services, Essbase Practice)1999 - 2001Hyperion (Currently: Oracle)
Technologies: Oracle, SQL Server, Hyperion Essbase, VBA, Informatica
- Led a practice for a consulting company covering for multiple clients.
- Developed Essbase satellite systems: relational data warehouses and data marts, reporting systems, ETL systems, CRM's, EPP's, ETL in and out of Essbase and with Essbase itself.
- Worked on multiple PL/SQL projects, by providing full support of the team's Oracle project pipeline.
- Helped to develop SQL servers for multiple Transact-SQL and analysis services projects.
- Developed a tool-based ETL for an Informatica project.
- Worked with Hyperion, Essbase, Enterprise, Pillar, planning, financial analyzers, and VBA projects.