Data Solutions Architect2020 - 2021Enterprise Client via Toptal
Technologies: Scala, Spark, Azure, Azure Data Factory, Azure Data Lake, Azure Databricks, deltalake, Data Engineering, ETL, Data Migration, Databricks
- Worked on orchestration and automation of the workflows via Azure Data Factory.
- Optimized and partitioned storage in Azure Data Lake Storage (ADLS) Gen2.
- Implemented complex strongly-typed Scala Spark workloads in Azure Databricks, along with dependency management and Git integration.
- Implemented real-time low cost and low latency streaming workflows which at their peak were processing >2MM raw JSON blobs per Second. Architected as Azure Blob Storage -> Azure Event Hubs -> Azure Queues via ABS-AQS.
- Created a multi-layered ELT platform which consisted of raw/bronze (Azure Blob Storage), current and silver (Azure Delta Lake), and mapped/gold (Azure Delta Lake) layers.
- Balanced the cost of computing by spinning up clusters on-demand vs persisting them.
- Made big data available for efficient and real-time analysis throughout the client via delta tables, which provided indexed and optimized stores, ACID transaction guarantees, and table level and row-level access controls.
- Tied all of this together in end-to-end workflows that were either refreshed with just a few clicks or automated as jobs.
- Led a team of five comprising of four developers, and one solutions architect to productionalize big data workflows in Azure Cloud that enabled the client to sunset its legacy applications and experience far more reliable and scalable Prod workflows.
- Enabled a wide diversity of use cases and future-proofed them by relying upon open source and open standards.
Lead Data Engineer2019 - 2020Stealth mode AI startup (Series A $20 Million)
Technologies: Data Engineering, Apache Hive, Apache Impala, SQL, Apache Spark, Scala, Bash, Linux, Spark Structured Streaming, Machine Learning, MLlib, Spark, Spark SQL, ETL
- Architected and implemented a distributed machine learning platform.
- Productionized 20+ machine learning models via Spark MLlib.
- Built products and tools to reduce time to market (TTM) for machine learning projects. Reduced the startup's TTM from the design phase to production by 50%.
- Productionalized 8 Scala Spark applications to transform the ETL layer to feed into the machine learning models downstream.
- Used Spark SQL for ETL and Spark Structured Streaming and Spark MLlib for analytics.
- Led a team of six comprising of three data scientists, two back-end engineers, and one front-end engineer. Delivered a solution that had a back-end layer that talked to the front end via REST API and launched and managed Spark jobs on demand.
Senior Data Engineer2018 - 2019Dow Chemical (Fortune 62)
Technologies: Data Engineering, Apache Hive, Apache Impala, SQL, Apache Spark, Scala, Hadoop, Bash, Linux, Oracle Database, Spark SQL, ETL
- Productionalized five Scala Spark apps for ETL. Wrote multiple Bash Scripts for the automation of these jobs.
- Architected and productionalized a Scala Spark app for validating the Oracle source tables with their ingested counterparts in HDFS. The user could dynamically choose to conduct either a high-level validation or a data level validation. The output of the app in case of a discrepancy was the exact columns and the exact rows that mismatched between source and destination.
- Reduced the engineer's manual debug workload by over 99%, reducing it to just running the app and then reading the human-readable output file.
- Delivered the entire ETL and validation project ahead of schedule.
Senior Data Engineer2018 - 2019Boston Scientific (Fortune 319)
Technologies: Data Engineering, Apache Hive, Apache Impala, SQL, Apache Spark, Scala, Hadoop, Bash, Linux, Kudu, Spark Structured Streaming, Apache Solr, Spark SQL, ETL
- Designed and implemented a Scala Spark application to build Apache Solr indices from Hive tables. The app was designed for a rollback on any failure and reduced the downtime for downstream consumers from ~three hrs to ~ten seconds.
- Implemented Spark Structured Streaming application to ingest data from Kafka streams and upsert into Kudu tables in a kerberized cluster.
- Implemented multiple Shell scripts to automate Spark jobs, Apache Sqoop jobs, Impala commands, and more.
Senior Data Engineer2017 - 2018General Mills (Fortune 200)
Technologies: Data Engineering, Apache Hive, Apache Impala, SQL, Apache Spark, Scala, Hadoop, Spark Structured Streaming, Spark SQL, ETL
- Consumed social marketing data from various sources. Namely Google Analytics API, Oracle Databases, various streaming sources, and more.
- Productionalized a Scala Spark application to ingest >100Gb of data as a daily batch job, partition, and store as parquet in HDFS, with corresponding Hive partitions at the query layer. App replaced legacy Oracle solution and reduced runtime by 90%.
- Used Spark SQL and Spark Structured Streaming for ETL.
Software Engineer2015 - 2016MetLife Insurance (Fortune 44)
Technologies: Model View Controller (MVC), Agile
- Acted as the product manager for a motorcycle insurance web app. The app grew into becoming the primary landing site for motorcycle insurance leads.
- Built master for deployment until production. Deployed all builds and was primary on the stability of the build.
- Led Scrum development for client teams of 30+ developers, testers, and analysts.
- Architected and supported the solution within the client organization.