Big Data Consultant2016 - PRESENTClients (via Toptal)
Technologies: Druid.io, Spark, Apache Kafka, Hadoop
- Consulted with startups and medium-scale organizations to build data lakes for analytics.
- Advised organizations on building real-time data pipelines using Kafka and Spark.
- Helped organizations to analyze and report on their datasets.
Senior Data Engineer2016 - 2020Morgan Stanley
Technologies: Amazon Web Services (AWS), Apache Airflow, Apache Hive, Apache Kafka, AWS EMR, AWS Glue, Hadoop, HDFS, Spark, Spark Structured Streaming, Spark Streaming
- Received consecutive promotions for four years for exceptional performance.
- Got MD recognition for exceptional deliverables for real-time data ingestion Initiative.
- Worked on analytics system that currently processing 10K records per minute on 10 node spark cluster.
- Managed and nurtured a team of six people to work on the next-generation real-time cyber analytics engine.