Big Data Consultant
2016 - PRESENTClients (via Toptal)- Consulted with startups and medium-scale organizations to build data lakes for analytics.
- Advised organizations on building real-time data pipelines using Kafka and Spark.
- Helped organizations to analyze and report on their datasets.
Technologies: Druid.io, Spark, Apache Kafka, HadoopSenior Data Engineer
2016 - 2020Morgan Stanley- Received consecutive promotions for four years for exceptional performance.
- Got MD recognition for exceptional deliverables for real-time data ingestion Initiative.
- Worked on analytics system that currently processing 10K records per minute on 10 node spark cluster.
- Managed and nurtured a team of six people to work on the next-generation real-time cyber analytics engine.
Technologies: Amazon Web Services (AWS), Apache Airflow, Apache Hive, Apache Kafka, AWS EMR, AWS Glue, Hadoop, HDFS, Spark, Spark Structured Streaming, Spark Streaming