Aniket Goel

This online event organization wanted to move from the Redshift data warehouse to an optimized data lake to save cost and enhance the platform with more capabilities, such as analyzing the data in a live streaming window of events, a new pipeline set up to implement SCD Type 2 to capture the complete history of changes using Spark Streaming, Apache Spark, AWS Database Migration Service (AWS DMS). The data lake was designed in three layers, bronze, silver, and gold, where each layer successively gets aggregated.

The client wanted to implement a user activity tracking system for data analytics. With the help of Segment.io and its plugins, the applications could send different events as the user uses the app. We built an enrichment pipeline to include attributes not part of row events. The frequency of events was around 200,000 per second on average, and each event was processed in around 70 milliseconds. This was possible by optimizing the pipeline using AWS Lambda cold start, Redis Cache, and RDS connection pool.

The online event organization wants to set up its data warehouse for data analysis and dashboarding. I helped the team set the Redshift instance through Terraform end-to-end and ETL pipeline development using Fivetran and AWS Services like DMS, Lambda, Glue, and others. The data was consumed from Qualtrics, Stripe, Chargebee, Salesforce, RDS, Zendesk, and other sources. The data warehouse was used by many stakeholders, like data analysts and engineers, ad-hoc queries, business insights, and dashboard development on Looker. I also contributed to setting up the Looker instance and dashboard development.

Strive VR LLC is a company with remote users to do workout sessions. The client aims to optimize the support structure and wants a remote support center to stream, record, break each video and store it on the server to have a full repository of these sessions. I implemented the requirements using Java Swing for the desktop application and JavaScript for the web page, and the whole streaming process was done using AWS Kinesis Video Stream.

Al Tayyar Group is a Saudi-based organization involved in many industries, including leisure, tourism, education, transport, real estate, retail trading, hospitality, aviation, and food and beverages. Having many different organizations with millions of customers, they wanted to analyze their data to understand better how to increase their revenue and productivity. I built a data lake that consumes data and incrementally puts that into a Hadoop Distributed File System (HDFS). It then uses a Hadoop ecosystem and other tools to analyze this massive amount of data and visualize it using D3.js, Qlik Sense, and Tableau after processing.

A US-based organization that invests in scientific innovation to create transformative medicines for people with serious diseases. The company has multiple approved medications that treat the underlying cause of cystic fibrosis (CF)—a rare, life-threatening genetic disease—and has several ongoing clinical and research programs in CF. Having many different medicines and research with millions of customers, they wanted to analyze their data to understand better how to increase their revenue and productivity.

I built a data lake that consumes data and incrementally puts that into AWS S3. It then uses AWS services and other tools to analyze this massive amount of data and visualize it using Tableau.

An application for Tata Sky enables greater accessibility for its viewers. Tata Sky is an Indian direct broadcast satellite television provider in India, using MPEG-4 digital compression technology and transmitting using INSAT-4A and GSAT-10 satellites. The company wanted subscribers to be able to watch select live TV channels, catch-up TV videos, and videos on demand (VOD) on their devices. The goal was for users to access these options through the Tata Sky application on any platform. The application was developed for Android, iOS, web, Dongle, and Smart TV platforms.