Paarth Kumar Gupta

A warehousing back-end solution aimed at the identification and mapping of customers.

• Customer identification is based on unique identifiers (UID and email) based on the customer's login preference.

• Identification was done using the APIs with C# utilities.

• Developed solutions to integrate big data streams into DWH for customer identification and purchase mapping.

• Enabled use cases for analytics based on customer purchase history, such as the propensity of a customer to purchase a to-be-launched product, target forecast, budget analysis, etc.

• Identified the areas of improvement at the architectural and code levels; optimization in these areas improved latency and resource consumption, thereby reducing the cost of running jobs on servers and clusters.

• The biggest challenge in this process was to optimize the architecture to reduce data latency. I approached this problem by applying indexes to fact tables and modularised the data flow instead of a single process.

An ETL-based project aimed to gather raw data from Postgres and transform it into a usable format. This project was completed using AWS Glue, PySpark (Python), S3, and Athena. The data gathered in the raw bucket had to be collected, processed, de-duplicated, and transformed back into a processed S3 bucket. The Athena layer on top of S3 was used to query the data points.

• Developed a mechanism to charge the trader/seller at a regular interval instead of on a per-purchase basis and provided the sellers with an option to choose their invoicing cycle.

• Worked on Hive to collect, load, and transform data points involved in the identification of such sellers who opted for this delay.

• Optimized the existing flow using Hive configurations and parameters to incorporate these changes.

• The project was meant to provide real-time insights and metrics to the store owners and product stakeholders during a certain period of a crucial sale.

• I designed and developed the end-to-end architecture of a refreshable pipeline to collate data from multiple sources (spreadsheets, databases, events, etc) and create a data warehouse.

• It was a completely cloud-based architecture, to be refreshed every 15 minutes, bringing in around 500 million records every run.

• It involved collecting transactions in real-time, to be made available to store owners and stakeholders in the form of reports (MIS and PowerBI) in 15 minutes. This involved creating a handshake between different environments that were not in sync.

• The complete architecture of this project was hosted on Azure Cloud, and the reporting was done through Power BI.

• The major challenge was bringing in some sources from outside the Azure environment into Azure pipelines since they will not be in sync. I solved this problem by implementing a look-back time interval for this new dataset so that no record is missed.