Nidhin Nandhakumar

Coursera wanted to create a centralized location for storing and managing all the metadata for various data assets. These data assets included:
• data warehouse metadata
• back-end database metadata
• job scheduling and automation metadata
• bi-platform metadata
• and various others, including SFMC, SFDc, etc.

I led this project and implemented an open-source solution called DataHub (formally developed by LinkedIn), which allowed Coursera to store and manage the above metadata components. Coursera's employees could log in through a two-factor authentication model, then browse and read the metadata information for all their data assets.

Some of the use cases included:
• understanding what each data-warehouse table contained, their column names, data types, description, tagging, domain categorization, documentation, etc.
• understanding the downstream and upstream impact of each table
• understanding the lineage of each table and airflow jobs
• understanding which dashboards are derived from what table in their data warehouse
• understanding each table's test results and analyzing if the table passes all the quality tests.

FreshBooks wanted to develop an end-to-end ELT and reporting system with Google BigQuery. The requirements were to ingest data coming from both streaming and nonstreaming offline databases regularly and to create an analytics reporting layer to be used by data analysts for looker dashboards.

I was a core developer in architecting and modeling the data warehouse system with Google BigQuery involving components such as Cloud Composer for scheduling the pipelines, Pub/Sub and Dataflow for streaming data and dbt, and Python for transforming and creating final reporting layers.
Also, I spearheaded the Terraform codebase to create and manage infrastructure as code and to organize and maintain it.

I worked on building an AWS end-to-end reporting platform from the ground up for one of the biggest educational platforms in the world.

The project involved organizing and ingesting data from various source systems, including real-time and batch models, transforming and cleaning data, and creating reporting tables for dashboards and data science models.

I used tools, such as Apache Airflow, for orchestrating the models and transformations, dbt for reporting and transformational logic, and Redshift as the data warehouse. An additional tool stack includes Terraform for IAC, Kinesis for real-time data ingestion, and Lambda for triggers.