Anish Chakraborty

A data product for Philips Healthcare. I was a data and back-end engineer for the Philips IoT division, working on products used as at-home baby monitors. My role involved implementing the data flow for processing data collected via sensors in devices and processing them to detect user patterns to create custom actions based on the results.

A Google Dataflow/Apache Beam application for real-time dashboards.

The application sourced event logs from Google Pubsub and processed them in Apache Beam, storing results in Google Cloud Datastore and used for real-time dashboards.

I was the back-end and data engineer for developing a microservice-based system for ingesting events from IoT devices (like Philips Hue) and storing them for analytical and alerting purposes. The system was initially designed on RabbitMQ, and I migrated it to Amazon SQS.

I wrote a developer-facing SDK in Scala for defining and managing alerts on data pipelines. This provides a custom DSL for managing metrics and defining alerts on these metrics on the Google Cloud Platform.

I led the migration of PyTorch-based DNN training models to run native on AWS Fargate and orchestrated it via Apache Airflow. This project involved migrating multiple stakeholders using an internal machine learning library for training models on on-prem infra to move to a wholly managed Dockerized environment in the cloud.

I was the lead data engineer for a project involving machine learning using TensorFlow to predict the outcome of payment retries and plan retry patterns leading to a decrease in payment failures. The project involved understanding the payment processing domain, designing warehouses for efficiently analyzing logs at scale using Big Query, and training models to serve the back ends.

I led the user-level data sourcing for financial payout processing for one of my clients. The project included migrating an existing sourcing infra from an API-based workflow to a data export-based workflow using Google Cloud Dataflow and Apache Beam for data processing.

As a contributor to Scio, which is the Scala API for Apache Beam, I've written sparse join modules and co-implemented the use of Bloom filters to improve the performance of certain join operations in Scio.

I scaled up and optimized an Apache Spark pipeline to run feature engineering on billions of records (25+ TB) in Bitcoin transactions. These features were used for training machine learning models for creating a risk score for transactions on the Bitcoin blockchain.

A data warehouse for a casino gaming business for real-time analytics and feature engineering based on data sourced from Kafka.

This was built on AWS using technologies like EMR, Apache Spark (Python), AWS Glue, and AWS Athena. The warehouse supported real-time queries on the data from Kafka with the infrastructure configured using Terraform.

A cloud-based data platform built with dbt, running on Snowflake, created for an IoT company with 30+ data scientists organized across multiple domains and teams. The project involved creating a data warehouse for the supply chain and IoT domain following the ELT method, with data processed using Snowflake and Snowpark.

The platform was built from scratch, then pipelines were migrated from a custom Scala framework onto Kafka, Fivetran, Snowpark, and dbt on Snowflake. The tech stacks used included the orchestration of Snowflake SQL via dbt Cloud with Amazon Managed Workflows for Apache Airflow and Terraform to manage the infrastructure. We used SnowPark for the heavily customized pipelines and AWS as the cloud platform.

Designed and rolled out the complete Edge layer infrastructure and API Gateways using Google Cloud Endpoints, Envoy, and Google ESP to a complex web app consisting of 80+ microservices and server-less applications.

This consisted of analyzing requirements around security concerns, user authentication, CDN setups, evaluating several options for implementing authentication and API Gateway design, monitoring traffic, understanding the use of microservices in the domain, and finally planning and rolling out this architecture over eight months across all services.

A system designed to process geo-spatially distributed data to create features for ML models. The system is designed to run on thousands of servers at a massively parallel scale and uses road network data from Open Street Maps for calculating routes and features related to routes at scale.