Anish Chakravartty

Designed and implemented a scalable retrieval-augmented generation (RAG) pipeline to power semantic search for a patent intelligence platform. The solution processed millions of patent documents by extracting, cleaning, and chunking text, followed by generating high-dimensional embeddings using the gte-large SentenceTransformer model on GPU-accelerated Databricks clusters. This approach achieved over 500 embeddings per second—a 5x speedup over the prior solution—significantly reducing time-to-index.

Integrated the pipeline with Pinecone using a custom microbatching architecture that ingested 100,000 vectors per batch with rate-limit handling, retries, and resume logic. Enriched all embeddings with metadata tags such as section, jurisdiction, and kind code to enable highly filtered semantic search.

End-to-end orchestration was automated using Databricks Jobs API, ensuring fault tolerance and batch-level traceability. This project enabled faster and more accurate prior art discovery, improving search relevance by 40% and reducing indexing latency by over 60%, directly aligning with the client’s goal to enhance innovation analysis and user productivity.

Developed a geospatial analytics pipeline to enrich crash and event datasets with location-based insights using Snowflake, GeoPandas, and Python. The system performed large-scale spatial joins between point-based crash data and geometry layers (e.g., road segments and intersections), enabling precise attribution of each incident to relevant infrastructure elements.

Designed and implemented functions to aggregate crash metrics at both segment and intersection levels, incorporating attributes such as crash severity, time-of-day, and roadway conditions. Created fallback logic to handle unmatched points, ensuring data completeness across the pipeline.

Engineered a robust ETL workflow to extract, process, and write enriched results back into Snowflake dynamic tables, supporting downstream analysis and reporting. Optimized spatial processing to reduce compute time and improve query performance for heavy geospatial workloads.

This solution empowered transportation analysts and planners to identify high-risk areas, evaluate safety countermeasures, and support data-driven decision-making for infrastructure improvements across city and state networks.

This project was done to create a data warehouse on Google BigQuery for a renewable energy provider. Data from the SCADA systems of different renewable energy power plants (solar, wind, etc.) were fetched in near real-time and loaded via REST APIs to BigQuery.

After loading the data into the warehouse, I created various materialized views with different aggregations to optimize performance. These views were utilized in a dashboard built in Looker for easy visualization and analysis. Additionally, I developed a web app dashboard that utilized data from these materialized views for real-time insights.

Additionally, I constructed a batch ETL framework using Apache Beam deployed on Cloud Dataflow, generating files on Cloud Storage for data models deployed on Vertex AI. The output of these models was leveraged on Looker dashboards and the web app.

I created the architecture of the entire system and worked on its implementation. Also, I set up and managed the initial phases of a data engineering team for the clients.

Developed a comprehensive adtech data pipeline for a mobile gaming company to integrate, analyze, and optimize performance across IronSource, AppsFlyer, and Google Analytics. Built scalable ETL workflows to ingest ad campaign data, including impressions, clicks, installs, cost, and revenue, enabling real-time insights into user acquisition and monetization.

Engineered transformation logic to normalize and reconcile cross-platform data, accounting for attribution differences and enhancing data accuracy. Implemented creative-to-campaign mapping, allowing the team to identify underperforming creatives and iterate rapidly on ad design and targeting.

Designed a unified reporting layer in Snowflake, combining cohort data and LTV projections to support ROI calculations and budget decisions. Reduced pipeline latency by over 70% through a modular architecture with retry handling and efficient batching for API calls.

This project empowered the marketing and UA teams to optimize ad spend, increase campaign ROI, and scale user acquisition strategies with confidence, directly impacting revenue and player engagement.

The project was a part of a client's group insurance vertical takeover by a different insurance company. The ETL jobs feeding data into the data warehouse and ODS for the client depended on the proprietary ETL framework used to manage parameterization, restartability, and load status. This ETL framework was not part of the takeover items, and as such, it enabled the ETL process to function in the target landscape, where existed a requirement for creating a new framework that would allow the execution of the jobs without making any changes.

I created the architecture of the new framework and worked as the technical lead in implementing the new framework and migrating jobs. The completed project enabled DataStage jobs migration without any changes, which resulted in all-around applause and appreciation.

A Wix plugin enables Wix store owners to track abandoned carts and send recovery messages via different channels such as Facebook, email, and WhatsApp.

I contributed to this project as a full-stack developer creating the front end for the Wix plugin and subscription modal script in Vue 2. The back end was hosted as multiple Google Cloud functions using Node.js and Express.js, which followed the REST architecture. We used Cloud Firestore as the operational database.

The project consisted of two parts. The first part involved creating a WhatsApp bot that served customer inquiries, lead generation, sample collection, and report distribution. This bot was developed using Google Dialogflow for NLU and context detection. A Node.js back end hosted on GCP App Engine and a Cloud Firestore DB was designed and developed, integrating with Twilio for communication over WhatsApp.

The second part of the project consisted of designing an ODS in BigQuery that used data from the WhatsApp bot, client website, and physical store data. Cloud functions were used to consume data in the form of files, REST API calls, and BigQuery stored procedures wrapped in pub/sub cloud functions for loading data into the ODS.