Shubham Agarwal, Developer in Bengaluru, Karnataka, India
Shubham is available for hire
Hire Shubham

Shubham Agarwal

Verified Expert  in Engineering

Bio

Shubham Agarwal is an AI engineer with 10+ years of experience in machine learning, NLP, and deep learning. He specializes in retrieval-augmented generation (RAG), multi-agent AI systems, and diffusion models. Proficient in Python, PyTorch, and scalable AI architectures, he has built LLM-powered autonomous agents, graph-based retrieval systems, and production AI workflows, optimizing inference and deploying AI at scale.

Portfolio

Empower Education Inc.
Artificial Intelligence (AI), Python, AI Agents, LangChain, FastAPI, Qdrant...
AB-InBev - Gen AI - India
Python, Large Language Models (LLMs), LangChain, Docker
Zust AI
Artificial Intelligence (AI), Generative Artificial Intelligence (GenAI)...

Experience

  • Python - 6 years
  • Data Science - 5 years
  • Machine Learning - 5 years
  • TensorFlow - 3 years
  • Generative Artificial Intelligence (GenAI) - 2 years
  • Stable Diffusion - 2 years
  • Large Language Models (LLMs) - 2 years

Availability

Part-time

Preferred Environment

Git, Linux, OS X

The most amazing...

...thing I’ve developed is a GraphRAG for E2E course generation, outperforming NotebookLM by structuring content with knowledge graphs from reference materials.

Work Experience

AI Developer

2024 - PRESENT
Empower Education Inc.
  • Designed and implemented a GraphRAG algorithm for course outline generation based on reference documents. The algorithm constructs a knowledge graph by extracting entities and relationships using small language models (SLMs).
  • Developed an agentic architecture similar to Ottogrid, enabling users to design AI workflows. Implemented asynchronous execution and query grouping to optimize search API calls and latency.
  • Built a multi-step agentic framework from scratch for orchestrating AI agents. The framework is model-agnostic, supports RAG with the Qdrant database, manages state transitions, and incorporates a human-in-the-loop mechanism.
  • Engineered the back-end architecture from scratch using FastAPI, MongoDB, and Beanie. Configured Docker for containerized deployment and deployed on GCP.
  • Developed an AI-powered chat agent inspired by Cursor Chat and Windsurf Agents, capable of routing queries to different agents, dynamically planning workflows, and handling ad-hoc queries.
Technologies: Artificial Intelligence (AI), Python, AI Agents, LangChain, FastAPI, Qdrant, Knowledge Graphs

LLM Specialist

2024 - 2024
AB-InBev - Gen AI - India
  • Designed and implemented a robust end-to-end RAG pipeline to process over 100,000 complex PDF documents containing tables, charts, and industrial images, ensuring seamless handling of unstructured data.
  • Established a comprehensive evaluation framework for analyzing various query complexities and all stages of RAG, including chunking, retrieval, reranking, and generation, ensuring optimal performance at every step.
  • Developed a cutting-edge mixture-of-embedding solution that increased retrieval accuracy by 11%. This hybrid approach combined two types of embeddings with BM25 search, significantly enhancing the retriever.
  • Trained and deployed a custom YOLO model for accurate table and figure detection on PDF pages, improving the parsing and processing of millions of PDF documents.
  • Reduced overall system latency from 25 seconds to 8 seconds by leveraging efficient open-source models (GLiNER, ColBERT small, BM25), alongside a custom-trained sequence classification model, ensuring faster and more responsive results.
  • Engineered an innovative search algorithm capable of running on edge CPU devices, using a combination of BM25 and answerai-colbert-small-v1, delivering performance on par with GPT-4 while maintaining low resource usage.
  • Built a robust pipeline to extract structured data from PDF pages and seamlessly integrate it into an SQL database, enabling insights and visualization through a dashboard.
Technologies: Python, Large Language Models (LLMs), LangChain, Docker

AI Researcher | Founder

2023 - 2024
Zust AI
  • Designed an advanced algorithm for background transformation, achieving flawless integration with precise adjustments in shadow, lighting, and subject composition.
  • Developed an innovative algorithm to seamlessly swap subject images into existing Midjourney or Stock photos, delivering realistic, high-quality results.
  • Recreated Adobe’s Generative Fill algorithm from the ground up, enabling automatic context awareness from the image itself, eliminating the need for user prompts.
  • Trained multiple LoRA models to cater to various image generation styles, enhancing versatility and creative output.
  • Deployed a containerized solution on AWS, enabling real-time inference and ensuring scalability and efficiency in production.
Technologies: Artificial Intelligence (AI), Generative Artificial Intelligence (GenAI), Python 3, Stable Diffusion, Large Language Models (LLMs)

Python Expert

2022 - 2022
HelloFresh - Data Science - Michael Johnson
  • Developed an embedding platform that includes embedding algorithms (graph neural network), recipe recommendation, and batch deployment from scratch.
  • Sped up the embedding algorithms and evaluation metrics for recipe recommendations to serve over a million customers.
  • Managed end-to-end deployment pipeline: unit testing, Prefect, MLFlow, Git automation, S3, Kubernetes, and extended documentation.
Technologies: Python, Kubernetes, Amazon Web Services (AWS), Apache Spark, TensorFlow, PySpark, DevOps, Machine Learning Operations (MLOps)

Senior Back-end Python Developer

2021 - 2022
Invicta Pte Ltd (Toptal Client)
  • Designed Python Flask back end for the Kynec application (https://kynec.com). The back end follows Okta-based authentication and multi-tenancy architecture.
  • Designed a Redis-based cache mechanism for fast retrieval of the back-end data.
  • Designed a mechanism for unit testing and code coverage.
Technologies: Python, Flask, PostgreSQL, Redis

Data Scientist (via Toptal)

2021 - 2021
Advertise Purple
  • Built a statistical model for sales and trend forecasting of eCommerce websites based on the affiliated data. It was an ARIMA-based model developed using the statsmodels Python library.
  • Developed a Node.js API to connect the Python back end with the front-end application.
  • Performed exploratory data analysis on affiliated data for eCommerce websites and supported the development of a front-end dashboard.
Technologies: Python, Sales Forecasting, Affiliate Marketing, Node.js, StatsModels

Quant Researcher

2021 - 2021
Valustat
  • Designed a pipeline to read financial data into a structured format that can be further filtered and processed on different factors.
  • Built factor models, like Fama French, Carhart, and more, for portfolio analysis and asset pricing.
  • Designed a Python API to create a custom portfolio based on industry and market data.
Technologies: Python, Quantitative Modeling

Data Scientist (via Toptal)

2020 - 2020
Truss Investments
  • Developed an algorithm to predict the Brazilian futures market using historic and brokers data. An LSTM-based algorithm was developed and implemented using TensorFlow.
  • Designed a backtesting algorithm to make decisions of buying or selling based on trade patterns and pricing constraints.
  • Developed a Python module to store and structure a large amount of historical pricing data directly used for machine learning models.
Technologies: TensorFlow, Python

Specialist Developer

2019 - 2020
EzAPI
  • Developed a network flow visualization of API inputs and outputs. This helped developers and managers to understand a higher level of API flow and speed up the development process.
  • Developed an MVP for API testing automation by parsing a Swagger and OpenAPI specification. Designed test cases based on API inputs and outputs to verify API endpoints.
  • Built a virtual mock service for API testing and integration.
Technologies: Generative Pre-trained Transformers (GPT), Natural Language Processing (NLP), Flask, MongoDB, Node.js, Python 3, Data Science

Lead Data Scientist

2017 - 2019
FlexiLoans Technologies, Pvt. Ltd.
  • Developed an identity document classifier, like KYC and business documents, using a two-step approach: the Naive Bayes text method and CNN-based object detection.
  • Designed a system for face detection and text information retrieval from digital documents using OCR (Optical Character Recognition) for identity verification and data enrichment.
  • Built an algorithm to understand the financial and credit-bureau data to construct a loan default prediction model. Many feature engineering, business context, and model ensembling were involved.
  • Designed a mathematical model to predict customer financial obligations using credit information. This optimization algorithm predicted two unknown variables from a single equation.
  • Automated the overall loan process with a custom-designed Python back end along with Flask and Restful APIs.
  • Improved the overall business effectiveness and user experience for lead prioritization and allocation and customer profiling by creating several data-driven applications.
  • Conducted an in-house workshop on data thinking and AI.
Technologies: TensorFlow, Scikit-learn, Flask, MongoDB, SQL, Python, Node.js

Data Scientist and Software Engineer

2015 - 2017
The Zero Games, Pvt. Ltd.
  • Designed an ad-server based on different demographics and time filters. It was a real-time bidding platform that automated the delivery of ads.
  • Engineered an ad recommendation system for user targeting and maximizing the click-through rate. The system was designed based on the demographic data and installed applications found on the smartphone.
  • Devised a Python back end for a client dashboard to check the daily reach and revenue generated.
Technologies: Scikit-learn, Redis, MongoDB, Node.js, Python

Experience

ysfit

ysfit is the simplest implementation of the PoseNet algorithm for the detection and analysis of workouts and counting repetitions of different exercises. The project uses ml5.js and a position movement-based algorithm for squats.

Facemask Detection

An end-to-end neural network that detects if a person is wearing a facemask or not. The idea is to design a memory-efficient and fast algorithm which can run on edge devices. Downloaded 180 images, approximately 500 faces, from Google images and hand-labeled them using LabelImg, an image annotation tool. The algorithm uses the TensorFlow object detection API with MobileNet-SSD pre-trained on the COCO dataset.

Political Mirror: R U Ready Hackathon (Winner for Best Code)

Designed a politically-focused news app in a 24-hour media hackathon. The application highlights relevant text from news articles using a text rank algorithm and a self-learning recommendation system implemented based on reading behaviors. The app won the award of best code category. Technologies used - Python, Flask, Cloud Firestore, TensorFlow, and Ionic.

Mailsy - Counter.app (Freelance Project)

https://counter.app
Mailsy is an automated email scraper for accommodations such as hotels and hostels. It extracts reservation information and sends outgoing emails through the counter dashboard.
Technologies used: Python, Django, PostgreSQL, and Oauth2.

Oerlikon Digital Hub Hackathon

An application to track the position of the manufactured components inside the industrial manufacturing pipeline. The application uses a vision-based segmentation algorithm for positional tracking and image processing for noise removal.
Technologies used: Python, OpenCV, and Flask.

Instapoker - Video Poker With Friends

A live multiplayer poker game with video conferencing. The application uses Node.js and Socket.IO for smooth server-to-client communication; it also uses Redis for storing game data and faster retrieval purposes.

RPC Benchmark for PyTorch

https://github.com/pytorch/pytorch/issues/43561
A PyTorch benchmarking application allows users to see throughput and latency if they use RPC for training in reinforcement learning. This solves the problem of slow training in single-agent, multiple-observer and considers users' custom neural network architecture.

Education

2010 - 2014

Bachelor of Technology Degree in Computer Science

National Institute of Technology, Raipur - Raipur, India

Certifications

MARCH 2020 - PRESENT

TensorFlow in Practice Specialization

Coursera

OCTOBER 2018 - PRESENT

Deep Learning Specialization

Coursera

Skills

Libraries/APIs

TensorFlow, Node.js, REST APIs, Scikit-learn, Pandas, NumPy, PyTorch, Matplotlib, Socket.IO, PySpark

Tools

Git, StatsModels

Languages

Python 3, Python, SQL, JavaScript

Frameworks

Flask, Django, Express.js, LlamaIndex, Apache Spark

Storage

Redis, PostgreSQL, MongoDB

Paradigms

DevOps

Platforms

OS X, Linux, Google Cloud Platform (GCP), Kubernetes, Amazon Web Services (AWS), Docker

Other

Machine Learning, Data Science, Data Analysis, Natural Language Processing (NLP), Analytics, Generative Pre-trained Transformers (GPT), Generative Artificial Intelligence (GenAI), Stable Diffusion, Large Language Models (LLMs), Deep Learning, Scraping, Computer Vision, LangChain, Quantitative Modeling, Sales Forecasting, Affiliate Marketing, Machine Learning Operations (MLOps), Artificial Intelligence (AI), AI Agents, FastAPI, Qdrant, Knowledge Graphs

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring