Lakshya Chadda
Verified Expert in Engineering
Data Engineering Developer
Pune, Maharashtra, India
Toptal member since April 2, 2026
Lakshya is a senior data engineer with nearly 6 years of experience architecting and delivering enterprise-scale data lakehouse platforms using Azure Databricks, PySpark, and Delta Lake across healthcare and financial services. He specializes in LLMs, RAGs, and knowledge-driven data automation frameworks, with proven expertise in designing Medallion architecture pipelines, orchestrating complex ELT workflows with ADF, and building governed enterprise data lakes supporting large-scale analytics.
Portfolio
Experience
- Python - 6 years
- Google Cloud Platform (GCP) - 6 years
- PySpark - 5 years
- Azure Databricks - 5 years
- Amazon Web Services (AWS) - 4 years
- RAG Architecture - 2 years
- Large Language Models (LLMs) - 2 years
- Deep Learning - 2 years
Preferred Environment
PySpark, Azure, SQL Server Management Studio (SSMS), MySQL, Azure Data Factory (ADF), Azure Data Lake, Azure Databricks, SQL, Cloud Storage, Python, Google Cloud Platform (GCP), Engineering, Excel Expert, Google Sheets API, Microsoft, Agentic AI, Artificial Intelligence (AI), Test Automation
The most amazing...
...work I've done is developing LLM automation to convert ADF pipelines to Databricks workflows, reducing migration effort and errors.
Work Experience
Senior Data and AI Engineer
UK Health
- Led end-to-end data engineering and generative AI initiatives, building cloud-native platforms, RAG applications, and scalable data pipelines that improved operational efficiency, accelerated insights, and supported enterprise growth.
- Architected and delivered cloud-scale data and AI solutions using Spark, Airflow, AWS, Azure, and LLMs; automated pipelines, optimized performance, implemented MLOps, and enabled data-driven business outcomes at enterprise scale.
- Led end-to-end data engineering and generative AI initiatives, building cloud-native platforms, RAG applications, and scalable data pipelines that improved operational efficiency, accelerated insights, and supported enterprise growth.
- Applied ontology frameworks and semantic data modeling to integrate enterprise data, strengthen knowledge representation, and enhance GenAI/RAG solutions, while leveraging data science and ML for actionable insights.
Senior Full-stack Data Engineer
UBS
- Built and optimized Azure Databricks and ADF ELT pipelines using PySpark and SQL.
- Developed unit testing and data validation scripts using prompt engineering.
- Automated DDL creation/alteration based on schema evolution, JSON flattening, and dynamic table generation for faster onboarding of datasets, saving over 100 hours.
- Designed a reusable parallel processing framework for Databricks via ADF.
- Delivered data science and AI solutions at UBS, applying ML, NLP, GenAI, advanced analytics, and ontology frameworks to build semantic models, connect enterprise data, uncover insights, and automate processes.
Senior Full-stack Engineer
Persistent
- Designed AI-assisted UX for data profiling and validation dashboards, enabling users to detect anomalies visually.
- Developed enterprise web applications and dashboards using React, Angular, and modern JavaScript frameworks.Designed responsive UI layouts and reusable design components, improving consistency across products.
- Built interactive dashboards and visualization components for analytics platforms. Implemented UI performance optimization.
AI/ML Intern
Hankernest
- Built ML/DL pipelines using TensorFlow, PyTorch, and Scikit-learn for structured and unstructured data. Designed workflows encompassing preprocessing, training, evaluation, and deployment on Azure. Optimized models via tuning, ensembling, quantization, and reducing latency.
- Developed CNN, LSTM, and Transformer solutions with explainability (SHAP/Grad-CAM).
- Built REST APIs and batch inference integrated with applications. Created Power BI/Tableau dashboards.
Experience
AI-powered Autonomous Data Platform – Medallion & LLM DataOps Engine
Modern data platforms suffer from:
• Manual data quality checks
• Slow onboarding of new datasets
• Complex pipeline migrations (from ADF to Databricks)
• Limited self-service analytics
Solution overview: You build a next-generation data platform that combines:
1. Scalable data engineering platform
• Azure Databricks and ADF pipelines
• Medallion architecture: bronze, then silver, then gold
• Delta Lake optimization (partitioning, incremental loads)
2. LLM-powered DataOps layer
• Auto data profiling and quality rule generation
• Schema evolution and DDL automation
• ADF to Databricks pipeline migration assistant
3. Intelligent analytics layer
• NL-to-SQL using GraphRAG and Unity Catalog
• Self-service analytics with guardrails
• Power BI dashboards for business users
Enterprise AI Copilot for Knowledge Management
Organizations struggle with:
• Knowledge silos across SharePoint, Confluence, PDFs, and databases
• Slow access to business-critical information
• Inconsistent responses from traditional search systems
• High support and operational overhead
Solution overview: Built an intelligent AI assistant platform that combines:
1. Agentic AI architecture
• Multi-agent workflows using LangGraph
• Specialized agents for search, analytics, and reporting
• Human-in-the-loop approval mechanisms
2. Enterprise RAG framework
• GraphRAG with Azure AI Search
• Vector search using embeddings
• Context-aware response generation with source citations
3. Business intelligence layer
• Natural language querying of enterprise data
• Automated report generation
• Conversational analytics and insights
Real-time Data Platform & MLOps Framework
Modern enterprises face:
• Delayed business insights from batch processing
• Inefficient model deployment processes
• Limited monitoring of data and model quality
• High operational costs for large-scale analytics
Solution overview: Built a cloud-native platform that combines:
Real-time data engineering
• Event-driven architecture using Kafka and Spark Streaming
• Delta Lake and Medallion architecture
• Near real-time analytics pipelines
Enterprise MLOps framework
• Automated model training and deployment
• MLflow experiment tracking
• CI/CD pipelines for machine learning
Monitoring and observability
• Data quality validation and alerting
• Model drift and performance monitoring
• Centralized operational dashboards
AI-powered Crop Intelligence & Agronomy Decision Platform
• Developed an agriculture domain ontology using OWL/RDF/SHACL to establish semantic relationships across datasets and created a knowledge graph for contextual discovery, semantic search, and relationship-aware reasoning. Applied machine learning, predictive analytics, NLP, and anomaly detection to derive crop health, disease risk, yield, and field-level insights.
• Built an ontology-aware RAG architecture combining knowledge graphs, vector search, embeddings, metadata filtering, and LLMs to deliver grounded, context-rich responses to natural-language agricultural queries. Implemented data quality, validation, lineage, and explainability mechanisms to improve trust in AI-generated insights.
The platform demonstrates how semantic data architecture, advanced analytics, and GenAI can be combined to create scalable, explainable decision intelligence for agriculture and enterprise.
Education
Master's Degree in Artificial Intelligence
IIT Patna - India
Bachelor of Technology Degree in Electronics
Shri Govindram Seksaria Institute of Technology and Science - Pune, India
Certifications
Databricks Certified Data Engineer Associate
Databricks
Skills
Libraries/APIs
PySpark, Pandas, NumPy, Node.js, React, Google Sheets API, SQLAlchemy, Vue, Azure API Management, Web API, REST APIs, SOAP APIs, TensorFlow, API Development, Sage, Amazon Marketplace Web Service (MWS), Playwright, QuickBooks API, Salesforce API, PyTorch, Matplotlib, OpenCV, Scikit-learn, OpenAI API, vLLM
Tools
Microsoft Power BI, Microsoft Excel, Microsoft Power Apps, Azure App Service, AWS Key Management Service (KMS), Azure Kubernetes Service (AKS), Terraform, Claude, ChatGPT, Microsoft AI, Microsoft Copilot, Power BI Desktop, Smartsheet, Tableau, Figma, Git, Apache Airflow, Claude Code, Blender, BigQuery, Amazon Elastic Container Service (ECS), Power Query, Seaborn, Jira, Azure OpenAI Service, Stencil.js
Languages
Python, SQL, JavaScript, Go, C#, Java, Scala, CSS, HTML5, CSS3, TypeScript, GraphQL, HTML, C++, Snowflake, Rust, OWL, RDF, SPARQL, Shapes Constraint Language (SHACL)
Frameworks
Angular, .NET, Django, Ontology Framework, Next.js, React Native, Bootstrap, LangGraph, NestJS, Express.js, Agentic Frameworks, Apache Spark, Selenium, Data Lakehouse
Paradigms
Azure DevOps, ETL, Testing, Test Automation, Back-end Architecture, Model Context Protocol (MCP), DevOps, Serverless Architecture, Business Intelligence (BI), Agile Product Management, Automation, Agile, Rule-based Programming, Event-driven Architecture, Microservices, HIPAA Compliance
Platforms
Azure, Databricks, Kubernetes, Amazon Web Services (AWS), Google Cloud Platform (GCP), Windows, Azure AI Studio, SharePoint, Microsoft, Apache Kafka, Azure Functions, AWS IoT, Fabric Lakehouse, Microsoft Fabric, Azure Service Fabric, Azure Event Hubs, Replit, Salesforce, Docker, AWS Lambda, LangSmith, Langfuse, Microsoft Power Automate, Kubeflow, TigerGraph, Azure AI Search, Salsify
Storage
SQL Server Management Studio (SSMS), MySQL, Azure Cloud Services, Azure SQL, PostgreSQL, JSON, Google Cloud, Data Validation, Databases, Database Modeling, Amazon S3 (AWS S3), Data Pipelines, Microsoft SQL Server, Redshift, Azure Cosmos DB, MongoDB, Redis, On-premise, Data Integration, Database Replication, Relational Databases, Neo4j
Industry Expertise
Insurance, Healthcare
Other
Azure Data Factory (ADF), Azure Data Lake, Azure Databricks, Microsoft Azure, Large Language Models (LLMs), Microsoft Azure Portal, CI/CD Pipelines, Data Analytics, Data Reporting, Data Science, Data Engineering, Cloud Infrastructure, Engineering, Excel Expert, Agentic AI, Artificial Intelligence (AI), FastAPI, Software Architecture, Architecture, SQL Server Express, SQL Server, Azure Function App, Multi-agent Systems, Agentic Workflow Design, Prompt Engineering, Workflows, Workflow, Third-party APIs, Third-party Integration, Troubleshooting, OpenAI, Web Scraping, Back-end, Data Infrastructure, GCP DevOps, Data Warehousing, Probability Theory, APIs, Data Modeling, Big Data, GitHub Actions, Full-stack Development, Solution Architecture, Azure AI Services, LLM Integration, AI Product Strategy, Agentic RAG Systems, Enterprise Software, Data Structures, WebSockets, Technical Leadership, Low Latency, Algorithmic Trading, API Design, Back-end Development, Web Application Design, Cloud, Cloud Architecture, Technical Architecture, Audits, Data Quality, Software Engineering, AWS DevOps, Infrastructure Automation, Cloud Engineering, Platform Engineering, Product Ownership, Data Processing, Scraping, Algorithms, Cloud Platforms, API Management, Data Migration, Data Management, Data Analysis, AI Tools, Microsoft 365, Data Mining, Process Mining, Ontology, RAG Architecture, Machine Learning, Cloud Storage, MERN Stack, Dashboards, Systems Design, Interaction Design (IxD), Data Visualization, Responsive UI, Quantitative Finance, Quantitative Analysis, Temporal, LangChain, MLflow, Amazon Bedrock AgentCore, Generative Artificial Intelligence (GenAI), API Gateways, Efficient Data Serialization, API Integration, Full-stack, AI Agents, Integration, ReAct Agents, Network Engineering, Censorship Measurement, Machine Learning Operations (MLOps), Retrieval-augmented Generation (RAG), Delta Lake, Containerization, Cursor AI, Large Data Sets, Agentic AI Systems, Minimum Viable Product (MVP), Agentic Coding, Terraform Cloud, Audio, Vite, Laboratory Information Management System (LIMS), Trading, Quantitative Trading, Forex, Anthropic, ChatGPT API, Healthtech, Devin, AI Architecture, Consulting, Infrastructure, Infrastructure Architecture, User Experience (UX), Azure AI Document Intelligence, AI Agent Orchestration, Life Science, AI-generated Code, Data Extraction, Data Mapping, Process Automation, QuickBooks Online, Fintech, DevOps Engineer, DevOps Automation, Reliability Engineering, Incident Response, Distributed Systems, Scalability, RAG Systems, Lead Generation, Site Reliability Engineering (SRE), Data Governance, Metadata, Web Development, AWS Cloud Architecture, Microsoft Lists, Deep Learning, RAG Pipelines, Data Architecture, Hugging Face, Vector Databases, Pinecone, ELT, Enterprise Architecture, Polars, Predictive Analytics, Natural Language Processing, Feature Engineering, RDFS, ontology mapping, semantic modeling, Graph Analytics, semantic reasoning, Embeddings from Language Models (ELMo), Vector Search
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring