Shivayogi Math Doddabasayya

• 83% — Accuracy
• 75% — Demand Forecast
• 25% — Churn Reduction

A mid-sized food delivery platform faced critical operational challenges with its legacy hosted system, which resulted in poor customer retention and high costs. Their on-time delivery rate was 65%, meaning 35% of orders arrived late. This caused customer dissatisfaction and a 5% monthly churn rate, leading to $250,000 in monthly lost revenue from 100,000 orders.

PAIN POINTS
• Operational Inefficiency: Manual delivery partner assignment took 10-15 minutes per order, creating peak-hour bottlenecks.
• Inaccurate Time Predictions: Rule-based ETA ignored real-time traffic, weather, and demand, causing late deliveries.
• No Demand Forecasting: The platform couldn’t predict order volume, causing idle partners or delayed orders in various zones.
• Slow Deployment: The legacy system took 4-6 months to onboard new clients, compared to 8-12 weeks for SaaS competitors.
• Limited Scalability: The monolithic architecture restricted expansion and updates, requiring downtime and custom development.

CONTEXT AND DATA
• 15,000+ historical orders across 22 cities
• 65% on-time delivery rate vs. 75-80% industry standard
• 5% churn rate with 500 customers lost per 10,000 base
• An average customer lifetime value of $500
• Customer acquisition cost was at $20-50

This set the stage for building an ML-powered SaaS solution to improve prediction accuracy, automate operations, and reduce time-to-value.

5-STEP SOLUTION

1. Analyzed 15,000+ orders from my IISc capstone, shadowed 5-6 operations managers, and reviewed support tickets. Discovered rule-based systems ignored traffic or weather, causing only a 65% on-time rate.

2. Built 20+ input features: distance, real-time traffic with Google Maps API, weather, delivery partner rating, time of day, and festivals. Used sin/cos transformations for cyclical patterns and created lag features for time trends.

3. Tested linear regression, random forest, and XGBoost won. Used Optuna for hyperparameter tuning across 100+ combinations. Did 5-fold cross-validation with time-based splits, and achieved a mean absolute error of 4.2 minutes.

4. Deployed a FastAPI microservice with an under 200-millisecond response, Redis caching of 1-hour TTL—reducing monthly API costs from $500 to $200— and Kubernetes auto-scaling for rush hours. A/B-tested the 20% to 100% gradual rollout and created a multi-layer fallback.

5. Ensured AI tracked model drift, weekly auto-retraining on Sundays with new data, MLflow tracked versions for rollback, and operations managers flagged bad predictions—the system learns from mistakes and continuously improves.

UNIQUE VALUE
• Real ML experience: 83% vs. 70% industry from the capstone project
• Cost-conscious: Redis saved $300 monthly per customer
• Reliability: 3-layer fallback never completely fails
• Business impact: 72% on-time rate, 30% churn reduction, $177,000 yearly savings

Achieved an 83% R² score using XGBoost with Optuna hyperparameter tuning, predicting food delivery times, a 75% driver demand forecast by building an XGBoost Classifier, and a 25% customer churn reduction via accurate delivery time estimates and proactive delay notifications.

RESULTS

1. Delivery Performance

• 11% improvement for on-time delivery from 65% to 72%
• 65% better prediction accuracy from 12 minutes to 4.2 minutes
• 35% monthly customer complaints reduction from 120 to 78

1. Business Metrics

• 30% drop in customer churn from 5% to 3.5% equivalent to $75,000 yearly savings
• 50% cut in monthly delivery waste from $20,000 to $10,000, equivalent to $120,000 yearly savings
• 60% faster deployment from 4-6 months to 16 weeks
• $237,000 total annual savings per customer

1. User Adoption

• Delivery app: 90% within 30 days
• Manager usage: 85% daily
• ML adoption: 75% within 60 days

1. Technical Excellence:

• 200-millisecond API response
• 99.9% uptime
• 83% R² score in model accuracy

LONG-TERM BENEFITS
A data flywheel—more orders, better data, accurate predictions, happier customers, more orders—an 83% ML accuracy that rule-based competitors can’t match, 18% higher lifetime value, and a cloud that supports 10x growth, brand loyalty—the 72% on-time rate driving 25% repeat orders and 40% more referrals.

• 15% — Answer Accuracy Improvement
• 40% — Faster Retrieval Workflow
• 60% — Troubleshooting Time Reduction
• 50% — Perceived Response Time Boost

Students and instructors in the AI/MLOps program struggled to efficiently extract specific information from multiple assignment instruction PDFs and course materials.

PAIN POINTS
• Time-consuming Manual Search: Students spend 15-20 minutes manually reading multiple PDF documents using Ctrl+F keyword search to find specific assignment requirements, submission guidelines, or grading rubrics, reducing productive learning time.
• Keyword Search Limitations: Traditional PDF search tools cannot understand semantic relationships. For example, searching for “deployment instructions” won’t find related information about “Hugging Face Spaces setup,” even though they refer to the same concept, resulting in 40% of relevant information missing.
• Repetitive Instructor Queries: Teaching assistants receive 50-100 repetitive questions weekly about information already documented in assignment PDFs, consuming 10-15 hours that could’ve been spent on meaningful mentorship.
• Context Loss Across Documents: Students lack a unified query interface to search across all documents simultaneously.
• No Contextual Understanding: There were no direct answers; documents had to be manually read.
• Late Discovery of Requirements: Students often discover critical requirements—team activity, mentor presentation timing, attendance rules—buried in the middle or end of PDFs only after starting work, leading to 30% of submissions having format or process errors.

As a solution, I developed an intelligent document query system using LangChain and LangGraph to orchestrate retrieval workflows with vector embeddings, semantic search, and LLMs.

1. Ingestion: Used PyPDFLoader and DirectoryLoader to load PDFs and extract text with metadata.
2. Chunking: Applied RecursiveCharacterTextSplitter with tuned chunk_size and chunk_overlap to retain context.
3. Embedding and Storage: Generated embeddings via HuggingFace MTEB models; stored in FAISS and tested Chroma and Pinecone.
4. RAG Pipeline: Built RetrievalQA chain combining LangChain retriever with Llama-2-7B/Mistral-7B. Added ConversationalRetrievalChain for multi-turn Q&A with memory, compared standard vs. MMR retrievers, and used PromptTemplate to ground answers in retrieved content.
5. Orchestration in LangGraph: Created state machine with query analysis, retrieval, reranking, generation, and validation nodes, and conditional edges reroute low-confidence queries. Added self-reflection for rephrasing and re-retrieval.
6. Validation: Tested with five realistic questions—grading, HF Spaces deployment, deadlines, project differences, and dataset choices—and evaluated groundedness via QAEvalChain.
7. Gradio Interface: Wrapped ConversationalRetrievalChain in Gradio with history and “show sources” to display chunks.
8. Deployment: Containerized and deployed on Hugging Face Spaces with GPU support and enabled streaming responses via callbacks.

Improved query answer accuracy by 15%, achieved a 40% faster retrieval workflow, and increased perceived response time by 50%.

Orchestration Efficiency:
• LangGraph State Management: Complex multi-step retrieval workflows were executed 40% faster than custom pipeline code.
• Conditional Logic: LangGraph’s conditional routing automatically retries with expanded search when initial retrieval confidence is low, improving answer accuracy by 15%.

Developer Productivity:
• LangChain Abstractions: Reduced development time from three weeks to one week using LangChain’s pre-built components.
• Prompt Engineering: LangChain’s PromptTemplate and FewShotPromptTemplate enabled rapid experimentation with 10+ prompt variations.

Maintainability:
• Modular Architecture: LangGraph nodes can be updated independently.
• LangSmith Integration: Logged all LangChain runs for debugging, showing exact chunks retrieved and LLM reasoning, reducing troubleshooting time by 60%.

Advanced Features:
• Conversational Memory: LangChain’s memory buffers enable follow-up questions.
• Agent-Based Routing: LangGraph agents can route queries to different knowledge bases based on query type.

Scalability:
• LangChain LCEL: Parallel retrieval execution from multiple vector stores reduced query latency from five seconds to two seconds.
• Streaming: LangChain streaming callbacks provide token-by-token LLM output, improving perceived response time by 50%.

• 0.88-0.94 — F1 Score
• 0.90 — Precision
• 85-92% — Multi-class Accuracy
• 5% — Model Drift Detection Accuracy

PROBLEM
Text classification challenges in ticket routing: fastText model limitations in the production environment.

1. Poor Contextual Understanding

• fastText was a bag-of-words with no semantic context
• “urgent server down” vs. “server down resolved” are in the same category
• Accuracy was at 72% and needed to be at 85%
• Misrouting was at 15% and caused a $180,000 yearly rework

1. Out-of-vocabulary Failures

• Product codes and error messages had a 20% error rate
• “SAP timeout” vs. “Oracle timeout” were treated identically
• Wrong routing caused 4-6-hour delays

1. Multilingual Gaps

• English 72%, Mandarin 58%, German 61%
• Separate models per language caused 3x training effort
• Code-mixed tickets failed

1. Intent Misclassification

• “Critical issue” vs. “routine query” had the same priority
• False escalations were at 25%, wasting senior engineers’ time

TRIED MISTRAL
• 800 milliseconds to 1.2 seconds latency, exceeding the under-500-millisecond SLA
• GPU costs were at $450 monthly per instance
• 8% hallucinations on technical jargon
• Long tickets truncated

WHAT WASN’T WORKING
• fastText: Fast at 50 milliseconds, but was 72% inaccurate
• Mistral: 88% accurate, but slow and expensive
• Need: Balance for 10,000+ tickets per day

SOLUTION
Optimized BERT for production text classification

Week 1-2: Model Selection
• Evaluated BERT-base, DistilBERT, ALBERT vs. fastText
• Selected DistilBERT, which was 40% faster and had 97% BERT accuracy
• Saw a 150-200-millisecond latency, which met the under-500-millisecond SLA at a 60% smaller rate
• Tested 5,000 labeled tickets and validated on real data

Week 3-4: Hybrid Architecture

• fastText pre-filter was at 50 milliseconds and handled 60% of simple cases
• BERT for complex tickets was at 200 milliseconds for semantic understanding
• The result was a 95-millisecond average latency vs. the 200-millisecond full BERT
• Infrastructure comprised of AWS EKS and GPU nodes, batch inference, ONNX Runtime speeding up 2.3x, and FP16 quantization reduced memory by 50%

Week 5-6: Training Approach
• 42,000 historical tickets cleaned
• Handled class imbalance: Weighted loss (40% zero-class)
• Pre-trained DistilBERT and three domain-specific epochs
• Created a multilingual mBERT single model covering English, Mandarin, and German

Week 8-10: Deployment and Validation
• A/B tested 10% BERT traffic, monitored accuracy and latency
• Results were 91% accuracy vs. 72% and 4% misrouting vs. 15%
• Used Prometheus/Grafana for real-time performance monitoring
• Auto-scaled with Kubernetes HPA

KEY INNOVATIONS
• A hybrid pipeline for speed and accuracy
• Production-ready ONNX optimization
• mBERT eliminated three separate models
• Applied food delivery zero-inflated data handling

Translated food delivery ML—92% R², 54 clusters, and zero-inflated data—to enterprise text classification. Achieved 91% accuracy and a latency of under 200 milliseconds through a hybrid architecture, delivering 3.1x ROI and 39% faster resolution.

BERT IMPLEMENTATION OUTCOMES

1. Classification Accuracy

• 26% accuracy improvement from 72% to 91%
• 73% misrouting reduction from 15% to 4%
• Multilingual gaps were unified to 91% and 87% from 72% for English and 58% for Mandarin 58%

1. Operational Efficiency

• $180,000 yearly savings in manual re-classification costs.
• 39% faster resolution time from 6.2 hours to 3.8 hours
• 450 engineer hours saved per year with false escalations dropping from 25% to only 6%
• 18% better support ticket routing, reducing ticket volume

1. System Performance

• 120-millisecond hybrid latency achieved vs. 50-millisecond fastText and 800-millisecond Mistral
• 2.4x throughput increase from 5,000 to 12,000 tickets daily
• SLA compliance improved from 71% to 94%

1. Business Impact

• 62% customer NPS increase from 42% to 68%
• Customer churn rate dropped from 28% to 19% with improved service quality
• 3.1x ROI with $96,000 infrastructure cost and $300,000 savings

LONG-TERM BENEFITS
• Scalability: Single mBERT supports new languages (no retraining)
• Adaptability: Client onboarding dropped from two weeks to three days
• Competitive edge: AI routing vs. competitors’ rule-based systems
• Proactive insights: Semantic clustering detects emerging issues

• 92% — Accuracy Ratio
• 100% — Validation
• 40% — Reduction in Ticket Resolution Time

Back in 2020, I worked as a traditional infrastructure engineer, managing IT service desk operations. Our team was drowning in repetitive tickets, password resets, system access requests, and basic troubleshooting that consumed 60% of our daily bandwidth. Talented engineers spent their time on mundane tasks instead of solving complex problems that could truly impact business outcomes.

I automated service desk operations by building AI-driven ticket classification and routing using NLP and MLOps pipelines. Repetitive tasks were handled by bots and self-service workflows, cutting manual effort by over 50%. Engineers moved to high-impact work, while monitoring, dashboards, and automated retraining kept models accurate. This improved SLA compliance, reduced backlog, and increased customer satisfaction through faster, more reliable resolutions. It also lowered costs, improved auditability, and built strong leadership trust in AI-driven operations overall value.

• Our BERT-based ticket classification system achieved 92% accuracy.
• Our AI-driven system reduces ticket resolution time by 40%, and our team focuses on strategic initiatives instead of repetitive work.