Karan Vijay Singh
Verified Expert in Engineering
Data Engineer and Software Developer
Vancouver, BC, Canada
Toptal member since June 12, 2026
Karan is a software engineer specializing in data and AI with over 8 years of experience in large-scale data engineering and analytics. His expertise spans Databricks, Snowflake, GCP, and Airflow, with notable work at DoorDash and LightspeedHQ. Karan led the IceX migration at DoorDash, automating conversions across 300+ tables and 100+ pipelines, reducing migration effort by 75% and deployment time to under 15 minutes.
Portfolio
Experience
- SQL - 10 years
- AWS IoT - 8 years
- Python - 8 years
- Snowflake - 6 years
- Spark - 6 years
- Databricks - 5 years
- Google Cloud Platform (GCP) - 5 years
- Claude Code - 2 years
Work Experience
Software Engineer – Data & AI
DoorDash
- Led the IceX migration initiative, architecting an automated framework to convert Snowflake-native tables to Snowflake-managed Iceberg by analyzing Airflow DAGs and SQL, scaling across 300+ tables and 100+ pipelines managing TBs of production data.
- Designed and built a Snowflake Target Load operator, enabling full lifecycle Iceberg table management (creation, schema evolution, metadata tagging, and governance), eliminating redundant storage and significantly reducing storage costs.
- Reduced migration effort by 75% and cut deployment time from hours to less than 15 minutes, while enforcing schema validation, automated query rewrites, and auditability at scale; enabled governed cross-engine analytics via Databricks Unity Catalog.
- Re-architected data pipelines by consolidating multi-stage Delta workflows into direct Delta to Snowflake processing and enabling Photon runtime, achieving 50% reduction in latency and significant improvements in compute efficiency.
- Built developer-facing tooling using Django and Databricks APIs to enable self-service job orchestration, reducing dependency on platform teams.
- Conducted performance benchmarking of Iceberg table formats, including managed, unmanaged, and Uniform configurations, to inform platform architecture decisions across Snowflake and Databricks environments.
- Established data governance and RBAC across Snowflake and Databricks (Unity Catalog), enabling secure data sharing, standardizing transformations with dbt, and driving self-service analytics via ThoughtSpot.
- Integrated pipelines with Chronosphere, enabling real-time monitoring of critical performance metrics and visualization of key insights into pipeline health and status; configured alerts for critical events.
- Improved data quality from approximately 80% to 99.99% by identifying and resolving root-cause issues in UDF logic in collaboration with product teams. Leveraged prompt engineering to develop AI-assisted tools for pipeline analysis and automation.
- Evaluated and applied advanced table optimization features (deletion vectors, Z-ordering, liquid clustering, predictive optimization) to improve query performance, data skipping efficiency, and reduce compute costs across large-scale datasets.
Senior Software Developer – Data
Lightspeed
- Optimized large-scale SQL transformations processing hundreds of gigabytes of data for Looker PDTs, reducing runtime and cutting costs by 50%, resulting in approximately $21,000 monthly savings across GCP, BigQuery, and SQL.
- Led migration of data warehouse from Snowflake to BigQuery, implementing automated validation using pytest and reducing manual testing effort by 80%.
- Drove cloud cost optimization initiatives by leveraging advanced query tuning and storage strategies to reduce spend by around $250,000 annually.
- Partnered with cross-functional teams to design data models and improve data accessibility for analytics and business use cases.
- Built a Python automation framework using Looker APIs to standardize and accelerate dashboard deployment across environments, eliminating manual effort and improving team productivity (Python, LookML).
- Designed and implemented REST APIs using Flask and SQLAlchemy within a containerized Docker architecture, improving scalability and reducing deployment time.
- Improved data pipeline robustness by optimizing merge logic and implementing retry mechanisms, reducing data inconsistencies and failures.
- Designed and implemented scalable ELT pipelines using Airflow (Composer), dbt, and BigQuery to enable reliable analytics and reporting for business intelligence and data science.
Data/ML Engineer
Ross Intelligence
- Optimized and enhanced the existing Apache Beam data pipeline involving thousands of XML files to include sort order in the final schema and reduced the running time from 6 hours to 30 minutes using GCP Dataflow.
- Processed data involving 13 million XML files using Databricks, PySpark, and Python.
- Leverages technologies such as Dataflow, Databricks, BigQuery, Python, SQL, XML, and GCS.
Data Scientist
Scotiabank
- Forecasted the trends in chequing account balances at the portfolio level for a 30-day window on daily balance data using deep learning-based ARIMA and LSTM models on detrended time series with an MAPE of 4.97%.
- Enhanced commercial banking recommendation engine for prospecting and deepening client relationships by decoupling some features using XG Boost library, PySpark, Python, NumPy, pandas, Hibernate Query Language (HQL), and Hive.
- Leveraged technologies such as Python, SQL, PySpark, Hive, machine learning, and NumPy.
Software Developer – Data
Ubilab
- Built pipelines to collect and process air quality data from IoT sensors.
- Designed a data ingestion pipeline using MongoDB to Elasticsearch to Kibana for visualization.
- Used technologies such as ETL, Python, Elasticsearch, Logstash, Kibana, and MongoDB.
Data Scientist
Amazon
- Processed and analyzed billions of rows of clickstream and order data of Amazon.com using Amazon S3, Redshift, EMR, EC2, PySpark, NumPy, pandas, Python, and Scala.
- Built logistic regression models to evaluate the impact of product messaging on user behavior, influencing shopping experience for over 200 million users.
- Leveraged Scala, Python, Spark SQL, AWS, S3, Redshift, EMR, EC2, and PySpark.
Product Development Engineer
MFS, Mahindra Comviva
- Built scalable back-end systems handling payments for over 150 million users.
- Reduced processing latency from 30 minutes to 60 seconds using asynchronous queue-based architecture (JMS).
- Automated CI/CD pipelines using Jenkins to reduce manual effort by 40%.
- Optimized CSV reports using Blocking Queue, which resulted in a reduction of report generation time of 100,000 records from 5 minutes to 40 seconds (Java).
Experience
Transformative Effects of IoT, Blockchain, & AI on Cloud Computing
https://scholar.google.com/citations?view_op=view_citation&hl=en&user=-_e6vI0AAAAJ&citation_for_view=-_e6vI0AAAAJ:u-x6o8ySG0sCEducation
Master's Degree in Computer Science
University of Waterloo - Waterloo, Canada
Bachelor's Degree in Computer Science and Engineering
Thapar University - Patiala, India
Certifications
Databricks Certified Data Engineer Associate
Databricks
Microsoft Technology Associate: Software Development Fundamentals
Microsoft
Skills
Libraries/APIs
PySpark, SQLAlchemy, NumPy, Pandas, LSTM, Java Message Service (JMS), Scikit-learn, Keras, TensorFlow
Tools
Apache Airflow, BigQuery, Claude Code, Codex, GitHub, Looker, Git, Tableau, Pytest, Apache Beam, Kibana, ARIMA, Jenkins, Jira, Cloud Dataflow, Amazon Athena, Terraform, Apache Iceberg, WinSCP, Toad, Crucible, IBM Rational ClearCase, Crystal Reports, PyCharm, Logstash, CircleCI, Claude, ChatGPT, Cloudability, Grafana
Languages
Snowflake, SQL, Python, Looker Modeling Language (LookML), Scala, Hibernate Query Language (HQL), Java, R, JavaScript, C++, C, YAML
Paradigms
ETL, Business Intelligence (BI), Role-based Access Control (RBAC), DevOps, Dimensional Modeling
Platforms
Databricks, Google Cloud Platform (GCP), Amazon Web Services (AWS), Salesforce, ThoughtSpot, Docker, Kubernetes, AWS IoT, Jakarta EE (Java EE or J2EE), webMethods, Azure, Dataform, Google Dataform
Storage
Redshift, Data Pipelines, Data Lakes, Relational Databases, MySQL, PostgreSQL, Google Cloud SQL, Google Cloud Storage, Data Integration, Master Data Management (MDM), Amazon S3 (AWS S3), MongoDB, Elasticsearch, Apache Hive, HDFS, Oracle SQL Developer, Dryad, DBeaver, Data Validation
Frameworks
Spark, Data Lakehouse, Django, Flask, Hadoop
Other
Data Build Tool (dbt), Delta Lake, Data Warehousing, Data Engineering, Google BigQuery, Data Modeling, Cloud Storage, ETL Pipelines, Data Analysis, Big Data, Database Normalization, Dashboards, Query Optimization, FinOps, Agentic AI, AI Engineering, Data Governance, Data Visualization, ELT, Artificial Intelligence (AI), Large Language Models (LLMs), Troubleshooting, Amazon Redshift, Generative Artificial Intelligence (GenAI), Prompt Engineering, APIs, OpenAI, AI Agents, Technology Management, Technical Leadership, Data Architecture, Data Quality, AI Tools, Unity Catalog, EC2, EMR, Machine Learning, Statistics for Data Science, Data Intensive Distributed Computing, Exploratory Data Analysis, Data Structures and Standards, Text Analytics, Uniformance Process History Database (PHD), HubbleStack, Docker Desktop, Cursor AI, Classification, Regression, Clustering, Software, Operating Systems, Research, Enterprise Data Warehouse (EDW), Data Security, Compliance, Identity & Access Management (IAM), CI/CD Pipelines, Advanced Analytics, ChatGPT API, Prometheus, Azure Data Factory (ADF), BI Reports
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring