Darshan is available for hire

Darshan Singh

Verified Expert in Engineering

Data Engineering Developer

Berlin, Germany

Toptal member since November 14, 2016

Expertise

Data Engineering Enterprise Development Data Warehouse Big Data Architecture Azure ETL SQL Python Sybase Database Snowflake AWS HDFS Spark Neo4j

Bio

For the past 16 years, Darshan has worked as a database developer, architect, and performance tuning expert, utilizing MS SQL, PostgreSQL, Redshift, and Snowflake. Since 2012, he’s been focusing on data and big data engineering projects using Spark, Hadoop, NoSQL, Python, Java 8, Kafka, and AWS, mainly building traditional ETL pipelines (Unix, Python, and SQL), big data ETL pipelines (Python, Spark, Hadoop, and HDFS), and real-time ETL pipelines (Kafka).

Portfolio

DKV

Snowflake, Apache Kafka, Databricks, SQL, ADF, Azure, Python 3, PySpark...

Tropicana Brands - Main

Databricks, Data Engineering, Leadership, Data Management, Azure, Azure Synapse...

BCG - Gamma

Data Engineering, Snowflake, Pandas, Python 3, SQL, ETL, Snowpark

Experience

Data Engineering - 20 years
SQL - 15 years
Python 3 - 15 years
ETL - 15 years
Database Modeling - 12 years
Snowflake - 7 years
Apache Kafka - 3 years
Apache Spark - 3 years

Preferred Environment

Linux, OS X, Windows

The most amazing...

...project I scaled was an eight-terabyte SQL server application from 4,000 requests per second to around 70,000 requests per second.

Work Experience

Senior Data Engineer

2025 - 2026

DKV

Designed and implemented end-to-end data pipelines ingesting real-time event streams from Kafka using Databricks Structured Streaming, persisting to Azure Data Lake Storage in Parquet format.
Orchestrated multi-source data ingestion workflows combining Kafka streaming, direct file ingestion, and ADF-based batch pipelines into a unified Snowflake.
Designed and implemented Snowpipe-based continuous ingestion from Azure Data Lake into Snowflake, enabling near real-time data availability for downstream transformation and analytics.
Built and maintained modular dbt transformation layers on Snowflake, implementing staging, intermediate, and mart models to deliver clean, analytics-ready datasets.
Implemented reverse ETL pipelines using Databricks to publish dbt-transformed Snowflake data back to Kafka topics, closing the data loop for real-time consumption by end-user applications.
Delivered a scalable unified data platform on Snowflake supporting both streaming and batch ingestion patterns, significantly reducing data latency and enabling self-serve analytics across multiple business domains.
Implemented Snowflake external stages pointing to Azure Data Lake for automated Snowpipe ingestion and leveraged Snowflake Streams for CDC-based change tracking to feed incremental dbt models.
Implemented automated dbt tests, Jinja macros, and dbt-docs to enforce data quality standards and provide full lineage from source to consumption.
Optimized warehouse performance by developing incremental dbt models powered by Snowflake Streams to process CDC data with minimal latency.
Engineered incremental dbt models and dbt snapshots to optimize warehouse performance for CDC data while capturing historical state changes.

Technologies: Snowflake, Apache Kafka, Databricks, SQL, ADF, Azure, Python 3, PySpark, Data Build Tool (dbt)

Data Engineer | Databricks

2024 - 2025

Tropicana Brands - Main

Built robust data pipelines in Azure Databricks using Delta Lake for Medallion architecture, automating ingestion and transformation of large-scale data, leading to 90% reduction in manual ETL efforts.
Developed modular and reusable dbt models to transform curated Delta tables into analytical datasets. Implemented testing, documentation, and CI/CD workflows to ensure data quality and faster development cycles.
Led the migration of legacy SQL-based workflows into Databricks SQL and Delta Lake, applying schema evolution and time-travel to support robust audit and rollback capabilities in production data.
Orchestrated a successful migration to Unity Catalog, restructuring permissions, catalogs, and schema ownership to enable fine-grained access control across teams while ensuring GDPR and compliance.
Optimized Synapse Serverless SQL views over curated Parquet data, achieving 70% faster query performance and powering cost-efficient ad-hoc analytics across large datasets with minimal compute overhead.
Developed GitHub Actions workflows utilizing the Databricks CLI to automate the deployment of Databricks Asset Bundles (DABs), streamlining the promotion of multi-environment jobs from development to production.
Integrated data from SQL Server and Parquet sources into Delta Lake, transforming it via dbt, exposing standardized tables for analytics, and supporting multi-source analytics with a consistent schema.
Developed a modular dbt project on Databricks to drive data flow through Bronze, Silver, and Gold layers, ensuring clear separation between raw data preservation and business-ready metrics.
Implemented automated dbt tests, Jinja macros, and dbt-docs to enforce high data quality standards and provide comprehensive lineage across the migrated Lakehouse environment.
Delivered interactive Power BI executive dashboards by connecting to dbt-transformed Databricks Gold layer, enabling self-service analytics and data-driven decision-making across business domains.

Technologies: Databricks, Data Engineering, Leadership, Data Management, Azure, Azure Synapse, DataWare, Data Build Tool (dbt)

Data Engineer

2022 - 2024

BCG - Gamma

Designed and developed the data model for Snowflake for a greenfield project. The data has lots of geospatial requirements like finding distances, covering by area, etc.
Created API endpoints for uploading and returning data.
Used Snowpark API while loading and querying the data, and Snowpark Python to transform and precalculate the formulas.
Created the entire history of the major tables. Separated the core database from the local and sandbox databases so that each user can have their own data and use the core one as well.
Used pandas for geospatial analysis, which was impossible with Snowflake due to the size of each geometry.

Technologies: Data Engineering, Snowflake, Pandas, Python 3, SQL, ETL, Snowpark

Senior Data Engineer

2022 - 2023

Eon

Architected a scalable Snowflake environment on Azure, utilizing External Stages, Storage Integrations, and Snowpipe to ingest SAP and Salesforce data from Azure Data Lake Storage (ADLS).
Optimized warehouse performance and costs by implementing Snowflake Streams and Tasks for CDC tracking and leveraging multi-cluster warehouses for high-concurrency workloads.
Designed a multi-layered dbt architecture (Staging, Intermediate, and Mart) on Snowflake to transform disparate ERP and CRM datasets into unified, business-ready data models.
Engineered incremental dbt models to process delta loads and utilized dbt Snapshots (SCD Type 2) to maintain historical accuracy for SAP financial and Salesforce sales data.
Standardized data governance by implementing automated dbt tests, custom Jinja macros, and dbt-docs to ensure 100% data integrity and provide end-to-end lineage.
Orchestrated complex data lifecycles using Azure Data Factory (ADF) and Python, automating the flow from source extraction to Snowflake ingestion and dbt-led transformation.
Delivered interactive Power BI executive dashboards by connecting to dbt-transformed Snowflake marts, enabling self-service analytics and data-driven decision-making across business domains.
Migrated data from Google Cloud Storage (GCS) to Azure Data Lake Storage Gen2 using Azure Data Factory (ADF) Copy Data activity with GCS linked service and HMAC key authentication.
Exported BigQuery tables to GCS in Parquet format and ingested into Azure Data Lake Storage Gen2 via ADF pipeline, validating data completeness by reconciling row counts and schemas post-migration.

Technologies: Snowflake, SQL, Python 3, Apache Airflow

Senior Data Engineer

2021 - 2022

Pfizer

Architected and deployed a centralized Amazon Redshift data warehouse, utilizing S3 and AWS Glue Catalog to ingest and manage TB-scale datasets from disparate sources.
Developed serverless AWS Glue ETL jobs using PySpark to perform high-scale transformations, loading optimized Parquet data from S3 into Redshift for downstream analytics.
Engineered high-performance Redshift tables using optimized distribution keys, sort keys, and advanced SQL to process complex datasets before migrating results to PostgreSQL.
Designed and managed complex Apache Airflow DAGs to orchestrate the end-to-end lifecycle between PySpark Glue jobs, Athena ad-hoc queries, and Redshift warehouse transformations.
Integrated Amazon Athena to query raw data residing in the S3 data lake, enabling serverless data exploration and reducing Redshift storage requirements for infrequently accessed data.
Built Python-based ETL workflows to migrate and transform Redshift and PostgreSQL data into Neo4j, implementing high-efficiency Cypher queries for material genealogy.
Optimized PySpark data frames and Redshift COPY commands to significantly reduce data ingestion latency and improve overall warehouse query performance.

Technologies: SQL, Neo4j, Cypher, Redshift, PostgreSQL, Python, Pandas, Amazon Web Services (AWS)

Senior Data Engineer

2020 - 2021

BCG

Led the migration of legacy SQL Server workloads to a centralized Snowflake data warehouse on AWS, designing optimized tables, JavaScript stored procedures, and UDFs for core business logic.
Architected end-to-end ETL flows using Fivetran, AWS Lambda, and AWS Glue to ingest disparate marketing data into Snowflake, ensuring high-frequency data availability.
Engineered a modular dbt transformation layer on Snowflake, centralizing all business logic in Git and leveraging environment-specific targets to ensure consistency across Dev, QA, and Prod.
Developed Apache Airflow DAGs to orchestrate complex data lifecycles, managing dependencies between AWS Lambda triggers, Glue ETL jobs, and dbt warehouse runs.
Automated the deployment of AWS serverless components using AWS SAM and CloudFormation, ensuring reproducible infrastructure across the cloud ecosystem.
Implemented automated deployment pipelines using Jenkins to manage the lifecycle of AWS Lambda functions and Glue jobs across multiple environments.
Migrated the reporting stack from Power BI to Tableau, optimizing underlying Snowflake SQL queries to deliver sub-second dashboard performance for the marketing team.
Integrated dbt tests and Airflow monitoring into the migration workflow to validate data parity between the legacy SQL Server and the new Snowflake warehouse.

Technologies: Amazon Web Services (AWS), Git, Fivetran, AWS Glue, Tableau, Amazon S3 (AWS S3), Microsoft SQL Server, Python, AWS Lambda, Snowflake, Data Build Tool (dbt), Azure SQL, Microsoft Power BI

Senior Data Engineer

2018 - 2021

Deutsche Börse Group

Developed and optimized on-premises SQL Server databases for the Financial Index team, implementing advanced indexing strategies, partitioning, and complex Stored Procedures to handle high-frequency market data.
Led the Azure migration of legacy SQL Server workloads using a lift-and-shift strategy, ensuring zero data loss and maintaining high availability for mission-critical index calculation engines.
Managed end-to-end data integrity and performance by auditing SQL execution plans and refactoring T-SQL logic, reducing query latency by 40% for the index product team.
Designed and implemented an end-to-end Azure Data Factory (ADF) and Databricks ecosystem to migrate the reporting stack from on-premises SQL Server to a scalable Lakehouse environment.
Developed high-performance ETL pipelines using Python, PySpark, and Delta Lake on Databricks to integrate multi-source financial data into Azure Synapse for centralized reporting.
Orchestrated the full data lifecycle from ADF ingestion to Power BI consumption, delivering automated executive dashboards that replaced manual legacy reporting processes.
Architected the Snowflake data platform for Qontigo, designing optimized tables, stages, and JavaScript stored procedures to support large-scale index product data ingestion.
Engineered a modular dbt transformation layer on Snowflake, utilizing Staging-Intermediate-Mart architecture and Git-based version control to standardize global index business logic.
Deployed Apache Airflow DAGs to manage complex dependencies between data ingestion and dbt runs, incorporating dbt tests to ensure 100% data accuracy for financial benchmarks.

Technologies: Azure Synapse, Azure SQL, Azure, Azure Data Factory (ADF), Snowflake, SQL, SQL Server 2016, Azure Databricks, Apache Spark

Data Engineer

2017 - 2019

Mobilityware

Developed an AWS data pipeline to execute AWS EMR, which then called an ETL process, which used Apache Spark (PySpark) to process data from S3 and finally loaded the processed data into S3.
Designed and developed AWS Redshift data warehouse to handle terabytes of data which was then used by the data analyst team for dashboards.
Designed and tuned Redshift queries for efficiency.
Designed the tables using proper distribution keys and sort keys for efficiency.
Built a solution in Python and AWS Athena for the GDPR based on users' requests to delete or return their data—the data was either deleted or returned to the users.
Developed a solution to find all the data for the users in S3 files using AWS Athena, then read the files, deleted the users' data, and rewrote the files (because all of the users' raw data was stored in S3).
Enabled the return of user data using AWS Athena queries and made sure that the data stored in Redshift was deleted or returned to the end-users, which made it much easier as the data was much more structured.
Designed and developed Tableau dashboards based on Redshift data for various KPIs.
Implemented real-time stream processing using Apache Kafka and AWS Kinesis for incoming data from various IOT devices. Finally saved the processed data in AWS S3 and created Athena tables for further querying and processing.

Technologies: Amazon Web Services (AWS), Apache Kafka, Amazon Elastic MapReduce (EMR), Amazon S3 (AWS S3), Amazon Kinesis, AWS Lambda, AWS Data Pipeline Service, Redshift, Python, Java 8, Flink, Apache Spark

Database Designer and Developer

2017 - 2017

Transparency AI

Designed and developed a database in PostgreSQL to collect the data from different car dealerships. The data was in XML, CSV, and JSON format.
Conceptualized and built a Python ETL process to transform the data into XML, CSV, and JSON formats as required per data model.
Wrote efficient PostgreSQL SQL, PL/pgSQL code, and other functions for reporting and loading data.
Built a proof of concept (POC) and developed dashboards using Power BI and Tableau to find which one suits better.

Technologies: PostgreSQL, Python

Senior Associate

2015 - 2017

JP Morgan Chase UK

Designed and developed columnar database systems using Sybase IQ for better performance.
Designed and developed Apache spark solution for handling complex business transformation for the profit and loss benefits where we had to generate the reports with almost 1,000 columns.
Used HDFS and parquet files to handle schema-less data with some rows having 100 columns and others with 1,000 columns for high performance.
Designed and developed an Apache Kafka solution for real-time processing of events and thus provide real-time updates on the profit and loss dashboards to various analysts.
Deployed and administrated Apache Hadoop, HDFS, Spark, and Kafka.
Used SQL for Sybase ASE and Sybase IQ related work.
Used Java, Python and Spark SQL for the big data work.

Technologies: Java, Python, Apache Kafka, Apache Spark, Hadoop, Sybase, Spark SQL, SQL

Principal Consultant

2014 - 2014

Genpact Singapore

Designed, modeled, and architected a new database system using Sybase ASE, MS SQL Server for scalability and performance.
Optimized and performance-tuned existing and new procedures using SQL DMVs Sybase Monitoring Tables to reduce queries that ran for hours to mere minutes.
Used SQL server trace, profiler, and extended events to troubleshoot the performance root causes (analysis and fixes).
Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase as well SQL server.
Conceptualized and implemented HA clustering and DR using database mirroring.
Partitioned the database table for maintenance and performance tuning.

Technologies: Sybase, Microsoft SQL Server

Database Architect

2012 - 2014

McAfee India Pvt Ltd.

Troubleshot the performance root causes by analyzing and implementing fixes. Used SQL server trace and profiler and extended events.
Designed and developed tables, stored procedures,, and indexes for new development and enhancement.
Worked on test-driven development and development using the agile methodology.
Worked on data modeling for changes and new development.
Monitored production server performance using DMVs and Perfmon; depending on the requirements for tuning the system, also the application and existing queries and objects.
Designed, tested, and tuned extensively on big data and NoSQL technologies like Cassandra and Hadoop, hive and pig stack to test different scenarios using Python to migrate the existing application onto a big data platform.
Designed and implemented HA clustering and DR using database mirroring.
Partitioned a database table for maintenance and performance tuning.

Technologies: Microsoft SQL Server

Associate

2011 - 2012

JP Morgan Chase India

Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase.
Used Sybase ASE’s XML to show plans, trace flags, and abstract query plans/statistics for performance root cause analysis.
Optimized and query performance-tuned existing and new procedures using monitoring tables to reduce queries running time by up to 2 to 30 times.
Worked on data modeling for changes and new development.
Developed SQL and T-SQL code using Sybase.
Developed Unix shell, Python, and Perl scripts for ETL and data analytics.
Partitioned database tables for maintenance and performance tuning.

Technologies: Python, Sybase

Experience

Scaling the McAfee Mobile Security App Database System

I worked on the McAfee Mobile Security application which used by various mobile devices for security and other activities. The data was being stored in a SQL Server 2008/2012. I worked on scaling the system from 4,000 transactions per minute to around 70,000 transactions per minute by removing various performance issues. I redesigned the HA and DR as well as the reporting systems using clustering, and database mirroring as well as replication.

Other Roles and Responsibilities:
• Used SQL Server trace, profiler, and extended events to troubleshoot the performance root cause (using analysis and fixes).
• Designed and developed tables, stored procedures, indexes for new development and enhancement.
• Worked on test-driven development and development using Agile.
• Implemented data modeling for changes and new development.
• Monitored the production server performance using DMVs and Perfmon; depending on requirements for tuning the system, also the application and existing queries/objects.
• Designed, tested, and tuned extensively on big data and NoSQL technologies like Cassandra and Hadoop, Hive, and Pig stack to test different scenarios using Python to migrate the existing application onto a big data platform.

Redesign and Architecture of a Compliant Web Database System

I redesigned the ETL as well as the database system to improve the performance of loading the data. It went from loading in more than eight hours to two hours. I designed the SQL server architecture to be a highly scalable system using various approaches including partitioning and partitioned views.

Other Roles and Responsibilities:
• Designed, modeled, and architected new database system using Sybase ASE, MS SQL server, and Oracle for scalability and performance.
• Optimized and performance-tuned existing and new procedures using SQL DMVs Sybase Monitoring Tables and Oracle performance views to reduce queries running in hours to minutes.
• Used SQL server trace, profiler, and extended events to troubleshoot the performance root causes (using analysis and fixes).
• Used Sybase ASE’s XML show plans, trace flags, abstract query plans and statistics for performance root cause analysis.
• Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase as well as the SQL server.

Sybase Database System Design and Development for the PB Credit System

I worked as a Sybase database designer as well as a developer for J.P. Morgan's PB credit application. I mainly worked with business analysts and gathered the requirements for new changes and reports. With this material, I then redesigned the existing database system as well as developed using SQL, TSQL, stored procedures views, and more.

Other Roles and Responsibilities:
• Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase.
• Used Sybase ASE’s XML show plans, trace flags, abstract query plans, and statistics for a performance root cause analysis.
• Optimized and query performance-tuned existing and new procedures using monitoring tables to reduce queries running time by up to 2 to 30 times.
• Developed Unix shell, Python, and Perl scripts for ETL and data analytics.
• Implemented data modeling for changes and new development.
• Wrote SQL and T-SQL code using Sybase.

Credit Suisse Swap Database System

I worked as the database designer and developer and administrator to the Credit Suisse Swap database system—I redesigned the database for new reports and migrated some of the reports from a SQL server to Sybase IQ. I also migrated the SQL server from 2000 to 2008. The main highlight for me was redesigning a C# FIFO trade matching in a SQL server for ad-hoc rematching.

Other Roles and Responsibilities:
• Created new stored procedure, functions, triggers, and views in a SQL Server.
• Optimized and query performance-tuned existing and new procedures using DMVs.
• ETL development using SSIS 2008 and report development using SSRS 2008.
• Using SQL server trace, profiler, and extended events to troubleshoot the performance root cause analysis and fixes.
• Developed Unix Shell, Python, and Perl scripts.
• Conducted an impact analysis for the migration of SQL server 2000 to the 2008 version.
• Defined the capacity planning and designed the migration to SQL server 2008 for performance improvement.
• Changed DTS packages to SSIS packages as well changed SQL and T-SQL code to be compatible with SQL Server 2008.

Real-time Analytics Platform

I have designed data pipelines for risk and PNL teams for an investment bank using technologies such as SQL, Sybase, Python, HDFS, Spark, and Kafka. The pipeline uses Kafka to move data from Sybase IQ to HDFS. Then we run a computing-intensive job against the HDFS and again move the output to Kafka and then it is sent to our reporting server in Sybase IQ and HDFS. Real-time data is then directly fed from Kafka to Spark and then processed and sent to reporting servers.

Data Warehouse and Data Lake for Transparency

I designed a data warehouse and data lake for transparency using AWS services like ETL, S3, Lambda, EMR, Redshift, SQL, Python, and Spark. The data is used to land in S3 buckets in CSV files and then a lambda used to launch an EMR Spark job to clean up and process data in parallel.

I cleaned up and enriched the data that was to be moved to Redshift and where we will run our reporting queries. I have used PySpark and Python.

Education

2014 - 2015

Master of Science Degree in Data Science

Goldsmiths, University of London - London, UK

Skills

Libraries/APIs

Spark Streaming, Snowpark, Pandas, PySpark

Tools

AWS CloudFormation, Amazon CloudWatch, Amazon Athena, Spark SQL, Amazon Elastic MapReduce (EMR), Kafka Streams, Azure Logic Apps, Amazon Kinesis Data Firehose, AWS Glue, Terraform, Microsoft Access, Apache Iceberg, Flink, Git, Tableau, Tableau Desktop Pro, Microsoft Power BI, Apache Airflow

Languages

Snowflake, Python, Transact-SQL (T-SQL), SQL, Python 3, Java 8, Java, R, Cypher

Frameworks

Presto, Hadoop, Apache Spark, ADF

Paradigms

ETL, Database Design, ETL Implementation & Design, Agile, Scrum

Platforms

Databricks, AWS Lambda, Spark Core, Apache Kafka, Azure Functions, Linux, Azure, Kubernetes, Azure Event Hubs, Azure Synapse, Windows, OS X, Amazon Web Services (AWS)

Storage

HDFS, Apache Hive, Redshift, Amazon S3 (AWS S3), PostgreSQL, Sybase, Microsoft SQL Server, Database Modeling, Azure SQL, Data Pipelines, Data Lakes, AWS Data Pipeline Service, Azure Blobs, Neo4j, SQL Server 2016, DataWare

Industry Expertise

Healthcare

Other

Big Data Architecture, Data Architecture, Data Engineering, ETL Development, Data Build Tool (dbt), Data Warehousing, Azure Data Lake, Azure Data Lake Analytics, Azure Databricks, Data Modeling, ELT, Enterprise, Medallion Architecture, Fivetran, Data Science, Azure Data Factory (ADF), Azure Event Grid, Amazon Kinesis, Machine Learning, Leadership, Data Management

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring