Dmitry Foshin, Developer in Porto, Portugal
Dmitry is available for hire
Hire Dmitry

Dmitry Foshin

Verified Expert  in Engineering

Bio

Dmitry is a seasoned senior lead data engineer with 12 years of experience in IT and seven years of hands-on expertise in Azure, Databricks, SQL, and Python. Known for leading a globally distributed team of data and BI engineers, he brings a strategic mindset and technical depth to deliver scalable, high-impact data solutions. Clients value Dmitry's ability to turn complex data challenges into clear, actionable results.

Portfolio

Mars
Databricks, ETL, Python, Azure, Continuous Integration (CI), DevOps, SQL, Spark...
Weir Group
Azure Data Factory (ADF), Azure Synapse Analytics, Python, Fabric...
Pandora
Databricks, Azure, Python, Apache Airflow, Azure Synapse Analytics, SQL, Spark...

Experience

  • SQL - 12 years
  • Python - 7 years
  • Databricks - 7 years
  • Data Engineering - 7 years
  • Microsoft Power BI - 7 years
  • Azure - 7 years
  • Spark - 5 years
  • Microsoft Fabric - 2 years

Availability

Full-time

Preferred Environment

Azure, Python, Databricks, Data Engineering, SQL, DevOps, Azure Data Factory (ADF), Apache Airflow, Spark, PySpark

The most amazing...

...projects I've authored include Azure Data Factory Cookbook (two editions), Jumpstart Snowflake (2nd edition), and Data Engineering with Azure Databricks.

Work Experience

Lead Data Engineer

2024 - 2025
Mars
  • Designed and architected scalable data solutions to support new products and deployment in 30 markets, ensuring feasibility and technical alignment using Azure's tech stack, Databricks, Spark, and Synapse.
  • Built and implemented end-to-end CI/CD pipelines in Azure pipelines using YAML, artifacts, variable groups, and Azure DevOps to automate deployment processes.
  • Conducted code review sessions with four vendor development teams to uphold code quality, compliance, and adherence to best practices.
  • Provided support and learning sessions to junior data engineers.
  • Developed and maintained ETL pipelines using Python object-oriented programming, Databricks, ADF, data lakes, Azure SQL, Azure Event Hubs, Azure Functions, Azure Service Bus, Synapse, Azure DevOps, SFTP services, and APIs.
  • Bridged the gap between development and production teams, resolving technical challenges to streamline the deployment of data products into production environments.
Technologies: Databricks, ETL, Python, Azure, Continuous Integration (CI), DevOps, SQL, Spark, Synapse, Data Lakes, PySpark, Microsoft SQL Server

Senior Data Engineer

2023 - 2024
Weir Group
  • Led the successful migration of a data product from SQL Server to Fabric, along with data modeling and CI/CD.
  • Built a Python package to improve and scale data engineering work within Synapse workspaces​.
  • Maintained the back-end database for the global website locations page, along with translations to six languages​.
  • Defined business requirement documents and reviewed statements of work and DataOps cost management for technical deliverables, ensuring alignment with organizational goals.
Technologies: Azure Data Factory (ADF), Azure Synapse Analytics, Python, Fabric, Microsoft Fabric, ETL, Data Engineering, SQL, Databricks, Data Visualization, OneLake, Data Lakes, PySpark, Azure Data Lake, Azure Synapse, Database Management, Microsoft SQL Server

Senior Data Engineer

2022 - 2023
Pandora
  • Migrated the SQL Server Integration Services (SSIS) brand tracker solution to the Synapse workspace.
  • Refactored and maintained a critical pipeline that managed 60 tables and around one TB of data, resulting in more efficient data management and processing.
  • Collaborated with the team to lead the development of a new Python framework based on Delta Lake for the data engineering area, improving efficiency and productivity.
  • Built and supported ETL pipelines in Azure Data Factory (ADF), Synapse, Apache Airflow, and Databricks workflows.
  • Defined and implemented a cluster tagging strategy and Databricks jobs cost management reporting.
Technologies: Databricks, Azure, Python, Apache Airflow, Azure Synapse Analytics, SQL, Spark, SQL Server Integration Services (SSIS), Data Lakes, PySpark, Azure Data Lake, Azure Synapse, Database Management, Microsoft SQL Server

Senior Data Engineer

2021 - 2022
JTI (Japan Tobacco International)
  • Built, supported, and resolved incidents for a global supply chain analytics platform built for 25 plants worldwide.
  • Optimized data pipelines and developed normalization of the current data model for better performance, partitioning for SAP data ingestion and reorganizing triggers.
  • Migrated the on-premise SQL database and SAP master data into an Azure SQL database.
  • Set up an Azure DevOps board for better task management in the Agile/Scrum environment.
  • Transferred to a modern Delta Lake architecture based on Databricks with Python codebase.
Technologies: Azure, SQL, Python, Spark, Databricks, Data Visualization, Data Lakes, PySpark, Azure Data Lake, Azure Synapse, Database Management, Microsoft SQL Server

Lead BI Engineer

2018 - 2021
Mars
  • Designed and created full-stack end-to-end data products in an Azure environment.
  • Extracted data using Azure Logic Apps, transformed it with DataBricks, orchestrated ADF pipelines, and prepared functional data marts using the SQL database.
  • Led a mid-sized team of data engineers, performing data-training sessions, consulting internal business customers with data-related questions, and helping with the development and designing of ETL procedures, BI reports, and dashboards.
  • Hired a team of engineers and organized working processes using Jira, Confluence, and Agile/Scrum principles.
  • Created over 20 basic and functional data marts in an Azure environment.
  • Built over 10 automated analytical tools for various business domains, leveraging Tableau and Microsoft Power BI.
Technologies: Azure, Databricks, SQL, Python, Microsoft Power BI, Tableau, Spark, Data Visualization, Data Lakes, PySpark, Database Management, Microsoft SQL Server

Experience

Azure Data Factory Cookbook

https://www.amazon.com/Azure-Data-Factory-Cookbook-integration/dp/1800565291/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1603711547&sr=8-1
As a co-author of the Azure Data Factory Cookbook, I contributed to creating a comprehensive, hands-on guide for data engineers and architects working with Microsoft's cloud data integration service. The book provides practical solutions, real-world use cases, and step-by-step recipes for building efficient data pipelines using Azure Data Factory (ADF). I also shared insights from my experience in production environments, helping readers understand how to implement features and why and when to use them. This project reflects my commitment to knowledge sharing and my deep expertise in building scalable, cloud-based data solutions.

Azure Data Factory Cookbook – 2nd Edition

https://www.amazon.com/Azure-Data-Factory-Cookbook-integration/dp/1803246596/
As a co-author of the Azure Data Factory Cookbook, 2nd edition, I contributed to delivering a practical, up-to-date guide tailored for modern data engineers and architects working in the Azure ecosystem. This edition goes beyond the fundamentals to cover advanced use cases, including configuring and running data flows with Synapse, integrating ADF with Azure ML Studio, Azure Logic Apps, and Azure Functions.

I authored and technically reviewed chapters focused on building scalable big data pipelines using Databricks and Delta tables, implementing industry-grade best practices, and integrating metadata-driven approaches with Azure Purview. My role involved hands-on development, scenario design, and knowledge sharing from real-world implementations, ensuring the content was actionable and aligned with current enterprise data needs. This project underscores my dedication to empowering data professionals with the tools and insights to build reliable, cloud-native data platforms using Azure technologies.

Jumpstart Snowflake – 2nd Edition

As a co-author of Jumpstart Snowflake, 2nd edition, I contributed to expanding and refining a comprehensive guide that helps data professionals adopt and master Snowflake, one of the most advanced cloud-native data warehouse platforms. With organizations increasingly moving their analytics to cloud providers like AWS, Azure, and GCP, this book equips readers with the knowledge to build modern, scalable analytics solutions using Snowflake.

I focused on real-world deployment best practices, cloud migration strategies from legacy data warehouses, and integration with tools like Matillion, Tableau, and Databricks. In this new edition, I also helped extend coverage to include cutting-edge features such as Snowpark for complex application development, Apache Iceberg for managing large datasets, and Streamlit for building interactive data apps. My contributions reflect deep hands-on experience and a practical approach to helping readers unlock the full potential of Snowflake in enterprise data platforms.

Education

2013 - 2017

Coursework for a PhD in in Mathematical and Instrumental Methods in Economics

Financial University under the Government of the Russian Federation - Moscow, Russia

2008 - 2013

Master's Degree in Mathematical Methods and Risk Analysis

Financial University under the Government of the Russian Federation - Moscow, Russia

Certifications

APRIL 2025 - PRESENT

Databricks Fundamentals

Databricks

SEPTEMBER 2021 - PRESENT

Microsoft Certified: Azure Data Fundamentals

Microsoft

SEPTEMBER 2021 - PRESENT

Microsoft Certified: Azure Fundamentals

Microsoft

JULY 2019 - PRESENT

Tableau Desktop Certified Associate

Tableau

Skills

Libraries/APIs

PySpark, Fabric, Snowpark

Tools

Microsoft Power BI, Tableau, Apache Airflow, Tableau Desktop, Azure Logic Apps, Azure ML Studio, SnowSQL, Synapse

Languages

SQL, Python, Visual Basic for Applications (VBA), Snowflake

Platforms

Azure, Databricks, Microsoft Fabric, Azure Synapse, Apache Kafka, Azure Synapse Analytics, Azure Functions, Amazon Web Services (AWS)

Storage

Data Lakes, Database Management, Microsoft SQL Server, Databases, Azure SQL Databases, Azure Cosmos DB, SQL Server Integration Services (SSIS)

Frameworks

Spark, Delta Live Tables (DLT), Spark Structured Streaming

Paradigms

DevOps, ETL, Continuous Integration (CI), Database Development, Database Design

Other

Data Engineering, Azure Data Factory (ADF), Azure Data Lake, Data Visualization, OneLake, Data Analytics, Excel 365, Data Cloud, Cloud, Storage, Azure Databricks, Tableau Server, Delta Lake, Delta Tables, Streaming, System Modeling, Snowpipe, Snowsight

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring