
Dmitry Foshin
Verified Expert in Engineering
Data Engineer and Developer
Porto, Portugal
Toptal member since April 15, 2025
Dmitry is a seasoned senior lead data engineer with 12 years of experience in IT and seven years of hands-on expertise in Azure, Databricks, SQL, and Python. Known for leading a globally distributed team of data and BI engineers, he brings a strategic mindset and technical depth to deliver scalable, high-impact data solutions. Clients value Dmitry's ability to turn complex data challenges into clear, actionable results.
Portfolio
Experience
- SQL - 12 years
- Python - 7 years
- Databricks - 7 years
- Data Engineering - 7 years
- Microsoft Power BI - 7 years
- Azure - 7 years
- Spark - 5 years
- Microsoft Fabric - 2 years
Availability
Preferred Environment
Azure, Python, Databricks, Data Engineering, SQL, DevOps, Azure Data Factory (ADF), Apache Airflow, Spark, PySpark
The most amazing...
...projects I've authored include Azure Data Factory Cookbook (two editions), Jumpstart Snowflake (2nd edition), and Data Engineering with Azure Databricks.
Work Experience
Lead Data Engineer
Mars
- Designed and architected scalable data solutions to support new products and deployment in 30 markets, ensuring feasibility and technical alignment using Azure's tech stack, Databricks, Spark, and Synapse.
- Built and implemented end-to-end CI/CD pipelines in Azure pipelines using YAML, artifacts, variable groups, and Azure DevOps to automate deployment processes.
- Conducted code review sessions with four vendor development teams to uphold code quality, compliance, and adherence to best practices.
- Provided support and learning sessions to junior data engineers.
- Developed and maintained ETL pipelines using Python object-oriented programming, Databricks, ADF, data lakes, Azure SQL, Azure Event Hubs, Azure Functions, Azure Service Bus, Synapse, Azure DevOps, SFTP services, and APIs.
- Bridged the gap between development and production teams, resolving technical challenges to streamline the deployment of data products into production environments.
Senior Data Engineer
Weir Group
- Led the successful migration of a data product from SQL Server to Fabric, along with data modeling and CI/CD.
- Built a Python package to improve and scale data engineering work within Synapse workspaces.
- Maintained the back-end database for the global website locations page, along with translations to six languages.
- Defined business requirement documents and reviewed statements of work and DataOps cost management for technical deliverables, ensuring alignment with organizational goals.
Senior Data Engineer
Pandora
- Migrated the SQL Server Integration Services (SSIS) brand tracker solution to the Synapse workspace.
- Refactored and maintained a critical pipeline that managed 60 tables and around one TB of data, resulting in more efficient data management and processing.
- Collaborated with the team to lead the development of a new Python framework based on Delta Lake for the data engineering area, improving efficiency and productivity.
- Built and supported ETL pipelines in Azure Data Factory (ADF), Synapse, Apache Airflow, and Databricks workflows.
- Defined and implemented a cluster tagging strategy and Databricks jobs cost management reporting.
Senior Data Engineer
JTI (Japan Tobacco International)
- Built, supported, and resolved incidents for a global supply chain analytics platform built for 25 plants worldwide.
- Optimized data pipelines and developed normalization of the current data model for better performance, partitioning for SAP data ingestion and reorganizing triggers.
- Migrated the on-premise SQL database and SAP master data into an Azure SQL database.
- Set up an Azure DevOps board for better task management in the Agile/Scrum environment.
- Transferred to a modern Delta Lake architecture based on Databricks with Python codebase.
Lead BI Engineer
Mars
- Designed and created full-stack end-to-end data products in an Azure environment.
- Extracted data using Azure Logic Apps, transformed it with DataBricks, orchestrated ADF pipelines, and prepared functional data marts using the SQL database.
- Led a mid-sized team of data engineers, performing data-training sessions, consulting internal business customers with data-related questions, and helping with the development and designing of ETL procedures, BI reports, and dashboards.
- Hired a team of engineers and organized working processes using Jira, Confluence, and Agile/Scrum principles.
- Created over 20 basic and functional data marts in an Azure environment.
- Built over 10 automated analytical tools for various business domains, leveraging Tableau and Microsoft Power BI.
Experience
Azure Data Factory Cookbook
https://www.amazon.com/Azure-Data-Factory-Cookbook-integration/dp/1800565291/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1603711547&sr=8-1Azure Data Factory Cookbook – 2nd Edition
https://www.amazon.com/Azure-Data-Factory-Cookbook-integration/dp/1803246596/I authored and technically reviewed chapters focused on building scalable big data pipelines using Databricks and Delta tables, implementing industry-grade best practices, and integrating metadata-driven approaches with Azure Purview. My role involved hands-on development, scenario design, and knowledge sharing from real-world implementations, ensuring the content was actionable and aligned with current enterprise data needs. This project underscores my dedication to empowering data professionals with the tools and insights to build reliable, cloud-native data platforms using Azure technologies.
Jumpstart Snowflake – 2nd Edition
I focused on real-world deployment best practices, cloud migration strategies from legacy data warehouses, and integration with tools like Matillion, Tableau, and Databricks. In this new edition, I also helped extend coverage to include cutting-edge features such as Snowpark for complex application development, Apache Iceberg for managing large datasets, and Streamlit for building interactive data apps. My contributions reflect deep hands-on experience and a practical approach to helping readers unlock the full potential of Snowflake in enterprise data platforms.
Education
Coursework for a PhD in in Mathematical and Instrumental Methods in Economics
Financial University under the Government of the Russian Federation - Moscow, Russia
Master's Degree in Mathematical Methods and Risk Analysis
Financial University under the Government of the Russian Federation - Moscow, Russia
Certifications
Databricks Fundamentals
Databricks
Microsoft Certified: Azure Data Fundamentals
Microsoft
Microsoft Certified: Azure Fundamentals
Microsoft
Tableau Desktop Certified Associate
Tableau
Skills
Libraries/APIs
PySpark, Fabric, Snowpark
Tools
Microsoft Power BI, Tableau, Apache Airflow, Tableau Desktop, Azure Logic Apps, Azure ML Studio, SnowSQL, Synapse
Languages
SQL, Python, Visual Basic for Applications (VBA), Snowflake
Platforms
Azure, Databricks, Microsoft Fabric, Azure Synapse, Apache Kafka, Azure Synapse Analytics, Azure Functions, Amazon Web Services (AWS)
Storage
Data Lakes, Database Management, Microsoft SQL Server, Databases, Azure SQL Databases, Azure Cosmos DB, SQL Server Integration Services (SSIS)
Frameworks
Spark, Delta Live Tables (DLT), Spark Structured Streaming
Paradigms
DevOps, ETL, Continuous Integration (CI), Database Development, Database Design
Other
Data Engineering, Azure Data Factory (ADF), Azure Data Lake, Data Visualization, OneLake, Data Analytics, Excel 365, Data Cloud, Cloud, Storage, Azure Databricks, Tableau Server, Delta Lake, Delta Tables, Streaming, System Modeling, Snowpipe, Snowsight
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring