Jagdish Wagh
Verified Expert in Engineering
Database Developer
Bayreuth, Bavaria, Germany
Toptal member since July 11, 2022
Jagdish is a data engineer who has worked in different domains, including retail, insurance, trading, and manufacturing. He has extensive experience implementing ETL pipelines (Databricks, Azure Data Factory, Informatica PC, Matillion, and Microsoft Fabric) and creating data models/warehouses using Data Vault, Medallion, and Kimball principles. Jagdish has worked on AWS and Azure cloud and implemented a cloud data warehouse using AWS Redshift, Snowflake, Azure Databricks, and Microsoft Fabric.
Portfolio
Experience
- Data Analytics - 12 years
- ETL Tools - 12 years
- Data Warehousing - 12 years
- Azure Data Factory (ADF) - 9 years
- Data Modeling - 9 years
- Python - 8 years
- Databricks - 6 years
- Microsoft Fabric - 3 years
Preferred Environment
Microsoft Power BI, Data Modeling, Terraform, Databricks, ETL Tools, Data Warehouse Design, Microsoft Fabric, Medallion Architecture, PySpark, Azure DevOps
The most amazing...
...thing: I've built cloud data platforms on AWS & Azure using Databricks, Fabric, Medallion, Data Vault, and Kimball models for scalable, hybrid data solutions.
Work Experience
Data Engineer
medi GmbH & Co. KG
- Migrated and developed data analytics and data warehouse using on-prem and cloud technologies and implemented hybrid architecture. Worked on the different use cases, stock controlling, production order, and sales management Cockpit.
- Created job auditing framework for monitoring data pipeline execution time and error message. Due to this process, we were able to fix the issue as soon as possible and saved a lot of time.
- Implemented an Azure analysis cube backup and recovery process using PowerShell script in automation. Due to this process, we can recover an old cube state in a minute.
- Created an HR management data mart for maintaining workflow throughout the organization. Because of this dashboard, the whole management can see sickness percentage, headcounts, and paternity leave, and they were able to make decisions.
Senior Software Engineer
TietoEVRY
- Built a data model and enterprise data warehouse from scratch using the Kimball approach and handed it over to the team; built the ETL data pipeline using Databricks and Azure Data Factory.
- Created automation using PowerShell and Unix shell script to copy the data files from the on-premises data center to Azure storage using the AzCopy utility.
- Contributed to constructing a cloud data warehouse using Matillion and Informatica cloud ETL tools.
- Worked on multiple POCs as part of a migration project from on-premises Teradata to a Snowflake database and Azure Synapse Analytics with the help of Informatica and Azure Data Factory.
- Integrated new Azure Network Services using Terraform and an ARM template in the existing platform.
Software Engineer
TietoEVRY
- Worked with clients to understand business needs and translate those business needs into actionable reports in QlikView and Power BI, saving 17 hours of manual work each week.
- Designed and implemented a real-time data pipeline to process semi-structured and unstructured data by integrating 100 million raw records from eight data sources using event Kafka and PySpark and stored processed data into Teradata.
- Analyzed the time an ETL pipeline took and improved performance by creating an index on the database and changing business logic.
- Worked on a migration project from AWS cloud to Azure using Azure Data Factory, data migration, Databricks, Azure storage, Container, and Event Hub.
Software Engineer
Trinus
- Migrated SSIS ETL code into Informatica PowerCenter using shell script.
- Created a job monitoring process to check all daily job statuses, fixed the process, and delivered the data to the end user.
- Used Informatica Power Center for ETL extraction, transformation, and loading data from heterogeneous source systems into the target database. Extracted data from the web service using Informatica web service transformation-personator.
Software Engineer
The Digital Group
- Worked legal and finance domain in a migration project from legacy applications. From Oracle Pro*C and SQL loader to data warehouse using the Informatica ETL tool and Unix shell scripting.
- Converted application Oracle Pro*C code into the pipeline using Informatica ETL and Unix shell scripting. Created automated data quality profile rules on staging layer to cleanse the data and capture in warehouse.
- Created reports using tables, and data delivery was high-speed. I also captured the customer's historical information so the client could see how customers changed their information.
Experience
Data Migration to Public Cloud
eCommerce and Manufacturing Data Warehouse
Outcomes:
• Extracted data from different source systems, including ERP, relational, real-time events, and files, and loaded Datalake, a database staging layer.
• Implemented various business rules based on the customer's requirement and delivered the data to the customer in a reporting layer.
• Created end-to-end data flow until the reporting layer.
• Built Delta warehouse in Databricks.
Tools and databases:
Datalake, staging, ods, and EDW.
GCM Trade Data Analysis - Data Engineer Warehouse Azure Databricks
• Developed a scalable fintech trading data platform to integrate and centralize data from trading sources such as MSCI, Bloomberg, and other third-party providers.
• Utilized Azure Blob Storage as the data lake and built PySpark-based ETL pipelines in Azure Databricks for data cleansing, transformation, and enrichment.
• Created multiple Unity Catalogs to organize datasets by domain and implemented row-level and catalog-level security for secure access.
• Established a Gold Layer as the final curated data model for business reporting.
• Integrated Sigma Computing for self-service BI and shared interactive workbooks with end users.
The entire infrastructure was provisioned using Terraform, with CI/CD pipelines built in Azure DevOps to automate deployments and promote code changes efficiently. The platform enabled high-performance analytics, secure data governance, and faster decision-making for trading operations.
Microsoft Fabrics Migration from Azure Stack
I rebuilt semantic models with Direct Lake, implemented RLS/OLS, and set up deployment pipelines via Azure DevOps YAML across DEV/TEST/PROD environments with environment variables and approvals. Utilizing OneLake Shortcuts, I facilitated a phased cutover, migrated workloads to Fabric Warehouse, and introduced a reusable data-quality and monitoring framework incorporating rules, watermarks, lineage, and alerts.
I authored migration runbooks and architectural documentation, completed security reviews (RBAC/PIM, sensitivity labels), and standardized governance. Additionally, I optimized performance using Delta best practices and F-SKU capacity planning, achieving faster refreshes, lower TCO, and increased data trust.
Education
Bachelor's Degree in Information Technology
Mumbai University - Mumbai, India
High School Diploma in Science
Amravati University - Amaravati, India
Certifications
Microsoft Certified: Fabric Analytics Engineer Associate
Microsoft
Hands On Essentials | Data Warehouse
Snowflake
Microsoft Certified | Azure Fundamentals
Microsoft
Skills
Libraries/APIs
PySpark, Pandas, REST APIs, Sigma.js
Tools
Microsoft Power BI, Jira, SQL Management Studio, Informatica PowerCenter, Microsoft Excel, GitHub, Terraform, Apache Airflow, Matillion ETL for Redshift, Informatica ETL, Control-M, Azure DevOps Services, Azure Logic Apps, Azure Automation, Microsoft Report Builder, Informatica Master Data Management (MDM), Kibana, Microsoft PowerPoint, Azure IoT Hub, Toad, Spark SQL, PyCharm, Microsoft Power Apps
Languages
SQL, Transact-SQL (T-SQL), SQL DDL, Python, Snowflake, Python 3
Frameworks
ADF, Windows PowerShell, Flask
Paradigms
Business Intelligence (BI), ETL, Database Design, Azure DevOps, DevOps
Platforms
Azure, Microsoft, Microsoft Fabric, Databricks, Unix, Azure Synapse, Oracle, Amazon EC2, Azure Event Hubs, Azure Functions, Azure SQL Data Warehouse, Visual Studio 2017, Linux, Dedicated SQL Pool (formerly SQL DW), Amazon Web Services (AWS), Visual Studio Code (VS Code), Fabric Lakehouse
Storage
Azure Cloud Services, Microsoft SQL Server, SQL Server Integration Services (SSIS), SSAS Tabular, Teradata, Data Integration, SQL Server Reporting Services (SSRS), SQL Stored Procedures, Data Pipelines, Database Architecture, Database Migration, Redshift, Azure SQL Databases, Database Security, SQL Server 2017, MySQL, JSON, Data Lakes, SQL Server Analysis Services (SSAS), Databases, Elasticsearch
Other
Data Warehousing, Data Analytics, ETL Tools, Data Modeling, Azure Data Lake, Azure Data Factory (ADF), Data Architecture, Data Vaults, Data Warehouse Design, Azure Analysis Services, Data Engineering, Analytics, Informatica, Data Analysis, Large Data Sets, Microsoft Azure Cloud Server, APIs, Azure Databricks, Medallion Architecture, Azure Blob Storage, Data Governance, Unix Shell Scripting, ServiceNow, ARM, MSBI, Cloud, Cloud Architecture, Delta Lake, Azure Stream Analytics, Azure Data Lake Analytics, SSRS Reports, HTTP REST, Workbench, Azure Administrator, SQL Server Administration, Cloud Infrastructure, Microsoft 365, Data Science, erwin Data Modeler, Informatica Data Quality, Azure Virtual Networks, Azure CLI, Fabric Warehouse, Semantic Model, Warehouses, Fabric Notebooks, Fabric Data Factory
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring