Verified Expert in Engineering
Jagdish is a data engineer who has worked in different domains, including retail, insurance, and manufacturing. He has extensive experience implementing ETL pipelines and creating data models using data vault and Kimball principles. Jagdish has worked on AWS and Azure cloud and implemented a cloud data warehouse using AWS Redshift and Azure Synapse Analytics.
Python, Microsoft Power BI, Data Modeling, Azure Data Factory, SQL, MSBI, Terraform, Databricks, ETL Tools, Data Warehouse Design
The most amazing...
...thing I've developed is a data warehouse and data mart application using Kimball and Data Vault 2.0 principles using on-prem and cloud environments.
medi GmbH & Co. KG
- Migrated and developed data analytics and data warehouse using on-prem and cloud technologies and implemented hybrid architecture. Worked on the different use cases, stock controlling, production order, and sales management Cockpit.
- Created job auditing framework for monitoring data pipeline execution time and error message. Due to this process, we were able to fix the issue as soon as possible and saved a lot of time.
- Implemented an Azure analysis cube backup and recovery process using PowerShell script in automation. Due to this process, we can recover an old cube state in a minute.
- Created an HR management data mart for maintaining workflow throughout the organization. Because of this dashboard, the whole management can see sickness percentage, headcounts, and paternity leave, and they were able to make decisions.
Senior Software Engineer
- Created a data model and enterprise data warehouse from scratch using the Kimball approach and handed it over to the team; built the ETL data pipeline using Databricks, Azure Data Factory.
- Created automation using PowerShell and Unix shell script to copy the data files from on-premises data center to Azure storage using AzCopy utility.
- Contributed to the construction of a cloud data warehouse using Matillion and Informatica cloud ETL tool.
- Worked on multiple POCs as part of a migration project from on-premises Teradata to a Snowflake database and Azure Synapse Analytics with the help of Informatica and Azure Data Factory.
- Integrated new Azure Network Services in the existing platform using Terraform and an ARM template.
- Worked with clients to understand business needs and translate those business needs into actionable reports in QlikView and Power BI, saving 17 hours of manual work each week.
- Designed and implemented a real-time data pipeline to process semi-structured and unstructured data by integrating 100 million raw records from eight data sources using event Kafka and PySpark and stored processed data into Teradata.
- Analyzed the time an ETL pipeline took and improved performance by creating an index on the database and changing business logic.
- Worked on a migration project from AWS cloud to Azure using Azure Data Factory, data migration, Databricks, Azure storage, Container, and Event Hub.
- Migrated SSIS ETL code into Informatica PowerCenter using shell script.
- Created a job monitoring process to check all daily job statuses, fixed the process, and delivered the data to the end user.
- Used Informatica Power Center for ETL extraction, transformation, and loading data from heterogeneous source systems into the target database. Extracted data from the web service using Informatica web service transformation-personator.
The Digital Group
- Worked legal and finance domain in a migration project from legacy applications. From Oracle Pro*C and SQL loader to data warehouse using the Informatica ETL tool and Unix shell scripting.
- Converted application Oracle Pro*C code into the pipeline using Informatica ETL and Unix shell scripting. Created automated data quality profile rules on staging layer to cleanse the data and capture in warehouse.
- Created reports using tables, and data delivery was high-speed. I also captured the customer's historical information so the client could see how customers changed their information.
eCommerce and Manufacturing Data Warehouse
• Extracted data from different source systems, including ERP, relational, real-time events, and files, and loaded Datalake, a database staging layer.
• Implemented various business rules based on the customer's requirement and delivered the data to the customer in a reporting layer.
• Created end-to-end data flow until the reporting layer.
• Built Delta warehouse in Databricks.
Tools and databases:
Datalake, staging, ods, and EDW.
Data Migration to Public Cloud
SQL, T-SQL (Transact-SQL), SQL DDL, Python, Snowflake, Python 3
ADF, Windows PowerShell, Flask
Jira, SQL Management Studio, Informatica PowerCenter, Microsoft Excel, Terraform, Apache Airflow, Microsoft Power BI, Matillion ETL for Redshift, Informatica ETL, Control-M, Azure DevOps Services, Azure Logic Apps, Azure Automation, Microsoft Report Builder, Informatica Master Data Management (MDM), Kibana, Microsoft PowerPoint, Azure IoT Hub, Toad, Spark SQL, PyCharm, Microsoft Power Apps
Business Intelligence (BI), Database Design, Azure DevOps, ETL, DevOps
Azure, Microsoft, Databricks, Unix, Oracle, Amazon EC2, Azure Event Hubs, Azure Functions, Visual Studio 2017, Linux, Amazon Web Services (AWS), Visual Studio Code (VS Code)
Azure Cloud Services, Microsoft SQL Server, SQL Server Integration Services (SSIS), SSAS Tabular, Teradata, Data Integration, SQL Server Reporting Services (SSRS), SQL Stored Procedures, Database Architecture, Database Migration, Redshift, Azure SQL Databases, Database Security, SQL Server 2017, MySQL, JSON, Data Lakes, SQL Server Analysis Services (SSAS), Databases, Elasticsearch
Data Warehousing, Data Analytics, ETL Tools, Data Modeling, Azure Data Lake, Azure Data Factory, Data Architecture, Data Vaults, Data Warehouse Design, Azure Analysis Services, Analytics, Informatica, Data Analysis, Large Data Sets, Microsoft Azure Cloud Server, APIs, Azure Synapse, Data Governance, Unix Shell Scripting, ServiceNow, ARM, MSBI, Cloud, Cloud Architecture, Delta Lake, Azure Stream Analytics, Azure SQL Data Warehouse (SQL DW), Azure Data Lake Analytics, Data Engineering, SSRS Reports, HTTP REST, Workbench, Azure Administrator, SQL Server Administration, Cloud Infrastructure, Microsoft 365, erwin Data Modeler, Informatica Data Quality
PySpark, Pandas, REST APIs
Bachelor's Degree in Information Technology
Mumbai University - Mumbai, India
High School Diploma in Science
Amravati University - Amaravati, India
Hands On Essentials | Data Warehouse
Microsoft Certified | Azure Fundamentals