Verified Expert in Engineering
Satya is a senior data engineer with over 15 years of IT experience designing and developing data warehouses for banking and insurance clients. He specializes in designing and building modern data pipelines and streams using AWS and Azure Data engineering stack. Satya is an expert in delivering modernization of enterprise data solutions using AWS and Azure cloud data technologies.
Azure Data Factory, Azure Databricks, Informatica ETL, Amazon Web Services (AWS), Apache Airflow, Redshift, AWS Glue, Python, PostgreSQL 10.1, Azure Synapse
The most amazing...
...project I've done is designing, developing, and supporting cloud-based and traditional data warehouse applications.
Heimstaden Services AB
- Acted as a senior data engineer with demonstrated analyst skills and worked on ETL architecture solutions.
- Performed requirements assessments and designed suitable data flows or data batches.
- Handled solutions optimization and end-to-end data pipelines with data integrity.
- Designed and developed ETL processes in AWS Glue to migrate campaign and API data with various file types (JSON, ORC, and Parquet) into Amazon RedShift.
- Designed and developed ETL processes to extract Salesforce data and load it into Amazon Redshift.
AWS Data Engineer
- Designed and implemented data pipelines using AWS services such as S3, Glue, and RedShift.
- Developed and maintained data processing and transformation scripts using Python and SQL. Optimized data storage and retrieval using AWS database services such as RDS and DynamoDB.
- Built and maintained data warehouses and data lakes using AWS Redshift and Athena.
- Implemented data security and access controls using AWS IAM and KMS. Monitored and troubleshot data pipelines and systems using AWS CloudWatch and other monitoring tools.
- Collaborated with data scientists and analysts to provide data insights and support their data needs.
- Automated data processing and deployment using AWS Lambda and other serverless technologies.
- Developed and maintained ETL workflows using AWS Step Functions and other workflow tools. Stayed up-to-date with the latest AWS data services and technologies and recommended new solutions to improve data engineering processes.
Azure Data Engineer and Data Warehouse Consultant
- Designed and developed data ingestion pipelines using ADF and processing layer using Databricks and notebooks with PySpark. Led the planning, development, testing, implementation, documentation, and support of data pipelines.
- Implemented various aspects of the project, including pause and resume Azure SQL data warehouse using ADF, ADF pipelines with business rules use cases as reusable asset Ingestion of CSV, fixed width, and excel files.
- Collaborated with a client and IBM ETL teams, analyzed on-premises Informatica-based ETL solutions, and designed ETL solutions using Azure Data Factory pipelines and Azure Databricks PySpark and Spark SQL.
- Worked with technical and product stakeholders to understand data-oriented project requirements and help implement the solution's Azure infrastructure components as part of the solution to create the first usable iteration of the CPD application.
- Orchestrated and automated the pipelines POCs with Apache Spark using PySpark and Spark SQL for various complex data transformation requirements.
- Used PowerShell scripts for automation of pipelines and Azure Data Factory and Azure Databricks for performance tuning of pipelines.
Senior ETL Consultant and Team Lead
- Developed solutions in a highly demanding environment and provided hands-on guidance to other team members. Headed complex ETL requirements and design and assessed requirements for completeness and accuracy.
- Implemented Informatica-based ETL solution fulfilling stringent performance requirements. Collaborated with product development teams and senior designers to develop architectural requirements to ensure client satisfaction with the product.
- Determined if requirements were actionable for the ETL team and conducted an impact assessment to determine the size of effort based on needs.
- Developed entire Software Development Lifecycle (SDLC) project plans to implement ETL solutions and identify resource requirements.
- Assisted and verified solutions design and production of all design phase deliverables. Managed the build phase and quality assurance code to fulfill requirements and adhere to ETL architecture. Resolved difficult design and development issues.
- Provided the team with the vision of the project's objectives, ensured discussions and decisions led toward closure, and maintained healthy group dynamics.
- Familiarized the team with customer needs, specifications, design targets, development process, design standards, techniques, and tools to support task performance.
- Performed an active, leading role in shaping and enhancing overall ETL Informatica architecture. Identified, recommended, and implemented ETL process and architecture improvements.
Senior ETL Developer
- Developed mapping for type two dimension for updating already existing rows and inserting new rows in targets. Worked on actuating for formatting reports related to different processes.
- Created and developed actuate reports like drill-up and drill-down, series, and parallel. Analyzed the number of reports generated, failed, waiting, and scheduled.
- Built dashboards for generated, failed, waiting, and scheduled reports concerning quarter-hour, hour, day, month, and year.
Senior ETL Developer
Magna Infotech Ltd
- Managed ETL development and data warehousing application support activities.
- Acquired hands-on experience in dimensional modeling up to ETL design.
- Developed mapping for type two dimension for updating existing rows and inserting new ones in targets.
Tool Client Rate (TCR) Desk
TCR Desk application migration solution leverages best practices of Azure's Well-architected framework in compliance with the client's Azure Service Governance rules to make the solution secure, resilient, highly available, and scalable. These design principles are for implementation in the client's Azure production environment. The same design will be implemented in disaster recovery and lower environments without high availability and disaster recovery.
• Designed and developed data ingestion pipelines using ADF and a processing layer using Databricks and notebooks with PySpark.
• Led the planning, design, development, testing, implementation, documentation, and support of data pipelines.
• Collaborated with ETL teams, both client and IBM.
• Analyzed on-premises Informatica-based ETL solutions and designed ETL solutions using Azure Data Factory pipelines, Azure Databricks, PySpark, and Spark SQL.
Customer Profitability Insights (CPI)
• Developed solutions in a highly demanding environment and provided hands-on guidance to other team members.
• Headed complex ETL requirements and design.
• Implemented Informatica-based ETL solution fulfilling stringent performance requirements.
• Collaborated with product development teams and senior designers to develop architectural requirements to ensure client satisfaction with the product.
• Assessed requirements for completeness and accuracy.
• Determined if requirements are actionable for the ETL team.
• Conducted impact assessment and determined the size of effort based on requirements.
• Developed complete SDLC project plans to implement ETL solutions and identify resource requirements.
• Performed an active, leading role in shaping and enhancing overall ETL Informatica architecture.
Achmea Solvency II
Solvency II enforces that all material risks of an insurer need to be more transparent in such a way that it can calculate what capital needs to be kept as coverage for unforeseen circumstances. Driven by these requirements and legislation, Achmea started the Value Management program.
A vital program result is the realization of an automated reporting facility by an integrated actuarial data warehouse.
• Release-1: Life 400 insurance
• Release-2: Non-life insurance
• Release-3: ALI/AMIS
• Release-4: VITALIS
• Headed in practical knowledge transfer sessions with modelers.
• Led technical design meetings for designing individual layers.
• Analyzed functional design documents and prepared analysis sheets for individual layers.
• Extensively worked on technical design generation set of documents and amended as suitable for the current release.
Data Analyst – Azure Data Factory Expertise
SQL, Python, T-SQL (Transact-SQL), Python 3, Snowflake, XML, C, C++, Pascal, R, Scala
Apache Spark, Spark
Informatica ETL, Autosys, Tableau, Postman, AWS Glue, Amazon Elastic MapReduce (EMR), Apache Airflow, AWS IAM, Amazon CloudWatch, Amazon QuickSight, GitHub, Excel 2016, Microsoft Power BI, Microsoft Excel, Control-M, Google Analytics, Power Query
ETL, Dimensional Modeling, Business Intelligence (BI), OLAP, Kanban, Database Design, DevOps, Data Science
Oracle, Azure, Amazon Web Services (AWS), Databricks, Amazon EC2, AWS Lambda, Google Cloud Platform (GCP)
Netezza, IBM Db2, Database Management Systems (DBMS), Data Pipelines, Relational Databases, Databases, PostgreSQL, SQL Stored Procedures, Data Integration, Database Architecture, Oracle PL/SQL, NoSQL, Database Transactions, MySQL, Azure SQL Databases, Azure SQL, JSON, Data Lakes, PL/SQL, Microsoft SQL Server, Redshift, Amazon Aurora, AWS Data Pipeline Service, PostgreSQL 10.1, Amazon DynamoDB, Database Administration (DBA), Database Migration, Database Structure, OLTP, Amazon S3 (AWS S3), Datadog, Elasticsearch
Unix Shell Scripting, Informatica, Data Engineering, Pipelines, Data Management, Data Architecture, Migration, Query Composition, Data Warehouse Design, Data Warehousing, CSV File Processing, CSV, ETL Tools, Scripting Languages, Orchestration, Technical Project Management, ETL Development, Data Transformation, Normalization, Shell Scripting, Architecture, Data Analysis, Data Analytics, Database Optimization, PL/SQL Tuning, Data Build Tool (dbt), DAX, Transactions, Data, Azure Data Factory, Azure Databricks, Azure Data Lake, Azure Synapse, Azure SQL Data Warehouse (SQL DW), Informatica Cloud, Data Modeling, Query Optimization, Performance Tuning, Analytics, XLSX File Processing, Data Migration, APIs, Data Wrangling, Amazon RDS, Excel Macros, Lambda Functions, Big Data Architecture, AWS Cloud Architecture, CI/CD Pipelines, Webhooks, BI Reporting, CDC, Data-driven Dashboards, Business Services, Identity & Access Management (IAM), Azure Data Explorer, Consulting, Machine Learning, Google Analytics 4, Big Data, Data Visualization, Microsoft Power Automate, Real Estate, Geospatial Data, AWS Certified Cloud Practitioner, Microsoft Azure
Bachelor's Degree in Technology and Electrical Engineering
Jawaharlal Nehru Technological University - Hyderabad, India
AWS Certified Cloud Practitioner
Azure Data Engineer
Microsoft Azure Fundamentals