Verified Expert in Engineering
Satya is a senior data engineer with over 15 years of IT experience designing and developing data warehouses for banking and insurance clients. He specializes in designing and building modern data pipelines and streams using AWS and Azure Data engineering stack. Satya is an expert in delivering modernization of enterprise data solutions using AWS and Azure cloud data technologies.
Apache Airflow, AWS Glue, Azure Synapse, ETL Implementation & Design, Amazon S3 (AWS S3), Databricks, AWS Lambda, Python 3, Data Engineering, Big Data, Python, APIs, REST APIs, SSH
The most amazing...
...project I've done is designing, developing, and supporting cloud-based and traditional data warehouse applications.
Millicom International Cellular SA - Main
- Orchestrated complex data workflows using AWS Glue and Apache Airflow, ensuring the efficient and timely execution of ETL processes.
- Implemented dynamic and scalable data pipelines that seamlessly adapt to fluctuations in data volume, enhancing system reliability and performance.
- Architected Lambda functions to enable real-time data processing, providing instant insights and analytics capabilities.
- Established event-driven architectures, allowing for automatic scaling and resource optimization, resulting in a responsive and cost-effective solution.
- Implemented S3 as a centralized data repository, optimizing storage costs and streamlining data accessibility. Utilized S3 features such as versioning and lifecycle policies to ensure data integrity and efficient data lifecycle management.
- Developed and applied intricate business rules within the data processing pipeline, enriching the analytical layer with meaningful insights.
- Collaborated closely with business stakeholders to understand and implement domain-specific rules, ensuring the processed data aligns precisely with business requirements.
- Conducted thorough performance optimizations, fine-tuning AWS Glue jobs and Airflow DAGs to maximize processing speed and resource efficiency.
- Implemented scalable solutions to accommodate future data growth, providing a foundation for long-term sustainability and adaptability.
Heimstaden Services AB
- Acted as a senior data engineer with demonstrated analyst skills and worked on ETL architecture solutions.
- Performed requirements assessments and designed suitable data flows or data batches.
- Handled solutions optimization and end-to-end data pipelines with data integrity.
- Designed and developed ETL processes in AWS Glue to migrate campaign and API data with various file types (JSON, ORC, and Parquet) into Amazon RedShift.
- Designed and developed ETL processes to extract Salesforce data and load it into Amazon Redshift.
AWS Data Engineer
- Designed and implemented data pipelines using AWS services such as S3, Glue, and RedShift.
- Developed and maintained data processing and transformation scripts using Python and SQL. Optimized data storage and retrieval using AWS database services such as RDS and DynamoDB.
- Built and maintained data warehouses and data lakes using AWS Redshift and Athena.
- Implemented data security and access controls using AWS IAM and KMS. Monitored and troubleshot data pipelines and systems using AWS CloudWatch and other monitoring tools.
- Collaborated with data scientists and analysts to provide data insights and support their data needs.
- Automated data processing and deployment using AWS Lambda and other serverless technologies.
- Developed and maintained ETL workflows using AWS Step Functions and other workflow tools. Stayed up-to-date with the latest AWS data services and technologies and recommended new solutions to improve data engineering processes.
Azure Data Engineer and Data Warehouse Consultant
- Designed and developed data ingestion pipelines using ADF and processing layer using Databricks and notebooks with PySpark. Led the planning, development, testing, implementation, documentation, and support of data pipelines.
- Implemented various aspects of the project, including pause and resume Azure SQL data warehouse using ADF, ADF pipelines with business rules use cases as reusable asset Ingestion of CSV, fixed width, and excel files.
- Collaborated with a client and IBM ETL teams, analyzed on-premises Informatica-based ETL solutions, and designed ETL solutions using Azure Data Factory pipelines and Azure Databricks PySpark and Spark SQL.
- Worked with technical and product stakeholders to understand data-oriented project requirements and help implement the solution's Azure infrastructure components as part of the solution to create the first usable iteration of the CPD application.
- Orchestrated and automated the pipelines POCs with Apache Spark using PySpark and Spark SQL for various complex data transformation requirements.
- Used PowerShell scripts for automation of pipelines and Azure Data Factory and Azure Databricks for performance tuning of pipelines.
Senior ETL Consultant and Team Lead
- Developed solutions in a highly demanding environment and provided hands-on guidance to other team members. Headed complex ETL requirements and design and assessed requirements for completeness and accuracy.
- Implemented Informatica-based ETL solution fulfilling stringent performance requirements. Collaborated with product development teams and senior designers to develop architectural requirements to ensure client satisfaction with the product.
- Determined if requirements were actionable for the ETL team and conducted an impact assessment to determine the size of effort based on needs.
- Developed entire Software Development Lifecycle (SDLC) project plans to implement ETL solutions and identify resource requirements.
- Assisted and verified solutions design and production of all design phase deliverables. Managed the build phase and quality assurance code to fulfill requirements and adhere to ETL architecture. Resolved difficult design and development issues.
- Provided the team with the vision of the project's objectives, ensured discussions and decisions led toward closure, and maintained healthy group dynamics.
- Familiarized the team with customer needs, specifications, design targets, development process, design standards, techniques, and tools to support task performance.
- Performed an active, leading role in shaping and enhancing overall ETL Informatica architecture. Identified, recommended, and implemented ETL process and architecture improvements.
Senior ETL Developer
- Developed mapping for type two dimension for updating already existing rows and inserting new rows in targets. Worked on actuating for formatting reports related to different processes.
- Created and developed actuate reports like drill-up and drill-down, series, and parallel. Analyzed the number of reports generated, failed, waiting, and scheduled.
- Built dashboards for generated, failed, waiting, and scheduled reports concerning quarter-hour, hour, day, month, and year.
Senior ETL Developer
Magna Infotech Ltd
- Managed ETL development and data warehousing application support activities.
- Acquired hands-on experience in dimensional modeling up to ETL design.
- Developed mapping for type two dimension for updating existing rows and inserting new ones in targets.
Tool Client Rate (TCR) Desk
TCR Desk application migration solution leverages best practices of Azure's Well-architected framework in compliance with the client's Azure Service Governance rules to make the solution secure, resilient, highly available, and scalable. These design principles are for implementation in the client's Azure production environment. The same design will be implemented in disaster recovery and lower environments without high availability and disaster recovery.
• Designed and developed data ingestion pipelines using ADF and a processing layer using Databricks and notebooks with PySpark.
• Led the planning, design, development, testing, implementation, documentation, and support of data pipelines.
• Collaborated with ETL teams, both client and IBM.
• Analyzed on-premises Informatica-based ETL solutions and designed ETL solutions using Azure Data Factory pipelines, Azure Databricks, PySpark, and Spark SQL.
Customer Profitability Insights (CPI)
• Developed solutions in a highly demanding environment and provided hands-on guidance to other team members.
• Headed complex ETL requirements and design.
• Implemented Informatica-based ETL solution fulfilling stringent performance requirements.
• Collaborated with product development teams and senior designers to develop architectural requirements to ensure client satisfaction with the product.
• Assessed requirements for completeness and accuracy.
• Determined if requirements are actionable for the ETL team.
• Conducted impact assessment and determined the size of effort based on requirements.
• Developed complete SDLC project plans to implement ETL solutions and identify resource requirements.
• Performed an active, leading role in shaping and enhancing overall ETL Informatica architecture.
Achmea Solvency II
Solvency II enforces that all material risks of an insurer need to be more transparent in such a way that it can calculate what capital needs to be kept as coverage for unforeseen circumstances. Driven by these requirements and legislation, Achmea started the Value Management program.
A vital program result is the realization of an automated reporting facility by an integrated actuarial data warehouse.
• Release-1: Life 400 insurance
• Release-2: Non-life insurance
• Release-3: ALI/AMIS
• Release-4: VITALIS
• Headed in practical knowledge transfer sessions with modelers.
• Led technical design meetings for designing individual layers.
• Analyzed functional design documents and prepared analysis sheets for individual layers.
• Extensively worked on technical design generation set of documents and amended as suitable for the current release.
Data Analyst – Azure Data Factory Expertise
SQL, Python, T-SQL (Transact-SQL), Python 3, Snowflake, XML, C, C++, Pascal, R, Scala
Apache Spark, Spark
PySpark, REST APIs, Pandas
Informatica ETL, Autosys, AWS Glue, Tableau, Spark SQL, Amazon Athena, Postman, Amazon Elastic MapReduce (EMR), Apache Airflow, AWS IAM, Amazon CloudWatch, Amazon QuickSight, GitHub, Excel 2016, Microsoft Excel, Jira, Control-M, Google Analytics, Power Query, Microsoft Power BI, Amazon SageMaker
ETL, Dimensional Modeling, Business Intelligence (BI), OLAP, ETL Implementation & Design, Kanban, Database Design, DevOps, Data Science
Oracle, Azure, Databricks, Amazon Web Services (AWS), Azure Synapse, Azure SQL Data Warehouse, Amazon EC2, Apache Kafka, Linux, Dedicated SQL Pool (formerly SQL DW), AWS Lambda, Google Cloud Platform (GCP), Microsoft Power Automate
Netezza, IBM Db2, Database Management Systems (DBMS), Data Pipelines, Relational Databases, Databases, PostgreSQL, SQL Stored Procedures, Data Integration, Database Architecture, Oracle PL/SQL, NoSQL, Database Transactions, MySQL, Database Lifecycle Management (DLM), Azure SQL Databases, Azure SQL, JSON, Data Lakes, PL/SQL, Microsoft SQL Server, Redshift, Amazon Aurora, AWS Data Pipeline Service, PostgreSQL 10, Amazon DynamoDB, Database Administration (DBA), Database Migration, Database Structure, OLTP, Apache Hive, SQL Server Integration Services (SSIS), Amazon S3 (AWS S3), Datadog, Elasticsearch
Azure Databricks, Unix Shell Scripting, Informatica, Data Engineering, Pipelines, Data Management, Data Modeling, Data Architecture, Migration, Query Composition, Data Warehouse Design, Data Warehousing, CSV File Processing, CSV, ETL Tools, Scripting Languages, Orchestration, Technical Project Management, ETL Development, Data Transformation, Normalization, Shell Scripting, Architecture, Data Analysis, Data Analytics, Database Optimization, PL/SQL Tuning, Data Build Tool (dbt), DAX, Transactions, Data, ETL Pipelines, Cloud, Data Feeds, Data Extraction, Leadership, Delta Lake, Azure Data Factory, Azure Data Lake, Informatica Cloud, Query Optimization, Performance Tuning, Analytics, XLSX File Processing, Data Migration, APIs, Data Wrangling, Big Data, Amazon RDS, Excel Macros, Lambda Functions, Big Data Architecture, AWS Cloud Architecture, CI/CD Pipelines, Webhooks, BI Reporting, CDC, Data-driven Dashboards, Business Services, Identity & Access Management (IAM), Entity Relationships, Message Queues, SSH, EMR Studio, Azure Data Explorer, Consulting, Machine Learning, Google Analytics 4, Data Visualization, Real Estate, Geospatial Data, AWS Certified Cloud Practitioner, Microsoft Azure, eCommerce, ETL Testing
Bachelor's Degree in Technology and Electrical Engineering
Jawaharlal Nehru Technological University - Hyderabad, India
AWS Certified Cloud Practitioner
Azure Data Engineer
Microsoft Azure Fundamentals
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.Start hiring