Raminder Singh Bagga
Verified Expert in Engineering
Data Engineer and Developer
Bengaluru, Karnataka, India
Toptal member since August 6, 2024
Raminder has over a decade of hands-on experience in end-to-end data engineering solutions, including cloud and on-premise services and API integrations. He specializes in AWS and Azure platforms, including EMR, Glue, Athena, and Amazon Managed Streaming for Apache Kafka (MSK). Raminder excels in guiding projects from initial estimations and design to implementing fault-tolerant, complex data pipelines for large-scale systems.
Portfolio
Experience
- Spark - 10 years
- Adobe Experience Platform - 10 years
- Data Engineering - 10 years
- Python - 10 years
- Amazon Web Services (AWS) - 10 years
- SQL - 10 years
- Apache Kafka - 4 years
- Apache Airflow - 4 years
Availability
Preferred Environment
Apache Airflow, AWS Glue, EMR, Snowflake, Spark, Adobe Experience Platform, Python, SQL
The most amazing...
...thing I've created and implemented is a highly scalable data ingestion framework on AWS, handling 2TB+ of data monthly through hundreds of orchestrated jobs.
Work Experience
Senior Data Engineer
BNY Mellon
- Created enterprise applications for data ingestion and processing using Spark with Scala and AWS services like EMR, Glue, Athena, and Amazon MSK. Orchestrated workflows using Apache Airflow, AWS Step Functions, Docker, and Workflow Engine.
- Managed the data engineering team on custom integration projects in EMEA and APAC and handled estimations, architecture, design, development, and deployment strategies.
- Led design and architectural discussions in markets around custom solutions that integrated applications and customers' enterprise applications while creating robust, fault-tolerant, real-time data processing.
- Collaborated closely with data analysts and data scientists, fostering a multidisciplinary approach to data projects.
Module Lead
Mindtree
- Crafted robust data pipelines for customer data platforms for companies' digital transformational goals using on-prem and cloud services for batch and real-time processing.
- Created Spark scripts using Scala on low-level and high-level APIs as required.
- Maintained multi-pipeline architecture design for data ingestion, ETL, and data analysis from different sources.
Experience
Custom Integration with Downstream Applications
• Understanding client requirements and preparing high-level designs.
• Creating Spark scripts using Scala with low- and high-level APIs, per the requirements.
• Maintaining a multi-pipeline architecture design for ingestion, ETL, and analysis of data from different sources.
• Leveraging Sqoop and Informatica for data ingestion for efficiency.
• Working collaboratively with the data science team to maintain data pipelines and automate consumption processes.
Integration with Adobe Experience Platform
Education
Bachelor's Degree in Computer Science
Nagpur University - Nagpur, India
Certifications
Microsoft Certified: Azure Data Engineer Associate
Microsoft
AWS Certified Solutions Architect – Associate
Amazon Web Services
Skills
Libraries/APIs
PySpark
Tools
AWS Glue, Apache Airflow
Languages
Python, SQL, Snowflake, Scala
Frameworks
Spark
Platforms
Adobe Experience Platform, AWS Lambda, Amazon Web Services (AWS), Apache Kafka
Storage
Amazon S3 (AWS S3)
Paradigms
ETL
Other
Data Engineering, Azure Data Factory (ADF), Amazon Kinesis, Amazon Redshift, EMR, Big Data
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring