Vijay Singh
Verified Expert in Engineering
Data Warehousing Developer
Noida, India
Toptal member since October 16, 2026
Across finance and retail, Vijay has spent seven years building scalable data engineering solutions. His toolkit centers on Spark, AWS, and Redshift. While at Amazon, he reduced annual storage and compute costs by 20-30% without impacting SLAs and improved query performance by up to 45%.
Portfolio
Experience
- Data Warehousing - 7 years
- Apache Airflow - 7 years
- Performance Tuning - 7 years
- Apache Spark - 7 years
- AWS Data Pipeline Service - 7 years
- Amazon Web Services (AWS) - 7 years
- Scala - 7 years
- Docker - 7 years
Work Experience
Senior Data Engineer
Dunnhumby
- Developed Spark-based ETL pipelines on EMR for large-scale financial transaction data, loading curated outputs into S3 and Redshift.
- Troubleshot job failures and performance regressions, improving reliability and reducing repeated failures by around 40%.
- Implemented data quality validations, including schema checks, reconciliation rules, and referential integrity, to improve reporting accuracy.
- Integrated pipelines into Git-based CI/CD workflows for traceability, rollback safety, and consistent deployments.
- Automated recurring data-processing and maintenance tasks with Python, reducing manual operations time by 50%.
- Managed secure storage and access patterns using S3 lifecycle policies, IAM roles, KMS encryption, and Redshift grants.
Big Data Engineer
Amazon
- Designed and built end-to-end financial data pipelines from scratch to production, processing 1-2 TB per day across S3, EMR, Redshift, and Athena.
- Architected scalable data models and ETL frameworks for finance and accounting teams, improving query performance by 30-45% and reducing onboarding effort by 40%.
- Optimized Spark/EMR workloads through partitioning, executor tuning, and shuffle optimization, reducing peak processing time from 4 hours to around 2.5 hours.
- Migrated workloads from Redshift to S3 and Athena/Parquet, reducing annual storage and compute costs by 20-30% without impacting SLAs.
- Built automated data quality, reconciliation, monitoring, and self-recovery mechanisms, reducing operational incidents and manual support by 60%.
- Productionized infrastructure using AWS CDK, IAM, KMS, CloudWatch, Lambda, and CI/CD, enabling automated deployments, secure access, and rollback support.
- Delivered QuickSight analytics and reporting pipelines for finance stakeholders, reducing manual reporting effort from 2+ hours to around 10 minutes.
- Built a real-time operational dashboard backed by ClickHouse, enabling low-latency analysis and faster decision-making.
- Led solution design, code reviews, and technical discussions, translating business requirements into scalable data engineering solutions.
Project History
D8Loop: Data Engineering Learning Platform
https://www.d8loop.com/aboutI worked across full-stack development, API integration, cloud deployment, and product architecture, using AI-assisted development tools to accelerate implementation. I also worked on designing interactive learning experiences that translate complex data engineering concepts into hands-on exercises. This project demonstrates my ability to combine data engineering expertise, software development, system design, and product thinking to build a platform from concept to deployment.
Education
Master's Degree in Business Administration (MBA)
Amity University - Noida, India
Postgraduate Degree in Internet of Things
CDAC-Acts - Pune, India
Bachelor's Degree in Instrumentation
A.P.J Technical University - Noida, India
Skills
Tools
Git, Apache Airflow, AWS IAM, AWS Key Management Service (KMS), Amazon Athena, Amazon CloudWatch, Amazon QuickSight, IntelliJ IDEA, Claude
Languages
Python, SQL, Scala, TypeScript
Frameworks
Spark, Apache Spark
Paradigms
ETL, Business Intelligence (BI)
Storage
Amazon S3 (AWS S3), Redshift, AWS Data Pipeline Service, ClickHouse
Platforms
Amazon Web Services (AWS), Docker, Apache Kafka, AWS Lambda, Linux, MacOS
Other
EMR, CI/CD Pipelines, SQL Programming, ELT, Data Warehousing, Performance Tuning, Data Engineering, Software Engineering, Cloud, Software Architecture, Data Processing, Dashboards, Data Visualization, Analytics, API Design, Parquet, Data Modeling, Boto3, Cursor AI, Analysis, Large Language Models (LLMs), OpenAI SDK, Data Science, Real-time Data Pipelines, Full-stack, Websites, Internet of Things (IoT), Database Algorithms, Data Structures, Programming
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring