Abhijeet Gupta
Verified Expert in Engineering
Data Engineer and Developer
Bengaluru, Karnataka, India
Toptal member since June 18, 2020
Abhijeet is a seasoned data engineer with more than 10 years of industry experience. He has considerable hands-on experience in Microsoft Azure tech stacks such as Azure Data Factory, Databricks, Azure SQL Data Warehouse, Azure Logic Apps, Key Vault, Delta Lake, and Data Lake. Abhijeet has completed several projects with Fortune 500 companies. He also has valuable knowledge in SQL, ETL, Unix shell scripting, and Agile.
Portfolio
Experience
- SQL - 7 years
- Data Engineering - 6 years
- Data Warehouse Design - 4 years
- Data Warehousing - 4 years
- Spark SQL - 3 years
- Azure - 3 years
- Spark - 3 years
- Unix Shell Scripting - 3 years
Availability
Preferred Environment
SQL, Unix, Azure, Spark, Azure Data Factory (ADF), Azure Data Lake, Azure SQL Data Warehouse, Dedicated SQL Pool (formerly SQL DW), Databricks, PySpark
The most amazing...
...thing I've coded are complex stored procedures where complex business logic is implemented to fulfill the user's requirements.
Work Experience
Lead Data Engineer
Johnson & Johnson
- Authored several complex Azure data factory pipelines to migrate the data from on-prem to Azure Cloud.
- Wrote complex Databricks notebooks to apply business transformation logic to build several KPIs.
- Used Informatica to load historical data from on-prem to the Azure Cloud.
- Wrote views in Azure SQL Data Warehouse to apply data governance rules.
- Completed Performance Tuning on tables and view in Azure SQL Data Warehouse.
- Used PolyBase mechanism to load data from Azure Data Lake Storage to Azure SQL Data Warehouse.
Senior Data Engineer
Maersk
- Created a unified data platform.
- Worked on Plantir Foundry to build business use cases.
- Created generic data bricks notebooks to cleanse or wrangle the data.
- Worked extensively with big data technologies like Spark, Scala, Pyspark, Hive, HDFS, and Sqoop.
- Developed with the following tools: MPP database (Teradata), ETL tool (Informatica) and Informatica Developer tool, Unix, shell scripting, Autosys, and Control-M.
- Contributed analytical skills, programming, problem-solving, troubleshooting, and mentoring to various projects.
Senior Data Engineer
Accenture Services
- Supported a master data management (MDM) application for a client.
- Wrote stored procedures to apply business logic to data.
- Worked on optimizing and tuning the Teradata views and SQL to improve the performance of batch and data response time for users.
- Created Informatica (ETL) mappings and workflows to extract and load source data to the target system.
- Developed Unix shell scripts to automate BTEQ scripts.
- Coordinated with the change management team in code deployments.
- Managed tickets and incidents by using an HP service manager.
- Provided quick production fixes and was proactively involved in fixing production support issues.
- Implemented Borland's StarTeam as a versioning tool.
- Prepared basic reports and supporting documents using Microsoft Excel.
Data Engineer
Capgemini India Pvt Ltd.
- Wrote stored procedures to apply business logic to data.
- Used Teradata-loading utilities to load data from the source to a Teradata database.
- Authored very complex Unix shell scripts to preprocess the staging data.
- Created database objects such as tables, views, and indexes.
- Used a BTEQ utility to access and query databases.
Associate Developer
Cognizant Technology Solution, India
- Improved stored procedures and enhanced their performance.
- Created database objects.
- Used BTEQ in a Unix environment.
- Tuned SQL queries to overcome spool space errors and improve performance.
Software Engineer
Exilant Technologies, India
- Wrote complex procedures to perform aggregations in a semantic layer and to written result sets to the front-end user.
- Created database objects like tables, view, and indexes.
- Contributed to different portions of the development lifecycle such as development, testing, UAT, and production deployment.
- Developed Informatica sessions and workflows.
- Wrote Unix shell scripts.
- Connected an AIX file system to the Teradata database in the network attached system.
- Developed AutoSys jobs (JIL).
- Extensively worked on Teradata utility tools like BTEQ, FastLoad, Multiload, TPUMP, FastExport, and TPT (Teradata Parallel Transporter).
- Used SQL assistant to query Teradata tables; worked with BTEQ in a Unix environment.
- Implemented performance tuning, bug fixing, and error handling; set up and supported, in a team, the data for a UAT environment.
Experience
Lead Data Engieer
• Migrated a considerable amount of data from on-prem to Azure Cloud.
• Authored several data pipelines for data movement.
• Wrote very complex Databricks notebooks to apply business transformation logic.
• Used Azure dedicated SQL pools as a Data warehouse.
• Completed performance tuning of SQL Data Warehouse tables and views.
Maestro (Unified Data Platform)
• Worked on the Azure stack of big data and its components.
• Developed Spark programs in Scala to perform data transformation, creating data frames, and running Spark SQL.
• Took part in converting Teradata queries into Spark transformations using Spark RDD and Scala.
• Created Hive tables, then loaded, and analyzed data using Hive queries.
• Designed and created data models for business using Hive and Spark SQL.
• Constructed and scheduled notebooks in Azure Databricks to perform data cleansing and to do transformations.
• Contributed to the creation and scheduling of data pipelines using Azure Data Factory.
• Created the L1 canonical model for GCSS sources.
SustainTeam at Accenture
Position: Senior Software Engineer
Technologies Used: Teradata, PL/SQL, Unix
Description:
I built a master data management (MDM) system that deals with various source systems and domains. It manages customers, materials, locations, finances, vendors, information, and reference data. There are various stages to manage and publish data to the down streams, respectively. The MDM provides the user interface to maintain the global attributes not found in any source.
Tasks Accomplished:
• Investigated production environment failure.
• Worked on user requests.
• Wrote stored procedures for tasks.
• Investigated QA environment failures.
Education
Bachelor's Degree in Electronics and Instrumentation Engineering
Shri Govindram Seksaria Institute of Technology and Science - Indore, India
Certifications
Azure 900
Microsoft
Skills
Libraries/APIs
Pandas, Azure Blob Storage API, PySpark
Tools
Spark SQL, Oracle GoldenGate, Informatica PowerCenter, Microsoft Excel, Subversion (SVN), Control-M, Autosys, Informatica ETL, BigQuery, Apache Airflow
Languages
Python, SQL, Scala
Storage
PL/SQL, Teradata, Oracle PL/SQL, Databases, JSON, Apache Hive, HDFS, Microsoft SQL Server, Azure Blobs, Master Data Management (MDM)
Frameworks
Spark
Paradigms
Agile Software Development, Dimensional Modeling, ETL
Platforms
Databricks, Windows, Windows XP, Unix, Azure, MacOS, Azure PaaS, Azure SQL Data Warehouse, Dedicated SQL Pool (formerly SQL DW)
Other
Data Warehouse Design, Data Warehousing, Data Engineering, Informatica, Big Data, UNIX Utilities, Unix Shell Scripting, Logistics, Blob Storage, Azure Data Factory (ADF), Migration, Azure Data Lake, Electronics, Engineering, Software Engineering, Data Modeling, Google BigQuery
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring