Wagdy El-Sherbiny, Developer in Salmiya, Hawalli Governorate, Kuwait
Wagdy is available for hire
Hire Wagdy

Wagdy El-Sherbiny

Verified Expert  in Engineering

Bio

Wagdy is a senior data engineer with over 10 years of experience designing and implementing scalable data solutions. He is highly skilled in SQL, databases, data warehouses, and big data technologies like Airflow, Spark, Hive, and BigQuery. Wagdy has successfully managed projects for major clients such as ADNOC and Emarat. He excels in developing data pipelines, ensuring data integrity, and delivering insights through innovative data solutions, with strong proficiency in Python.

Portfolio

Vodafone Group
Agile, Google Cloud Platform (GCP), Google Cloud SDK, BigQuery, Apache Airflow...
Susquehanna Technology Management, Inc. (STMI) - Main
Python, SQL, PL/SQL, Linux, C#.NET, Oracle, C#, Microsoft SQL Server, Shell...
SIG - Main
SQL, Python, PL/SQL, Oracle, Linux, C#.NET, C#, Data Warehousing...

Experience

Availability

Full-time

Preferred Environment

Data Engineering, Data Warehouse Design, Python, Data Analysis, ETL Tools, Google Cloud Platform (GCP), PySpark, Apache Airflow, Data Modeling, Data Visualization

The most amazing...

...thing I've developed is the self-service payment and operation station used by ADNOC in the UAE, the world's 12th largest oil company by production.

Work Experience

Senior Cloud Data Engineer

2023 - PRESENT
Vodafone Group
  • Collaborated with data scientists and analysts to understand their data requirements and provide solutions to their needs.
  • Designed and implemented data pipelines using GCP services such as Dataflow, Dataproc, and Pub/Sub.
  • Developed and maintained data ingestion and transformation processes using tools like Apache Beam and Apache Spark.
  • Created and managed data storage solutions using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, and Cloud SQL.
  • Automated data processing tasks using scripting languages such as Python and Bash.
Technologies: Agile, Google Cloud Platform (GCP), Google Cloud SDK, BigQuery, Apache Airflow, Big Data, Google Cloud Dataproc, PySpark, Google Cloud Storage, ETL, Data Pipelines, GitHub, Azure Boards, Data Warehousing, Data Modeling, Data Engineering, Python, Dimensional Modeling, Data Transformation, Databricks

Python and PL/SQL Developer

2023 - 2024
Susquehanna Technology Management, Inc. (STMI) - Main
  • Implemented a Python program to replace a legacy system, drastically reducing processing time for calculating trading volumes per symbol and exchange messages. Reduced processing duration from 7 to 10 hours to approximately 20 to 30 minutes.
  • Engineered a Python script to extract data from URLs, parse JSON information, and seamlessly store it within an SQL server database for efficient data management.
  • Created a shell script designed to retrieve data from the FIDO trading system. Implemented filtering to extract essential information, saving it into a CSV file. Utilized external tables to facilitate the loading of data into Oracle tables.
  • Migrated all legacy Python scripts initially written in Python 2 to the latest Python 3.7 version, ensuring compatibility and optimization for current development standards.
  • Managed bug tracking and resolution processes utilizing Jira, ensuring seamless production support and maintenance.
  • Crafted a robust database logging and debugging utility for streamlined troubleshooting. Also, designed a Python module for easy integration into scripts, bolstering debugging capabilities.
  • Revamped and reimplemented outdated systems prone to failure when processing large files with efficient and swift Python programs.
Technologies: Python, SQL, PL/SQL, Linux, C#.NET, Oracle, C#, Microsoft SQL Server, Shell, Bash, Workflow Automation, Data Pipelines, ETL, Data Engineering, Big Data, PySpark, Dashboards, Microsoft Power BI, Business Intelligence (BI), Trading, Shell Scripting, MySQL, BigQuery, Google Cloud Platform (GCP), Dimensional Modeling, Data Analytics, Data Transformation, Entity Framework

Python and PL/SQL Developer

2022 - 2023
SIG - Main
  • Developed a regression testing program that validates the outcomes of one of the leading trading solutions at SIG. That application was handy for top management to validate the data outcome.
  • Built a real-time data pipeline that monitors three trading symbols' order trades and extracts data as of the ACT messages protocol. The script extracts and parses data and then loads it into the Oracle database.
  • Participated in PySpark script development, converting legacy reports into new ones that optimized execution time and processed the vast amount of daily data, 500+ million transactions.
  • Developed an ETL, where parsing XML files loads data into an Oracle database table.
  • Created many reports using SSRS and a modern visualization dashboard using Qlik Sense.
  • Converted a legacy Perl program into Python. The scripts aim to process trading orders and calculate the five-minute sliding average for each symbol.
Technologies: SQL, Python, PL/SQL, Oracle, Linux, C#.NET, C#, Data Warehousing, Data Engineering, Database Modeling, Data Visualization, Data Reporting, Optimization, Data Pipelines, PySpark, Big Data, Qlik Sense, Dashboards, Business Intelligence (BI), ETL, Trading, Shell Scripting, MySQL, Microsoft Power BI, BigQuery, Google Cloud Platform (GCP), Dimensional Modeling, Data Analytics, Data Transformation, Entity Framework

Senior Data Engineer

2014 - 2022
Diyar United Company
  • Collaborated with the project management team to analyze and consolidate request requirements to improve the decision-planning process and meet project deadlines.
  • Led medium projects from scope definition to execution and implementation.
  • Worked with cross-functional teams to structure problems, identify appropriate data sources, extract data, and develop integrated information delivery solutions.
  • Designed, implemented, and maintained various database management systems (DBMSs), such as Oracle, Microsoft SQL Server, Microsoft Azure SQL, and PostgreSQL.
  • Analyzed and compiled an itemized list of components needed for data migration or integration with various systems, such as Oracle Enterprise Resource Planning (ERP), SAP, and Comarch Loyalty Management (CLM).
  • Participated in designing and implementing a data warehouse, data marts for analysis, data mining projects, and implementing changes to existing BI systems.
  • Used the SSIS tool to gather data from various resources, followed the data-wrangling cycle to store and deliver cleaned data for data analysis, and developed a customized pipeline for data ingestion and loading from an external system.
  • Built a fast and reliable invoicing system that efficiently processes millions of transactions and integrates with the Oracle ERP or SAP system. Reduced the invoice generation time from six hours to 20 minutes.
  • Identified the key reports and dashboards to satisfy the client's requirements and developed different kinds of reports like parameterized reports, ad hoc reports, and drill down and drill through reports using SSRS and Power BI.
  • Troubleshooted data issues, analyzed and corrected error and performance logs, created backup routines, and prepared system deployments and upgrades. Trained various business unit teams on effectively using processes, tools, and resources.
Technologies: Azure SQL, C#, C#.NET, Database, Data Engineering, Data Analysis, Agile, SQL Server Reporting Services (SSRS), SQL Server Integration Services (SSIS), Microsoft Power BI, Qlik Sense, Oracle, PL/SQL, T-SQL (Transact-SQL), Microsoft SQL Server, PostgreSQL, Python, Data Pipelines, Dashboards, Business Intelligence (BI), ETL, Shell Scripting, MySQL, Dimensional Modeling, Data Analytics, Data Transformation, Entity Framework

Full-stack Developer

2012 - 2014
Management Development Center (MDC)
  • Participated in developing the Rhazes solution that includes all the medical, clinical, and administrative modules needed to manage and operate a hospital in a smooth and user-friendly way.
  • Developed SCHOOLIZER software used to efficiently manage all the systems and processes related to running any educational institute.
  • Troubleshot and fixed the production issues related to an e-learning system called MUST.
Technologies: Active Server Pages (ASP), ASP.NET, Windows Services, Web Services, Crystal Reports, SQL, Microsoft SQL Server, HTML, CSS, C#, Audacity, Business Intelligence (BI), Entity Framework

Teaching Assistant

2012 - 2014
Misr University for Science and Technology
  • Managed, planned, and prepared the learning materials and courses related to computer science for different classes and levels.
  • Used different educational methods to promote a positive learning environment and organized group work projects and brainstorming sessions.
  • Set requirements for the different assessments and kept records of the students' development.
  • Kept up to date with changes and developments in the field of computer science.
Technologies: C#, C++, Data Structures, Operating Systems, Education, Communication, University Teaching, Algorithms, Applied Mathematics, Statistics, Database, Java, Entity Framework

ADNOC Smart Wallet

https://smart.adnocdistribution.ae/
ADNOC Distribution is the UAE's leading operator of retail fuel service stations. ADNOC Wallet makes ADNOC service station payments quick and easy. It's fully secure and keeps track of everyday spending at ADNOC service stations. It includes refueling, shopping, lube change, and car washes. In addition, customers can register multiple cars or beneficiaries to an account.

I was the software engineer who participated in and developed all objects related to the data area, including databases such as triggers, stored procedures, jobs, data warehouse, pipelines, and reporting. I created a set of systems, including the following:

• A card management system (CMS).
• A comprehensive centralized software solution providing our clients with a user-friendly interface that integrates tightly with other back-end and external systems, such as customer management, beneficiary management, token management, transaction management, system settings, BI reports and dashboards, differential pricing, notification management, ERP integration, loyalty system, e-vouchers, coupons, data migration, and card printing system that personalizes smart cards, vehicle depot system, and invoicing.

Emarat Petroleum Company Station Management Service

https://www.emarat.ae/businesses/fleet-payment-solutions
Emarat is a multi-channel power and energy company with a network of service stations and fuel depots across Dubai and the Northern Emirates. In this project, I worked on the following:

• Implementing and customizing a turnkey solution to meet the company's station management and retail services needs.
• Participating in the system requirement gathering and analysis phases.
• Analyzing and compiling components needed for data migration and integration with various systems, such as Oracle ERP, NCR, and NAMOS.
• Developing a data migration workflow to move data into a new system using SSIS.
• Building a differential pricing system and NCR discounts.
• Developing various reports and dashboards using SSRS and Power BI.

Kuwait E-passport

I collaborated with a team to design and implement a set of reports and dashboards for executives and operation users of Kuwait’s Ministry of the Interior e-passport system. I performed requirements analysis, made system designs, oversaw implementation, and drafted documentation. Also, I created numerous processes using Microsoft BI tools to develop and implement a data warehouse and designed an SSIS package that extracts citizen data from an SQL Server database using lookup transformation and the Foreach Loop container. Finally, I built dashboards to demonstrate the overall performance of the passport printing center.

KNPC Petrol Filling Stations

The Kuwait National Petroleum Company (KNPC) is Kuwait's national oil refining company. As a software engineer, I participated in building a turnkey solution for retail service stations from scratch to meet KNPC's business needs and operations. I participated in all of the systems development lifecycle (SDLC) to build a centralized solution that manages customers' profiles, consumption, transactions, invoicing, loyalty points, e-vouchers, pre-printed vouchers, and data integrations with Oracle ERP and NCR systems, in addition to building modules related to managing users permissions and system setting and data backup routines.

Kuwait Digital Oil Field Project

I worked on the first Digital Oil Field (DOF) project in Kuwait, using Petroleum Experts (PETEX) technology to enhance the oil well production and operations by providing frameworks and dashboards for decision-makers. I served as the first line of support for the PETEX solution, assisting petroleum engineers during daily operations, overseeing troubleshooting data issues, and helping users interpret and understand report data. I used the following software solutions: Integrated Visualisation Management (IVM), Integrated Production Modeling (IPM), IFM, PROSPER, and Model Catalogue.

Kuwait National Guard's Fleet Management

This web application manages all the Kuwait National Guard's vehicles fleet operations. As a software engineer, I oversaw gathering requirements, analyzing the client's needs, and providing detailed design documents to the team. I worked closely with the development team to design and implement the database's objects and the Windows Communication Foundation (WCF) service to provide the front-end developers with the required info for a web app. Also, I managed the production deployment activities and troubleshooting issues and conducted training and user acceptance testing (UAT) sessions for end-users.

The Ford GoBike System

https://github.com/wagdySamy/DataAnalysis
The Ford GoBike System dataset contains information about individual rides made in a bike-sharing system covering the greater San Francisco Bay area. There are approximately 183,400 records with attributes in the dataset that describe rides, including start station, end station, and duration.

WeRateDogs Analysis

I illustrated the steps and procedures used for wrangling a real-world dataset known as WeRateDogs. WeRateDogs is a humorous Twitter account that rates dogs with commentary. I collected data from different resources, such as ZIP and CSV files, and queried the Twitter API to fetch data.

A Real-time Data Pipeline for ACT Messages

Built a real-time data pipeline that monitors three trading symbols' order trades and extracts data as of the ACT messages protocol. The script extracts and parses data and then loads it into the Oracle database.

PySpark for G1x Reports

• Participated in PySpark script development, converting legacy reports into new ones that optimized execution time and processed the vast amount of daily data, 500+ million transactions.
• Developed a regression testing program that validates the outcomes of one of the leading trading solutions at SIG. That application was handy for top management to validate the data outcome.

CVIEurex Reports

• Developed an ETL, where parsing XML files loads data into an Oracle database table. The pipeline automatically parses any newly added XML tags into the DB.
• Developed an interactive dashboard using Qlik Sense after creating a data model suited for reporting.

Five-minute Sliding Average

Converted a legacy Perl program into Python. The scripts aim to process trading orders and calculate the five-minute sliding average for each symbol.
The program steams the data from Fido API, then aggregates data by calculating the one-minute average, then calculates a five-minute average and stores the result in the Oracle DB table.

Revamping a Legacy System

Implemented a Python program to replace a legacy system, drastically reducing processing time for calculating trading volumes per symbol and exchange messages. Reduced processing duration from 7 to 10 hours to approximately 20 to 30 minutes.

Logging and Debugging Utility

Designed and developed a robust database logging and debugging utility for streamlined troubleshooting. Also, created a Python module for easy integration into scripts, bolstering debugging capabilities.

Data Pipelines

• Engineered a Python script to extract data from URLs, parse JSON information, and seamlessly store it within an SQL server database for efficient data management.
• Created a shell script designed to retrieve data from the FIDO trading system. - Implemented filtering to extract essential information, saving it into a CSV file. - Utilized external tables to facilitate the loading of data into Oracle tables.
2008 - 2012

Bachelor's Degree in Computer Science

Misr University for Science and Technology - Giza, Egypt

MARCH 2021 - PRESENT

Spark and Python for Big Data with PySpark

Udemy

NOVEMBER 2020 - PRESENT

HCIA-Big Data Course

Huawei Academy

JULY 2020 - PRESENT

AWS Machine Learning

Udacity

MAY 2020 - PRESENT

Data Analysis Professional Nanodegree Program

Udacity

MAY 2020 - PRESENT

Data Engineer with Python

DataCamp

FEBRUARY 2020 - PRESENT

MongoDB Basics

MongoDB University

JANUARY 2020 - PRESENT

Data Analysis Advanced Nanodegree Program

Udacity

AUGUST 2019 - PRESENT

Career Track Data Analyst with SQL Server

DataCamp

MARCH 2019 - PRESENT

Big Data Introduction Course

Coursera

JUNE 2017 - JUNE 2020

MCPS: Microsoft Certified Professional

Microsoft

SEPTEMBER 2014 - PRESENT

Computer Science Diploma

Mansoura University

Libraries/APIs

Pandas, NumPy, Entity Framework, Matplotlib, X (formerly Twitter) API, PySpark

Tools

Microsoft Report Builder, Microsoft Power BI, BigQuery, Apache Airflow, Qlik Sense, Flink, Flume, Apache ZooKeeper, SQL Server BI, Oracle ERP, Crystal Reports, Shell, Audacity, Apache Sqoop, Google Cloud Dataproc, GitHub

Languages

SQL, Python, T-SQL (Transact-SQL), C#, C++, C#.NET, Scala, Active Server Pages (ASP), HTML, CSS, Bash, Java

Paradigms

Business Intelligence (BI), ETL, Dimensional Modeling, ETL Implementation & Design, Agile, MapReduce, User Acceptance Testing (UAT)

Platforms

Amazon Web Services (AWS), Google Cloud Platform (GCP), Oracle, Apache Kafka, Linux, Google Cloud SDK, Databricks

Storage

Database, SQL Server Reporting Services (SSRS), PL/SQL, PL/SQL Developer, Oracle PL/SQL, Database Modeling, SQL Server Integration Services (SSIS), PostgreSQL, Data Pipelines, MySQL, NoSQL, MongoDB, Azure SQL, Microsoft SQL Server, HBase, Apache Hive, Database Administration (DBA), SQL Server Analysis Services (SSAS), Google Cloud Storage

Frameworks

Spark, Hadoop, ASP.NET

Other

Data Engineering, Data Warehouse Design, PL/SQL Tuning, Data Modeling, Data Reporting, Data Transformation, Data Analysis, ETL Tools, ELT, Data Wrangling, DAX, Data Warehousing, Dashboards, Trading, Shell Scripting, Data Analytics, Big Data, Azure Data Factory, Software Engineering, Data Structures, Operating Systems, Algorithms, Windows Communication Foundation (WCF), Artificial Intelligence (AI), Neural Networks, Streaming, Machine Learning, Data Visualization, PETEX, Data, IIS, Analysis, Teamwork, Team Management, Data & Backup Management, Production, Deployment, Software Troubleshooting, Troubleshooting, Education, Communication, University Teaching, Applied Mathematics, Statistics, Windows Services, Web Services, Linear Regression, Multidimensional Expressions (MDX), Star Schema, Big Data Architecture, Optimization, Workflow Automation, Workflow, Reporting, Analytics, Data Extraction, Web Scraping, Azure Boards

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring