Wagdy El-Sherbiny
Verified Expert in Engineering
Data and Database Developer
Salmiya, Hawalli Governorate, Kuwait
Toptal member since April 18, 2022
Wagdy is a senior data engineer with over 10 years of experience designing and implementing scalable data solutions. He is highly skilled in SQL, databases, data warehouses, and big data technologies like Airflow, Spark, Hive, and BigQuery. Wagdy has successfully managed projects for major clients such as ADNOC and Emarat. He excels in developing data pipelines, ensuring data integrity, and delivering insights through innovative data solutions, with strong proficiency in Python.
Portfolio
Experience
Availability
Preferred Environment
Data Engineering, Data Warehouse Design, Python, Data Analysis, ETL Tools, Google Cloud Platform (GCP), PySpark, Apache Airflow, Data Modeling, Data Visualization
The most amazing...
...thing I've developed is the self-service payment and operation station used by ADNOC in the UAE, the world's 12th largest oil company by production.
Work Experience
Senior Cloud Data Engineer
Vodafone Group
- Collaborated with data scientists and analysts to understand their data requirements and provide solutions to their needs.
- Designed and implemented data pipelines using GCP services such as Dataflow, Dataproc, and Pub/Sub.
- Developed and maintained data ingestion and transformation processes using tools like Apache Beam and Apache Spark.
- Created and managed data storage solutions using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, and Cloud SQL.
- Automated data processing tasks using scripting languages such as Python and Bash.
Python and PL/SQL Developer
Susquehanna Technology Management, Inc. (STMI) - Main
- Implemented a Python program to replace a legacy system, drastically reducing processing time for calculating trading volumes per symbol and exchange messages. Reduced processing duration from 7 to 10 hours to approximately 20 to 30 minutes.
- Engineered a Python script to extract data from URLs, parse JSON information, and seamlessly store it within an SQL server database for efficient data management.
- Created a shell script designed to retrieve data from the FIDO trading system. Implemented filtering to extract essential information, saving it into a CSV file. Utilized external tables to facilitate the loading of data into Oracle tables.
- Migrated all legacy Python scripts initially written in Python 2 to the latest Python 3.7 version, ensuring compatibility and optimization for current development standards.
- Managed bug tracking and resolution processes utilizing Jira, ensuring seamless production support and maintenance.
- Crafted a robust database logging and debugging utility for streamlined troubleshooting. Also, designed a Python module for easy integration into scripts, bolstering debugging capabilities.
- Revamped and reimplemented outdated systems prone to failure when processing large files with efficient and swift Python programs.
Python and PL/SQL Developer
SIG - Main
- Developed a regression testing program that validates the outcomes of one of the leading trading solutions at SIG. That application was handy for top management to validate the data outcome.
- Built a real-time data pipeline that monitors three trading symbols' order trades and extracts data as of the ACT messages protocol. The script extracts and parses data and then loads it into the Oracle database.
- Participated in PySpark script development, converting legacy reports into new ones that optimized execution time and processed the vast amount of daily data, 500+ million transactions.
- Developed an ETL, where parsing XML files loads data into an Oracle database table.
- Created many reports using SSRS and a modern visualization dashboard using Qlik Sense.
- Converted a legacy Perl program into Python. The scripts aim to process trading orders and calculate the five-minute sliding average for each symbol.
Senior Data Engineer
Diyar United Company
- Collaborated with the project management team to analyze and consolidate request requirements to improve the decision-planning process and meet project deadlines.
- Led medium projects from scope definition to execution and implementation.
- Worked with cross-functional teams to structure problems, identify appropriate data sources, extract data, and develop integrated information delivery solutions.
- Designed, implemented, and maintained various database management systems (DBMSs), such as Oracle, Microsoft SQL Server, Microsoft Azure SQL, and PostgreSQL.
- Analyzed and compiled an itemized list of components needed for data migration or integration with various systems, such as Oracle Enterprise Resource Planning (ERP), SAP, and Comarch Loyalty Management (CLM).
- Participated in designing and implementing a data warehouse, data marts for analysis, data mining projects, and implementing changes to existing BI systems.
- Used the SSIS tool to gather data from various resources, followed the data-wrangling cycle to store and deliver cleaned data for data analysis, and developed a customized pipeline for data ingestion and loading from an external system.
- Built a fast and reliable invoicing system that efficiently processes millions of transactions and integrates with the Oracle ERP or SAP system. Reduced the invoice generation time from six hours to 20 minutes.
- Identified the key reports and dashboards to satisfy the client's requirements and developed different kinds of reports like parameterized reports, ad hoc reports, and drill down and drill through reports using SSRS and Power BI.
- Troubleshooted data issues, analyzed and corrected error and performance logs, created backup routines, and prepared system deployments and upgrades. Trained various business unit teams on effectively using processes, tools, and resources.
Full-stack Developer
Management Development Center (MDC)
- Participated in developing the Rhazes solution that includes all the medical, clinical, and administrative modules needed to manage and operate a hospital in a smooth and user-friendly way.
- Developed SCHOOLIZER software used to efficiently manage all the systems and processes related to running any educational institute.
- Troubleshot and fixed the production issues related to an e-learning system called MUST.
Teaching Assistant
Misr University for Science and Technology
- Managed, planned, and prepared the learning materials and courses related to computer science for different classes and levels.
- Used different educational methods to promote a positive learning environment and organized group work projects and brainstorming sessions.
- Set requirements for the different assessments and kept records of the students' development.
- Kept up to date with changes and developments in the field of computer science.
Experience
ADNOC Smart Wallet
https://smart.adnocdistribution.ae/I was the software engineer who participated in and developed all objects related to the data area, including databases such as triggers, stored procedures, jobs, data warehouse, pipelines, and reporting. I created a set of systems, including the following:
• A card management system (CMS).
• A comprehensive centralized software solution providing our clients with a user-friendly interface that integrates tightly with other back-end and external systems, such as customer management, beneficiary management, token management, transaction management, system settings, BI reports and dashboards, differential pricing, notification management, ERP integration, loyalty system, e-vouchers, coupons, data migration, and card printing system that personalizes smart cards, vehicle depot system, and invoicing.
Emarat Petroleum Company Station Management Service
https://www.emarat.ae/businesses/fleet-payment-solutions• Implementing and customizing a turnkey solution to meet the company's station management and retail services needs.
• Participating in the system requirement gathering and analysis phases.
• Analyzing and compiling components needed for data migration and integration with various systems, such as Oracle ERP, NCR, and NAMOS.
• Developing a data migration workflow to move data into a new system using SSIS.
• Building a differential pricing system and NCR discounts.
• Developing various reports and dashboards using SSRS and Power BI.
Kuwait E-passport
KNPC Petrol Filling Stations
Kuwait Digital Oil Field Project
Kuwait National Guard's Fleet Management
The Ford GoBike System
https://github.com/wagdySamy/DataAnalysisWeRateDogs Analysis
A Real-time Data Pipeline for ACT Messages
PySpark for G1x Reports
• Developed a regression testing program that validates the outcomes of one of the leading trading solutions at SIG. That application was handy for top management to validate the data outcome.
CVIEurex Reports
• Developed an interactive dashboard using Qlik Sense after creating a data model suited for reporting.
Five-minute Sliding Average
The program steams the data from Fido API, then aggregates data by calculating the one-minute average, then calculates a five-minute average and stores the result in the Oracle DB table.
Revamping a Legacy System
Logging and Debugging Utility
Data Pipelines
• Created a shell script designed to retrieve data from the FIDO trading system. - Implemented filtering to extract essential information, saving it into a CSV file. - Utilized external tables to facilitate the loading of data into Oracle tables.
Education
Bachelor's Degree in Computer Science
Misr University for Science and Technology - Giza, Egypt
Certifications
Spark and Python for Big Data with PySpark
Udemy
HCIA-Big Data Course
Huawei Academy
AWS Machine Learning
Udacity
Data Analysis Professional Nanodegree Program
Udacity
Data Engineer with Python
DataCamp
MongoDB Basics
MongoDB University
Data Analysis Advanced Nanodegree Program
Udacity
Career Track Data Analyst with SQL Server
DataCamp
Big Data Introduction Course
Coursera
MCPS: Microsoft Certified Professional
Microsoft
Computer Science Diploma
Mansoura University
Skills
Libraries/APIs
Pandas, NumPy, Entity Framework, Matplotlib, X (formerly Twitter) API, PySpark
Tools
Microsoft Report Builder, Microsoft Power BI, BigQuery, Apache Airflow, Qlik Sense, Flink, Flume, Apache ZooKeeper, SQL Server BI, Oracle ERP, Crystal Reports, Shell, Audacity, Apache Sqoop, Google Cloud Dataproc, GitHub
Languages
SQL, Python, T-SQL (Transact-SQL), C#, C++, C#.NET, Scala, Active Server Pages (ASP), HTML, CSS, Bash, Java
Paradigms
Business Intelligence (BI), ETL, Dimensional Modeling, ETL Implementation & Design, Agile, MapReduce, User Acceptance Testing (UAT)
Platforms
Amazon Web Services (AWS), Google Cloud Platform (GCP), Oracle, Apache Kafka, Linux, Google Cloud SDK, Databricks
Storage
Database, SQL Server Reporting Services (SSRS), PL/SQL, PL/SQL Developer, Oracle PL/SQL, Database Modeling, SQL Server Integration Services (SSIS), PostgreSQL, Data Pipelines, MySQL, NoSQL, MongoDB, Azure SQL, Microsoft SQL Server, HBase, Apache Hive, Database Administration (DBA), SQL Server Analysis Services (SSAS), Google Cloud Storage
Frameworks
Spark, Hadoop, ASP.NET
Other
Data Engineering, Data Warehouse Design, PL/SQL Tuning, Data Modeling, Data Reporting, Data Transformation, Data Analysis, ETL Tools, ELT, Data Wrangling, DAX, Data Warehousing, Dashboards, Trading, Shell Scripting, Data Analytics, Big Data, Azure Data Factory, Software Engineering, Data Structures, Operating Systems, Algorithms, Windows Communication Foundation (WCF), Artificial Intelligence (AI), Neural Networks, Streaming, Machine Learning, Data Visualization, PETEX, Data, IIS, Analysis, Teamwork, Team Management, Data & Backup Management, Production, Deployment, Software Troubleshooting, Troubleshooting, Education, Communication, University Teaching, Applied Mathematics, Statistics, Windows Services, Web Services, Linear Regression, Multidimensional Expressions (MDX), Star Schema, Big Data Architecture, Optimization, Workflow Automation, Workflow, Reporting, Analytics, Data Extraction, Web Scraping, Azure Boards
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring