Big Data Architect
2019 - PRESENTVodafone- Designed data pipelines for different data sources types using GCP cloud technologies.
- Developed and implemented ETL jobs using Spark.
- Transformed data sources using Spark.
- Developed and implemented analytical jobs using Spark.
- Developed and built geospatial analysis models with Spark to do parallel geoprocessing.
- Implemented and developed data pipelines to ingest data from on-premise clusters into a cloud data lake.
- Developed dashboards for businesses using Tableau.
- Worked and developed use cases on on-premise clusters.
- Migrated data and jobs from on-premise to cloud clusters.
- Designed and apply modeling for data stores to be used for reporting.
Technologies: Spark, Python, GeoPandas, Airflow, BigQuery, Tableau, Scala, Nifi, BeamSenior Python Developer
2018 - 2019Rio Tinto (via Toptal)- Built a data processing platform to process seismic events.
- Created a RESTful API to store and retrieve seismic data and files.
- Used Kafka as a message bus between all modules.
- Implemented Redis as a cache to store data needed to accessed frequently by the pipeline.
- Built an admin UI by Django to administer configurations and saved objects.
- Integrated the API with different processing pipeline stages to trigger sync-and-async processing of data.
- Migrated and converted a Flask API to a Django RESTful API.
- Worked with docker containerized environments for different pipeline modules.
- Worked with automated deployment pipelines on Kubernetes.
- Developed and ran components on the Microsoft Azure cloud platform.
Technologies: Python, Django RESTful, Flask, Kafka, Redis, MongoDB, Docker, Kubernetes, AzureSenior Big Data Engineer
2017 - 2019Orange Business Services- Developed new business use cases with big data technologies.
- Created analytical and ETL jobs using Spark.
- Built data pipelines to ingest data into different data lakes like Azure DataLake.
- Developed new PoCs for customers to build big data platforms over cloud environments.
- Constructed a real-time monitoring platform to monitor all customers servers hosted on cloud.
- Implemented a new centralized Elasticsearch to collect metrics from all customers servers.
- Designed and built multiple dashboards for systems monitoring use cases using Tableau and Power BI.
- Developed multiple automated scripts for most day-to-day tasks.
- Handled and optimized the performance of the big data platforms.
- Managed the Hadoop clusters with all included services.
- Developed scripts and modules that automate day-to-day tasks.
- Led a squad for automation and self-monitoring activities.
- Upgraded on-premise Hadoop cluster version.
- Managed and added new nodes and disks to on-premise Hadoop.
- Installed and built the security of Hadoop clusters using Kerberos, Knox, and Ranger.
- Worked on different cloud platforms like Azure and AWS.
Technologies: Hadoop, Spark, Nifi, Kafka, Hive, Elasticsearch, Cassandra, MongoDB, Tableau, Power BI, Azure, AWS, DataLake, S3DWH and Campaigns Senior Developer
2014 - 2017Etisalat- Developed analysis and segmentation models to build customer profiles.
- Created offering and campaign applications to create targeted and non-targeted campaigns that reach millions of customers daily.
- Built real-time engines that serve and fulfill millions of customer requests per hour.
- Designed and developed massive complex platforms that interact with many different systems.
- Developed real-time location based advertising platform to send users offers based on their current location.
- Developed multiple data monetization solutions to be used by third-party advertisers.
- Developed and integrated the campaigns applications with many channels to empower business to reach users using there preferred channels.
- Built many web applications to empower business users to easily interact with campaigns platform.
- Designed and put architecture of DWH models for reporting and segmentations.
- Developed ETL and Integration jobs from different sources to DWH.
Technologies: Python, Java, JSF, Spring, Spark, Teradata, Oracle, SQL Server, SSIS and PrimeFacesMIS Specialist
2013 - 2014ADIB- Designed and implemented new database models for reporting purposes.
- Developed extraction jobs and stored procedures.
- Implemented Business Objects universes and developed Business Objects reports.
- Developed custom Crystal Reports.
- Performed data transformation.
Technologies: Business Objects, Crystal Reports, SQL, Sybase, MS SQL ServerDWH Support Analyst
2012 - 2013Etisalat- Deployed and fixed issues for production ETL jobs, data mining, and analytic models.
- Developed new shell scripts for automatic monitoring and alarms for production issues.
Technologies: Teradata, Oracle, SQL, Datastage, Teradata Warehouse Miner, Unix Shell ScriptingSoftware Developer
2011 - 2012ITS- Developed new modules in core banking applications.
- Handled a full migration of the trade finance applications from Sybase to a SQL server.
- Implemented a full-service interface for a trade finance application.
- Developed custom reports using Crystal Reports.
Technologies: MS SQL Server, Oracle, Sybase, Java
Big Data Architect and Developer in Cairo, Cairo Governorate, Egypt
Member since May 13, 2016
Hanee' is a data expert who enjoys working on data analytics and segmentation to better target customers with campaigns. He is an experienced Java developer who has built enterprise applications that interact with millions of customers daily. Hanee' also has experience working with big data, Spark, and Python.
Hanee' is now available for hire
Portfolio
- VodafoneSpark, Python, GeoPandas, Airflow, BigQuery, Tableau, Scala, Nifi, Beam
- Rio Tinto (via Toptal)Python, Django RESTful, Flask, Kafka, Redis, MongoDB, Docker, Kubernetes, Azure
- Orange Business ServicesHadoop, Spark, Nifi, Kafka, Hive, Elasticsearch, Cassandra, MongoDB, Tableau...
Experience
- SQL 9 years
- Big Data 5 years
- Python 5 years
- Apache Spark 4 years
- Tableau 2 years
- Django REST Framework 2 years
- Apache Kafka 2 years
- Machine Learning 2 years
Availability
Part-time
Preferred Environment
OS X, Linux, Git
The most amazing...
...project that I've implemented is a platform that optimized campaigns scripts by analyzing user responses to identify the best way to interact with the users.
Employment
Experience
- Certified CCA Spark and Hadoop Developer (CCA175) (Other amazing things)http://certification.cloudera.com/verify/
I gained a certification from Cloudera.
License No: 100-019-596. - Big Data Development | Mastery Award for Professionals 2016 (Other amazing things)https://www.youracclaim.com/badges/da6c7070-8fde-4799-b04e-f9d8719a49a3/linked_in_profile
I won an award from IBM.
- Big Data Specialist with IBM BigInsights V2.1 Certificate (Other amazing things)
I gained the certificate IBM Big Data Specialist in March 2016 with license # 0717-1458-8215-5644.
- Introduction to Big Data Certificate (Other amazing things)https://www.coursera.org/account/accomplishments/certificate/CPH7HZ6TDEZN
I completed this Coursera course from the University of California, San Diego.
- Hadoop Platform and Application Framework Certificate (Other amazing things)https://www.coursera.org/account/accomplishments/certificate/Y6QNGTJMQFVV
I completed this Coursera course from the University of California, San Diego.
- Introduction to Data Science in Python (Other amazing things)https://www.coursera.org/account/accomplishments/certificate/3ZD9CU8S32TK
I completed and earned a certificate from the Michigan University via Coursera.
- Apache Spark Streaming Tutorial: Identifying Trending Twitter Hashtags (Publication)Social networks are among the biggest sources of data today, and this means they are an extremely valuable asset for marketers, big data specialists, and even individual users like journalists and other professionals. Harnessing the potential of real-time Twitter data is also useful in many time-sensitive business processes. In this article, Toptal Freelance Software Engineer Hanee' Medhat explains how you can build a simple Python application to leverage the power of Apache Spark, and then use it to read and process tweets to identify trending hashtags.
Skills
Languages
SQL, Java, Python, Scala, C++Frameworks
Apache Spark, Django REST Framework, Spark, Hadoop, AWS EMR, Django, Flask, SpringLibraries/APIs
Pandas, NumPy, D3.js, Charts.jsTools
Azure HDInsight, Tableau, Cloudera, Google Cloud Dataproc, GIS, GitHub, Apache Impala, Apache Sqoop, Apache Avro, Apache NiFi, GitLab CI/CD, Jenkins, Microsoft Power BI, Apache Airflow, Qlik Sense, Grafana, IBM InfoSphere (DataStage), Crystal Reports, KibanaParadigms
Business Intelligence (BI), ETL, REST, Continuous Integration (CI), Automation, Data SciencePlatforms
Databricks, Jupyter Notebook, Apache Kafka, Hortonworks Data Platform (HDP), Unix, Google Cloud Platform (GCP), Azure, Oracle, Docker, Kubernetes, Amazon Web Services (AWS)Storage
MySQL, Teradata, Apache Hive, PostgreSQL, Microsoft SQL Server, Data Pipelines, AWS S3, MongoDB, HBase, Sybase, Elasticsearch, PostGIS, Redis, SQL Server Integration Services (SSIS), Cassandra, DatastageOther
Azure Data Lake, Big Data, Data Warehousing, Aprimo, AWS, APIs, Data Engineering, Data Architecture, Big Data Architecture, Data Analysis, Data Modeling, SAP BusinessObjects (BO), Parquet, Machine Learning, Google BigQuery, GeoPandas, GeoSpark, Scraping, Prometheus, Apache Flume, Statistics
Education
- Diploma in Business Intelligence and Software Development2010 - 2011Information Technology Institute - Cairo, Egypt
- Bachelor of Engineering degree in Computer Engineering2005 - 2010Benha University - Banha, Egypt
Certifications
- CCA Spark and Hadoop Developer CCA175DECEMBER 2017 - DECEMBER 2019Cloudera
