Verified Expert in Engineering
Data Engineer and ETL Developer
Mohamed has a master's degree in data science. He worked as a data engineer and full-stack developer for over seven years. Building cloud-native big data applications are his strongest suit. Mohamed served as the TL on multiple projects in the eCommerce industry, including delivering a Google partner SaaS app used by more than 400,000 monthly users. He showed technical and organizational excellence as TL building military software. He enjoys mentoring developers and presenting technology talks.
JVM, Node.js, Scala, Kubernetes, Google Cloud Platform (GCP), Apache Kafka, ETL, Identity & Access Management (IAM), Software Architecture, Domain-driven Design (DDD)
The most amazing...
...work I've done is designing and implementing a multi-TB data warehouse on GCP leveraging data meshes architecture with E2E security to isolate the 450+ tenants.
Kafka System Engineer
- Industrialized the installation and DevOps of the Kafka cluster on AWS EKS and Alibaba Cloud ACK. Benchmarked and improved throughput, reliability, and fault tolerance in preparation for the production rollout.
- Learned to deploy and operate a multi-cloud architecture set up on AWS and Alibaba Cloud and to deliver and deploy software in China to serve Chinese clients.
- Aligned with the engineering lead and team on the best practices for managing Kafka clusters at scale.
- Analyzed and implemented GitOps POCs using different technologies, such as ArgoCD or Flux.
Smarter Ecommerce GmbH
- Spearheaded the integration of Kafka streams used for different data processing pipelines. Reduced running costs by 70% and improved throughput by 350%.
- Worked on a generic cloud-native big data platform and data pipeline framework on top of Kafka, Akka Streams, Kubernetes, and Scala to ingest and transform hundreds of datasets from various sources.
- Led the design, implementation, and integration of the data warehouse Google Cloud BigQuery. Defined and used idiomatic cloud patterns for reading/writing data and building ETL/data pipelines in the data platform.
- Managed the design and adoption of data mesh architecture and data governance. Oversaw the delivery of four data products and their data pipelines.
- Oversaw the design and implementation of securing and isolating the 450+ tenants on the compute, storage, and billing levels in Google Cloud.
- Deployed, integrated, and operated Kafka to handle the communication between the different data pipelines. Designed all the operations processes, disaster recovery, and automation for the cluster deployed on Kubernetes.
- Spearheaded the definition and implementation of the domain data model. Aligned the requirements and constraints of at least five teams that use the storage-agnostic model in many data pipelines.
- Led the Google Cloud infrastructure automation setup handling cloud projects, Kubernetes clusters, cloud functions, storage (GCS, BigQuery, Kafka), and different security topics like Identity and Access Management (IAM) and workload identity.
Smarter Ecommerce GmbH
- Worked on a SaaS for comparison shopping service used by more than 400,000 monthly users and part of the Google CSS Program.
- Led the research, design, and implementation of a clustering algorithm and data pipeline that processes more than 100 million products daily from different data sources like Google Merchant Center and Amazon Marketplace.
- Optimized client performance for a smooth, frictionless user experience, achieving a score of 92/100 on Google PageSpeed Insights.
- Implemented server-side rendering for the Angular client, generating and serving the sitemap. Enabled Googlebot to crawl 10+ million pages.
- Implemented a search engine and indexing data pipeline using Elasticsearch and reactive programming. Improved search accuracy and latency by 20%.
Dirty Paws NGO
- Created a web app (front and back end) where users can report different animals (stray, lost, or missing) to get help from rescuers.
- Implemented a notification feature for the nearby rescuers using geolocation.
- Executed a feature-complete authentication system.
- Applied table listing and map listing for the reported animals.
Egyptian Ministry Of Defence
- Led the design and implementation of a CMS with real-time document administration features and a notification system.
- Oversaw the integration with the existing legacy system. Minimized operating cost and processing time.
- Deployed and operated different virtual machines, web applications, databases, file servers, and data back-ups.
Cloud Native Multitenant Data Warehouse
• Aligned with top management and product owners on the requirements and use cases for the data warehouse and the high-level architecture. Led the research on the adoption of data mesh architecture.
• Prepared all the ADRs, design, and architecture sketches to choose the best fit solution given the different trade-offs.
• Implemented the back-end services and data pipelines that continuously load the data into the warehouse using BigQuery APIs, Kubernetes, Scala, and Cloudflow.
• Created and implemented the solution for tenant isolation, securing the data and computation of the different tenants along with costs and blast-radius isolation.
• Designed the organization and structure of the Google Cloud projects, cloud resources and cloud products to use, and I reviewed the solution with Google engineers.
• Implemented the automation, CI, and CD for provisioning all the needed cloud resources and components supporting the overall architecture using Kubernetes, operators, and ArgoCD.
Real-time Data Processing Pipeline
• I started by deep diving into Kafka Streams, reading a 250-page book and many other resources that helped me understand the ecosystem and become an expert in data streaming.
• Implemented three POCs to analyze and guide the adoption in production.
• Given the previous research, I designed the final architecture, flow of data, the different components needed, and blast-radius isolation.
• I implemented the Kafka streams pipeline using the scala DSL and deployed it to Kubernetes with CI/CD. The pipeline processes 20 GB of data daily from multiple datasets performing different aggregations and joins, and delivers the result to Google Cloud Storage and BigQuery.
The pipeline reduced costs by 77% and improved throughout by 350% compared to the previous approach, so it was a very successful project. As a result, I was promoted to senior software engineer.
Comparison Shopping Servicehttp://smec.shopping
It should be noted that the main clustering algorithm displays grouped products to the end user. This also involved implementing a search engine.
I also developed the Angular web client and did extensive work on SEO, including server-side rendering.
I was involved in planning, designing the different epics, and setting the direction for the project.
Spring, Angular, Spring Boot, Ruby on Rails 5
Akka Streams, API Development, RxJS, RxJava, REST APIs, Node.js
IntelliJ, Kafka Streams, BigQuery, Git, Grafana, Bitbucket, Google Kubernetes Engine (GKE), ScalaTest, RabbitMQ, Terraform, TeamCity, Helm, Confluence, Amazon EKS
Microservices Architecture, Continuous Integration (CI), REST, Event-driven Architecture, Continuous Deployment, ETL, Test-driven Development (TDD), Data Science
JVM, Kubernetes, Docker, Apache Kafka, Google Cloud Platform (GCP), Cloud Native, Amazon Web Services (AWS), AWS Lambda
Google Cloud, Databases, Data Pipelines, Google Cloud Storage, Elasticsearch, MySQL, Google Cloud SQL, Alibaba Cloud
Google Data Studio, Networking, API Design, Prometheus, Google Cloud Functions, Serverless, Reactive Streams, Identity & Access Management (IAM), Kustomize, Cloudflow, Argo CD, Domain-driven Design (DDD), Software Architecture, Cloud Architecture, Google BigQuery, GSM, Data Modeling, Big Data, System Design, Data Processing, SaaS, Data Warehouse Design, Data Warehousing, Data Products, Data Mesh, Cloud
Master's Degree in Data Science
Johannes Kepler University - Linz, Austria
Bachelor's Degree in Computer Science Engineering
German University in Cairo - Cairo, Egypt
AWS Certified Developer