Kafka System Engineer
2021 - 2022KM.ON- Industrialized the installation and DevOps of the Kafka cluster on AWS EKS and Alibaba Cloud ACK. Benchmarked and improved throughput, reliability, and fault tolerance in preparation for the production rollout.
- Learned to deploy and operate a multi-cloud architecture set up on AWS and Alibaba Cloud and to deliver and deploy software in China to serve Chinese clients.
- Aligned with the engineering lead and team on the best practices for managing Kafka clusters at scale.
- Analyzed and implemented GitOps POCs using different technologies, such as ArgoCD or Flux.
Technologies: Apache Kafka, Helm, Docker, Bitbucket, Continuous Integration (CI), Alibaba Cloud, Kubernetes, Git, JVM, Google Docs, Visual Studio Code, Monitoring, Software Design Patterns, Programming, Java, Databases, Networking, Microservices Architecture, API Design, Grafana, Prometheus, Event-driven Architecture, Identity & Access Management (IAM), Kustomize, Software Architecture, Confluence, Cloud Architecture, Realtime, Continuous Deployment, System Design, Cloud Native, Big Data, Data Modeling, SaaS, Slack, Amazon EKS, Amazon Web Services (AWS)Data Engineer
2020 - 2022Smarter Ecommerce GmbH- Spearheaded the integration of Kafka streams used for different data processing pipelines. Reduced running costs by 70% and improved throughput by 350%.
- Worked on a generic cloud-native big data platform and data pipeline framework on top of Kafka, Akka Streams, Kubernetes, and Scala to ingest and transform hundreds of datasets from various sources.
- Led the design, implementation, and integration of the data warehouse Google Cloud BigQuery. Defined and used idiomatic cloud patterns for reading/writing data and building ETL/data pipelines in the data platform.
- Managed the design and adoption of data mesh architecture and data governance. Oversaw the delivery of four data products and their data pipelines.
- Oversaw the design and implementation of securing and isolating the 450+ tenants on the compute, storage, and billing levels in Google Cloud.
- Deployed, integrated, and operated Kafka to handle the communication between the different data pipelines. Designed all the operations processes, disaster recovery, and automation for the cluster deployed on Kubernetes.
- Spearheaded the definition and implementation of the domain data model. Aligned the requirements and constraints of at least five teams that use the storage-agnostic model in many data pipelines.
- Led the Google Cloud infrastructure automation setup handling cloud projects, Kubernetes clusters, cloud functions, storage (GCS, BigQuery, Kafka), and different security topics like Identity and Access Management (IAM) and workload identity.
Technologies: Akka Streams, Apache Kafka, BigQuery, Kafka Streams, Kubernetes, Google Cloud Functions, Serverless, Event-driven Architecture, Reactive Streams, Data Warehouse Design, Data Wrangling, Data Warehousing, Scala, Identity & Access Management (IAM), Terraform, Kustomize, Google Cloud Storage, Prometheus, Grafana, Cloudflow, Google Cloud Platform (GCP), Argo CD, TeamCity, Domain-driven Design (DDD), Software Architecture, Data Pipelines, Continuous Integration (CI), Git, REST APIs, JVM, IntelliJ, Google Cloud, Google Docs, Docker, Visual Studio Code, Data Science, Google Data Studio, Monitoring, Software Design Patterns, Programming, Java, Databases, Networking, Microservices Architecture, API Design, API Development, Helm, Confluence, Cloud Architecture, Google BigQuery, Google Secret Manager, Data Mesh, Cost Cutting, Realtime, Data Processing, ETL, ScalaTest, Continuous Deployment, Google Kubernetes Engine (GKE), System Design, Cloud Native, Big Data, Data Modeling, Test-driven Development (TDD), SaaS, SQL, Slack, PythonSoftware Engineer
2018 - 2020Smarter Ecommerce GmbH- Worked on a SaaS for comparison shopping service used by more than 400,000 monthly users and part of the Google CSS Program.
- Led the research, design, and implementation of a clustering algorithm and data pipeline that processes more than 100 million products daily from different data sources like Google Merchant Center and Amazon Marketplace.
- Optimized client performance for a smooth, frictionless user experience, achieving a score of 92/100 on Google PageSpeed Insights.
- Implemented server-side rendering for the Angular client, generating and serving the sitemap. Enabled Googlebot to crawl 10+ million pages.
- Implemented a search engine and indexing data pipeline using Elasticsearch and reactive programming. Improved search accuracy and latency by 20%.
Technologies: Scala, Spring, Kubernetes, BigQuery, Google Data Studio, Google Cloud, Angular, Elasticsearch, Monitoring, Node.js, GraphQL, Software Design Patterns, RabbitMQ, HTML, CSS, SCSS, Grafana, Prometheus, TypeScript, Express.js, Bootstrap, API Design, API Development, Docker, MySQL, Google Cloud SQL, Google Cloud Storage, REST, RxJS, RxJava, Server-side Rendering, Data Pipelines, NgRx, Sass, Google Cloud Platform (GCP), Continuous Integration (CI), Git, JavaScript, REST APIs, JVM, IntelliJ, Google Docs, Visual Studio Code, Data Science, Programming, Java, Databases, Networking, Spring Boot, Microservices Architecture, Serverless, Event-driven Architecture, Reactive Streams, Data Wrangling, Identity & Access Management (IAM), Terraform, Kustomize, Argo CD, TeamCity, Domain-driven Design (DDD), Software Architecture, Helm, Confluence, Cloud Architecture, Google BigQuery, Cost Cutting, Realtime, Data Processing, ETL, Continuous Deployment, Google Kubernetes Engine (GKE), System Design, Cloud Native, Big Data, Data Modeling, Test-driven Development (TDD), SaaS, SQL, Slack, PythonCo-founder
2017 - 2017Dirty Paws NGO- Created a web app (front and back end) where users can report different animals (stray, lost, or missing) to get help from rescuers.
- Implemented a notification feature for the nearby rescuers using geolocation.
- Executed a feature-complete authentication system.
- Applied table listing and map listing for the reported animals.
Technologies: Node.js, Express.js, Angular, MongoDB, Bootstrap, Sass, REST APIs, Git, HTML, CSS, SCSS, JavaScript, Docker, Visual Studio Code, Programming, Databases, Networking, Microservices Architecture, API Design, API Development, Domain-driven Design (DDD), Software Architecture, Bitbucket, System Design, Data Modeling, SQLTech Lead
2016 - 2017Egyptian Ministry Of Defence- Led the design and implementation of a CMS with real-time document administration features and a notification system.
- Oversaw the integration with the existing legacy system. Minimized operating cost and processing time.
- Deployed and operated different virtual machines, web applications, databases, file servers, and data back-ups.
Technologies: Ruby on Rails 5, .NET, Bootstrap, Sass, Ruby on Rails (RoR), REST APIs, Git, HTML, CSS, SCSS, JavaScript, JVM, Node.js, Docker, Visual Studio Code, Programming, Java, Databases, Networking, Microservices Architecture, API Design, API Development, Domain-driven Design (DDD), Software Architecture, System Design, Data Modeling, SQL