Mohamed Ahmed, Big Data Architect and Back-end Developer in Berlin, Germany
Mohamed Ahmed

Big Data Architect and Back-end Developer in Berlin, Germany

Member since November 4, 2022
Mohamed is a big data platform architect with 15 years of experience in the IT industry. He excels with distributed systems, data engineering, machine learning, and DevOps. Mohamed builds robust batch and real-time data platforms for stakeholders and moves companies to state-of-the-art data platforms. He designed GDPR-compliant data lakes and optimized an ETL pipeline, which saved the client hundreds of thousands of dollars annually. Mohamed is pragmatic and has an agile mindset.
Mohamed is now available for hire

Portfolio

  • Mobile.de
    Apache Spark, Scala, Apache Kafka, BigQuery, Google Cloud Platform (GCP)...
  • Careem Networks FZ
    Apache Spark, Apache Airflow, Scala, Python, Apache Livy, Jupyter Notebook...
  • Searchmetrics gmbh
    Amazon Web Services (AWS), RabbitMQ, Apache Spark, Apache Zeppelin, Java...

Experience

Location

Berlin, Germany

Availability

Full-time

Preferred Environment

Linux, Slack, Jira, GitHub, Google Cloud Platform (GCP), Apache Kafka, Data Engineering, Amazon Web Services (AWS), Big Data

The most amazing...

...project I've delivered required tuning a microservice to serve over 100,000 requests per second in 20 milliseconds per request.

Employment

  • Big Data Architect

    2020 - PRESENT
    Mobile.de
    • Designed, led, and guided the company data platform migration from on-premises infrastructure to public cloud (GCP) and saved 30% of stakeholders' daily business hours.
    • Built and designed a data lake that complies with GDPR requirements with minimal effect on downstream users.
    • Planned and monitored the GCP budget across dozens of projects, which resulted in an accurate spend cost.
    Technologies: Apache Spark, Scala, Apache Kafka, BigQuery, Google Cloud Platform (GCP), Python, Delta Lake, Data Processing, Kafka Streams, GDPR, Apache Airflow, Google Cloud SQL, Google Cloud Functions, Kubernetes, Catalog Data Entry Services, Apache Cassandra, PostgreSQL, Cloud Run, Apache Flume, Linux, Spark Streaming, MongoDB, Data Engineering, Data Pipelines, ETL, ELT, SQL, Jira, GitHub, Redshift, Pub/Sub, Data Modeling, Data Architecture, Spark, User-defined Functions (UDF), Data Governance, Data Strategy, Big Data
  • Staff Data Engineer

    2018 - 2020
    Careem Networks FZ
    • Developed an ETL framework that automates data processing pipelines and runs hundreds of ETL daily.
    • Optimized an ETL pipeline, which saved hundreds of thousands of dollars yearly.
    • Established a data academy to help full-stack engineers grow their data engineering knowledge.
    Technologies: Apache Spark, Apache Airflow, Scala, Python, Apache Livy, Jupyter Notebook, Apache Zeppelin, Amazon Elastic MapReduce (EMR), Apache Hive, Presto DB, Kafka Streams, Data Processing, Linux, Data Engineering, Data Pipelines, ETL, ELT, Amazon S3 (AWS S3), Data Structures, SQL, Deep Learning, Jira, GitHub, AWS EMR, Data Modeling, Data Architecture, Spark, User-defined Functions (UDF), Big Data, Pandas, Amazon Web Services (AWS), AWS Lambda
  • Senior Software Engineer and Big Data

    2017 - 2018
    Searchmetrics gmbh
    • Designed and developed a challengeable data pipeline for billions of messages and records.
    • Devised the ETL framework to work with many sources and sinks.
    • Presented new technologies and discussed them with my team.
    Technologies: Amazon Web Services (AWS), RabbitMQ, Apache Spark, Apache Zeppelin, Java, Apache Kafka, Spark Streaming, Apache Flink, MySQL, Hadoop, Apache Sqoop, Apache Hive, Scala, Spring Boot, Data Processing, Linux, Data Engineering, Data Pipelines, ETL, ELT, Amazon S3 (AWS S3), Data Structures, Amazon Elastic MapReduce (EMR), SQL, Jira, GitHub, AWS EMR, Data Modeling, Spark, User-defined Functions (UDF), Big Data
  • Senior Software Engineer and Big Data

    2015 - 2017
    Agoda
    • Developed and tuned recommendation microservices to accept millions of requests per second with a success rate of 99.99 in 20 milliseconds for the whole request trip.
    • Designed and developed a reactive DAG framework to build any logical flow over Akka actors and futures.
    • Assisted the data scientist team with ETL pipelines to apply ML offline training.
    Technologies: Apache Spark, Hadoop, Scala, Akka, spray, Data Processing, Apache Cassandra, PostgreSQL, Linux, Data Engineering, Data Pipelines, ETL, ELT, Data Structures, SQL, Jira, Django, HDFS, Data Modeling, Spark, User-defined Functions (UDF), Big Data
  • Back-end Specialist

    2014 - 2015
    CIT global
    • Developed a logging service that tracks all app actions on MongoDB with AspectJ.
    • Developed an e-payment workflow using Mule ESB that controls payment steps.
    • Created a wallet payment microservice that transfers payments across bank accounts.
    Technologies: Java, Mule ESB, ActiveMQ, Spring, MongoDB, JAX-WS, EJB3, Hibernate, Oracle Database, AspectJ, SQL, Apache Cassandra, Stored Procedure
  • Senior Java Developer

    2010 - 2014
    E-Finance
    • Developed back-end and front-end payment services using multiple frameworks; ADF, Struts, and ICEfaces.
    • Created business reports using the Jasper Reporting tool.
    • Built and automated administration pages created from a DB ER diagram.
    Technologies: Oracle ADF, Apache Struts, ICEfaces, EJB3, JPA 2, JAX-WS, Oracle PL/SQL, JasperReports, Quartz, QR Codes, Oracle Database, JavaScript, Ajax, SQL, Web Services, Data Modeling, Stored Procedure
  • Service Information Developer

    2009 - 2010
    HP Inc
    • Built the endpoint of the sales (EPOS) client app validator using Servlet and JSP, which can validate big XML files and return invalid tags.
    • Wrote a user tutorial that guided users to new features and increased the customer acceptance rate.
    • Contributed to the internal development community that helped new users get familiar with internal tools.
    Technologies: Servlet, JSP, JAX-RPC, Java API for XML Processing (JAXP), Hibernate, Ant, Ajax, Web Services, JAX-WS
  • Java Developer

    2007 - 2009
    Networks Valley
    • Created a custom payroll desktop app that handled complicated payroll logic and generated company payroll reports.
    • Devised an innovative home service that monitored smart homes and sent mobile notifications to homeowners.
    • Built a PCL interface app that controlled devices in an electricity plant.
    Technologies: Servlet, JSP, Hibernate, JavaScript, Ajax, Microsoft SQL Server, Java, SQL, JPA 2

Experience

  • On-premises to Public Cloud (GCP) Migration

    We experienced various limitations in the private cloud, so we decided to move to the public cloud (GCP). I took the initiative to design, lead, and build migration across the company and be an excellent example for new data platform infrastructure.

    I collected and discussed pain points with stakeholders, created general architecture ADR, reviewed the new design with my team and stakeholders, and collected feedback. Next, I estimated the budget and discussed it with the head of technology, removed obstacles to implementation, and modified open-source frameworks to fit our needs; for example, I added a new feature to the Atlas data catalog framework to support delta-lake. I reviewed the road map with my team and broke it down into epics and parallel stories, jumped in to help when blocks arose, and discussed best practices with different teams in the company from a data point of view.

  • Real-time Analytics Service

    This real-time analytics tool extracts user-tracking metrics from event streams depending on configurable input using Kafka and Spark-streaming frameworks.

    I designed a real-time solution that fulfilled stakeholders' requirements, tuned the reader service to achieve <100 milliseconds latency in the 99.99 success rate percentile, and introduced network solutions as the project ran in a hybrid cloud environment.

  • Building Marketplace Data Platform

    I worked across teams in three countries to define common pain points and introduce new solutions. This included creating the POC/RFC for new solutions, discussing solutions with teams, collaborating with teams and project managers to set the execution plan, mentoring data academy members, and building standard tools that accelerate development time.

  • Keyword Ranking

    I designed and developed innovative stream and batch data processing projects to improve the ranking of our clients with millions of keywords for many search engines in many countries with many languages.

    I created a changeable data pipeline for billions of messages and records and designed the ETL framework to work with many sources and sinks. I presented new technologies and discussed them with the team. I tuned the jobs to fit our cluster, reviewed the code, and took ownership of the project.

  • Reactive Framework (Jarvis)

    Jarvis is a reactive DAG framework to build logical flow over Akka actors and futures. I reviewed the requirements with all teams involved to simplify and remove repeated functions. I designed and built a DAG solution to streamline and service our business logic units in reactive mode. I presented the solution and helped teams use it.

  • Hotel Recommendation

    I designed and developed an innovative project to rank hotels based on user preferences. I assisted data scientists in collecting the data to apply the offline training and designed and built a solution replicating the ALS model to five data centers. I created a distributed and local cache solution to hold the ML model in memory. This achieved a four-millisecond response time in the worst-case scenarios. I developed a load balancer between servers in the same data center, applied our DAG framework (Jarvis) to build our ranking service, and tuned the microservice to accept millions of requests with a success rate of 99.99. I then audited the customer interaction with the microservice to use it for model evaluation and configured the deployment scripts for production and stage servers.

  • Bidding Channel ROI Manager

    This is a back-end project to manage the bidding channel jobs for companies such as Google and TripAdvisor. I reviewed the design with the software architect, and built the dynamic implementation for channels, accounts, sync data, and the Oozie and HDFS clients. I built the migration scripts and configured the deployment environment for production.

Skills

  • Languages

    Java, Scala, SQL, HTML, Python, C, C#, VHDL, Pascal, Snowflake, Stored Procedure
  • Frameworks

    Apache Spark, Spark, Hadoop, Akka, Presto DB, Spring Boot, Hibernate, Oracle ADF, Apache Struts, Django, AWS EMR
  • Tools

    Apache Airflow, Kafka Streams, BigQuery, Slack, Jira, GitHub, Servlet, RabbitMQ, Apache Beam
  • Paradigms

    ETL
  • Platforms

    Apache Kafka, Amazon Web Services (AWS), Google Cloud Platform (GCP), Linux, Jupyter Notebook, Apache Flink, Oracle Database, AWS Lambda, Kubernetes
  • Storage

    Data Pipelines, Google Cloud SQL, MySQL, PostgreSQL, Apache Hive, MongoDB, Oracle PL/SQL, Microsoft SQL Server, HDFS, Amazon S3 (AWS S3), Redshift
  • Other

    Data Processing, GDPR, Data Engineering, ELT, Data Architecture, Big Data, Ajax, Web Services, Algorithms, Data Structures, Graph Algorithms, Delta Lake, Google Cloud Functions, Catalog Data Entry Services, Apache Cassandra, Apache Livy, Apache Zeppelin, EJB3, Atlas, Pub/Sub, Data Modeling, User-defined Functions (UDF), Data Governance, Data Strategy, Microprocessors, Microcontroller Programming, Java Card OpenPlatform (JCOP), Deep Learning, Cloud Run, Apache Flume
  • Libraries/APIs

    JPA 2, Spark Streaming, JAX-WS, JAX-RPC, Pandas, Quartz, Java API for XML Processing (JAXP)

Education

  • Bachelor's Degree in Electrical Engineering
    2002 - 2007
    Fayoum University - Egypt

Certifications

  • Algorithms on Graphs
    DECEMBER 2018 - PRESENT
    Coursera
  • Deep Learning Specialisation
    JUNE 2018 - PRESENT
    Coursera
  • Data Structures
    FEBRUARY 2017 - PRESENT
    Coursera
  • Algorithmic Toolbox
    JANUARY 2017 - PRESENT
    Coursera
  • OCE Java EE 6 EJB 3.x (1Z0-895)
    JULY 2014 - PRESENT
    Oracle
  • OCE Java EE 6 Web Service (1Z0-897)
    MAY 2014 - PRESENT
    Oracle
  • OCE Java Persistence API 2.0 - EE 6 (1Z0-898)
    NOVEMBER 2013 - PRESENT
    Oracle
  • Sun Certified Web Component Developer SCWCD 5 (310-083)
    MAY 2008 - PRESENT
    Sun Microsystems
  • Sun Java5 Certified SCJP 5 (310-055)
    AUGUST 2006 - PRESENT
    Sun Microsystems

To view more profiles

Join Toptal
Share it with others