Big Data Architect
2020 - PRESENTMobile.de- Designed, led, and guided the company data platform migration from on-premises infrastructure to public cloud (GCP) and saved 30% of stakeholders' daily business hours.
- Built and designed a data lake that complies with GDPR requirements with minimal effect on downstream users.
- Planned and monitored the GCP budget across dozens of projects, which resulted in an accurate spend cost.
Technologies: Apache Spark, Scala, Apache Kafka, BigQuery, Google Cloud Platform (GCP), Python, Delta Lake, Data Processing, Kafka Streams, GDPR, Apache Airflow, Google Cloud SQL, Google Cloud Functions, Kubernetes, Catalog Data Entry Services, Apache Cassandra, PostgreSQL, Cloud Run, Apache Flume, Linux, Spark Streaming, MongoDB, Data Engineering, Data Pipelines, ETL, ELT, SQL, Jira, GitHub, Redshift, Pub/Sub, Data Modeling, Data Architecture, Spark, User-defined Functions (UDF), Data Governance, Data Strategy, Big DataStaff Data Engineer
2018 - 2020Careem Networks FZ- Developed an ETL framework that automates data processing pipelines and runs hundreds of ETL daily.
- Optimized an ETL pipeline, which saved hundreds of thousands of dollars yearly.
- Established a data academy to help full-stack engineers grow their data engineering knowledge.
Technologies: Apache Spark, Apache Airflow, Scala, Python, Apache Livy, Jupyter Notebook, Apache Zeppelin, Amazon Elastic MapReduce (EMR), Apache Hive, Presto DB, Kafka Streams, Data Processing, Linux, Data Engineering, Data Pipelines, ETL, ELT, Amazon S3 (AWS S3), Data Structures, SQL, Deep Learning, Jira, GitHub, AWS EMR, Data Modeling, Data Architecture, Spark, User-defined Functions (UDF), Big Data, Pandas, Amazon Web Services (AWS), AWS LambdaSenior Software Engineer and Big Data
2017 - 2018Searchmetrics gmbh- Designed and developed a challengeable data pipeline for billions of messages and records.
- Devised the ETL framework to work with many sources and sinks.
- Presented new technologies and discussed them with my team.
Technologies: Amazon Web Services (AWS), RabbitMQ, Apache Spark, Apache Zeppelin, Java, Apache Kafka, Spark Streaming, Apache Flink, MySQL, Hadoop, Apache Sqoop, Apache Hive, Scala, Spring Boot, Data Processing, Linux, Data Engineering, Data Pipelines, ETL, ELT, Amazon S3 (AWS S3), Data Structures, Amazon Elastic MapReduce (EMR), SQL, Jira, GitHub, AWS EMR, Data Modeling, Spark, User-defined Functions (UDF), Big DataSenior Software Engineer and Big Data
2015 - 2017Agoda- Developed and tuned recommendation microservices to accept millions of requests per second with a success rate of 99.99 in 20 milliseconds for the whole request trip.
- Designed and developed a reactive DAG framework to build any logical flow over Akka actors and futures.
- Assisted the data scientist team with ETL pipelines to apply ML offline training.
Technologies: Apache Spark, Hadoop, Scala, Akka, spray, Data Processing, Apache Cassandra, PostgreSQL, Linux, Data Engineering, Data Pipelines, ETL, ELT, Data Structures, SQL, Jira, Django, HDFS, Data Modeling, Spark, User-defined Functions (UDF), Big DataBack-end Specialist
2014 - 2015CIT global- Developed a logging service that tracks all app actions on MongoDB with AspectJ.
- Developed an e-payment workflow using Mule ESB that controls payment steps.
- Created a wallet payment microservice that transfers payments across bank accounts.
Technologies: Java, Mule ESB, ActiveMQ, Spring, MongoDB, JAX-WS, EJB3, Hibernate, Oracle Database, AspectJ, SQL, Apache Cassandra, Stored ProcedureSenior Java Developer
2010 - 2014E-Finance- Developed back-end and front-end payment services using multiple frameworks; ADF, Struts, and ICEfaces.
- Created business reports using the Jasper Reporting tool.
- Built and automated administration pages created from a DB ER diagram.
Technologies: Oracle ADF, Apache Struts, ICEfaces, EJB3, JPA 2, JAX-WS, Oracle PL/SQL, JasperReports, Quartz, QR Codes, Oracle Database, JavaScript, Ajax, SQL, Web Services, Data Modeling, Stored ProcedureService Information Developer
2009 - 2010HP Inc- Built the endpoint of the sales (EPOS) client app validator using Servlet and JSP, which can validate big XML files and return invalid tags.
- Wrote a user tutorial that guided users to new features and increased the customer acceptance rate.
- Contributed to the internal development community that helped new users get familiar with internal tools.
Technologies: Servlet, JSP, JAX-RPC, Java API for XML Processing (JAXP), Hibernate, Ant, Ajax, Web Services, JAX-WSJava Developer
2007 - 2009Networks Valley- Created a custom payroll desktop app that handled complicated payroll logic and generated company payroll reports.
- Devised an innovative home service that monitored smart homes and sent mobile notifications to homeowners.
- Built a PCL interface app that controlled devices in an electricity plant.
Technologies: Servlet, JSP, Hibernate, JavaScript, Ajax, Microsoft SQL Server, Java, SQL, JPA 2