Mahmoud Mehdi

I had the chance to contribute to the open-source Delta Lake project (maintained by Databricks), which is a layer on top of Apache Spark. This framework offers the possibility to make ACID transactions on stored data.

Sometimes different jobs made the same transformations of specific read data—in my case, it was daily sales—that resulted in a lot of duplicate code and different ways to deal with data.
In order to avoid such issues, I took the initiative to develop a custom Apache Spark Data source that easily reads data and transforms it.
My team members used that library and had their data ready for analysis after calling only one line.

When I joined the AXA team for fraud detection, the project contained a basic Node.js graph representation that allowed users to detect fraud between the different nodes (representing the claims). Once the data started to grow, we started having performance issues. To solve the issue, I took the initiative to develop a distributed solution based on Apache Spark and its library for graphs that takes advantage of Spark DataFrames and the possibility of parallelizing data processing.

Once the project was completed, we were able to detect fraud rapidly compared to the old solution.

2016 was an interesting year for football fans since we had the Euro 2016 and the Copa America in the same year. We decided to create a big data application that predicts different game scores. We extracted data from various sources, processed the data with Apache Spark jobs (the data was massive—we had historical football datasets starting from 1930), and then implemented machine learning algorithms with Spark ML to train our models and predict different games.

We had a good prediction rate: 77% of the predictions were correct.