Baris Ergun

In recent years, this is the most exciting project I've worked on.

I worked on the disease detection project to analyze and report candidates most likely to be affected by a specific disease or condition. I introduced the fastest and most reliable algorithm among three different people in three other teams working on the same problem. I achieved this by thinking out of the box and implementing the required solution, abiding functional programming concepts such as establishing an algorithm in line with associativity and identity laws.

The project was handed over to me by two permanent employees who were leaving the company. In nine months, I established the complete code pipeline and solved the most complex parts of the project, which were required to bring the performance up. Measured according to the same reference data, the overall solution I have provided was at least 20 times faster than any other solution.

I heavily used functional programming concepts, algorithmic thinking, and comprehensive testing strategies for this project.

I migrated a complex big data ETL pipeline from the MSSQL pipeline to IQVIA's Hadoop-based P360G platform for pharmaceutical clients in France. This was a project where basic extracts were implemented with Impala queries and complex extracts handled with Spark jobs. I delivered and productionized the first two phases (R1 and R2) of the project successfully. There was no up-to-date documentation about the legacy project, and since every complex logic was handled with SQL, the solutions that needed migration were complex.

I overcame all the difficulties by:
- Reverse engineering the legacy code both by interpreting it and pulling features out of the resulting User Acceptance Test data with an input-output analysis.
- Keeping up-to-date documentation of new Spark and Impala jobs.
- Communicating with all clients frequently and efficiently.
- Using the property, unit, and integration tests, which were implemented with random data generators and data generators with imposed conditions.
- Revolutionizing complex extracts with Spark jobs utilizing both dataset SQLs and more with Scala algorithms on each partition.
- Conducting a meticulous UAT discrepancy analysis.

Data ingestion of ShopDirect's general business data into a data lake in the AWS environment. For this project, the major achievement was to drive whole data pipeline integration end to end on different data storage zones by overcoming the challenge of realizing the need for snapshotting the mutable database sources that were not clear in the requirements. Another challenge I had was joining big data with a smaller but still large data source by custom partitioning and grouping techniques in Apache Spark.