Selahattin Gungormus

A visual data integration product designed to run on a web application with drag-n-drop components. Any data professional without coding experience can use it to build data pipelines with a 100% visual experience. It leverages the Apache Spark execution engine and works on top of Hadoop platforms.

I created my startup with two developers in 2015 to launch the Integer8 product both on local and international marketplaces. I designed and led the development effort to make the product feasible for local SMEs. At the end of the first year, we deployed our platform to two different retail companies.

I became a cloud partner for Microsoft Azure in Turkey and spent one more year making Integer8 eligible for Azure Marketplace. At the end of this effort, Integer8 successfully became an official Azure Marketplace product.

Over 300 data pipeline tasks were transformed from Talend into Airflow on a Python/Spark data architecture running on distributed Celery. The daily denormalized payment dataset is refreshed on Azure Blob Storage. The daily ETL duration was reduced by 60%.

I designed and implemented whole data pipeline processes as the responsible data engineer for the new data platform. I built a CDC mechanism from MySQL database into Kafka to provide a pub/sub-event system for near real time integration. I then prepared live Spark Streaming jobs to consume Kafka topics to refresh target data-stores. That helped the marketing and operations team to monitor the workload on the system and detect anomalies.

All data sources were consolidated into two main data marts for the Tableau reporting layer. Daily pre-aggregated tables helped live reports to perform 400% faster than the previous implementation. That also increased the motivation of using reporting tools by power-users all over the organization.

I prepared a cloud ETL automation solution by using AWS Lambda along with Python for a client project. In this project, I was responsible for connecting REST APIs and event sources for new events and collecting the CRM information from AWS S3 Buckets. I used AWS EC2 layers to provide additional support for S3 and Pandas in Python environments. Individual Lambda functions were created to collect, cleanse and transform data sources on serverless architecture and writing them into the destination database.

As the target database I used Amazon Redshift. So individual events are emitted from Amazon EventBridge into Lambda functions and accumulated into Redshift database for further analysis.