Burak Uyar

This project aimed to tackle the challenge of changing databases, which might affect a data warehouse that periodically loads data from them. As a solution, we developed a microservice that integrates into the configuration items of those source applications and checks the database structure in their test environments. Those structures are sent to the microservice and compared with the expected schema structure in the data warehouse. If any part of the new schema in the test environment is unexpected, the microservice makes the check fail, blocking the pull request until that conflict is fixed. The project has been live in production since it was developed and has saved many breaking changes before deployment.

I designed and implemented a DWH in the cloud and BI solutions on top of it to enable self-service analytics. I was the head of data in the company and managed 2-4 team members. My role was to receive the brief, prepare the solution, and ensure that our Data team delivered the results.

The company had different data sources, all of which needed to be used for different purposes, including self-service analytics, training, and testing ML models, and feeding the application database back.

The sources included PostgreSQL, MongoDB, APIs, web-scraping data, spreadsheets, and flat files.

The solution included having the brief from the internal teams, designing the structure, preparing the implementation phases, preparing the individual tasks, and resource allocation from our data team.

The solution mainly involves utilizing Python, SQL, NoSQL, Crontab, AWS S3, AWS Glue, AWS Athena, AWS IAM, Tableau, and Tableau Bridge.

An end-to-end business intelligence solution for self-service analytics for the client.

The client had their data on a PostgreSQL database and wanted to use Tableau for self-service analytics to improve the efficiency of their investments, hence ROI.

The project was split into two phases. The first part was to create a PoC without using an OLAP, use smaller data, and deliver the first version of the Tableau Cloud. The second part was to improve the solution using a proper OLAP, data pipeline, and ETL setup.

The first part is delivered quickly using PostgreSQL directly as the source, Tableau extracts, and Tableau Cloud.

For the second part, ClickHouse was selected as the OLAP. The implementation included using the MaterializedPostgreSQL database engine and replicating the relevant source tables from Postgres to ClickHouse. On top of the raw replicated tables, materialized views were built to create analytics layers. That approach allowed easy-to-manage infrastructure and lower maintenance effort for the client at later steps. With that implementation, near real-time data flow was provided into ClickHouse, hence into Tableau Cloud, via a Tableau Bridge instance running on Linux.

A containerized web-scraping service was designed from scratch and integrated into GCP Cloud Composer (Airflow). The process runs on the Kubernetes cluster and is fault-tolerant, so if it fails at any point, it can find where it was left and continue to process from that point. GCS, BigQuery, Cloud Composer, Cloud Vision, and Secret Manager are some of the services used in this solution.