Zohaa Qamar

Worked on optimizing the data lake solution of adtech company, reducing the overall cost by around 50%. I orchestrated the pipeline using Apache Airflow to integrate jobs executed on Amazon EMR. In this process, I used AWS cloud services—Kinesis Data Stream, Lambda, S3, Athena, EMR—and Python and Spark to manage the processing of huge datasets. Further, I developed jobs to dump the data from the data lake into Snowflake and create Tableau dashboards on top of it.

This complete data lake solution covers the ingestion of different data sources. The indigested structured data include Oracle and MySQL, while the unstructured data are in the form of files or data fetched via an API. This was done using the Python language. Then, I converted them to Parquet format, which is efficient and columnar. Next, I used PySpark with Spark SQL feature to transform the data into facts and dimensions, which was executed on AWS Glue.

Finally, the data were dumped into Amazon S3. Amazon Athena was used on top of S3 for the ad-hoc querying on the transformed data. Also, I created a process to load the transformed data into Amazon Redshift, making it consumable for the end users. The whole pipeline was orchestrated using AWS Step Function with complete CloudWatch logging and monitoring.

I designed and developed a reporting system for multiple clients where data from the source databases, Microsoft SQL hosted on Microsoft Azure, are brought into a single database with a flat report-friendly schema.

This helps clients query this database for their own reporting purposes instead of directly accessing the source database. It also increases the overall query performance as the SQL joins are no longer needed. I used Pentaho Spoon to create the ETL process, Microsoft Azure SQL as a database, and developed dashboards from reporting database using Microsoft Power BI.