Faisal Falah

Created a centralized data platform with batch and stream pipeline support for a leading media house in India. Major components of the UDP are a data lake on top of S3 (time partitioned in Parquet format) and a data warehouse in Redshift and BigQuery.

Data collected as events from different properties using SDKs (JavaScript, Android, and iOS). AWS Kinesis was used as a message queue. Set up batch ETL using Spark on EMR. Used Python, Go, and Shell scripts for production table data load, Apache Airflow for orchestration, and Athena and Spectrum for ad-hoc queries on top of the data lake.

Data lake design and implementation in lakehouse architecture for a multi-billion supply chain provider. S3 (time partitioned, Parquet format) and Redshift are its main components. AWS services like DataSync, Lambda, and Step Functions are used for a batch pipeline. Used Kinesis Data Firehorse for near-real-time ingestions, AWS Glue for catalog, and Amundsen for metadata discovery with OIDC integration with Keycloack. I also used AWS Step Functions for orchestration and SQL on Athena for ETL. Also, Jupiter notebooks are automated for advanced data transformation and modeling.

View26 was a SaaS analytic product on AWS to get a complete view of testing progress from multiple tools like HP ALM and Jira.

As a lead engineer in an early startup, I had hands-on experience in different systems. I mainly designed and implemented a highly concurrent data collection system using Go, MongoDB, Postgres, and Redshift.

Data is inserted to Pub/Sub topics from App Engine web servers. SDKs send messages from different devices to API endpoint in App Engine webserver. Pub and Sub messages are consumed via dataflow using Apache Beam and inserted into BigQuery. Dataflow Python SDK is also used.

Created a chatbot that acts as a virtual real estate broker using AWS Lex. AWS Lambda in Python is used to satisfy intents. The existing search API on Solr is utilized for getting relevant results. It was created for a classified website and is still running in production.

Data migration ETL development for projects like credit account information sharing (CAIS) and FATCA for a world-leading bank. Converted complex business logic from mapping documents to SQL. ETL tools like SAP BODS are used along with UNIX text editing tools like SED and AWS with shell scripting. Source data was from Oracle tables. Target was a data warehousing solution from IBM on top of Oracle DB.

Originally, AWS Redshift was used for data warehousing, but only around one thousand queries were running in the cluster per month. We compared the cost with GCP BigQuery, and it was way lesser, as costing is based on data volume scanned in BigQuery. Moved data from Redshift to S3, then S3 to GCS, and then to BigQuery. Other than the cost-benefit, complex queries were 10-20 times faster in BigQuery.

Developed a scaling recommendation engine in Spark for a leading real estate provider in the US. Original PySpark code ran for a very limited subset of properties, and it's currently on sale in the market. They wanted to scale it and run for all the properties, even off-market. The code was complicated to scale as it contains multiple nested loops through all properties. Still, I've successfully done that using Spark tuning and implementing some minor changes in the code.

While doing software testing, especially while doing unit testing for ETL projects, it is tricky to generate dates. For example, we might need random weekend days for the last year. I was facing this issue and created a small module in Go for this.

Snowflake has published my article on data migration on their official community blog. This is one among the many of my well-accepted data engineering articles. Many other blogs are among the top Google search results for respective topics.