Sam Rogers

Total Home Score is a product designed to help prospective home buyers and renters understand what it is like to live at a particular property prior to making a decision.

In order to scale this product and calculate scores for millions of properties, I constructed a large scale data pipeline to perform complex geospatial calculations and aggregations.

This pipeline involved the use of Spark and EMR to run calculations on road traffic data and produce aggregations on how drivers typically drive on a particular length of roadway. Addresses are then loaded into Dask and calculations are performed across thousands of partitions to determine how much "dangerous" roadway exists within a particular radius of a given address.

Additionally, I developed a pipeline to process aircraft location data (10 billion+ points) and determine the level of aircraft noise expected at a particular property.

A Python-based application running on AWS Lambda and Redis that enabled the millisecond-level record retrieval of aggregated product usage data.

While working at a marketing technology provider, our product team wanted the ability to surface product usage data to our customers. Customers do not want to have to wait for a query to run in our data warehouse a return a result. The solution that I devised was to run a predefined set of aggregations and place them into a cache so that results could be received and visualized by a customer almost instantly.

Not only was this more rapid than running aggregations on demand, but it was also more cost-effective, instead of running thousands of aggregation queries in Snowflake per day, only one query needed to run to generate the output data and place it into our cache.

Airflow is an open-source tool designed for orchestration, scheduling, and execution of ETL jobs. The tool was originally designed to run these processes within it as well. However, many problems can arise from Airflow performing actual processing and not just coordinating between resources. For example, all dependencies need to be installed on the same instance, memory leaks can bring down the entire cluster, and additional security measures may be necessary if customer data is being touched by the cluster.

My solution was to have Airflow serve solely as a container execution tool—no processing actually takes place within the application. In addition, all the configurations of what job to execute and what parameters to pass to it are still contained within the Airflow code, but executed elsewhere. This makes for a simple interface for other data engineers to implement new pipelines.

For example, if an engineer has a file in S3 that they want to be loaded to a database on a schedule, they simply utilize the loader operator class that already exists in the Airflow repository. When executed, the class provisions a task to run in AWS Fargate which executes a process using the configurations passed to it.