Sebastian Brestin

Context:
We wanted to improve the user and developer experience. The Spark application was presenting performance problems, and tech debt became a blocker for adding new functionality.

Solution:
Redesigned Spark application to make it more robust and flexible for data scientists and software engineers:
Partitioned data to avoid shuffles and data skew
Decreased number of partitions while increasing partition size to avoid data skew and driver congestion
Used strategic caching in order avoid recomputation and improve computation by cutting down the query plan
Refactored the application as a pipeline for engineers to add features as plugins to the pipeline
Generated deterministic Spark output in case of rows with equal comparison values

Results:
Reduced execution time by 50% and increased configuration flexibility, which in the second turn improves data scientists productivity
Reduced tech debt, which improved development time and improved development experience
Improved functional tests by having a deterministic output

Context:
We wanted to increase cluster availability, which was being blocked by long-running Spark jobs.

Solution:
Researched to increase Spark S3 parquet write performance. To analyze Spark jobs, I have used Spark History Server and Ganglia to pinpoint where time is spent and compare configuration fixes.

Results:
Increased write performance by a factor of three.

Context:
We wanted to increase the product stability and our development process.

Solution:
Implemented robust and generic functional test framework for Spark pipelines:
Allowed multiple instances of a test to run in parallel
Allowed tests to run in a pipeline manner as part of a suite
Simulated production runs by using a representative dataset

Results:
Faster POC development
Reduced development time by 50%
Increased product stability
Reduced AWS costs by simulating production runs with fewer resources

Context:
Some of our Spark applications were long-running applications up to nine hours. In this timeframe, nodes could go out of memory or lose connectivity. We wanted our applications to run faster and to be easier to recover in case of failure.

Solution:
Redesigned Spark application from a monolith into a modular Spark application which writes intermediate results and can be run in parallel.

Results:
No failures due to out of memory
Easier to recover by re-running only the failed module

Context:
We wanted a churn model that we can use to predict the future for our members and also integrate it into other Spark applications.

Solution:
Built a Spark application for ingesting 1TB of data from multiple sources and generate 40k possible features based on which the data science team can perform EDA and test models.

Context:
The data science team has to use the data and our systems most efficiently to be productive.

Solution:
Implemented new application APIs to help other teams improve their productivity:
1. Wrote better API for complex Spark or Cloud functionality
2. Wrote wrappers for complex application configuration
3. Implemented integration with adjacent applications
4. Wrote documentation and Unit Tests
5. Ran data analysis and data quality checks

For a minimum viable product, I designed and implemented a custom Django MapReduce system for matching an object with other million objects. The system was running on 60 Docker nodes and one match task was running under 60 seconds. In order to test the system I have extended Django test framework in order to test locally the Django custom MapReduce system without spawning virtual machines only by using multiple processes and cores.

This project aimed to match mutable objects (which users can create/update) with millions of immutable objects in real time (or as close as possible).

We chose Apache Spark because of its Python support, rich analytics toolkit, and streaming capabilities.

The solution was split into three Spark apps:
01. Retrieves mutable data from the sources using Kafka and Spark streaming, extracting features and saving the result to Cassandra.
02. Retrieves immutable data from sources using Kafka and Spark streaming, extracting features and saving the result to disc as Parquet files.
03. Loads data from Parquet files, computing a match percentage between all immutable objects and a single mutable object from Cassandra.

In order to query data in a reasonable amount of time, I denormalized the PostgreSQL tables (billions of records) using Elasticsearch. This improved the read performance by two orders of magnitude but added a write penalty. Because only parts of the documents were changing frequently, the problem was solved using Elasticsearch bulk partial updates with Groovy scripts for complex fields.

I led the implementation of a single-sign-on platform, using OpenID Connect, Django, PostgreSQL, Angular/TypeScript and AWS (which integrated the customer's entire product suite).

While working as a contractor, I used NLTK, Scikit-learn, AWS Transcribe, and AWS Lambda to run a sentiment analysis to improve customer support.

I led the implementation of a high-content-streaming platform using Django, PostgreSQL, Elasticsearch, and AWS. The system allowed pushing/pulling content efficiently using a RESTful API.

While working with Spark, I noticed the lack of proper documentation concerning PySpark. This situation motivated me to put together this article aimed at thoroughly explaining the secondary sort design pattern using PySpark.

The article covers two solutions. The first solution uses Spark's groupByKey which does a sort in the reducer phase while the second solution uses Spark's repartitionAndSortWithinPartitions which leverages the shuffle phase and iterator-to-iterator transformation to sort more efficiently and not run out of memory.

I wrote this article about PostgreSQL and data partitioning in general with practical examples using the Django web framework.

The first part of the article goes on about the reasons to partition data and also how partitioning can be implemented in PostgreSQL.

The second part describes a few solutions one can use to take advantage of PostgreSQL partitioning while using a web framework such as Django.

Working with PostgreSQL for many years has inspired me to create an article about indexes that any developer should know.

In the first section of the article, I explain about the fundamentals of the PostgreSQL B-tree index structure with a focus on B-tree data structure and its main components—leaf nodes and internal nodes, what are their roles, and how they are accessed while executing a query (a process also known as index lookup). The section ends with the index classification, with a broader view over the index key arity which helps explain the impact on a query’s performance.

In the second section, there is a quick introduction to the query plan to better understand the examples that follow.

In the third section, I introduce the concept of predicates, and the explanations cover the process PostgreSQL uses to classify predicates depending on index definition and feasibility.

The last section goes deeper into the mechanics of scans and how an index definition, data distribution, or even predicate usage can influence the performance of a query.