Jiri Pokorny

This project was responsible for applying patterns to large-scale data. I was the main developer on this project. Patterns themselves were a moving target, updated several times a day, and had a deadline when they had to be applied to all the historical data from the time of their publication.

As updates mutated the resulting data, there had to be a safe publishing mechanism on HDFS and safe synchronization to other clusters. There was a service to expose the current status of patterns for consumers. The pattern application was costly—there was an algorithm to limit the recalculation to just specific parts. Another interesting aspect of this service was the resiliency and the effort to minimize the chances of disrupting the computation by a single bad batch/pattern.

The service was a Scala-based application with multiple concurrent Spark jobs spawned into the Cloudera cluster. It used Scala futures and Akka Actors for concurrency handling and MongoDB to persist global computation state and track job states. An HTTP API and a CLI tool controlled various aspects of the running system.

A small application to notify users of the current station in Prague metro in the notification bar on Android phone. Prague metro was specific as it had cell signal just in stations—this application exploited that and used a list of GSM cell ids to track and predict the movement of the user. Now the application is outdated as metro tunnels are being covered with signals, and I did not continue with development, but still, it was a nice little hobby project.