Concurrency Developer in Prague, Czech Republic

This project was responsible for the application of patterns to large-scale data. I was the main developer on this project. Patterns themselves were a moving target, were updated several times a day, and there was a deadline when the pattern has to be applied to all the historical data from the time of its publication.

As updates mutated the resulting data, there had to be a safe publishing mechanism on HDFS and safe synchronization to other clusters. There was a service to expose the current status of patterns for consumers. The pattern application was costly—there was an algorithm to limit the recalculation to just specific parts. Another interesting aspect of this service was the resiliency and the effort to minimize the chances of disrupting the computation by a single bad batch/pattern.

The service itself was a Scala-based application with multiple concurrent Spark jobs spawned into the Cloudera cluster. It used Scala futures and Akka Actors for handling concurrency and MongoDB to persist global computation state and track job states. There was an HTTP API and CLI tool to control various aspects of the running system.

A small application to notify users of the current station in Prague metro in the notification bar on Android phone. Prague metro was specific as it had cell signal just in stations—this application exploited that and used a list of GSM cell ids to track and predict the movement of the user. Now the application is outdated as metro tunnels are being covered with signals, and I did not continue with development, but still, it was a nice little hobby project.