Vivek Ramaswamy

This project aimed to move away from end-of-day, batch-style feeds from a data warehousing system to a data lake system updated in real-time where the system acted as a data source for all downstream applications that needed live data for functional and analytical use cases.

We used the CDC tool to move database feed updates to Kafka and wrote Spark streaming applications to process and store data in HBase. There was also a caching system outside the data lake to facilitate faster access to data.

OUTCOME
As a result of this migration, critical downstream applications had access to data in real-time. They no longer had to wait T+1 days for a feed or deal with stale data in case of batch processing failure. Due to near-real-time data feeds, it also created new opportunities to identify misuse and potential cross-sellable products.

An on-premise data lake based on Hortonworks Data Platform facilitated storing data for corporate and investment banks dealing in capital markets and this data lake provided the necessary data for computing, tweaking, testing, and calibrating risk models and strategies. Different front office application systems would send in data and files via Kafka to the platform. The Spark-based batch and streaming applications would process and transform the data and store it in ORC formats in HDFS and expose them via hive tables for consumption. The application logs were also integrated into ELK to enable live monitoring of the applications.

OUTCOME
The project led to the collation of organization-wide trade data and provided an environment for scientists, actuaries, and risk modelers to analyze, test, and tweak their existing and new models.

Evaluation, POC, and benchmark comparison of different databases capable of handling "ticker" data. TimescaleDB, Kdb+, eXtremeDB, and OneTick databases were evaluated. The databases were made to run on common hardware and were loaded with sample data. Then, they were benchmarked for the query performance, ease of use, scalability, and cost. The evaluation and POC gave all the necessary data points and enabled the business to make a more informed decision.