Nan Gu

I recently wrote an cluster computing scheduler in Python to maximize the utilization of cluster computing resources for parallel computing. It is designed with a minimalist approach to ensure we have as much physical resources available as possible to be committed to actual computing jobs. It relies on SSH for communication and command line interface to execute tasks. The only pre-requisites are a few standard command line utility programs available out-of-the-box in any Unix-like system. It currently schedules resources based on logical CPUs. It has been used as the new parallel computing infrastructure for our massive model simulation and optimization and has greatly improved our utilization. I may consider the open-source part of it in the future.

We use R to pull and merge trading-related data from different data sources, calculate important statistics, and generate daily email reports with PDF attachments using RMarkdown to send to a list of authorized subscribers.

We implemented a set of stored procedures to calculate highly configurable pair-wise price correlations between financial instruments and set up a process to update the calculation on a daily basis. All the fine-tuned calculation algorithm, as well as the data ETL process, was entirely written in SQL. The result was widely used within the firm, including the crucial real-time automated trading system.