Peixian Wang

At Bloomberg, we ran a monolithic service that called out to over 30 other services. Each downstream service was responsible for a single asset type, such as commodities, mortgages, and corporate bonds.

With so many requests and services in flight, we would run into pricing instability errors; the numbers returned from a downstream service were not deterministic, where the same request would yield different results. This presented a problem for us since our service had to call out to multiple asset types, then compute a price based on the prices of the underlying assets; for example, pricing a corn future requires a call to the downstream commodities prices service.

We opted for a full request-tracing mechanism to track down and debug non-deterministic requests. I handled the technical architecture and the first implementation of tracking all messages.

We had a team that had just experienced attrition from six to two people within a month and had a new team lead as part of that process. The team needed to quickly rebuild and scale up, as they maintained services that parsed information from real-time news feeds. However, the services were legacy and were built upon a combination of parser-based logic and machine learning information extraction tools. Most of the maintenance information had also been left with the former team members. I was brought in to help reduce tech debt, document all information, refactor as necessary, and help out any way I could.

Given that I didn't know the code, I decided it might be best to treat it as a black box. Since all the services were self-contained binaries with some configuration INIs, I sketched out a rough design of how each service interacted with another and which datastores were called by each service. I compiled this information into a variety of Helm deployment charts. I then set up a system to sample a portion of each day's incoming requests and responses, for each service, into a log that could be replayed.