Eduardo Kitamura Manrique

At S1mbi0se, we needed to provide a multidimensional BI tool. To achieve this, we used Elasticsearch aggregations at the beginning.

But we had some problems with the models. They were quick to be indexed but slow on the aggregation or the contrary. After realizing ES would not work, we decided to create our search engine.

I was responsible for the POC (proof of concept). The POC consisted of a custom binary-tree data structure (with memory-mapped files) where every stored value would be hashed (64 bits) and added if not existing to the tree. Every leaf had the hash, the pointers to the left and right leaves, followed by an array of bytes. In this array, we had the compressed user ids that have that value.

The array was extendable if necessary. The user ID was a long (64 bits), but as it was sharded, we could compress the user ID into 14 bits (1,5 bytes). The shard ID itself would give the user ID range.

The POC was successful, and we created the first version with a clustered server. The clients determined the coordination, and I left the company to move to Europe, and now it has become a product.

For more information, you can contact Gabriel Meneghati, the CEO and founder of S1mbi0se.

My most important experience with Go was at PionerLabs.

We had two critical microservices that would be used from all other systems and services. For this particular reason, we chose Go with GRPC for those two services (the other services were Node.js with Express.js). I was responsible for the stock service: a service that exposed operations for consultations and some to update snapshots and add stock movements.

Some of the snapshot operations were quite heavy in terms of processing due to the amount of data (the customer was a large retail company). We also fully covered the stock service with tests. However, we chose to have only a few unit tests along with full integration BDD tests with Cucumber (Gherkin).

Having only unit tests for more complex code was a good choice because it allowed more flexibility in refactoring. As the stock service doesn't call external services—and we used docker (with Docker-compose)—nothing needed to be mocked in the tests.

The service also used NATs to publish messages to notify stock changes in stock for any service that needs to be notified. The database used was Google Cloud Database (and PostgreSQL for development).