Attila Pápai

Redesigned and completely rewrote the old and slow Catalog and Search experience. The old system wasn't scaling well and became unreliable on larger datasets. In the new system design, we chose Elasticsearch as the search engine. We used Debezium to look for changes in other databases to store asynchronously and index relevant or searchable information in Elasticsearch. The performance of regular search improved to only milliseconds on hundreds of millions of data rows. Later, we added a similarity search for an even better user experience.

Census is a data activation and reverse ETL platform that makes it easy to connect customers' data warehouses to other services.

My role in this project was to design and create a back-end service called Connector to receive and transform data from Census into Labelbox's data warehouse. With this, customers were able to import their data stored in different warehouses into Labelbox without writing a single line of code.

After setting up the sync pipeline in Census UI, Census would call the connector service's APIs with batches of data to be imported or updated (upserted). This new feature enabled our customers to import data to Labelbox in an even easier and faster way to speed up their AI and ML pipelines.

When labeling PDF documents, a text layer is essential so labeling becomes even more accurate. At Labelbox, customers usually upload the text layer with the PDF assets. This requires them to generate this file independently, taking precious time and resources.

I took the initiative to work on this mini-project. At first, I evaluated different off-the-shelves solutions: AWS Textract, Google Document AI, and some Python libraries (pymupdf, extract, etc.). I picked Document AI at the end because of its ease of use and sync/async support, and I added image enhancements before running the OCR, which made processing scanned PDFs more reliable.

After an agreement with the tech lead engineer, I implemented this feature into our data ingestion pipeline, bulletproofing the solution with many automated tests. Ultimately, customers were not required to upload their text layer, speeding up their document labeling process.

I was part of the engineering team that completely rebuilt the data ingestion system to achieve ten times faster uploads, enhancing performance and efficiency. The data rows, including assets like images, videos, texts, and more, are the core of Labelbox. They were stored in MySQL, but the table didn't scale well, slowing down ingestion daily.

I created two new and separate microservices to handle data ingestion and stored the data rows in Spanner, which scales automatically infinitely. The old and new ingestion systems ran in parallel for months to ensure consistency, and then we switched to the new system with a flip of a switch, as pretty much no maintenance window was required.

This was a major breakthrough that enabled large enterprises to upload hundreds of millions of new data rows, also leading to an increase in revenues.

The old export system was often unreliable and unable to export data for longer intervals of time. It also exported all available (relevant) data, hence generating super-large outputs. The output format was also not structured well, making it complicated to add new features to it.

In the Exports V2 project, we first defined a public JSON schema of the exported data to ensure consistency across version changes. Then, we implemented a robust, scalable streaming export system that allowed granular control over the data to be exported. The cleverly designed export APIs allowed the front-end team to enhance the export experience across the Labelbox UI.

SophosLabs is the core team behind Sophos, which offers the best anti-malware protection, real-time web protection, mobile security, network security, and more for mid-sized companies. I worked on various microservices, refactored legacy systems, added systems monitoring with LogicMonitor metrics, served as a technical lead for various tasks and projects, and became a go-to person for any technical questions.

TIA Portal from Siemens is an engineering framework for creating, testing, and deploying robust automation systems, as well as simulation, diagnostics, and remote management. It is one of the largest C# projects in the world.