Pablo Mainar Jovaní

A full speech-to-text (STT) hybrid I developed between cloud and on-device processing.

A controller was designed to differentiate between simple voice commands, such as "play the next song," and complex commands, such as "play something from the Stones." Simple commands were processed on-device to improve availability, privacy, and battery life, while complex ones were sent to the cloud for more powerful processing.

I handled the technical development of the on-device STT—designing the model and training it on a limited vocabulary. I was also in charge of the controller deciding how to route the command. This required deep knowledge of speech processing, including the classical and more modern approaches for STT. The STT was developed in C++ for embedding in Raspberry Pi, and most of the experimentation was done in Python. A patent was filed based on this project.

A deep learning-based metric to measure the quality of speech.

Traditionally, quality has been measured by asking a group of experts and averaging their opinions. In this project, we developed an automatic way of doing it based on a deep-learning model. We used crowdsourcing to collect a large dataset that we used—among other publicly available data—to train our model. This model is now in use in the production line of many audio products.

I led the project and supervised the student working on it. This included writing code for training the model and suggesting ideas to improve the performance. The model was written in Python using PyTorch. We participated in a speech quality metric competition at a top-tier scientific conference, and the model ranked among the best. The scientific paper can be found in the attached project URL.

A deep neural model we developed to predict the state of flow based on wearable EEG.

Flow is a mental state where the user is very focused and productive. We developed a deep neural model to predict the state of flow based on sensors placed on the scalp that measured electrical brain activity, known as EEG. This involved a big experimental design considering many confounding variables, data collection, data analysis, and modeling. All the coding was done in Python, including the experimental design—with the synchronization of all sensors—and modeling, which was done using TensorFlow.

I led this project and was responsible for the experimental design and the modeling. I also supervised the student working on the project, providing insights and suggestions on how to proceed. This project was published at the IEEE EMBS conference in 2022. The paper can be found in the attached project URL.