Derek Thomas

Our team of four non-Arabic speakers created an Arabic/English translation engine that was quantitatively competitive with Google and outperformed Google qualitatively. This performance was especially notable because we used only open data. The reproducible pipeline handled 24 billion sentences.

Key Activities
• Handled all the data engineering and trained some models.
• Implemented a back-translation data generator and improved BLEU by 78% over the baseline.
• Developed a trie-based cleaning technique for scraped data.
• Used Bicleaner to detect noisy sentence pairs based on mutual translation likelihood.

This application was developed to aggregate and process social media data. The data was collected in AWS but processed on a private cloud. The processing consisted of data routing, running models, and ingestion in Elastic.

The routing was deployed in Docker containers that subscribed to AWS and worked in batches. They would then call the models. I used TorchServe to deploy the models because it's lightweight and easily configurable, allowing quick optimization for the computational capacity. The results were ingested into Elastic, which worked well because it's flexible and scales well. Kibana was a good choice for visualizations.

Relation extraction is a way to obtain the semantic relationship between entities in text. The state-of-the-art methods use linguistic tools to build a graph for the text in which the entities appear, then a graph convolutional network (GCN) is employed to encode the pre-built graphs.

Although their performance is promising, the reliance on linguistic tools results in a non-end-to-end process. In this work, we proposed a novel model, the self-determined graph convolutional network (SGCN), which determines a weighted graph using a self-attention mechanism rather than using any linguistic tool. Then, the self-determined graph is encoded using a GCN.

We tested our model on the TACRED dataset and achieved the state-of-the-art result. Our experiments show that SGCN outperforms the traditional GCN, which uses dependency parsing tools to build the graph.

I was the second author, and this work was accepted into CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management.

Document clustering requires a deep understanding of the complex structure of LONGTEXT; in particular, the intra-sentential (local) and inter-sentential features (global). Existing representation learning models do not fully capture these features. To address this, we presented a novel graph-based representation for document clustering that builds a graph autoencoder (GAE) on a keyword correlation graph.

The graph was constructed with topical keywords as nodes and multiple local and global features as edges. A GAE was employed to aggregate the two sets of features by learning a latent representation, which could jointly reconstruct them. Clustering was then performed on the learned representations, using vector dimensions as features for inducing document classes.

Extensive experiments on two datasets showed that the features learned by our approach could achieve better clustering performance than other existing features, including term frequency-inverse document frequency and average embedding.

I was the third author, and this work was accepted into ACL 2020.

Text classification requires a deep understanding of the linguistic features in text; in particular, the intra-sentential (local) and inter-sentential features (global). Models that operate on word sequences have been used successfully to capture the local features, yet they are not effective in capturing the global features in LONGTEXT.

We investigated graph-level extensions to such models and proposed a novel architecture for combining alternative text features. It used an attention mechanism to dynamically decide how much information to use from a sequence- or graph-level component. We evaluated different architectures on a range of text classification datasets, and graph-level extensions were found to improve performance on most benchmarks. In addition, the attention-based architecture adaptively learned from the data outperforms the generic and fixed-value concatenation ones.

I was the fourth author, and this work was accepted into SIGIR 2020.