Yanchuan Sim

Get instantaneous responses to clinical questions on drugs, diseases, guidelines, and medical calculators. Bot M.D. also helps to transcribe your dictated case notes. He knows 50+ languages.

Bot MD is created by me and my team.

I wrote an open source software for extracting text from binary documents scalably using AWS Lambda. With AWS Lambda's serverless architecture, we can perform OCR on hundreds of pages of text within minutes.

Some key features of the software are:

• Out-of-the-box support for many common binary document formats
• Scalable PDF parsing using OCR in parallel using AWS Lambda and asyncio
• Creation of text searchable PDFs after OCR
• Serverless architecture makes deployment quick and easy
• Detailed instruction for preparing libraries and dependencies necessary for processing binary documents
• Sensible Unicode handling

We built a state of the art 3-class (positive, neutral, negative) sentiment classifier for tweets. We performed in-depth analysis to evaluate the performance of the system, including incorporating crowdsourcing to achieve the best performance gains with least cost. The system is exposed using JSON and REST API.

We built a system for automatically extracting relevant soundbites from text corpora using a hybrid of multiple frequency measures—TF-IDF, PMI, SAGE, WAPMI

We implemented a system for automatically scoring and ranking named entities within an article by the salience. A "salient" entity is one that is highly relevant to the document and is the main entity that is being discussed. These features are incorporated into Elasticsearch to improve the relevance scoring algorithm.

Ark-SAGE is a Java library that implements the L1-regularized version of Sparse Additive GenerativE models of Text (Einsenstein et al, 2011). SAGE is an algorithm for learning sparse representations of text (you can read more about it here).

In NLP, entity linking is the task of determining the identity of entities mentioned in text. For example, given the sentence "Paris is the capital of France," the idea is to determine that "Paris" refers to the city of Paris and not to Paris Hilton or any other entity that could be referred as "Paris." Hence, EL maps mentions in the documents to entries in a knowledge base through resolving the name variations and ambiguities. In this project, we proposed 3 advancements for entity linking.

1. Expanding acronyms can effectively reduce the ambiguity of the acronym mentions. Rule-based approaches rely heavily on the presence of text markers. Here, we propose a supervised learning algorithm to expand more complicated acronyms which leads to 15.1% accuracy improvement over state-of-the-art acronym expansion methods.
2. Entity linking annotation is expensive and labor intensive. We propose an instance selection strategy to effectively utilize the automatically generated annotation. In our selection strategy, an informative and diverse set of instances are selected for effective disambiguation.
3. Topic modeling is used to model the semantic topics of the articles, significantly improving entity linking performance.

We present a joint probabilistic model of who cites whom in computational linguistics, and also of the words they use to do the citing. The model reveals latent factions, or groups of individuals whom we expect to collaborate more closely within their faction, cite within the faction using language distinct from citation outside the faction, and be largely understandable through the language used when cited from without. We conduct an exploratory data analysis on the ACL Anthology. We extend the model to reveal changes in some authors’ faction memberships over time.

We explore Debatepedia, a community authored encyclopedia of socio-political debates, as evidence for inferring a low dimensional, human-interpretable representation in the domain of issues and positions. We introduce a generative model positing latent topics and cross-cutting positions that gives special treatment to person mentions and opinion words. We evaluate the resulting representation's usefulness in attaching opinionated documents to arguments and its consistency with human judgements about positions.

We seek to measure political candidates’ ideological positioning from their speeches. To accomplish this, we infer ideological cues from a corpus of political writings annotated with known ideologies. We then represent the speeches of U.S. presidential candidates as sequences of cues and lags (filler distinguished only by its length in words). We apply a domain-informed Bayesian HMM to infer the proportions of ideologies each candidate uses in each campaign. The results are validated against a set of preregistered, domain expert authored hypotheses.

We improved the state of the art on a Supreme Court vote prediction task.

We explored the idea that authoring a piece of text is an act of maximizing one’s expected utility. To make this idea concrete, we consider the societally important decisions of the Supreme Court of the United States. Extensive past work in quantitative political science provides a framework for empirically modeling the decisions of justices and how they relate to text. We incorporate into such a model texts authored by amici curiae (“friends of the court” separate from the litigants) who seek to weigh in on the decision, then explicitly model their goals in a random utility model. We demonstrate the benefits of this approach in improved vote prediction and the ability to perform counterfactual analysis.

Online debate forums are important social media for people to voice their opinions and debate with each other. Mining user stances or viewpoints from these forums has been a popular research topic. However, most current work does not address an important problem: For a specific issue, there may not be many users participating and expressing their opinions. Despite the sparsity of user stances, users may provide rich side information—for example, users may write arguments to back up their stances, interact with each other, and provide biographical information. In this work, we propose an integrated model to leverage side information. Our proposed method is a regression-based latent factor model which jointly models user arguments, interactions, and attributes. Our method can perform stance prediction for both warm-start and cold-start users. We demonstrate in experiments that our method has promising results on both micro-level and macro-level stance prediction.

Authoring a scientific paper is a complex process involving many decisions. We introduce a probabilistic model of some of the important aspects of that process: that authors have individual preferences, that writing a paper requires trading off among the preferences of authors as well as extrinsic rewards in the form of community response to their papers, that preferences (of individuals and the community) and tradeoffs vary over time. Variants of our model lead to improved predictive accuracy of citations given texts and texts given authors. Further, our model’s posterior suggests an interesting relationship between seniority and author choices.

This is an assortment of utilities/functions that I have written and found useful for data manipulation and NLP. It is written mostly in C++, along with a mish-mash of scripts, libraries, modules, headers, etc., written in Python and Java.