Artificial Intelligence Developer in San Francisco, CA, United States

We introduced Codex, a generative pre-trained transformer (GPT) language model fine-tuned on publicly available code from GitHub, and studied its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot.

On HumanEval, a new evaluation set we released to measure functional correctness for synthesizing programs from docstrings, our model solved 28.8% of the problems, while GPT-3 solved 0% and GPT-J solved 11.4%. Furthermore, we found that repeated sampling from the model was a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solved 70.2% of our problems with 100 samples per problem.

Careful investigation of our model revealed its limitations, including difficulty with docstrings describing long chains of operations and binding operations to variables.

Finally, we discussed the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

I codeveloped the first Copilot extension at OpenAI.

Deep reinforcement learning has had many significant successes, including a superhuman performance at Dota and Go. However, there are several challenges ahead if we want to apply it in the real world, including sample efficiency, task specification, and exploration. We believe addressing these challenges will require an open-world environment and human data.

To spur research on open-world environments with human data, we released MineRL, a suite of environments within Minecraft, alongside a large-scale dataset of human gameplay within those environments.

Besides the challenges discussed above, these environments also highlight a variety of other research challenges, including open-world multi-agent interactions, long-term planning, vision, control, navigation, and explicit and implicit subtask hierarchies. We also released a flexible framework to define new Minecraft tasks.

GhostWrite is an AI-based email automation solution built on large-language models. Aimed at automating away the inbox, this tool enables users to write emails instantly with just a few instructions. The application comprises a central web service and distributed web-browser extensions that augment the user's webmail client functionality.

The learnability of different neural architectures can be characterized directly by computable measures of data complexity.

In this paper, we reframed the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data beyond inductive bias. After suggesting algebraic topology as a measure for data complexity, we showed that the power of a network to express the topological complexity of a dataset in its decision region is a strictly limiting factor in its ability to generalize. We then provided the first empirical characterization of the topological capacity of neural networks.

Our empirical analysis showed that neural networks exhibit topological phase transitions at every level of dataset complexity. This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for fully-connected neural networks.

In this line of work, I answered the open question of universal approximation of nonlinear operators F: X -> Y when X and Y are both infinite-dimensional.

We showed that for a large class of different infinite analogues of neural networks, any continuous map can be approximated arbitrarily closely with some mild topological conditions on X. Additionally, we provided the first lower-bound on the minimal number of input and output units required by a finite approximation to an infinite neural network to guarantee that it can uniformly approximate any nonlinear operator using samples from its inputs and outputs.