The Mysterious AI Reading List: Ilya Sutskever's Recommendations

There's a mysterious list of research papers that Ilya Sutskever reportedly gave to John Carmack in 2020. While everyone talks about it, no one has ever seen it. Here’s the story, an update on it, and the purported list →

John Carmack, the renowned game developer, rocket engineer, and VR visionary, shared in an interview that he asked Ilya Sutskever, OpenAI co-founder and former Chief Scientist, for a reading list about AI. Ilya responded with a list of approximately 40 research papers, saying:

This elusive list became a topic of search and discussion, amassing 131 comments on Ask HN. So many people wanted it that Carmack posted on Twitter, expressing his hope that Ilya would make it public and noting that “a canonical list of references from a leading figure would be appreciated by many”:

We agree. However, Ilya has yet to publish such a list, leaving us to speculate. Recently, an OpenAI researcher reignited the conversation by claiming to have compiled this list, and the post went viral.

Here’s what was inside (grouped for your convenience)

Core Neural Network Innovations

1. Recurrent Neural Network Regularization - Enhancement to LSTM units for better overfitting prevention.

2. Pointer Networks - Novel architecture for solving problems with discrete token outputs.

3. Deep Residual Learning for Image Recognition - Improvements for training very deep networks through residual learning.

4. Identity Mappings in Deep Residual Networks - Enhancements to deep residual networks through identity mappings.

5. Neural Turing Machines - Combining neural networks with external memory resources for enhanced algorithmic tasks.

6. Attention Is All You Need - Introducing the Transformer architecture solely based on attention mechanisms.