Geoffrey Hinton's AI Legacy

Exploring the Research, Mathematics, and Ideas That Shaped Modern Artificial Intelligence

An independent, open-access educational archive exploring Geoffrey Hinton's scientific contributions through original research papers, mathematical explanations, PyTorch implementations, experiments, and video lectures.

Turn on JavaScript to read each paper with its equations, figures, PyTorch code and video lectures.

Papers covered

Relaxation and Its Role in Vision (PhD Thesis) (1977, Hinton). Geoffrey E. Hinton, PhD thesis, University of Edinburgh, 1977 — keeps many competing interpretations alive as continuous supposition values, links them with numerical constraints, and uses parallel relaxation to settle on the best consistent interpretation.
A Learning Algorithm for Boltzmann Machines (1985, Ackley, Hinton, Sejnowski). David H. Ackley, Geoffrey E. Hinton & Terrence J. Sejnowski, 1985 — introduces the Boltzmann Machine and a general learning rule for shaping its internal generative model to match the statistics of observed data.
Learning Representations by Back-Propagating Errors (1986, Rumelhart, Hinton, Williams). David E. Rumelhart, Geoffrey E. Hinton & Ronald J. Williams, 1986 — introduces back-propagation, the algorithm that let multilayer networks train their own hidden units.
Distributed Representations (1986, Hinton, McClelland, Rumelhart). Geoffrey E. Hinton, James L. McClelland & David E. Rumelhart, 1986 — represents each concept as a pattern of activity over many shared units, and shows why this gives content-addressable memory, automatic generalization, efficient coarse codes and graceful degradation.
Learning Internal Representations by Error Propagation (1986, Rumelhart, Hinton, Williams). David E. Rumelhart, Geoffrey E. Hinton & Ronald J. Williams, 1986 — derives the generalized delta rule (back-propagation) in full, shows the hidden-unit representations it discovers on XOR, parity, symmetry, addition and T-C, and extends it to sigma-pi units and recurrent networks.
Adaptive Mixtures of Local Experts (1991, Jacobs, Jordan, Nowlan, Hinton). Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan & Geoffrey E. Hinton, 1991 — expert networks compete for training cases under a learned softmax gate, giving automatic task decomposition and a link between supervised learning and competitive learning.
How Neural Networks Learn from Experience (1992, Hinton). Geoffrey E. Hinton, Scientific American, September 1992 — explains back-propagation step by step, weighs it as a model of the brain, and recasts unsupervised learning as minimizing code cost plus reconstruction cost, with population codes as the route to hierarchical representations.
Autoencoders, Minimum Description Length and Helmholtz Free Energy (1994, Hinton, Zemel). Geoffrey E. Hinton & Richard S. Zemel, NIPS 6 (1994) — derives an autoencoder objective from Minimum Description Length, shows through the bits-back argument that it is the Helmholtz free energy, and uses a factorial recognition distribution as a tractable upper bound to learn factorial codes.
The Helmholtz Machine (1995, Dayan, Hinton, Neal, Zemel). Peter Dayan, Geoffrey E. Hinton, Radford M. Neal & Richard S. Zemel, 1995 — pairs a hierarchical generative network with a learned recognition network, trains both on a free-energy lower bound on the data likelihood, and introduces the wake-sleep algorithm.
The Wake-Sleep Algorithm for Unsupervised Neural Networks (1995, Hinton, Dayan, Frey, Neal). Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey & Radford M. Neal, Science 1995 — trains a multilayer stochastic network with two local phases: wake, where recognized data train the generative weights, and sleep, where generated fantasies train the recognition weights.
Products of Experts (1999, Hinton). Geoffrey E. Hinton, ICANN 1999 — combines probabilistic models by multiplying and renormalizing instead of mixing, trains them with the difference between derivatives on data and on one-step Gibbs reconstructions, and shows that a restricted Boltzmann machine is a product of experts.
Training Products of Experts by Minimizing Contrastive Divergence (2002, Hinton). Geoffrey E. Hinton, Neural Computation 2002 — replaces the likelihood with the contrastive divergence, whose gradient cancels the intractable equilibrium term, giving the one-step CD-1 rule that made restricted Boltzmann machines practical to train.
Stochastic Neighbor Embedding (2002, Hinton, Roweis). Geoffrey E. Hinton & Sam T. Roweis, NIPS 2002 — embeds objects in a low-dimensional space by matching neighbour probability distributions with KL divergence, calibrated by perplexity, and lets ambiguous objects have several images.
A Fast Learning Algorithm for Deep Belief Nets (2006, Hinton, Osindero, Teh). Geoffrey E. Hinton, Simon Osindero & Yee-Whye Teh, Neural Computation 2006 — uses complementary priors to show that a deep belief net can be learned one RBM at a time, fine-tunes it with the up-down algorithm, and beats discriminative methods on permutation-invariant MNIST.
Reducing the Dimensionality of Data with Neural Networks (2006, Hinton, Salakhutdinov). Geoffrey E. Hinton & Ruslan R. Salakhutdinov, Science 2006 — pretrains a stack of RBMs, unrolls it into a deep autoencoder and fine-tunes it with backpropagation, learning low-dimensional codes that far outperform PCA on images and text.
Restricted Boltzmann Machines for Collaborative Filtering (2007, Salakhutdinov, Mnih, Hinton). Ruslan Salakhutdinov, Andriy Mnih & Geoffrey Hinton, ICML 2007 — models each user's movie ratings with an RBM over softmax rating units, sharing weights across users, and shows on 100 million Netflix ratings that RBMs rival SVD and combine with it to beat Netflix's system by over 6%.
Visualizing Data using t-SNE (2008, van der Maaten, Hinton). Laurens van der Maaten & Geoffrey Hinton, JMLR 2008 — symmetrizes SNE and uses a heavy-tailed Student-t distribution in the map, solving the crowding problem and producing maps that reveal structure at many scales.
Deep Boltzmann Machines (2009, Salakhutdinov, Hinton). Ruslan Salakhutdinov & Geoffrey Hinton, AISTATS 2009 — trains Boltzmann machines with many hidden layers by combining mean-field inference for data statistics with persistent Markov chains for model statistics, pretrains them with modified RBMs, and evaluates them with annealed importance sampling.
Rectified Linear Units Improve Restricted Boltzmann Machines (2010, Nair, Hinton). Vinod Nair & Geoffrey E. Hinton, ICML 2010 — builds stepped sigmoid units from tied binary units with offset biases, approximates them with noisy rectified linear units, and shows they learn better features than binary units for object recognition and face verification.
Deep Neural Networks for Acoustic Modeling in Speech Recognition (2012, Hinton, Deng, Yu). Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath & Brian Kingsbury, IEEE Signal Processing Magazine 2012 — the shared view of four research groups that deep neural networks, pre-trained as DBNs and fine-tuned to predict HMM states, outperform Gaussian mixture models for acoustic modeling.
ImageNet Classification with Deep Convolutional Neural Networks (2012, Krizhevsky, Sutskever, Hinton). Alex Krizhevsky, Ilya Sutskever & Geoffrey E. Hinton, NIPS 2012 — AlexNet: an eight-layer, 60-million-parameter CNN trained with ReLUs, two GPUs, data augmentation and dropout that cut ILSVRC-2012 top-5 error to 15.3% against 26.2% for the runner-up.
On the Importance of Initialization and Momentum in Deep Learning (2013, Sutskever, Martens, Dahl, Hinton). Ilya Sutskever, James Martens, George Dahl & Geoffrey Hinton, 2013 — full breakdown coming soon. (coming soon)
Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014, Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov). Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever & Ruslan Salakhutdinov, 2014 — full breakdown coming soon. (coming soon)
Distilling the Knowledge in a Neural Network (2015, Hinton, Vinyals, Dean). Geoffrey Hinton, Oriol Vinyals & Jeff Dean, 2015 — full breakdown coming soon. (coming soon)
Layer Normalization (2016, Ba, Kiros, Hinton). Jimmy Lei Ba, Jamie Ryan Kiros & Geoffrey E. Hinton, 2016 — full breakdown coming soon. (coming soon)
Using Fast Weights to Attend to the Recent Past (2016, Ba, Hinton, Mnih, Leibo, Ionescu). Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo & Catalin Ionescu, 2016 — full breakdown coming soon. (coming soon)
Dynamic Routing Between Capsules (2017, Sabour, Frosst, Hinton). Sara Sabour, Nicholas Frosst & Geoffrey E. Hinton, 2017 — full breakdown coming soon. (coming soon)
Matrix Capsules with EM Routing (2018, Hinton, Sabour, Frosst). Geoffrey Hinton, Sara Sabour & Nicholas Frosst, 2018 — full breakdown coming soon. (coming soon)
How to Represent Part-Whole Hierarchies in a Neural Network (2021, Hinton). Geoffrey Hinton, 2021 — proposes GLOM, an imaginary architecture in which weight-shared columns hold an embedding per level and islands of identical vectors, formed by attention-based agreement, represent the nodes of an image's parse tree.
The Forward-Forward Algorithm: Some Preliminary Investigations (2022, Hinton). Geoffrey Hinton, 2022 — replaces backpropagation's forward and backward passes with two forward passes on positive and negative data, giving every layer its own goodness objective, and shows it works on small problems.

Interactive playgrounds

The Puppet-Finding Playground: Parallel LP relaxation in visual perception (from “Relaxation and Its Role in Vision (PhD Thesis)”)
Joint Zone Tester: Space occupancy and separating borders (from “Relaxation and Its Role in Vision (PhD Thesis)”)
Hyperspace Vector & Polytope Explorer: Continuous relaxation in linear programming (from “Relaxation and Its Role in Vision (PhD Thesis)”)
Gomory Cutting Plane & Fractional Vertex Slicer: Integer linear programming in perception (from “Relaxation and Its Role in Vision (PhD Thesis)”)
Master Relaxation Equation Dynamics Lab: Numerical stability and phase transitions (from “Relaxation and Its Role in Vision (PhD Thesis)”)
Generators & Lazy Search Expansion: Pruning combinatorial growth in relaxation (from “Relaxation and Its Role in Vision (PhD Thesis)”)
The Search Arena: Waltz filtering vs. the simplex method vs. parallel relaxation (from “Relaxation and Its Role in Vision (PhD Thesis)”)
SETTLE: Schema Demons & Gated Forward Chaining: Rules that only fire when relaxation trusts their premises (from “Relaxation and Its Role in Vision (PhD Thesis)”)
The Stochastic Neuron & Simulated Annealing Chamber: Escaping local minima (from “A Learning Algorithm for Boltzmann Machines”)
The Two-Phase Hebbian Machine: Learning from clamped and free-running statistics (from “A Learning Algorithm for Boltzmann Machines”)
The Communication Bottleneck: Solving the encoder problem with a Boltzmann machine (from “A Learning Algorithm for Boltzmann Machines”)
Code Repulsion & Territory Competition: How the learning rule pushes clashing codes apart in hidden space (from “A Learning Algorithm for Boltzmann Machines”)
Hinton Diagrams: The recursive weight matrix of a Boltzmann machine (from “A Learning Algorithm for Boltzmann Machines”)
The G-Divergence & Information Landscape: Kullback–Leibler optimization in a Boltzmann machine (from “A Learning Algorithm for Boltzmann Machines”)
Weight Divergence & the Noisy Clamping Lab: Why rare patterns demand infinite energies, and how noise prevents it (from “A Learning Algorithm for Boltzmann Machines”)
The Speed–Accuracy Trade-Off: Annealing speed, completion errors and reaction times in a Boltzmann machine (from “A Learning Algorithm for Boltzmann Machines”)
The Forward–Backward Chain Rule Lab: Back-propagation in a 2-2-1 network, one micro-step at a time (from “Learning Representations by Back-Propagating Errors”)
The Mirror Symmetry Resolver: Two hidden units, opposite weights, and the 1 : 2 : 4 cancellation (from “Learning Representations by Back-Propagating Errors”)
Family Trees & Distributed Representations: What a five-layer network learns from 104 kinship triples (from “Learning Representations by Back-Propagating Errors”)
Receptive Fields & Weight Decay: Reading the six person units of the family-tree network (from “Learning Representations by Back-Propagating Errors”)
Momentum & Ravine Dynamics: Changing the velocity of the point in weight space, not its position (from “Learning Representations by Back-Propagating Errors”)
The Unrolled Recurrent Net: An iterative net, the layered net it equals, and back-propagation through time (from “Learning Representations by Back-Propagating Errors”)
Local Minima & Extra Dimensions: Why a few more connections let gradient descent get around the barriers (from “Learning Representations by Back-Propagating Errors”)
Saturation & the Tolerance Window: Why the paper counted any output above 0.8 or below 0.2 as correct (from “Learning Representations by Back-Propagating Errors”)
Coarse Coding & the Hypercylinder: Why broad, overlapping receptive fields pinpoint a feature more accurately (from “Distributed Representations”)
The Binding Problem: Which x goes with which y, and what it costs to know (from “Distributed Representations”)
Two Pathways: Recognition and Generation: Bottom-up inference and a top-down world model, with different weights (from “The Helmholtz Machine”)

An independent educational project by Mohammed Fahd Abrah. Not affiliated with or endorsed by Geoffrey Hinton or the University of Toronto. All research is credited to Geoffrey Hinton and his collaborators.