Rumelhart, Hinton & Williams, Learning representations by back-propagating errors, Nature 1986 · p. 535
Local Minima & Extra Dimensions
Why a few more connections let gradient descent get around the barriers
Run
hidden units H
ε
α
Perturb
Add a hidden unit here
sweep
0
· E
–
· |∇E|
–
A slice through weight space
range
Re-centre here
Reset view
drag to orbit · scroll to zoom · the sheet is E on a plane through the weights; the ball is the run
Many random starts
Every H from 1 to 8
This run
error over the sweeps