Rumelhart, Hinton & Williams, Learning representations by back-propagating errors, Nature 1986 · fig. 4, p. 535

Receptive Fields & Weight DecayReading the six person units of the family-tree network

Receptive fields weights from the 24 people to each person unit · English row above, Italian twins below

Weight decay Δw(t) = −ε ∂E/∂w + α Δw(t−1), then every weight shrinks by λ

Across λ one stored run per point

■ training answers ranked right ■ feature clarity (mean |r| of units 1, 2, 6) ■ largest |w|, scaled

Decay against gradient each of the 144 person weights

Perturb the weights Gaussian noise on the 144 person weights