Learning and Backpropagation

This database covers the roots of modern neural-network learning: differentiable multi-layer training, early convolutional systems, sequence memory, and the deep-learning revival.

Core Learning Mechanisms

Year	Paper	Topic	Note
1986	Learning Representations by Back-Propagating Errors	Backpropagation	Practical procedure for training multi-layer networks by gradient propagation.
1989	Backpropagation Applied to Handwritten Zip Code Recognition	CNN training	Early successful CNN trained with backpropagation.
1997	Long Short-Term Memory	LSTM	Gated recurrent memory for long-range sequence dependencies.
1998	Gradient-Based Learning Applied to Document Recognition (PDF)	LeNet	End-to-end convolutional document recognition.

Deep Learning Revival

Year	Paper	Topic	Note
2006	A Fast Learning Algorithm for Deep Belief Nets	Deep belief nets	Layer-wise unsupervised pretraining revived interest in deep architectures.
2006	Reducing the Dimensionality of Data with Neural Networks	Deep autoencoders	Deep autoencoders for nonlinear dimensionality reduction.
2009	Learning Deep Architectures for AI	Deep learning survey	Early synthesis of representation learning and deep architectures.
2015	Deep Learning	Survey	High-level landmark review of deep learning.

Reading Path

Step	Read
1	Backpropagation and the 1989 zip-code CNN.
2	LeNet for end-to-end convolutional learning.
3	LSTM for gated sequence memory.
4	Deep belief nets, deep autoencoders, and the 2015 Nature review for the deep-learning revival.