Before deep learning became standard, training many layers at once was hard. Deep belief networks offered a way forward by stacking restricted Boltzmann machines, training each layer greedily, then fine-tuning the whole stack.

Each layer learns a representation of the layer below. The top layers capture abstract features. Because training happens layer by layer, gradients do not vanish as they did in earlier deep architectures.

Structure and training

• Stacked restricted Boltzmann machines
• Unsupervised pretraining layer by layer
• Supervised fine-tuning on labeled data
• Generative capabilities at the top

DBNs were influential in the mid-2000s and helped revive interest in deep architectures. Modern practice uses ReLU networks, batch normalization, and better optimizers, which train end-to-end without the greedy layerwise approach.