During the training of deep neural networks, we use optimization algorithms like gradient descent to adjust weights. This involves calculating gradients – essentially, how much a change in a weight affects the network's output error. These gradients are propagated backward through the network (backpropagation) to update weights in each layer. "Vanishing" and "Exploding Gradients" are two critical problems that can severely disrupt this process, particularly in very deep architectures.
Vanishing gradients occur when the gradients become extremely small as they propagate backward through many layers. Imagine multiplying many small numbers together; the product quickly approaches zero. When gradients vanish, the weight updates for the initial layers become negligible. This means those layers learn extremely slowly, or stop learning altogether, preventing the network from capturing long-range dependencies or complex patterns. This issue was very prominent with activation functions like sigmoid and tanh, whose derivatives are very small over large ranges of their input.
Conversely, exploding gradients happen when gradients become excessively large. Instead of shrinking, the product of derivatives grows exponentially large during backpropagation. This leads to massive updates to the network's weights, causing training to become highly unstable. The model weights might oscillate wildly, diverge, or even result in NaN (Not a Number) values, making further training impossible. Exploding gradients are often seen in very deep networks, networks with poor weight initialization, or during long sequences in recurrent neural networks.
Key Takeaways
- Gradients are the engine of neural network learning; their instability halts progress.
- Vanishing gradients: Earlier layers learn too slowly or stop, hindering deep pattern recognition.
- Exploding gradients: Unstable training, extreme weight updates, potential
NaNvalues. - Both problems are amplified in very deep networks and can make training impossible.
- Solutions like ReLU activations, Batch Normalization, and Gradient Clipping directly address these issues.
Code Example
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def sigmoid_derivative(x):
s = sigmoid(x)
return s * (1 - s)
print("--- Demonstrating Vanishing Gradients ---")
gradient_from_loss = 1.0 # Assumed gradient from the subsequent layer or loss
# Simulate pre-activation values for 5 deep layers
# Values far from 0 lead to small sigmoid derivatives
pre_activation_inputs = [5.0, -6.0, 7.0, -8.0, 9.0]
for i, val in enumerate(pre_activation_inputs):
deriv_at_layer = sigmoid_derivative(val)
gradient_from_loss *= deriv_at_layer
print(f"Layer {len(pre_activation_inputs)-i}: d(sigmoid)/dx at {val:.1f} = {deriv_at_layer:.6f}, Current overall gradient: {gradient_from_loss:.9f}")
print("\nNotice how the gradient rapidly approaches zero.")How this code works
This code visually demonstrates the "vanishing gradient" problem, a challenge in training deep neural networks, especially when using the sigmoid activation function. It simulates how gradients diminish as they backpropagate through multiple layers. The script first defines helper functions for sigmoid and its sigmoid_derivative, which are fundamental for understanding how activation functions influence gradient flow. It then initializes a gradient_from_loss to 1.0, representing the initial gradient signal from the network's output or subsequent layers. A list of pre_activation_inputs is set up, simulating the raw inputs to the sigmoid function across five hypothetical deep layers.
The core demonstration happens within the for loop, simulating backpropagation. For each layer, the code calculates the sigmoid_derivative using the val from pre_activation_inputs. A subtle but crucial choice is that these pre_activation_inputs (like 5.0, -6.0) are large positive or negative values. This causes the sigmoid_derivative to be very small. This tiny derivative is then multiplied with the gradient_from_loss accumulated from previous layers. Due to this repeated multiplication of small numbers, the gradient_from_loss rapidly shrinks towards zero, as shown by the print statement for each layer. This rapid decrease highlights how earlier layers might receive an insignificant gradient signal, hindering their ability to learn effectively.