When you feed data into a neural network, it first undergoes a Forward Pass. This is the process where your input data travels from the input layer, through all the hidden layers, and finally reaches the output layer. At each step, neurons perform calculations (weighted sum of inputs + bias, followed by an activation function) to transform the data. The ultimate goal of the forward pass is to generate a prediction or an output based on the network's current set of weights and biases. Think of it as the network making its best guess given what it currently "knows."
Once the forward pass provides a prediction, we compare it against the actual desired output (the "ground truth"). The difference between the prediction and the ground truth is quantified by a Loss Function, which tells us how "wrong" the network's guess was. To make the network learn and improve its predictions, we then perform a Backward Pass, also known as backpropagation. During this phase, the error (loss) is propagated backward through the network, from the output layer to the input layer. Using calculus (specifically, the chain rule), the backward pass calculates the gradient of the loss with respect to each weight and bias in the network, effectively determining how much each parameter contributed to the error.
The calculated gradients from the backward pass are crucial because they indicate the direction and magnitude by which each weight and bias needs to be adjusted to reduce the loss. An optimizer (like Stochastic Gradient Descent) then uses these gradients to update the network's parameters. This entire cycle – a forward pass to predict, calculate loss, and a backward pass to adjust weights – is repeated thousands or millions of times over many "epochs" and "batches" of data. This iterative process is how a neural network continuously learns, gradually improving its ability to make accurate predictions by minimizing its errors.
Key Takeaways
- Forward Pass: The process of feeding input data through the network to generate a prediction.
- Backward Pass (Backpropagation): The process of propagating the error backward to calculate gradients for each weight and bias.
- Gradients: Essential for determining how to adjust network parameters to reduce prediction error.
- Learning Cycle: Forward and backward passes iteratively train the network, minimizing loss over time.
Code Example
import numpy as np
# Simplified single-neuron example
x_input = 2.0
y_true = 4.0 # Target output
# Initial random weight and bias
weight = 0.5
bias = 0.1
learning_rate = 0.01
# === Forward Pass ===
y_pred = x_input * weight + bias
loss = (y_true - y_pred)**2
print(f"Input: {x_input}, True Y: {y_true}")
print(f"Predicted Y (Forward Pass): {y_pred:.2f}")
print(f"Loss: {loss:.2f}")
# === Backward Pass (Conceptual Gradient Calculation) ===
# How much y_pred needs to change: d_loss/d_y_pred
error_gradient = 2 * (y_pred - y_true)
# How much weight contributed to error: d_loss/d_weight = (d_loss/d_y_pred) * (d_y_pred/d_weight)
grad_weight = error_gradient * x_input
# How much bias contributed to error: d_loss/d_bias = (d_loss/d_y_pred) * (d_y_pred/d_bias)
grad_bias = error_gradient * 1
# === Parameter Update ===
new_weight = weight - learning_rate * grad_weight
new_bias = bias - learning_rate * grad_bias
print(f"Gradient W: {grad_weight:.2f}, Gradient B: {grad_bias:.2f}")
print(f"Updated W: {new_weight:.2f}, Updated B: {new_bias:.2f}")How this code works
This code demonstrates a single training step for a basic neural network: performing a "forward pass" to make a prediction and measure its error, then a "backward pass" to adjust its internal parameters (weight and bias) to reduce that error. It begins by setting up an x_input, a target y_true, and initial guesses for weight and bias, along with a learning_rate to control adjustment size. The "Forward Pass" section computes y_pred using x_input * weight + bias, similar to a linear equation. Subsequently, it calculates the loss using the squared difference between y_true and y_pred, quantifying how far off the prediction was.
The "Backward Pass" then conceptually calculates gradients, which indicate the direction and magnitude to change weight and bias to decrease the loss. error_gradient first determines how y_pred needs to change. This gradient is then propagated backward: grad_weight incorporates x_input to reflect its influence, while grad_bias is simply error_gradient multiplied by 1, as bias directly shifts the output without being scaled by input. A subtle point for beginners is that d_y_pred/d_bias is 1 because bias is an additive term, directly impacting y_pred one-to-one. Finally, the "Parameter Update" section adjusts new_weight and new_bias by subtracting learning_rate times their respective gradients, moving them closer to optimal values for the next learning step.