ANN — Sensei Learning Card

🎯 Core Concept

An ANN is a stack of layers of simple units (neurons). Each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation. Stacking these layers lets the network compose simple transformations into arbitrarily complex functions — it learns the weights by minimizing a loss via gradient descent and backpropagation. Mental model: a chain of adjustable dimmer switches. Each connection dims or brightens a signal; training turns the knobs until the output pattern matches reality.

z = Σ wᵢxᵢ + b → a = σ(z) · w := w − η ∂L/∂w

🔑 Key Components

Weights & Biases

The learnable parameters. Weights scale each input's influence; the bias shifts the activation threshold. Training = finding good values for these.

Activation Function

Non-linearity (ReLU, sigmoid, tanh) applied per neuron. Without it, stacked layers collapse into a single linear map — no expressive power.

Forward Propagation

Inputs flow layer-by-layer to produce a prediction. Each layer transforms the previous layer's activations into a new representation.

Backpropagation

The chain rule run backward: compute how the loss changes with each weight, then step opposite the gradient. This is how the network learns.

⚙️ How It Works

Initialize weights randomly and feed a batch of inputs through the network (forward pass).

At each neuron, compute z = Σwᵢxᵢ + b, then apply the activation σ(z) to get the output.

Compare the final output to the target using a loss function (e.g. cross-entropy, MSE).

Backpropagate: use the chain rule to compute ∂L/∂w for every weight.

Update weights: w := w − η·∂L/∂w. Repeat over many epochs until the loss converges.

🌎 Real-World Applications

Image classification Speech recognition Fraud detection Recommendation systems Medical diagnosis Tabular prediction Time-series forecasting

🧪 Checkpoint Questions

1. Why does a neural network need non-linear activation functions? What happens without them? Hint: compose two linear maps — what kind of function do you get? Think about the collapse of depth.

2. Backpropagation is often called "just the chain rule." Why is the chain rule the natural tool for computing gradients in a layered network? Hint: the loss depends on a weight only through a chain of intermediate activations — each layer's output feeds the next.

3. If your learning rate η is too large, training may diverge; too small and it crawls. Why does this single knob have such outsized effect? Hint: η sets the step size along the gradient — picture overshooting vs. inching down a valley.