Neural networks explained

A neural network is a function with many adjustable numbers (weights). You show it examples, it measures how wrong it is, and it nudges the weights to be a little less wrong. Repeat this many times and the function becomes useful: it can recognise a face, translate a sentence or predict a price.

This lesson explains the idea with plain arithmetic and a few lines of NumPy. You need only school-level maths. In the next lesson, *Deep learning with PyTorch: your first model*, you will build and train one.

Think of a neuron as a tiny decision maker. It takes some numbers in, gives each input an importance (a weight), adds them up, adds a bias (a starting nudge) and passes the result through an activation function.

Example: should Brew & Bean's app suggest a hot drink? Inputs: temperature outside (in °C) and whether it is raining (1 or 0). A neuron might give a negative weight to temperature (hotter weather, fewer hot drinks) and a positive weight to rain.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

x = np.array([24.0, 1.0])        # 24 degrees C, raining
w = np.array([-0.2, 1.5])        # weights (importance of each input)
b = 3.0                          # bias

z = np.dot(w, x) + b             # weighted sum plus bias
a = sigmoid(z)                   # activation: squashes z into 0..1
print("z =", round(z, 2), " output =", round(a, 3))

Computes z = (-0.2 x 24) + (1.5 x 1) + 3 = -0.3, then squashes it with the sigmoid.

The output 0.426 can be read as a score between 0 and 1. Learning means finding weights and a bias that make these scores match the real answers in your data.

One neuron can only draw a straight line between two groups. A layer is many neurons that all look at the same inputs, each with its own weights. A network stacks layers: the outputs of one layer become the inputs of the next.

- Input layer: your features (numbers). - Hidden layers: the layers in between. They build up useful intermediate features. - Output layer: the prediction (one number for regression, one score per class for classification).

Deep learning just means networks with many hidden layers.

import numpy as np

rng = np.random.default_rng(0)

def relu(z):
    return np.maximum(0, z)

x = np.array([24.0, 1.0])              # 2 inputs

W1 = rng.normal(size=(2, 4))           # layer 1: 2 inputs -> 4 neurons
b1 = np.zeros(4)
W2 = rng.normal(size=(4, 1))           # layer 2: 4 neurons -> 1 output
b2 = np.zeros(1)

h = relu(x @ W1 + b1)                  # hidden layer
y = h @ W2 + b2                        # output layer
print("hidden:", h.round(2))
print("output:", y.round(2))

A forward pass through a 2-4-1 network with random weights. The exact numbers depend on the random weights, but hidden has 4 values and output has 1. Untrained, the output is meaningless.

Count the learnable numbers: layer 1 has 2x4 weights plus 4 biases, layer 2 has 4x1 weights plus 1 bias, so 17 parameters. Large language models have billions. The idea is the same.

If you stack layers without an activation function, the whole network collapses into one straight-line formula, however deep it is. The activation adds a bend. Bends let the network fit curves and complicated patterns.

Training is a loop with four ideas:

1. Forward pass: feed inputs through the network to get predictions.

2. Loss: one number that says how wrong the predictions are. For prices, mean squared error; for classes, cross-entropy.

3. Backpropagation: work out, for every weight, how the loss would change if that weight went up a little. This is the gradient. It uses the chain rule from calculus, applied layer by layer backwards. Libraries like PyTorch do this for you automatically.

4. Gradient descent: move every weight a small step in the direction that reduces the loss. The step size is the learning rate.

The analogy is walking down a foggy hill. You cannot see the valley, but you can feel which way the ground slopes under your feet, so you take a small step downhill, and repeat.

# Gradient descent on the simplest possible model: y = w * x
# True relationship is y = 3 * x. Can we discover w = 3?
import numpy as np

x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([3.0, 6.0, 9.0, 12.0])

w = 0.0                  # start with a bad guess
lr = 0.02                # learning rate

for step in range(50):
    pred = w * x
    loss = np.mean((pred - y) ** 2)            # mean squared error
    grad = np.mean(2 * (pred - y) * x)         # d loss / d w
    w = w - lr * grad                          # step downhill
    if step % 10 == 0:
        print(f"step {step:2d}  w = {w:.3f}  loss = {loss:.3f}")

print("final w:", round(w, 3))

Prints w moving from 0 towards 3 while the loss shrinks. The final w is very close to 3.0.

This is exactly what PyTorch does, but for millions of weights at once and with the gradients calculated automatically.

You will meet CNNs in *Computer vision with CNNs and transfer learning* and transformers in *NLP and transformers with Hugging Face*.

Neural networks shine on unstructured data (images, audio, text) and when you have lots of data. For a normal spreadsheet with a few thousand rows, gradient boosting or random forests (see *Decision trees, random forests and gradient boosting*) are usually as good or better, faster to train and easier to explain. Start simple, and move to deep learning only when the simple model is not enough.