Neural Networks Explained Simply

Students arrive at this lesson braced for calculus. Then we show them the whole building block, a weighted sum followed by a simple "squash", and someone always says "that's it?" That's it. The network that recognises your face, transcribes Urdu speech or runs a large language model is billions of that block, stacked and trained with the loop from last lesson. We'll build one up from a single neuron, in runnable plain Python.

One neuron

An artificial neuron takes some inputs, multiplies each by a weight, adds them up with a bias, and pushes the total through an activation function. Nothing else.

Written out, y = f(w1·x1 + w2·x2 + w3·x3 + b). The weights say how much each input matters, the bias shifts the threshold, and the activation f adds a bend so the network can model curves instead of only straight lines. You'll meet three activations over and over.

Activation	Formula	Used for
Sigmoid	1 / (1 + e^-z), outputs 0–1	Probabilities in the output layer
ReLU	max(0, z)	Hidden layers; simple and fast
Softmax	Turns a list of scores into probabilities that sum to 1	Multi-class outputs (which of 10 categories)

Layers

One neuron can draw one straight boundary, and that's all. Put several side by side and you have a layer. Stack layers and each one builds more abstract features out of what the layer below hands it. In a fraud model the input layer holds the raw features (amount, hour, city), a hidden layer might learn a combination like "large amount, at night, from a new city", and the output layer turns that into a probability.

"Deep" just means many hidden layers. A modern LLM has around a hundred of them and billions of weights, but the sum inside each neuron is the one you saw above.

Why does depth help? Because each layer can only combine what the previous one gives it. The first hidden layer turns raw inputs into simple features, the second turns those into more abstract ones, and so on up. A shallow, very wide network can in theory represent the same functions, but depth learns them with far fewer weights and far less data, because low-level features get built once and reused by everything above.

How it learns: backpropagation

Training is the loop from last lesson, with one extra trick. After measuring the loss at the output, backpropagation works backwards through the layers, figures out how much each weight contributed to the error, and nudges every one a little. Underneath that's the chain rule from calculus, but PyTorch does it for you. Understand the idea. Nobody will ask you to do the algebra, not even in an interview.

A neuron that learns AND

Standard-library Python, nothing to install. We train a single sigmoid neuron to output 1 only when both inputs are 1.

import math, random

random.seed(1)
data = [((0, 0), 0), ((0, 1), 0), ((1, 0), 0), ((1, 1), 1)]   # logical AND
w1, w2, b = random.random(), random.random(), random.random()
lr = 0.5

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for epoch in range(3000):
    for (x1, x2), target in data:
        y = sigmoid(w1 * x1 + w2 * x2 + b)        # forward pass
        d = (y - target) * y * (1 - y)             # backprop through sigmoid
        w1 -= lr * d * x1                          # update weights
        w2 -= lr * d * x2
        b  -= lr * d

for (x1, x2), target in data:
    y = sigmoid(w1 * x1 + w2 * x2 + b)
    print(f"{x1} AND {x2} -> {y:.2f} (target {target})")
print(f"weights: w1={w1:.2f} w2={w2:.2f} b={b:.2f}")

After training, the neuron gives close to 0 for three cases and close to 1 for (1, 1). Look at the learned numbers. Both weights are positive and the bias is strongly negative, so the sum only crosses zero when both inputs are on. Nobody told it "both must be true". It found that from four examples.

Now change the targets to XOR (1 when the inputs differ). It can't do it, however long you train, because XOR isn't separable by one straight line. Add a hidden layer of two neurons and it becomes possible. That failure and that fix are the whole reason depth exists, and they held the field back for nearly twenty years.

Why they suddenly became practical

• Data. Networks are hungry, and the internet plus digitised business records finally fed them.
• Compute. GPUs multiply big matrices in parallel, which is exactly what a layer does.
• Tricks. ReLU, better initialisation, normalisation and dropout made deep training stable instead of a lottery.
• Pre-training. You rarely start from scratch. You download a model trained on general data and fine-tune it on yours.

A small network doing a big job at K-Electric. The demand-planning team needed hourly load forecasts for Karachi. A linear model caught the daily rhythm but missed the interactions, like the way a heatwave during Ramadan shifts the evening peak. A modest network with two hidden layers, fed hour, day, temperature, humidity and a Ramadan flag, learned those interactions on its own and cut the forecast error substantially. It runs on an ordinary server. Deep learning doesn't have to mean huge, and in our experience the small ones ship.

Before you move on

• A neuron is a weighted sum plus a bias, pushed through an activation function.
• Layers stack neurons so the network can learn combinations and curves. "Deep" means many layers.
• Backpropagation carries the output error backwards to update every weight.
• Data, GPUs, training tricks and pre-training are what made all this practical.

Homework

1. Run the AND example, then change the data to OR. Do the learned weights make sense?
2. Try XOR with the single neuron and describe what happens to the outputs.
3. For a Rawalpindi school's exam-result predictor, list the inputs you would feed the input layer and what a hidden neuron might learn to detect.