Your Neural Network Is Just Matrix Multiply and Regret

I finally trained a small neural network from scratch — no framework, just NumPy — on a rainy weekend mostly to settle an argument with myself about whether I actually understood what was happening inside the models I’d been running on a spare GPU for months. Watching the loss number tick down, epoch after epoch, from a wildly wrong prediction toward a merely mediocre one, the mystique evaporated fast. There’s no understanding in there, no representation of meaning in any sense a person would recognise. There’s a stack of matrices, a number describing how wrong the current output was, and an algorithm nudging every weight a tiny amount in the direction that made that number slightly smaller last time. Repeat that a few million times and you get something that looks eerily intelligent from the outside while being, underneath, a genuinely mechanical process you can trace by hand on paper.

That gap between the outputs and the mechanism is exactly where most of the confusion around AI comes from — both the overselling (“it understands language”) and the dismissiveness (“it’s just autocomplete”). Both framings skip past the actual mechanism, which is more interesting and more limited than either. Once you’ve traced the arithmetic through by hand, at least once, you stop needing either metaphor.

What a single neuron actually computes

Forget layers and networks for a moment. One artificial neuron computes exactly this:

1
2
3
4
5

import numpy as np

def neuron(inputs, weights, bias):
    z = np.dot(inputs, weights) + bias   # weighted sum plus a bias term
    return max(0, z)                      # ReLU activation: clip negatives to zero

That’s the entire unit. Multiply each input by a learned weight, sum them, add a learned bias, then pass the result through a simple nonlinear function — ReLU here, which just zeroes out negative values. A “layer” is a bunch of these computed in parallel, which is exactly why frameworks express it as one matrix multiplication instead of a loop of individual neurons:

1
2

def layer(inputs, weight_matrix, biases):
    return np.maximum(0, inputs @ weight_matrix + biases)

inputs @ weight_matrix is the matrix multiply doing every neuron’s weighted sum simultaneously. Stack several of these layers, feeding each one’s output into the next one’s input, and you have a “deep” network — deep meaning nothing more mystical than “more than a couple of these layers stacked in sequence.”

Forward pass: guessing, badly, on purpose

Running an input through the whole stack of layers to produce an output is the forward pass, and early in training it produces near-random garbage, because the weights start as small random numbers:

1
2
3
4
5
6
7
8
9

np.random.seed(0)
W1 = np.random.randn(4, 8) * 0.1
b1 = np.zeros(8)
W2 = np.random.randn(8, 1) * 0.1
b2 = np.zeros(1)

x = np.array([0.2, 0.9, 0.1, 0.5])
h = np.maximum(0, x @ W1 + b1)   # hidden layer
y_pred = h @ W2 + b2             # output

y_pred at this point has no relationship to anything meaningful — it’s an artefact of whatever random numbers np.random.randn happened to produce. The entire remainder of training is the process of making that number less arbitrary, one gradient step at a time.

The loss function: turning “wrong” into a number you can differentiate

Training needs a single scalar measuring how wrong y_pred was compared to the actual target y_true. Mean squared error is the simplest version:

1
2

def mse_loss(y_pred, y_true):
    return np.mean((y_pred - y_true) ** 2)

That single number is the entire feedback signal the network gets. It has no concept of “the model misunderstood the sentence” or “the classifier confused a cat for a dog” — it has a number, and the number is currently 4.7, and the entire rest of the training loop exists to make that number smaller.

Backpropagation: assigning blame, layer by layer, with the chain rule

This is the part that sounds most like magic and is, mechanically, the least mysterious step in the whole pipeline: the chain rule from first-year calculus, applied repeatedly. Every weight in the network contributed something to y_pred, and therefore contributed something to how wrong the loss was. Backpropagation computes exactly how much each individual weight is to blame, by working backwards from the loss through each layer:

1
2
3
4
5
6
7
8
9

# gradient of loss with respect to the output layer's weights
d_loss_d_ypred = 2 * (y_pred - y_true) / y_true.size
d_ypred_d_W2 = h
grad_W2 = d_ypred_d_W2.T @ d_loss_d_ypred

# gradient flows backward into the hidden layer through W2
d_loss_d_h = d_loss_d_ypred @ W2.T
d_h_d_z1 = (h > 0).astype(float)   # derivative of ReLU
grad_W1 = x.reshape(-1, 1) @ (d_loss_d_h * d_h_d_z1).reshape(1, -1)

Every line here is an application of the chain rule: “how much does the loss change if this specific weight changes by a tiny amount.” Nothing in this computation involves meaning, comprehension or reasoning about the input — it’s calculus, distributed across every parameter in the network, computed mechanically and identically every single time.

Gradient descent: the regret part

Once you know how much each weight is to blame, the update rule is almost anticlimactically simple:

1
2
3

learning_rate = 0.01
W2 -= learning_rate * grad_W2
W1 -= learning_rate * grad_W1

Nudge every weight a small step in the direction that reduces the loss. That’s it. That’s the entire “learning” in machine learning — a weighted average of blame, applied as a small correction, repeated across millions of examples until the loss settles somewhere low. There is no point in this loop where the network “realises” anything. Every single step is the same arithmetic: forward pass, compute loss, backpropagate blame, nudge weights, repeat. The “regret” in the title is doing real work here — gradient descent is, quite literally, an automated process of taking a small step away from whatever the last mistake was, forever, without ever fully arriving.

Why this mechanism explains what LLMs actually are, and aren’t

A large language model is this exact loop scaled to billions of parameters and trained on a next-token-prediction loss instead of mean squared error, with a transformer’s attention mechanism replacing the plain matrix layers above to let each token’s representation attend to every other token in context. None of that scaling introduces a fundamentally different mechanism. What an LLM’s context window actually costs you and what a token actually is when a model reads, reasons and bills you for it are both direct consequences of this same forward-pass-and-loss architecture, just at a scale that makes the arithmetic impossible to trace by hand the way the toy example above is.

This is also the honest answer to why these models confidently produce wrong answers: there’s no internal check for truth anywhere in the loop above, only a check for “does this reduce the loss against training data.” A model that’s never seen a fact has no mechanism for knowing it doesn’t know — it just produces whatever the trained weights compute, confidently, because confidence was never part of what the loss function measured. RAG explains why retrieval helps a model stop making things up precisely because it works around this gap rather than fixing it: it supplies the facts externally instead of hoping gradient descent memorised them correctly.

Why nonlinearity is the one ingredient that can’t be skipped

The ReLU step exists for a reason that’s easy to skip past, and skipping it reveals something genuinely load-bearing rather than cosmetic. Stack several plain linear layers — matrix multiply followed by matrix multiply, with no nonlinear function between them — and the composition collapses mathematically into a single linear transformation, since multiplying matrices together is itself just another matrix. A hundred linear layers stacked without nonlinearity between them compute exactly what one linear layer computes; the depth buys nothing.

1
2

# two "layers" with no nonlinearity between them
combined = W1 @ W2  # this single matrix does exactly what both layers do together

ReLU, sigmoid, tanh and their modern variants (GELU, SiLU) all exist purely to break that collapse — inserting a nonlinear function between layers is the only reason stacking layers produces something more expressive than a single matrix multiplication could. Every architectural innovation in deep learning since the 1980s, transformers included, is downstream of this one requirement: without a nonlinearity somewhere in the stack, “deep” is a meaningless word, because the whole network reduces algebraically to a shallow one.

Epochs, batches and why training takes hours instead of one pass

The toy example above ran one input through the network once. Real training runs the entire dataset through repeatedly — each full pass is an epoch — and rarely one example at a time. Instead, examples are grouped into batches, and the gradient is averaged across the whole batch before a single weight update happens:

1
2
3
4
5
6
7
8

batch_size = 32
for epoch in range(num_epochs):
    for batch_x, batch_y in get_batches(training_data, batch_size):
        y_pred = forward(batch_x)
        loss = mse_loss(y_pred, batch_y)
        grads = backprop(loss)
        W1 -= learning_rate * grads["W1"]
        W2 -= learning_rate * grads["W2"]

Batching exists mainly for hardware efficiency — a GPU computing one matrix multiply across 32 examples simultaneously is dramatically faster per example than 32 separate smaller multiplications — and secondarily because averaging the gradient across several examples produces a less noisy, more stable update direction than any single example would on its own. Training “for ten epochs” simply means running this entire loop over the full dataset ten times, watching the loss curve to decide when further passes stop helping.

Why GPUs matter: the arithmetic is embarrassingly parallel

None of the matrix multiplications above have any sequential dependency between the individual multiply-add operations inside them — every element of the output matrix can be computed independently of every other element. That property, embarrassingly parallel arithmetic, is the entire reason GPUs dominate this workload: a GPU is thousands of simple arithmetic cores built for exactly this shape of problem, where a CPU’s handful of powerful, sequentially-optimised cores are comparatively wasted on it. Running local LLM inference on a spare GPU is faster than CPU inference for the identical structural reason training is faster on a GPU: the operation at the heart of both is the same matrix multiply, and matrix multiply is exactly the shape of problem a GPU’s architecture was built to devour.

Troubleshooting: debugging training the way you’d debug any numerical code

• Loss goes to NaN partway through training. Almost always exploding gradients — the learning rate is too high, or the weight initialisation is too large, and repeated multiplication through many layers blows the numbers up. Lower the learning rate, or clip gradients to a maximum magnitude before applying the update.
• Loss stops decreasing early and plateaus. Check whether ReLU units have “died” — a weight update that pushes a neuron’s output permanently negative means its gradient is permanently zero from then on, since ReLU’s derivative is zero for negative inputs, so that neuron never updates again. Leaky ReLU or a smaller learning rate on early layers both address this directly.
• Training loss looks great, real-world performance doesn’t. This is overfitting: the network has adjusted its weights to the specific noise in the training examples rather than the general pattern behind them. A validation set that isn’t used for gradient updates is the only reliable way to catch this before deploying.
• Two runs with identical code produce different results. Random weight initialisation and, on GPU, non-deterministic parallel reduction order both introduce genuine randomness. Seeding numpy, your framework’s RNG, and setting deterministic CUDA flags removes most of it, though bit-for-bit reproducibility across GPU architectures remains genuinely hard.

Is it worth reaching for

None of this arithmetic makes neural networks less useful — matrix multiplication at sufficient scale, trained against enough data with enough compute, produces genuinely capable systems, and pretending otherwise is its own kind of dishonesty. What tracing the mechanism by hand does buy you is calibration: knowing precisely which claims about “AI understanding” or “AI reasoning” describe an emergent, useful pattern in the weights, and which ones are marketing language borrowed from a mechanism the system never actually implements. The forward pass, the loss, the backward pass and the update rule are the whole machine. Everything else — attention, transformers, RLHF, fine-tuning — is refinement built on top of that same four-step loop, not a departure from it.