Gradients, loss functions, and gradient descent

From a simple derivative to parameter updates in a learning model.

Mathematics for AIIntermediate

Learning goal

Compute a gradient-descent step by hand and explain the roles of the gradient and learning rate.

Prerequisites

Single-variable derivatives
Basic vectors

From derivative to direction

A derivative measures local change. For f(x)=(x-3)^2, the derivative tells us which direction decreases or increases the function near a point.

Function and derivative

f (x) = {(x - 3)}^{2}, f' (x) = 2 (x - 3)

One numerical step

One-dimensional gradient descent

x_{t + 1} = x_{t} - η f' (x_{t})

If x₀=0 and η=0.1, then f'(0)=-6 and x₁=0.6. Moving against the derivative moves toward the minimum.

Many parameters

Multivariable update

θ_{t + 1} = θ_{t} - η \nabla J (θ_{t})

In a neural network, θ contains weights and biases and J is usually the loss. Backpropagation computes gradients and the optimizer uses them for updates.

Exercises

For f(x)=(x-5)^2, start at x=1 with η=0.2 and compute three steps.
Compare two learning rates.
Explain local versus global minimum for a non-convex function.

Related projects

HowDeepLearningWorks

Optimization mathematics behind deep learning.

Open source

HowLLMsWork

Where optimization fits into language-model training.