Activation Functions Explorer

Activation functions are the nonlinearity that lets neural networks learn anything beyond straight lines. Pick a function, move the input slider, and watch both the function and its derivative update live.

Function Plot ReLU

f(x) — function value f′(x) — derivative (slope) tangent line at the selected x

Controls

Function

α0.10

Input x0.00

f(x) = max(0, x)

The derivative tells gradient descent which direction to nudge the weights. Where the slope is 0, learning stops.

Values at the selected input

f(x)

—

f′(x)

—

Output range

—

Monotonic

—

How it works: Each neuron computes σ(W · x + b) — a weighted sum passed through an activation function. ReLU (Rectified Linear Unit) is the default in most modern networks: it keeps positive values and zeroes out negatives. The derivative is what backpropagation uses to decide how much each weight should change.

What's happening here? Think of an activation function as a gatekeeper on a pipe. ReLU is a one-way gate: water (signal) flows freely forward but never backward. Sigmoid is a valve that smoothly squeezes everything into 0–1 — useful for probabilities, but its slope flattens at the extremes, so learning slows down there (the "vanishing gradient" problem). Tanh is the same shape centered on zero. GELU and SiLU are modern smooth gates that keep a little signal flowing even for negative inputs. Try sliding x across the plot and watch the blue derivative curve — where it is flat, gradient descent has nothing to push on.