Why does a neural network need an activation function?
Because linear layers compose into one linear layer. Each neuron in a neural network computes a weighted sum plus a bias; two layers of that multiply out to a single matrix. The activation bends the signal between layers, so the network can represent thresholds, curves and interactions. It must be differentiable almost everywhere, because training sends the gradient of the loss back through every activation on each gradient descent step. The shape of the activation decides how well gradients flow: sigmoid and tanh flatten at both ends, so in deep stacks their gradients shrink toward zero, the vanishing gradient problem that pushed the field toward ReLU.
What are the common activation functions?
| Function | Formula | Output | Typical use |
|---|---|---|---|
| ReLU | max(0, x) | Zero or positive | Hidden layers of feed-forward and convolutional networks |
| Sigmoid | 1 / (1 + e^-x) | Between 0 and 1 | A single yes or no probability |
| Tanh | (e^x - e^-x) / (e^x + e^-x) | Between -1 and 1 | Recurrent networks |
| GELU | x · Φ(x), Φ the normal CDF | Smooth, dips slightly below zero | BERT and GPT-style transformers |
| SwiGLU | Swish(xW) ⊗ xV | Unbounded, gated | Feed-forward layers of LLaMA |
GELU weights inputs by their value rather than gating them by sign as ReLU does (Hendrycks and Gimpel, 2016, read 5 November 2026). For a small network built from scratch, ReLU in the hidden layers is still the sensible default.
Which activation functions do LLMs use?
The activation sits in the feed-forward sublayer of every transformer block. The 2017 original used ReLU there (Attention Is All You Need, read 5 November 2026). BERT used “a gelu activation (Hendrycks and Gimpel, 2016) rather than the standard relu, following OpenAI GPT” (Devlin et al., 2018, read 5 November 2026). Shazeer tested gated variants, which use three weight matrices instead of two, and found “The GEGLU and SwiGLU variants produce the best perplexities” (GLU Variants Improve Transformer, February 2020, read 5 November 2026). LLaMA then replaced “the ReLU non-linearity by the SwiGLU activation function” (Touvron et al., February 2023, read 5 November 2026). In a modern LLM the activation is an architectural choice about the feed-forward layer, not a per-neuron detail, and in a mixture of experts model each expert is one such block.
How do you choose an activation function?
By position. The output layer follows the task: sigmoid for one probability, softmax across classes or tokens, none at all for a regression output. Hidden layers get ReLU in small networks, GELU or SwiGLU when you follow a transformer recipe. When you fine-tune or serve a pretrained model, its activation is already fixed: the weights were learned with that function, and swapping it means training again.