How does a neural network work?
Each neuron computes activation(w · x + b): a weighted sum of its inputs plus a bias, passed through a nonlinear function. Neurons are arranged in layers: an input layer that receives numbers, hidden layers that transform them, and an output layer that produces a score, a class probability or a vector. Data flows forward through the layers to make a prediction. Training compares that prediction with the label using a loss, then backpropagation and gradient descent adjust every weight to reduce it. Stack many hidden layers and you have deep learning. Everything the network knows lives in its weights; nobody writes its rules.
What does an activation function do in a neural network?
It supplies the nonlinearity. Without one, any stack of layers collapses into a single linear transformation, no more expressive than the linear models of classic machine learning.
| Activation | Formula | Typical use |
|---|---|---|
| ReLU | max(0, x) | Hidden layers; the long-standing default |
| Sigmoid | 1 / (1 + e^-x) | Output for a yes or no probability |
| Tanh | (e^x - e^-x) / (e^x + e^-x) | Hidden layers in older and recurrent networks |
| Softmax | e^xi / Σ e^xj | Output over several classes or tokens |
Smooth ReLU variants such as GELU are common inside transformers. The output activation decides what the number means: a softmax score looks like a probability whether or not it behaves like one.
What are the main types of neural network?
The architectures differ in what structure they assume in the input.
| Type | Built for | Typical use |
|---|---|---|
| Feedforward (multilayer perceptron) | Fixed-size inputs | Tabular scoring, simple classifiers |
| Convolutional (CNN) | Grids such as images | Computer vision, defect detection |
| Recurrent (RNN, LSTM) | Sequences read step by step | Older speech and text models, small streaming audio models |
| Transformer | Sequences related through attention | LLMs, vision-language models |
The transformer architecture now dominates language, and increasingly vision and audio, because attention parallelizes well during training.
What should a team running AI know about neural networks?
Three operating facts. Outputs are scores, not certainties: a network’s confidence is often miscalibrated and needs checking through model calibration before a threshold relies on it. The weights do not explain themselves, so reasons for a decision must come from explainability methods or from the system around the model, never from reading the network. And behavior is fixed at training time: a neural network changes only when someone retrains or fine-tunes it, so every new version needs the same held-out evaluation as the first.