What does regularization actually do?
It makes the simple explanation of the data the cheap one. Explicit regularization adds a penalty term to the loss function, scaled by a strength setting usually called lambda or alpha, so large weights cost something even when they fit the training set. Gradient descent then trades a little training error for smaller, smoother weights. The target is a smaller gap between training and test performance, not a lower training score. Set the strength too high and the model underfits instead.
What are the main regularization techniques?
| Technique | How it works | Typical home |
|---|---|---|
| L1 (lasso) | Adds the sum of absolute weights to the loss | Linear models with many features |
| L2 (ridge) | Adds the sum of squared weights to the loss | Linear models, small networks |
| Weight decay | Shrinks weights by a fraction at each update | Neural network and LLM training |
| Dropout | Randomly switches off units during training | Deep neural networks |
| Early stopping | Halts training once validation loss stops improving | Any iterative training |
| Data augmentation | Trains on altered copies of real examples | Images, audio, text |
The lasso “is able to set coefficients exactly to zero” (scikit-learn 1.9.1 linear models guide, read 5 November 2026), which makes L1 a built-in feature selector. Dropout stops units from co-adapting (Srivastava et al., JMLR 2014, read 5 November 2026). Most production models stack two or three of these rather than relying on one.
Is weight decay the same as L2 regularization?
Only under plain stochastic gradient descent. Loshchilov and Hutter showed the two “are equivalent for standard stochastic gradient descent” but not for adaptive optimizers such as Adam (Decoupled Weight Decay Regularization, revised January 2019, read 5 November 2026). Their fix is AdamW, which in PyTorch defaults to weight_decay=0.01 (PyTorch 2.15 AdamW docs, read 5 November 2026). BERT added “a dropout probability of 0.1 on all layers” (Devlin et al., 2018, read 5 November 2026). In modern deep learning the regularization knobs are usually weight decay and dropout, not an L2 term in the loss.
How do you know your model needs regularization?
Read the learning curves. Training loss keeps falling while validation loss flattens or climbs: that is overfitting. Other warning signs are more parameters than examples, fine-tuning on a small internal dataset, or predictions that swing when you retrain on a slightly different sample. If both scores are poor, the model underfits, and more regularization makes it worse. Tune the strength on a validation set and judge the result on a separate test set, the split model validation describes.