Adam is a popular training method that adjusts each weight's step size automatically; AdamW is a fixed-up version that handles weight decay properly.
In plain words
Training nudges a model's weights downhill on its error, step by step. Plain gradient descent uses one step size for everything. Adam keeps running averages of recent gradients and of their size, and uses them to pick a separate step for each weight. It works well with noisy data and needs little tuning. AdamW separates weight decay, a pull that keeps weights small, from those adaptive steps, which helps models generalise better.
A quick example
In PyTorch, swapping plain SGD for AdamW is usually a one-line change: create torch.optim.AdamW with the model's parameters and a learning rate, and the training loop stays the same.
Where you'll meet it
- Training neural networks, including language models.
- Fine-tuning pretrained models.
Related
Sources
- PaperAdam: A Method for Stochastic Optimization, Kingma and Ba (arXiv) · read 29 Sept 2026
- PaperDecoupled Weight Decay Regularization, Loshchilov and Hutter (arXiv) · read 29 Sept 2026
- Docstorch.optim.AdamW, PyTorch docs · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.