LSTMConcepts
Long short-term memory
1 · In one line
An LSTM is a recurrent unit whose multiplicative gates regulate access to its memory path.
1 · What it is
LSTMCell accepts a hidden state and a cell state as inputs. The new cell state is the gated old state plus the gated candidate. The output gate then scales a transformed version of that new cell state to produce the next hidden state.
LSTM was introduced to address decaying error flow in recurrent learning. It later became a key recurrent architecture for learning long-range dependencies.
Ordinary recurrent training can lose the learning signal before it reaches a distant time step.
Fading signalGradients can shrink as they pass backward through many recurrent steps.
Unstable signalGradients can also grow too large during recurrent training.
Selective memoryMultiplicative gates learn to open and close access to the memory path.
Follow information through one LSTM cell.
- 1 · inspectReceive the current input together with the previous hidden and cell states.
- 2 · forgetUse the forget gate to scale which parts of the previous cell state remain.
- 3 · proposeCreate candidate content and use the input gate to scale what may be written.
- 4 · updateAdd the retained old state and selected candidate to form the new cell state.
- 5 · exposeUse the output gate to turn part of the new cell state into the next hidden state.
The cell-state update adds a retained old state and a selected candidate.
| Who | What they ask | What it works with |
|---|---|---|
| Speech recognizer | “Which earlier sounds still matter for this word?” | Ordered audio features |
| Forecasting team | “Which past measurements should remain in memory?” | A time series of sensor values |
| Translation system | “What information from the source sequence should guide the next output?” | Encoded words from the source sentence |
solves
- Multiplicative gates learn to open and close access to the memory path.
- The cell-state update provides a more direct path for information and gradients through time.
- An LSTM encoder can map an input sequence to a fixed-dimensional vector.
doesn't solve
- The encoder compresses its input sequence into a fixed-dimensional vector.
- The reference LSTMCell example iterates over the input with a loop.
6 · Go deeper
Sources used
This explainer is written in original language. The links below support its factual claims.
- paperLong Short-Term Memory, Hochreiter and Schmidhuber · read 27 Sept 2026
- docsLSTMCell, PyTorch · read 27 Sept 2026
- paperOn the difficulty of training Recurrent Neural Networks, Pascanu, Mikolov and Bengio · read 27 Sept 2026
- paperSequence to Sequence Learning with Neural Networks, Sutskever, Vinyals and Le · read 27 Sept 2026
- paperDeep learning, LeCun, Bengio and Hinton · read 27 Sept 2026