Long short-term memory

Variants

In the equations below, the lowercase variables represent vectors. Matrices ${\displaystyle W_q}$ and ${\displaystyle U_q}$ contain, respectively, the weights of the input and recurrent connections, where the subscript ${\displaystyle {}_q}$ can either be the input gate ${\displaystyle i}$, output gate ${\displaystyle o}$, the forget gate ${\displaystyle f}$ or the memory cell ${\displaystyle c}$, depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, ${\displaystyle c_t\in {\mathbb{R}}^h}$ is not just one unit of one LSTM cell, but contains ${\displaystyle h}$ LSTM cell's units.

LSTM with a forget gate (crucial)

The compact forms of the equations for the forward pass of an LSTM cell with a forget gate are:

${\displaystyle \begin{array}{cc}{f}_t& ={\sigma }_g(W_f{x}_t+U_f{h}_{t-1}+b_f)\\ i_t& ={\sigma }_g(W_i{x}_t+U_i{h}_{t-1}+b_i)\\ o_t& ={\sigma }_g(W_o{x}_t+U_o{h}_{t-1}+b_o)\\ {\tilde{c}}_t& ={\sigma }_c(W_c{x}_t+U_c{h}_{t-1}+b_c)\\ c_t& =f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t\\ h_t& =o_t\odot {\sigma }_h(c_t)\end{array}}$

where the initial values are ${\displaystyle c_0=0}$ and ${\displaystyle h_0=0}$ and the operator ${\displaystyle \odot }$ denotes the Hadamard product (element-wise product). The subscript ${\displaystyle t}$ indexes the time step.

Variables

Letting the superscripts ${\displaystyle d}$ and ${\displaystyle h}$ refer to the number of input features and number of hidden units, respectively:

• ${\displaystyle x_t\in {\mathbb{R}}^d}$: input vector to the LSTM unit
• ${\displaystyle f_t\in {(0,1)}^h}$: forget gate's activation vector
• ${\displaystyle i_t\in {(0,1)}^h}$: input/update gate's activation vector
• ${\displaystyle o_t\in {(0,1)}^h}$: output gate's activation vector
• ${\displaystyle h_t\in {(-1,1)}^h}$: hidden state vector also known as output vector of the LSTM unit
• ${\displaystyle {\tilde{c}}_t\in {(-1,1)}^h}$: cell input activation vector
• ${\displaystyle c_t\in {\mathbb{R}}^h}$: cell state vector
• ${\displaystyle W\in {\mathbb{R}}^{h\times d}}$, ${\displaystyle U\in {\mathbb{R}}^{h\times h}}$ and ${\displaystyle b\in {\mathbb{R}}^h}$: weight matrices and bias vector parameters which need to be learned during training

Activation functions

• ${\displaystyle {\sigma }_g}$: sigmoid function.
• ${\displaystyle {\sigma }_c}$: hyperbolic tangent function.
• ${\displaystyle {\sigma }_h}$: hyperbolic tangent function or, as the peephole LSTM paper suggests, ${\displaystyle {\sigma }_h(x)=x}$.

Peephole LSTM

The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM). Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state. ${\displaystyle h_{t-1}}$ is not used, ${\displaystyle c_{t-1}}$ is used instead in most places.

${\displaystyle \begin{array}{cc}{f}_t& ={\sigma }_g(W_f{x}_t+U_f{c}_{t-1}+b_f)\\ i_t& ={\sigma }_g(W_i{x}_t+U_i{c}_{t-1}+b_i)\\ o_t& ={\sigma }_g(W_o{x}_t+U_o{c}_{t-1}+b_o)\\ c_t& =f_t\odot c_{t-1}+i_t\odot {\sigma }_c(W_c{x}_t+b_c)\\ h_t& =o_t\odot {\sigma }_h(c_t)\end{array}}$

Each of the gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. ${\displaystyle i_t,o_t}$ and ${\displaystyle f_t}$ represent the activations of respectively the input, output and forget gates, at time step ${\displaystyle t}$.

The 3 exit arrows from the memory cell ${\displaystyle c}$ to the 3 gates ${\displaystyle i,o}$ and ${\displaystyle f}$ represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. the contribution of ${\displaystyle c_{t-1}}$ (and not ${\displaystyle c_t}$, as the picture may suggest). In other words, the gates ${\displaystyle i,o}$ and ${\displaystyle f}$ calculate their activations at time step ${\displaystyle t}$ (i.e., respectively, ${\displaystyle i_t,o_t}$ and ${\displaystyle f_t}$) also considering the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. ${\displaystyle c_{t-1}}$.

The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes ${\displaystyle c_t}$.

The little circles containing a ${\displaystyle \times }$ symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum.

Peephole convolutional LSTM

Peephole convolutional LSTM. The ${\displaystyle \ast }$ denotes the convolution operator.

${\displaystyle \begin{array}{cc}{f}_t& ={\sigma }_g(W_f\ast x_t+U_f\ast h_{t-1}+V_f\odot c_{t-1}+b_f)\\ i_t& ={\sigma }_g(W_i\ast x_t+U_i\ast h_{t-1}+V_i\odot c_{t-1}+b_i)\\ c_t& =f_t\odot c_{t-1}+i_t\odot {\sigma }_c(W_c\ast x_t+U_c\ast h_{t-1}+b_c)\\ o_t& ={\sigma }_g(W_o\ast x_t+U_o\ast h_{t-1}+V_o\odot c_t+b_o)\\ h_t& =o_t\odot {\sigma }_h(c_t)\end{array}}$

Training

An RNN using LSTM units can be trained in a supervised fashion on a set of training sequences, using an optimization algorithm like gradient descent combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.

A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to ${\displaystyle \underset{n\to \mathrm{\infty }}{lim}{W}^n=0}$ if the spectral radius of ${\displaystyle W}$ is smaller than 1.

However, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.

CTC score function

Many applications use stacks of LSTM RNNs and train them by connectionist temporal classification (CTC) to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

Alternatives

Sometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution or by policy gradient methods, especially when there is no "teacher" (that is, training labels).

Applications

Applications of LSTM include:

• Robot control
• Time series prediction
• Speech recognition
• Rhythm learning
• Hydrological rainfall–runoff modeling
• Music composition
• Energy forecasting
• Grammar learning
• Handwriting recognition
• Human action recognition
• Sign language translation
• Protein homology detection
• Predicting subcellular localization of proteins
• Time series anomaly detection
• Several prediction tasks in the area of business process management
• Prediction in medical care pathways
• Semantic parsing
• Object co-segmentation
• Airport passenger management
• Short-term traffic forecast
• Drug design
• Financial forecasting
• Activity classification in video

2015: Google started using an LSTM trained by CTC for speech recognition on Google Voice. According to the official blog post, the new model cut transcription errors by 49%.

2016: Google started using an LSTM to suggest messages in the Allo conversation app. In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.

Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype in the iPhone and for Siri.

Amazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.

2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.

Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".

2018: OpenAI used LSTM trained by policy gradients to beat humans in the complex video game of Dota 2, and to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.

2019: DeepMind used LSTM trained by policy gradients to excel at the complex video game of Starcraft II.

History

Development

Aspects of LSTM were anticipated by "focused back-propagation", cited by the LSTM paper.

Sepp Hochreiter's 1991 German diploma thesis analyzed the vanishing gradient problem and developed principles of the method. His supervisor, Jürgen Schmidhuber, considered the thesis highly significant.

An early version of LSTM was published in 1995 in a technical report by Sepp Hochreiter and Jürgen Schmidhuber, then published in the NIPS 1996 conference.

The most commonly used reference point for LSTM was published in 1997 in the journal Neural Computation. By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.

Felix Gers, Jürgen Schmidhuber, and Fred Cummins introduced the forget gate (also called "keep gate") into the LSTM architecture in 1999, enabling the LSTM to reset its own state. This is the most commonly used version of LSTM nowadays. They added peephole connections in 2000. Additionally, the output activation function was omitted.

Development of variants

In 2005 Graves and Schmidhuber published LSTM with full backpropagation through time and bidirectional LSTM.

In 2006 Graves, Fernandez, Gomez, and Schmidhuber introduced a new error function for LSTM: Connectionist Temporal Classification (CTC) for simultaneous alignment and recognition of sequences.

In 2014 Kyunghyun Cho an others published a simplified variant of the forget gate LSTM called Gated recurrent unit (GRU).

In 2015 Srivastava, Greff, and Schmidhuber used LSTM principles to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks. Concurrently, the ResNet architecture was developed. It is equivalent to an open-gated or gateless highway network.

A modern upgrade of LSTM called xLSTM is published by a team led by Sepp Hochreiter. One of the 2 blocks (mLSTM) of the architecture are parallelizable like the Transformer architecture, the other ones (sLSTM) allow state tracking.

Applications

2001: Gers and Schmidhuber trained LSTM to learn languages unlearnable by traditional models such as Hidden Markov Models.

Hochreiter et al. used LSTM for meta-learning (i.e. learning a learning algorithm).

2004: First successful application of LSTM to speech Alex Graves et al.

2005: Daan Wierstra, Faustino Gomez, and Schmidhuber trained LSTM by neuroevolution without a teacher.

Mayer et al. trained LSTM to control robots.

2007: Wierstra, Foerster, Peters, and Schmidhuber trained LSTM by policy gradients for reinforcement learning without a teacher.

Hochreiter, Heuesel, and Obermayr applied LSTM to protein homology detection the field of biology.

2009: Justin Bayer et al. introduced neural architecture search for LSTM.

2009: An LSTM trained by CTC won the ICDAR connected handwriting recognition competition. Three such models were submitted by a team led by Alex Graves. One was the most accurate model in the competition and another was the fastest. This was the first time an RNN won international competitions.

2013: Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton used LSTM networks as a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.

2017: Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference. Their time-aware LSTM (T-LSTM) performs better on certain data sets than standard LSTM.

2020: Kaplan and McCandlish et al. explored the scaling law of LSTM architecture in language modeling and compared it with Transformers, showing that Transformers scale better than LSTM on text data.