Gated recurrent unit

Architecture

There are several variations on the full gated unit, with gating done using the previous hidden state and the bias in various combinations, and a simplified form called minimal gated unit.

In the following, the operator ${\displaystyle \odot }$ denotes the Hadamard product.

Fully gated unit

Initially, for ${\displaystyle t=0}$, the output vector is ${\displaystyle h_0=0}$.

${\displaystyle \begin{array}{cc}{z}_t& =\sigma (W_z{x}_t+U_z{h}_{t-1}+b_z)\\ r_t& =\sigma (W_r{x}_t+U_r{h}_{t-1}+b_r)\\ {\hat{h}}_t& =\varphi (W_h{x}_t+U_h(r_t\odot h_{t-1})+b_h)\\ h_t& =(1-z_t)\odot h_{t-1}+z_t\odot {\hat{h}}_t\end{array}}$

Variables (${\displaystyle d}$ denotes the number of input features and ${\displaystyle e}$ the number of output features):

• ${\displaystyle x_t\in {\mathbb{R}}^d}$: input vector
• ${\displaystyle h_t\in {\mathbb{R}}^e}$: output vector
• ${\displaystyle {\hat{h}}_t\in {\mathbb{R}}^e}$: candidate activation vector
• ${\displaystyle z_t\in (0,1{)}^e}$: update gate vector
• ${\displaystyle r_t\in (0,1{)}^e}$: reset gate vector
• ${\displaystyle W\in {\mathbb{R}}^{e\times d}}$, ${\displaystyle U\in {\mathbb{R}}^{e\times e}}$ and ${\displaystyle b\in {\mathbb{R}}^e}$: parameter matrices and vector which need to be learned during training

Activation functions

• ${\displaystyle \sigma }$: The original is a logistic function.
• ${\displaystyle \varphi }$: The original is a hyperbolic tangent.

Alternative activation functions are possible, provided that ${\displaystyle \sigma (x)\in [0,1]}$.

Alternate forms can be created by changing ${\displaystyle z_t}$ and ${\displaystyle r_t}$

• Type 1: each gate depends only on the previous hidden state and the bias.

  ${\displaystyle \begin{array}{cc}{z}_t& =\sigma (U_z{h}_{t-1}+b_z)\\ r_t& =\sigma (U_r{h}_{t-1}+b_r)\end{array}}$

• Type 2: each gate depends only on the previous hidden state.

  ${\displaystyle \begin{array}{cc}{z}_t& =\sigma (U_z{h}_{t-1})\\ r_t& =\sigma (U_r{h}_{t-1})\end{array}}$

• Type 3: each gate is computed using only the bias.

  ${\displaystyle \begin{array}{cc}{z}_t& =\sigma (b_z)\\ r_t& =\sigma (b_r)\end{array}}$

Minimal gated unit

The minimal gated unit (MGU) is similar to the fully gated unit, except the update and reset gate vector is merged into a forget gate. This also implies that the equation for the output vector must be changed:

${\displaystyle \begin{array}{cc}{f}_t& =\sigma (W_f{x}_t+U_f{h}_{t-1}+b_f)\\ {\hat{h}}_t& =\varphi (W_h{x}_t+U_h(f_t\odot h_{t-1})+b_h)\\ h_t& =(1-f_t)\odot h_{t-1}+f_t\odot {\hat{h}}_t\end{array}}$

Variables

• ${\displaystyle x_t}$: input vector
• ${\displaystyle h_t}$: output vector
• ${\displaystyle {\hat{h}}_t}$: candidate activation vector
• ${\displaystyle f_t}$: forget vector
• ${\displaystyle W}$, ${\displaystyle U}$ and ${\displaystyle b}$: parameter matrices and vector

Light gated recurrent unit

The light gated recurrent unit (LiGRU) removes the reset gate altogether, replaces tanh with the ReLU activation, and applies batch normalization (BN):

${\displaystyle \begin{array}{cc}{z}_t& =\sigma (\mathrm{BN}(W_z{x}_t)+U_z{h}_{t-1})\\ {\tilde{h}}_t& =\mathrm{ReLU}(\mathrm{BN}(W_h{x}_t)+U_h{h}_{t-1})\\ h_t& =z_t\odot h_{t-1}+(1-z_t)\odot {\tilde{h}}_t\end{array}}$

LiGRU has been studied from a Bayesian perspective. This analysis yielded a variant called light Bayesian recurrent unit (LiBRU), which showed slight improvements over the LiGRU on speech recognition tasks.