Mathematics of neural networks in machine learning

Neural networks as functions

Neural network models can be viewed as defining a function that takes an input (observation) and produces an output (decision) ${\displaystyle f:X\to Y}$ or a distribution over ${\displaystyle X}$ or both ${\displaystyle X}$ and ${\displaystyle Y}$. Sometimes models are intimately associated with a particular learning rule. A common use of the phrase "ANN model" is really the definition of a class of such functions (where members of the class are obtained by varying parameters, connection weights, or specifics of the architecture such as the number of neurons, number of layers or their connectivity).

Mathematically, a neuron's network function ${\displaystyle f(x)}$ is defined as a composition of other functions ${\displaystyle g_i(x)}$, that can further be decomposed into other functions. This can be conveniently represented as a network structure, with arrows depicting the dependencies between functions. A widely used type of composition is the nonlinear weighted sum, where ${\displaystyle f(x)=K\left(\sum _i{w}_i{g}_i(x)\right)}$, where ${\displaystyle K}$ (commonly referred to as the activation function) is some predefined function, such as the hyperbolic tangent, sigmoid function, softmax function, or rectifier function. The important characteristic of the activation function is that it provides a smooth transition as input values change, i.e. a small change in input produces a small change in output. The following refers to a collection of functions ${\displaystyle g_i}$ as a vector ${\displaystyle g=(g_1,g_2,\dots ,g_n)}$.

This figure depicts such a decomposition of ${\displaystyle f}$, with dependencies between variables indicated by arrows. These can be interpreted in two ways.

The first view is the functional view: the input ${\displaystyle x}$ is transformed into a 3-dimensional vector ${\displaystyle h}$, which is then transformed into a 2-dimensional vector ${\displaystyle g}$, which is finally transformed into ${\displaystyle f}$. This view is most commonly encountered in the context of optimization.

The second view is the probabilistic view: the random variable ${\displaystyle F=f(G)}$ depends upon the random variable ${\displaystyle G=g(H)}$, which depends upon ${\displaystyle H=h(X)}$, which depends upon the random variable ${\displaystyle X}$. This view is most commonly encountered in the context of graphical models.

The two views are largely equivalent. In either case, for this particular architecture, the components of individual layers are independent of each other (e.g., the components of ${\displaystyle g}$ are independent of each other given their input ${\displaystyle h}$). This naturally enables a degree of parallelism in the implementation.

Networks such as the previous one are commonly called feedforward, because their graph is a directed acyclic graph. Networks with cycles are commonly called recurrent. Such networks are commonly depicted in the manner shown at the top of the figure, where ${\displaystyle f}$ is shown as dependent upon itself. However, an implied temporal dependence is not shown.

Backpropagation

Backpropagation training algorithms fall into three categories:

• steepest descent (with variable learning rate and momentum, resilient backpropagation);
• quasi-Newton (Broyden–Fletcher–Goldfarb–Shanno, one step secant);
• Levenberg–Marquardt and conjugate gradient (Fletcher–Reeves update, Polak–Ribiére update, Powell–Beale restart, scaled conjugate gradient).

Algorithm

Let ${\displaystyle N}$ be a network with ${\displaystyle e}$ connections, ${\displaystyle m}$ inputs and ${\displaystyle n}$ outputs.

Below, ${\displaystyle x_1,x_2,\dots }$ denote vectors in ${\displaystyle {\mathbb{R}}^m}$, ${\displaystyle y_1,y_2,\dots }$ vectors in ${\displaystyle {\mathbb{R}}^n}$, and ${\displaystyle w_0,w_1,w_2,\dots }$ vectors in ${\displaystyle {\mathbb{R}}^e}$. These are called inputs, outputs and weights, respectively.

The network corresponds to a function ${\displaystyle y=f_N(w,x)}$ which, given a weight ${\displaystyle w}$, maps an input ${\displaystyle x}$ to an output ${\displaystyle y}$.

In supervised learning, a sequence of training examples ${\displaystyle (x_1,y_1),\dots ,(x_p,y_p)}$ produces a sequence of weights ${\displaystyle w_0,w_1,\dots ,w_p}$ starting from some initial weight ${\displaystyle w_0}$, usually chosen at random.

These weights are computed in turn: first compute ${\displaystyle w_i}$ using only ${\displaystyle (x_i,y_i,w_{i-1})}$ for ${\displaystyle i=1,\dots ,p}$. The output of the algorithm is then ${\displaystyle w_p}$, giving a new function ${\displaystyle x\mapsto f_N(w_p,x)}$. The computation is the same in each step, hence only the case ${\displaystyle i=1}$ is described.

${\displaystyle w_1}$ is calculated from ${\displaystyle (x_1,y_1,w_0)}$ by considering a variable weight ${\displaystyle w}$ and applying gradient descent to the function ${\displaystyle w\mapsto E(f_N(w,x_1),y_1)}$ to find a local minimum, starting at ${\displaystyle w=w_0}$.

This makes ${\displaystyle w_1}$ the minimizing weight found by gradient descent.

Learning pseudocode

To implement the algorithm above, explicit formulas are required for the gradient of the function ${\displaystyle w\mapsto E(f_N(w,x),y)}$ where the function is ${\displaystyle E(y,y^{\prime })=|y-y^{\prime }{|}^2}$.

The learning algorithm can be divided into two phases: propagation and weight update.

Propagation

Propagation involves the following steps:

• Propagation forward through the network to generate the output value(s)
• Calculation of the cost (error term)
• Propagation of the output activations back through the network using the training pattern target to generate the deltas (the difference between the targeted and actual output values) of all output and hidden neurons.

Weight update

For each weight:

• Multiply the weight's output delta and input activation to find the gradient of the weight.
• Subtract the ratio (percentage) of the weight's gradient from the weight.

The learning rate is the ratio (percentage) that influences the speed and quality of learning. The greater the ratio, the faster the neuron trains, but the lower the ratio, the more accurate the training. The sign of the gradient of a weight indicates whether the error varies directly with or inversely to the weight. Therefore, the weight must be updated in the opposite direction, "descending" the gradient.

Learning is repeated (on new batches) until the network performs adequately.

Pseudocode

Pseudocode for a stochastic gradient descent algorithm for training a three-layer network (one hidden layer):

initialize network weights (often small random values). <b id="mw3g">do</b> <b id="mw3w">for each</b> training example named ex <b id="mw4A">do</b> prediction = <u id="mw4Q">neural-net-output</u>(network, ex) <i id="mw4g">// forward pass</i> actual = <u id="mw4w">teacher-output</u>(ex) compute error (prediction - actual) at the output units <span class="nowrap" about="#mwt90" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"compute <math>\\Delta w_h</math> for all weights from hidden layer to output layer"}},"i":0}}]}' id="mw5A">compute <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt91" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"\\Delta w_h"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \Delta w_{h}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi mathvariant="normal">Δ<!-- Δ --></mi> <msub> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>h</mi> </mrow> </msub> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \Delta w_{h}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/0f84dc6434bf1fcc9fc997866e03b7ddfc5a5234" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.779ex; height:2.509ex;" alt="{\displaystyle \Delta w_{h}}"/></span> for all weights from hidden layer to output layer</span> <i id="mw5Q">// backward pass</i> <span class="nowrap" about="#mwt92" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"compute <math>\\Delta w_i</math> for all weights from input layer to hidden layer"}},"i":0}}]}' id="mw5g">compute <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt93" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"\\Delta w_i"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \Delta w_{i}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi mathvariant="normal">Δ<!-- Δ --></mi> <msub> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>i</mi> </mrow> </msub> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \Delta w_{i}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/238dcf3322345baf33ca8d6ccb1a129fd18809ac" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.4ex; height:2.509ex;" alt="{\displaystyle \Delta w_{i}}"/></span> for all weights from input layer to hidden layer</span> <i id="mw5w">// backward pass continued</i> update network weights <i id="mw6A">// input layer not modified by error estimate</i> <b id="mw6Q">until</b> error rate becomes acceptably low <b id="mw6g">return</b> the network

The lines labeled "backward pass" can be implemented using the backpropagation algorithm, which calculates the gradient of the error of the network regarding the network's modifiable weights.