Partial least squares regression

PLS is used to find the fundamental relations between two matrices (X and Y), i.e. a latent variable approach to modeling the covariance structures in these two spaces. A PLS model will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space. PLS regression is particularly suited when the matrix of predictors has more variables than observations, and when there is multicollinearity among X values. By contrast, standard regression will fail in these cases (unless it is regularized).

Partial least squares was introduced by the Swedish statistician Herman O. A. Wold, who then developed it with his son, Svante Wold. An alternative term for PLS is projection to latent structures, but the term partial least squares is still dominant in many areas. Although the original applications were in the social sciences, PLS regression is today most widely used in chemometrics and related areas. It is also used in bioinformatics, sensometrics, neuroscience, and anthropology.

Core idea

We are given a sample of $n$ paired observations $({\vec{x}}_{i}, {\vec{y}}_{i}), i \in 1, \dots, n$ . In the first step $j = 1$ , the partial least squares regression searches for the normalized direction ${\vec{p}}_{j}$ , ${\vec{q}}_{j}$ that maximizes the covariance

max_{{\vec{p}}_{j}, {\vec{q}}_{j}} E [\underset{t_{j}}{\underset{⏟}{({\vec{p}}_{j} \cdot \vec{X})}} \underset{u_{j}}{\underset{⏟}{({\vec{q}}_{j} \cdot \vec{Y})}}] .

Note below, the algorithm is denoted in matrix notation.

4 more chapters are free with a Lably account.

Underlying model

The general underlying model of multivariate PLS with $ℓ$ components is

X = T P^{T} + E

Y = U Q^{T} + F

where

X is an $n \times m$ matrix of predictors
Y is an $n \times p$ matrix of responses
T and U are $n \times ℓ$ matrices that are, respectively, projections of X (the X score, component or factor matrix) and projections of Y (the Y scores)
P and Q are, respectively, $m \times ℓ$ and $p \times ℓ$ loading matrices
and matrices E and F are the error terms, assumed to be independent and identically distributed random normal variables.

The decompositions of X and Y are made so as to maximise the covariance between T and U.

Note that this covariance is defined pair by pair: the covariance of column i of T (length n) with the column i of U (length n) is maximized. Additionally, the covariance of the column i of T with the column j of U (with $i \neq j$ ) is zero.

In PLSR, the loadings are thus chosen so that the scores form an orthogonal basis. This is a major difference with PCA where orthogonality is imposed onto loadings (and not the scores).

Algorithms

A number of variants of PLS exist for estimating the factor and loading matrices T, U, P and Q. Most of them construct estimates of the linear regression between X and Y as $Y = X \tilde{B} + {\tilde{B}}_{0}$ . Some PLS algorithms are only appropriate for the case where Y is a column vector, while others deal with the general case of a matrix Y. Algorithms also differ on whether they estimate the factor matrix T as an orthogonal (that is, orthonormal) matrix or not. The final prediction will be the same for all these varieties of PLS, but the components will differ.

PLS is composed of iteratively repeating the following steps k times (for k components):

finding the directions of maximal covariance in input and output space
performing least squares regression on the input score
deflating the input $X$ and/or target $Y$

PLS1

PLS1 is a widely used algorithm appropriate for the vector Y case. It estimates T as an orthonormal matrix. (Caution: the t vectors in the code below may not be normalized appropriately; see talk.) In pseudocode it is expressed below (capital letters are matrices, lower case letters are vectors if they are superscripted and scalars if they are subscripted).

1 <span class="nowrap" about="#mwt86" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''function''' PLS1({{mvar|X, y, ℓ}})"}},"i":0}}]}' id="mwtQ"><b>function</b> PLS1(<span class="texhtml mvar" style="font-style:italic;">X, y, ℓ</span>)</span> 2 <span class="nowrap" about="#mwt87" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>X^{(0)} \\gets X</math>"}},"i":0}}]}' id="mwtg"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt88" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"X^{(0)} \\gets X"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle X^{(0)}\gets X}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mn>0</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <mi>X</mi> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle X^{(0)}\gets X}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/94e440e81bf537c6e748552d001fcbe35c8191e2" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:9.925ex; height:2.843ex;" alt="{\displaystyle X^{(0)}\gets X}"/></span></span> 3 <span class="nowrap" about="#mwt89" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>w^{(0)} \\gets X^\\mathrm{T} y/\\|X^\\mathrm{T}y\\|</math>"}},"i":0}}]}' id="mwtw"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt90" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"w^{(0)} \\gets X^\\mathrm{T} y/\\|X^\\mathrm{T}y\\|"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle w^{(0)}\gets X^{\mathrm {T} }y/\|X^{\mathrm {T} }y\|}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mn>0</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mo>/</mo> </mrow> <mo fence="false" stretchy="false">‖<!-- ‖ --></mo> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <mi>y</mi> <mo fence="false" stretchy="false">‖<!-- ‖ --></mo> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle w^{(0)}\gets X^{\mathrm {T} }y/\|X^{\mathrm {T} }y\|}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/d8225e7f141df6f8ff5b1cd2b61c2916dacf8f10" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:20.242ex; height:3.343ex;" alt="{\displaystyle w^{(0)}\gets X^{\mathrm {T} }y/\|X^{\mathrm {T} }y\|}"/></span></span>, an initial estimate of <span class="texhtml mvar" style="font-style:italic;" about="#mwt91" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"mvar","href":"./Template:Mvar"},"params":{"1":{"wt":"w"}},"i":0}}]}' id="mwuA">w</span>. 4 <span class="nowrap" about="#mwt92" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''for''' <math>k = 0</math> '''to''' <math>\\ell-1</math>"}},"i":0}}]}' id="mwuQ"><b>for</b> <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt93" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"k = 0"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k=0}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo>=</mo> <mn>0</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k=0}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/6307c8a99dad7d0bcb712352ae0a748bd99a038b" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.472ex; height:2.176ex;" alt="{\displaystyle k=0}"/></span> <b>to</b> <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt94" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"\\ell-1"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \ell -1}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>ℓ<!-- ℓ --></mi> <mo>−<!-- − --></mo> <mn>1</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \ell -1}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/8144f37f93f6f504a155fa04a07f0da8917fab1d" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:4.973ex; height:2.343ex;" alt="{\displaystyle \ell -1}"/></span></span> 5 <span class="nowrap" about="#mwt95" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>t^{(k)} \\gets X^{(k)}w^{(k)}</math>"}},"i":0}}]}' id="mwug"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt96" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"t^{(k)} \\gets X^{(k)}w^{(k)}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle t^{(k)}\gets X^{(k)}w^{(k)}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle t^{(k)}\gets X^{(k)}w^{(k)}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/04625f88ebf1ee2aa87307c172b887a20eea6040" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:15.219ex; height:2.843ex;" alt="{\displaystyle t^{(k)}\gets X^{(k)}w^{(k)}}"/></span></span> 6 <span class="nowrap" about="#mwt97" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>t_k \\gets {t^{(k)}}^\\mathrm{T} t^{(k)}</math> (note this is a scalar)"}},"i":0}}]}' id="mwuw"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt98" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"t_k \\gets {t^{(k)}}^\\mathrm{T} t^{(k)}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle t_{k}\gets {t^{(k)}}^{\mathrm {T} }t^{(k)}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>k</mi> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <msup> <mrow class="MJX-TeXAtom-ORD"> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle t_{k}\gets {t^{(k)}}^{\mathrm {T} }t^{(k)}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/79fec236018c9d668f0c40e8e6a644367b17f3a0" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:13.377ex; height:3.676ex;" alt="{\displaystyle t_{k}\gets {t^{(k)}}^{\mathrm {T} }t^{(k)}}"/></span> (note this is a scalar)</span> 7 <span class="nowrap" about="#mwt99" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>t^{(k)} \\gets t^{(k)} / t_k</math>"}},"i":0}}]}' id="mwvA"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt100" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"t^{(k)} \\gets t^{(k)} / t_k"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle t^{(k)}\gets t^{(k)}/t_{k}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <mrow class="MJX-TeXAtom-ORD"> <mo>/</mo> </mrow> <msub> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>k</mi> </mrow> </msub> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle t^{(k)}\gets t^{(k)}/t_{k}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/fc467ca5ec12be797c75b3a0f6468f9af306b904" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:13.12ex; height:3.343ex;" alt="{\displaystyle t^{(k)}\gets t^{(k)}/t_{k}}"/></span></span> 8 <span class="nowrap" about="#mwt101" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>p^{(k)} \\gets {X^{(k)}}^\\mathrm{T} t^{(k)}</math>"}},"i":0}}]}' id="mwvQ"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt102" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"p^{(k)} \\gets {X^{(k)}}^\\mathrm{T} t^{(k)}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle p^{(k)}\gets {X^{(k)}}^{\mathrm {T} }t^{(k)}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>p</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mrow class="MJX-TeXAtom-ORD"> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle p^{(k)}\gets {X^{(k)}}^{\mathrm {T} }t^{(k)}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/2a3b53ad2c160b02d190c258f1d79f984c768c2b" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; margin-left: -0.089ex; width:16.232ex; height:3.676ex;" alt="{\displaystyle p^{(k)}\gets {X^{(k)}}^{\mathrm {T} }t^{(k)}}"/></span></span> 9 <span class="nowrap" about="#mwt103" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>q_k \\gets {y}^\\mathrm{T} t^{(k)}</math> (note this is a scalar)"}},"i":0}}]}' id="mwvg"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt104" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"q_k \\gets {y}^\\mathrm{T} t^{(k)}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle q_{k}\gets {y}^{\mathrm {T} }t^{(k)}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>q</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>k</mi> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <msup> <mrow class="MJX-TeXAtom-ORD"> <mi>y</mi> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle q_{k}\gets {y}^{\mathrm {T} }t^{(k)}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/267ca338482a198a0d159d89813075833b370a27" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.527ex; height:3.176ex;" alt="{\displaystyle q_{k}\gets {y}^{\mathrm {T} }t^{(k)}}"/></span> (note this is a scalar)</span> 10 <span class="nowrap" about="#mwt105" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''if''' <math>q_k = 0</math>"}},"i":0}}]}' id="mwvw"><b>if</b> <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt106" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"q_k = 0"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle q_{k}=0}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>q</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>k</mi> </mrow> </msub> <mo>=</mo> <mn>0</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle q_{k}=0}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/04a4c55d9f595e321c11cbe0545306b0e1871f5c" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.387ex; height:2.509ex;" alt="{\displaystyle q_{k}=0}"/></span></span> 11 <span class="nowrap" about="#mwt107" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>\\ell \\gets k</math>, '''break''' the '''for loop'''"}},"i":0}}]}' id="mwwA"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt108" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"\\ell \\gets k"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \ell \gets k}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>ℓ<!-- ℓ --></mi> <mo stretchy="false">←<!-- ← --></mo> <mi>k</mi> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \ell \gets k}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/0744df12a2af22cb9d4285f29c138078ce5ebad6" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.795ex; height:2.176ex;" alt="{\displaystyle \ell \gets k}"/></span>, <b>break</b> the <b>for loop</b></span> 12 <span class="nowrap" about="#mwt109" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''if''' <math>k < (\\ell-1)</math>"}},"i":0}}]}' id="mwwQ"><b>if</b> <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt110" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"k < (\\ell-1)"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k<(\ell -1)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo><</mo> <mo stretchy="false">(</mo> <mi>ℓ<!-- ℓ --></mi> <mo>−<!-- − --></mo> <mn>1</mn> <mo stretchy="false">)</mo> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k<(\ell -1)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/83fa4fa2bfa68e9594e77c8d361cec99e935fe30" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:11.092ex; height:2.843ex;" alt="{\displaystyle k<(\ell -1)}"/></span></span> 13 <span class="nowrap" about="#mwt111" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>X^{(k+1)} \\gets X^{(k)} - t_k t^{(k)} {p^{(k)}}^\\mathrm{T}</math>"}},"i":0}}]}' id="mwwg"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt112" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"X^{(k+1)} \\gets X^{(k)} - t_k t^{(k)} {p^{(k)}}^\\mathrm{T}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle X^{(k+1)}\gets X^{(k)}-t_{k}t^{(k)}{p^{(k)}}^{\mathrm {T} }}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo>+</mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <mo>−<!-- − --></mo> <msub> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>k</mi> </mrow> </msub> <msup> <mi>t</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> <msup> <mrow class="MJX-TeXAtom-ORD"> <msup> <mi>p</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo stretchy="false">)</mo> </mrow> </msup> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle X^{(k+1)}\gets X^{(k)}-t_{k}t^{(k)}{p^{(k)}}^{\mathrm {T} }}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/23cc346134227fe369f66c5aec3b5da468518ee3" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:27.377ex; height:3.676ex;" alt="{\displaystyle X^{(k+1)}\gets X^{(k)}-t_{k}t^{(k)}{p^{(k)}}^{\mathrm {T} }}"/></span></span> 14 <span class="nowrap" about="#mwt113" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>w^{(k+1)} \\gets {X^{(k+1)}}^\\mathrm{T} y </math>"}},"i":0}}]}' id="mwww"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt114" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"w^{(k+1)} \\gets {X^{(k+1)}}^\\mathrm{T} y "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle w^{(k+1)}\gets {X^{(k+1)}}^{\mathrm {T} }y}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo>+</mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mrow class="MJX-TeXAtom-ORD"> <msup> <mi>X</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>k</mi> <mo>+</mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </msup> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <mi>y</mi> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle w^{(k+1)}\gets {X^{(k+1)}}^{\mathrm {T} }y}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/c06fbbd8f6cabfabeb9ec01ffe45247d91a44adb" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:18.787ex; height:3.676ex;" alt="{\displaystyle w^{(k+1)}\gets {X^{(k+1)}}^{\mathrm {T} }y}"/></span></span> 15 <span class="nowrap" about="#mwt115" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''end''' '''for'''"}},"i":0}}]}' id="mwxA"><b>end</b> <b>for</b></span> 16 <b id="mwxQ">define</b> <span class="texhtml mvar" style="font-style:italic;" about="#mwt116" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"mvar","href":"./Template:Mvar"},"params":{"1":{"wt":"W"}},"i":0}}]}' id="mwxg">W</span> to be the matrix <span class="nowrap" about="#mwt117" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"with columns <math>w^{(0)},w^{(1)},\\ldots,w^{(\\ell-1)}</math>."}},"i":0}}]}' id="mwxw">with columns <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt118" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"w^{(0)},w^{(1)},\\ldots,w^{(\\ell-1)}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle w^{(0)},w^{(1)},\ldots ,w^{(\ell -1)}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mn>0</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo>,</mo> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </msup> <mo>,</mo> <mo>…<!-- … --></mo> <mo>,</mo> <msup> <mi>w</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mi>ℓ<!-- ℓ --></mi> <mo>−<!-- − --></mo> <mn>1</mn> <mo stretchy="false">)</mo> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle w^{(0)},w^{(1)},\ldots ,w^{(\ell -1)}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/4a5c74cb8b8ae48af7dd1f1d2dae7d25d41c4ab9" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:20.17ex; height:3.176ex;" alt="{\displaystyle w^{(0)},w^{(1)},\ldots ,w^{(\ell -1)}}"/></span>.</span> Do the same to form the <span class="texhtml mvar" style="font-style:italic;" about="#mwt119" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"mvar","href":"./Template:Mvar"},"params":{"1":{"wt":"P"}},"i":0}}]}' id="mwyA">P</span> matrix and <span class="texhtml mvar" style="font-style:italic;" about="#mwt120" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"mvar","href":"./Template:Mvar"},"params":{"1":{"wt":"q"}},"i":0}}]}' id="mwyQ">q</span> vector. 17 <span class="nowrap" about="#mwt121" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>B \\gets W {(P^\\mathrm{T} W)}^{-1} q</math>"}},"i":0}}]}' id="mwyg"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt122" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"B \\gets W {(P^\\mathrm{T} W)}^{-1} q"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle B\gets W{(P^{\mathrm {T} }W)}^{-1}q}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>B</mi> <mo stretchy="false">←<!-- ← --></mo> <mi>W</mi> <msup> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <msup> <mi>P</mi> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <mi>W</mi> <mo stretchy="false">)</mo> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mo>−<!-- − --></mo> <mn>1</mn> </mrow> </msup> <mi>q</mi> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle B\gets W{(P^{\mathrm {T} }W)}^{-1}q}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/47a18a64eb25d19f17290ce1ec2fc830d7c00af4" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.701ex; height:3.676ex;" alt="{\displaystyle B\gets W{(P^{\mathrm {T} }W)}^{-1}q}"/></span></span> 18 <span class="nowrap" about="#mwt123" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"<math>B_0 \\gets q_0 - {P^{(0)}}^\\mathrm{T} B</math>"}},"i":0}}]}' id="mwyw"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt124" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"B_0 \\gets q_0 - {P^{(0)}}^\\mathrm{T} B"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle B_{0}\gets q_{0}-{P^{(0)}}^{\mathrm {T} }B}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>B</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <msub> <mi>q</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <mo>−<!-- − --></mo> <msup> <mrow class="MJX-TeXAtom-ORD"> <msup> <mi>P</mi> <mrow class="MJX-TeXAtom-ORD"> <mo stretchy="false">(</mo> <mn>0</mn> <mo stretchy="false">)</mo> </mrow> </msup> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi mathvariant="normal">T</mi> </mrow> </mrow> </msup> <mi>B</mi> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle B_{0}\gets q_{0}-{P^{(0)}}^{\mathrm {T} }B}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/6e7d6c6ad5a8a2de4add0673b6c3d9159797a0a3" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:18.702ex; height:3.676ex;" alt="{\displaystyle B_{0}\gets q_{0}-{P^{(0)}}^{\mathrm {T} }B}"/></span></span> 19 <span class="nowrap" about="#mwt125" typeof="mw:Transclusion" data-mw='{"parts":[{"template":{"target":{"wt":"nowrap","href":"./Template:Nowrap"},"params":{"1":{"wt":"'''return''' <math>B, B_0</math>"}},"i":0}}]}' id="mwzA"><b>return</b> <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt126" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"B, B_0"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle B,B_{0}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>B</mi> <mo>,</mo> <msub> <mi>B</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle B,B_{0}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/5c5a6a6cfea606288ed06bcb6adcfe962bb51d93" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:5.616ex; height:2.509ex;" alt="{\displaystyle B,B_{0}}"/></span></span>

This form of the algorithm does not require centering of the input X and Y, as this is performed implicitly by the algorithm. This algorithm features 'deflation' of the matrix X (subtraction of $t_{k} t^{(k)} {p^{(k)}}^{T}$ ), but deflation of the vector y is not performed, as it is not necessary (it can be proved that deflating y yields the same results as not deflating). The user-supplied variable l is the limit on the number of latent factors in the regression; if it equals the rank of the matrix X, the algorithm will yield the least squares regression estimates for B and $B_{0}$

Extensions

OPLS

In 2002 a new method was published called orthogonal projections to latent structures (OPLS). In OPLS, continuous variable data is separated into predictive and uncorrelated (orthogonal) information. This leads to improved diagnostics, as well as more easily interpreted visualization. However, these changes only improve the interpretability, not the predictivity, of the PLS models. Similarly, OPLS-DA (Discriminant Analysis) may be applied when working with discrete variables, as in classification and biomarker studies.

The general underlying model of OPLS is

X = T P^{T} + T_{Y-orth} P_{Y-orth}^{T} + E

Y = U Q^{T} + F

or in O2-PLS

X = T P^{T} + T_{Y-orth} P_{Y-orth}^{T} + E

Y = U Q^{T} + U_{X-orth} Q_{X-orth}^{T} + F

L-PLS

Another extension of PLS regression, named L-PLS for its L-shaped matrices, connects 3 related data blocks to improve predictability. In brief, a new Z matrix, with the same number of columns as the X matrix, is added to the PLS regression analysis and may be suitable for including additional background information on the interdependence of the predictor variables.

3PRF

In 2015 partial least squares was related to a procedure called the three-pass regression filter (3PRF). Supposing the number of observations and variables are large, the 3PRF (and hence PLS) is asymptotically normal for the "best" forecast implied by a linear latent factor model. In stock market data, PLS has been shown to provide accurate out-of-sample forecasts of returns and cash-flow growth.

Partial least squares SVD

A PLS version based on singular value decomposition (SVD) provides a memory efficient implementation that can be used to address high-dimensional problems, such as relating millions of genetic markers to thousands of imaging features in imaging genetics, on consumer-grade hardware.

PLS correlation

PLS correlation (PLSC) is another methodology related to PLS regression, which has been used in neuroimaging and sport science, to quantify the strength of the relationship between data sets. Typically, PLSC divides the data into two blocks (sub-groups) each containing one or more variables, and then uses singular value decomposition (SVD) to establish the strength of any relationship (i.e. the amount of shared information) that might exist between the two component sub-groups. It does this by using SVD to determine the inertia (i.e. the sum of the singular values) of the covariance matrix of the sub-groups under consideration.

Literature

Kramer, R. (1998). Chemometric Techniques for Quantitative Analysis. Marcel-Dekker. ISBN 978-0-8247-0198-7.
Frank, Ildiko E.; Friedman, Jerome H. (1993). "A Statistical View of Some Chemometrics Regression Tools". Technometrics. 35 (2): 109–148. doi:10.1080/00401706.1993.10485033.
Haenlein, Michael; Kaplan, Andreas M. (2004). "A Beginner's Guide to Partial Least Squares Analysis". Understanding Statistics. 3 (4): 283–297. doi:10.1207/s15328031us0304_4.
Henseler, Jörg; Fassott, Georg (2010). "Testing Moderating Effects in PLS Path Models: An Illustration of Available Procedures". In Vinzi, Vincenzo Esposito; Chin, Wynne W.; Henseler, Jörg; Wang, Huiwen (eds.). Handbook of Partial Least Squares: Concepts, Methods and Applications. Springer. pp. 713–735. doi:10.1007/978-3-540-32827-8_31. ISBN 978-3-540-32827-8.
Lingjærde, Ole-Christian; Christophersen, Nils (2000). "Shrinkage Structure of Partial Least Squares". Scandinavian Journal of Statistics. 27 (3): 459–473. doi:10.1111/1467-9469.00201. S2CID 121489764.
Tenenhaus, Michel (1998). La Régression PLS: Théorie et Pratique. Paris: Technip.
Rosipal, Roman; Krämer, Nicole (2006). "Overview and Recent Advances in Partial Least Squares". In Saunders, Craig; Grobelnik, Marko; Gunn, Steve; Shawe-Taylor, John (eds.). Subspace, Latent Structure and Feature Selection: Statistical and Optimization Perspectives Workshop, SLSFS 2005, Bohinj, Slovenia, February 23–25, 2005, Revised Selected Papers. Lecture Notes in Computer Science. Springer. pp. 34–51. doi:10.1007/11752790_2. ISBN 978-3-540-34138-3.
Helland, Inge S. (1990). "PLS regression and statistical models". Scandinavian Journal of Statistics. 17 (2): 97–114. JSTOR 4616159.
Wold, Herman (1966). "Estimation of principal components and related models by iterative least squares". In Krishnaiaah, P.R. (ed.). Multivariate Analysis. New York: Academic Press. pp. 391–420.
Wold, Herman (1981). The fix-point approach to interdependent systems. Amsterdam: North Holland.
Wold, Herman (1985). "Partial least squares". In Kotz, Samuel; Johnson, Norman L. (eds.). Encyclopedia of statistical sciences. Vol. 6. New York: Wiley. pp. 581–591.
Wold, Svante; Ruhe, Axel; Wold, Herman; Dunn, W.J. (1984). "The collinearity problem in linear regression. the partial least squares (PLS) approach to generalized inverses". SIAM Journal on Scientific and Statistical Computing. 5 (3): 735–743. doi:10.1137/0905052.
Garthwaite, Paul H. (1994). "An Interpretation of Partial Least Squares". Journal of the American Statistical Association. 89 (425): 122–7. doi:10.1080/01621459.1994.10476452. JSTOR 2291207.
Wang, H., ed. (2010). Handbook of Partial Least Squares. ISBN 978-3-540-32825-4.
Stone, M.; Brooks, R.J. (1990). "Continuum Regression: Cross-Validated Sequentially Constructed Prediction embracing Ordinary Least Squares, Partial Least Squares and Principal Components Regression". Journal of the Royal Statistical Society, Series B. 52 (2): 237–269. doi:10.1111/j.2517-6161.1990.tb01786.x. JSTOR 2345437.