Chambolle–Pock algorithm

The Chambolle–Pock algorithm is specifically designed to efficiently solve convex optimization problems that involve the minimization of a non-smooth cost function composed of a data fidelity term and a regularization term. This is a typical configuration that commonly arises in ill-posed imaging inverse problems such as image reconstruction, denoising and inpainting.

The algorithm is based on a primal-dual formulation, which allows for simultaneous updates of primal and dual variables. By employing the proximal operator, the Chambolle–Pock algorithm efficiently handles non-smooth and non-convex regularization terms, such as the total variation, specific in imaging framework.

Problem statement

Let be $X, Y$ two real vector spaces equipped with an inner product $⟨ \cdot, \cdot ⟩$ and a norm $‖ \cdot ‖ = ⟨ \cdot, \cdot ⟩^{\frac{1}{2}}$ . From now on, a function $F$ is called simple if its proximal operator ${prox}_{τ F}$ has a closed-form representation or can be accurately computed, for $τ > 0$ , where ${prox}_{τ F}$ is referred to

x = {prox}_{τ F} (\tilde{x}) = arg min_{x^{'} \in X} {\frac{‖ x^{'} - \tilde{x} ‖^{2}}{2 τ} + F (x^{'})}

Consider the following constrained primal problem:

min_{x \in X} F (K x) + G (x)

where $K : X \to Y$ is a bounded linear operator, $F : Y \to [0, + \infty), G : X \to [0, + \infty)$ are convex, lower semicontinuous and simple.

The minimization problem has its dual corresponding problem as

max_{y \in Y} - (G^{*} (- K^{*} y) + F^{*} (y))

where $F^{*}, G^{*}$ and $K^{*}$ are the dual map of $F, G$ and $K$ , respectively.

Assume that the primal and the dual problems have at least a solution $(\hat{x}, \hat{y}) \in X \times Y$ , that means they satisfy

$\begin{matrix} K \hat{x} & \in \partial F^{*} (\hat{y}) \\ - (K^{*} \hat{y}) & \in \partial G (\hat{x}) \end{matrix}$

where $\partial F^{*}$ and $\partial G$ are the subgradient of the convex functions $F^{*}$ and $G$ , respectively.

The Chambolle–Pock algorithm solves the so-called saddle-point problem

min_{x \in X} max_{y \in Y} ⟨ K x, y ⟩ + G (x) - F^{*} (y)

which is a primal-dual formulation of the nonlinear primal and dual problems stated before.

4 more chapters are free with a Lably account.

Algorithm

The Chambolle–Pock algorithm primarily involves iteratively alternating between ascending in the dual variable $y$ and descending in the primal variable $x$ using a gradient-like approach, with step sizes $σ$ and $τ$ respectively, in order to simultaneously solve the primal and the dual problem. Furthermore, an over-relaxation technique is employed for the primal variable with the parameter $θ$ .

<b>Algorithm</b> Chambolle-Pock algorithm <span class="nowrap">Input: <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt70" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" F, G, K, \\tau, \\sigma >0, \\, \\theta \\in[0,1],\\, (x^0,y^0)\\in\\mathcal{X}\\times\\mathcal{Y}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle F,G,K,\tau ,\sigma >0,\,\theta \in [0,1],\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>F</mi> <mo>,</mo> <mi>G</mi> <mo>,</mo> <mi>K</mi> <mo>,</mo> <mi>τ<!-- τ --></mi> <mo>,</mo> <mi>σ<!-- σ --></mi> <mo>></mo> <mn>0</mn> <mo>,</mo> <mspace width="thinmathspace"></mspace> <mi>θ<!-- θ --></mi> <mo>∈<!-- ∈ --></mo> <mo stretchy="false">[</mo> <mn>0</mn> <mo>,</mo> <mn>1</mn> <mo stretchy="false">]</mo> <mo>,</mo> <mspace width="thinmathspace"></mspace> <mo stretchy="false">(</mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo>,</mo> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo stretchy="false">)</mo> <mo>∈<!-- ∈ --></mo> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi class="MJX-tex-caligraphic" mathvariant="script">X</mi> </mrow> </mrow> <mo>×<!-- × --></mo> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi class="MJX-tex-caligraphic" mathvariant="script">Y</mi> </mrow> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle F,G,K,\tau ,\sigma >0,\,\theta \in [0,1],\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/11b98c28c274409ea67b6aa8e6c267a446710429" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:44.645ex; height:3.176ex;" alt="{\displaystyle F,G,K,\tau ,\sigma >0,\,\theta \in [0,1],\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}"/></span> and set <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt71" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\overline{x}^0 = x^0"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle {\overline {x}}^{0}=x^{0}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo>=</mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle {\overline {x}}^{0}=x^{0}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/5e4dac3d5d6a1435b7dcc710906e86040402d3c6" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:7.981ex; height:2.843ex;" alt="{\displaystyle {\overline {x}}^{0}=x^{0}}"/></span>,</span> <i>stopping criterion</i>.

<span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt73" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" k \\leftarrow 0 "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k\leftarrow 0}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo stretchy="false">←<!-- ← --></mo> <mn>0</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k\leftarrow 0}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/eba4e1dbc3c8f3ccb6b0f50e5791996d25a9855b" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.988ex; height:2.176ex;" alt="{\displaystyle k\leftarrow 0}"/></span></span> <span class="nowrap"><b>do while</b> <i>stopping criterion</i> <b>not satisfied</b></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt76" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" y^{n+1} \\leftarrow \\text{prox}_{\\sigma F^*}\\left(y^{n} + \\sigma K\\overline{x}^{n}\\right) "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma F^{*}}\left(y^{n}+\sigma K{\overline {x}}^{n}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msub> <mrow class="MJX-TeXAtom-ORD"> <mtext>prox</mtext> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>σ<!-- σ --></mi> <msup> <mi>F</mi> <mrow class="MJX-TeXAtom-ORD"> <mo>∗<!-- ∗ --></mo> </mrow> </msup> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> <mo>+</mo> <mi>σ<!-- σ --></mi> <mi>K</mi> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma F^{*}}\left(y^{n}+\sigma K{\overline {x}}^{n}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/63401ff8b37c650d04a7b2c1ff3862c494d22211" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:29.77ex; height:3.343ex;" alt="{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma F^{*}}\left(y^{n}+\sigma K{\overline {x}}^{n}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt78" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" x^{n+1} \\leftarrow \\text{prox}_{\\tau G}\\left(x^{n} - \\tau K^*y^{n+1}\\right) "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau G}\left(x^{n}-\tau K^{*}y^{n+1}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msub> <mrow class="MJX-TeXAtom-ORD"> <mtext>prox</mtext> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>τ<!-- τ --></mi> <mi>G</mi> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> <mo>−<!-- − --></mo> <mi>τ<!-- τ --></mi> <msup> <mi>K</mi> <mrow class="MJX-TeXAtom-ORD"> <mo>∗<!-- ∗ --></mo> </mrow> </msup> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau G}\left(x^{n}-\tau K^{*}y^{n+1}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/c07f3a756dce2bcfee0c1fe8fa37d92b07afae29" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:31.966ex; height:3.343ex;" alt="{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau G}\left(x^{n}-\tau K^{*}y^{n+1}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt80" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\overline{x}^{n+1} \\leftarrow x^{n+1} + \\theta\\left( x^{n+1} -x^{n}\\right)"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta \left(x^{n+1}-x^{n}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>+</mo> <mi>θ<!-- θ --></mi> <mrow> <mo>(</mo> <mrow> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>−<!-- − --></mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta \left(x^{n+1}-x^{n}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/29f8e50c78368c4fe6b6606aadae7523c1a2d8b3" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:29.511ex; height:3.509ex;" alt="{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta \left(x^{n+1}-x^{n}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt82" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"k\\leftarrow k+1"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k\leftarrow k+1}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo stretchy="false">←<!-- ← --></mo> <mi>k</mi> <mo>+</mo> <mn>1</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k\leftarrow k+1}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/469b0d14d5eff3c19bde9957713c939411776b66" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:10.039ex; height:2.343ex;" alt="{\displaystyle k\leftarrow k+1}"/></span></span> <b>end do</b>

"←" denotes assignment. For instance, "largest ← item" means that the value of largest changes to the value of item.
"return" terminates the algorithm and outputs the following value.

Chambolle and Pock proved that the algorithm converges if $θ = 1$ and $τ σ ‖ K ‖^{2} \leq 1$ , sequentially and with $O (1 / N)$ as rate of convergence for the primal-dual gap. This has been extended by S. Banert et al. to hold whenever $θ > 1 / 2$ and $τ σ ‖ K ‖^{2} < 4 / (1 + 2 θ)$ .

The semi-implicit Arrow-Hurwicz method coincides with the particular choice of $θ = 0$ in the Chambolle–Pock algorithm.

Acceleration

There are special cases in which the rate of convergence has a theoretical speed up. In fact, if $G$ , respectively $F^{*}$ , is uniformly convex then $G^{*}$ , respectively $F$ , has a Lipschitz continuous gradient. Then, the rate of convergence can be improved to $O (1 / N^{2})$ , providing a slightly changes in the Chambolle–Pock algorithm. It leads to an accelerated version of the method and it consists in choosing iteratively $τ_{n}, σ_{n}$ , and also $θ_{n}$ , instead of fixing these values.

In case of $G$ uniformly convex, with $γ > 0$ the uniform-convexity constant, the modified algorithm becomes

<b>Algorithm</b> Accelerated Chambolle-Pock algorithm <span class="nowrap">Input: <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt114" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" F, G, \\tau_0, \\sigma_0 >0"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle F,G,\tau _{0},\sigma _{0}>0}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>F</mi> <mo>,</mo> <mi>G</mi> <mo>,</mo> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <mo>,</mo> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <mo>></mo> <mn>0</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle F,G,\tau _{0},\sigma _{0}>0}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/81b9a2b5b87ccf116beb9de16d9467ba5e09d984" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:15.382ex; height:2.509ex;" alt="{\displaystyle F,G,\tau _{0},\sigma _{0}>0}"/></span> such that <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt115" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\tau_0\\sigma_0 L^2 \\leq 1,\\, (x^0,y^0)\\in\\mathcal{X}\\times\\mathcal{Y}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \tau _{0}\sigma _{0}L^{2}\leq 1,\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msub> <msup> <mi>L</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>2</mn> </mrow> </msup> <mo>≤<!-- ≤ --></mo> <mn>1</mn> <mo>,</mo> <mspace width="thinmathspace"></mspace> <mo stretchy="false">(</mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo>,</mo> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo stretchy="false">)</mo> <mo>∈<!-- ∈ --></mo> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi class="MJX-tex-caligraphic" mathvariant="script">X</mi> </mrow> </mrow> <mo>×<!-- × --></mo> <mrow class="MJX-TeXAtom-ORD"> <mrow class="MJX-TeXAtom-ORD"> <mi class="MJX-tex-caligraphic" mathvariant="script">Y</mi> </mrow> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \tau _{0}\sigma _{0}L^{2}\leq 1,\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/2fe447305d83b62777c847094a0c7b84e95644fc" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:29.429ex; height:3.176ex;" alt="{\displaystyle \tau _{0}\sigma _{0}L^{2}\leq 1,\,(x^{0},y^{0})\in {\mathcal {X}}\times {\mathcal {Y}}}"/></span> and set <span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt116" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\overline{x}^0 = x^0."}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle {\overline {x}}^{0}=x^{0}.}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo>=</mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mn>0</mn> </mrow> </msup> <mo>.</mo> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle {\overline {x}}^{0}=x^{0}.}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/6f214fc65483ce1410a8d946a4dba38bccce2678" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:8.628ex; height:2.843ex;" alt="{\displaystyle {\overline {x}}^{0}=x^{0}.}"/></span>,</span> <i>stopping criterion</i>.

<span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt118" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" k \\leftarrow 0 "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k\leftarrow 0}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo stretchy="false">←<!-- ← --></mo> <mn>0</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k\leftarrow 0}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/eba4e1dbc3c8f3ccb6b0f50e5791996d25a9855b" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:5.988ex; height:2.176ex;" alt="{\displaystyle k\leftarrow 0}"/></span></span> <span class="nowrap"><b>do while</b> <i>stopping criterion</i> <b>not satisfied</b></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt121" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" y^{n+1} \\leftarrow \\text{prox}_{\\sigma_n F^*}\\left(y^{n} + \\sigma_n K\\overline{x}^{n}\\right) "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma _{n}F^{*}}\left(y^{n}+\sigma _{n}K{\overline {x}}^{n}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msub> <mrow class="MJX-TeXAtom-ORD"> <mtext>prox</mtext> </mrow> <mrow class="MJX-TeXAtom-ORD"> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <msup> <mi>F</mi> <mrow class="MJX-TeXAtom-ORD"> <mo>∗<!-- ∗ --></mo> </mrow> </msup> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> <mo>+</mo> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <mi>K</mi> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma _{n}F^{*}}\left(y^{n}+\sigma _{n}K{\overline {x}}^{n}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/86e68ebc6f0fa83085c8dce9cdf450fb7aec1ec1" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:31.949ex; height:3.343ex;" alt="{\displaystyle y^{n+1}\leftarrow {\text{prox}}_{\sigma _{n}F^{*}}\left(y^{n}+\sigma _{n}K{\overline {x}}^{n}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt123" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" x^{n+1} \\leftarrow \\text{prox}_{\\tau_n G}\\left(x^{n} - \\tau_n K^*y^{n+1}\\right) "}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau _{n}G}\left(x^{n}-\tau _{n}K^{*}y^{n+1}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msub> <mrow class="MJX-TeXAtom-ORD"> <mtext>prox</mtext> </mrow> <mrow class="MJX-TeXAtom-ORD"> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <mi>G</mi> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> <mo>−<!-- − --></mo> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <msup> <mi>K</mi> <mrow class="MJX-TeXAtom-ORD"> <mo>∗<!-- ∗ --></mo> </mrow> </msup> <msup> <mi>y</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau _{n}G}\left(x^{n}-\tau _{n}K^{*}y^{n+1}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/726803787cbe3a3b8740ada6fc8b203378013619" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:33.832ex; height:3.343ex;" alt="{\displaystyle x^{n+1}\leftarrow {\text{prox}}_{\tau _{n}G}\left(x^{n}-\tau _{n}K^{*}y^{n+1}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt125" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\theta_n \\leftarrow \\frac{1}{\\sqrt{1+2\\gamma \\tau_n}}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \theta _{n}\leftarrow {\frac {1}{\sqrt {1+2\gamma \tau _{n}}}}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>θ<!-- θ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <mrow class="MJX-TeXAtom-ORD"> <mfrac> <mn>1</mn> <msqrt> <mn>1</mn> <mo>+</mo> <mn>2</mn> <mi>γ<!-- γ --></mi> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> </msqrt> </mfrac> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \theta _{n}\leftarrow {\frac {1}{\sqrt {1+2\gamma \tau _{n}}}}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/f40897b992fc3a8434106f0cbe63f890c499174d" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.171ex; width:17.745ex; height:6.509ex;" alt="{\displaystyle \theta _{n}\leftarrow {\frac {1}{\sqrt {1+2\gamma \tau _{n}}}}}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt127" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\tau_{n+1} \\leftarrow \\theta_n \\tau_n"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \tau _{n+1}\leftarrow \theta _{n}\tau _{n}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <msub> <mi>θ<!-- θ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <msub> <mi>τ<!-- τ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \tau _{n+1}\leftarrow \theta _{n}\tau _{n}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/6d5a30a66a51278f1af7d4d2956a2eed2cc4b02a" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:12.493ex; height:2.509ex;" alt="{\displaystyle \tau _{n+1}\leftarrow \theta _{n}\tau _{n}}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt129" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\sigma_{n+1} \\leftarrow \\frac{\\sigma_n}{\\theta_n}"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle \sigma _{n+1}\leftarrow {\frac {\sigma _{n}}{\theta _{n}}}}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msub> <mo stretchy="false">←<!-- ← --></mo> <mrow class="MJX-TeXAtom-ORD"> <mfrac> <msub> <mi>σ<!-- σ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <msub> <mi>θ<!-- θ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> </mfrac> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle \sigma _{n+1}\leftarrow {\frac {\sigma _{n}}{\theta _{n}}}}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/c1cc155dd4cb5515b0eff8c00008d170fe962f99" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.338ex; width:11.642ex; height:5.176ex;" alt="{\displaystyle \sigma _{n+1}\leftarrow {\frac {\sigma _{n}}{\theta _{n}}}}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt131" data-mw='{"name":"math","attrs":{},"body":{"extsrc":" \\overline{x}^{n+1} \\leftarrow x^{n+1} + \\theta_n\\left( x^{n+1} -x^{n}\\right)"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta _{n}\left(x^{n+1}-x^{n}\right)}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <msup> <mrow class="MJX-TeXAtom-ORD"> <mover> <mi>x</mi> <mo accent="false">¯<!-- ¯ --></mo> </mover> </mrow> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo stretchy="false">←<!-- ← --></mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>+</mo> <msub> <mi>θ<!-- θ --></mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> <mo>+</mo> <mn>1</mn> </mrow> </msup> <mo>−<!-- − --></mo> <msup> <mi>x</mi> <mrow class="MJX-TeXAtom-ORD"> <mi>n</mi> </mrow> </msup> </mrow> <mo>)</mo> </mrow> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta _{n}\left(x^{n+1}-x^{n}\right)}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/e9acb40f903f3ee3262d95878e64e096ed64004c" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:30.73ex; height:3.509ex;" alt="{\displaystyle {\overline {x}}^{n+1}\leftarrow x^{n+1}+\theta _{n}\left(x^{n+1}-x^{n}\right)}"/></span></span> <span class="nowrap"><span class="mwe-math-element mwe-math-element-inline" typeof="mw:Extension/math" about="#mwt133" data-mw='{"name":"math","attrs":{},"body":{"extsrc":"k\\leftarrow k+1"}}'><span class="mwe-math-mathml-a11y" style="display: none;"><math xmlns="http://www.w3.org/1998/Math/MathML" class="mathjax_ignore" alttext="{\displaystyle k\leftarrow k+1}"> <semantics> <mrow class="MJX-TeXAtom-ORD"> <mstyle displaystyle="true" scriptlevel="0"> <mi>k</mi> <mo stretchy="false">←<!-- ← --></mo> <mi>k</mi> <mo>+</mo> <mn>1</mn> </mstyle> </mrow> <annotation encoding="application/x-tex">{\displaystyle k\leftarrow k+1}</annotation> </semantics> </math></span><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/469b0d14d5eff3c19bde9957713c939411776b66" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:10.039ex; height:2.343ex;" alt="{\displaystyle k\leftarrow k+1}"/></span></span> <b>end do</b>

"←" denotes assignment. For instance, "largest ← item" means that the value of largest changes to the value of item.
"return" terminates the algorithm and outputs the following value.

Moreover, the convergence of the algorithm slows down when $L$ , the norm of the operator $K$ , cannot be estimated easily or might be very large. Choosing proper preconditioners $T$ and $Σ$ , modifying the proximal operator with the introduction of the induced norm through the operators $T$ and $Σ$ , the convergence of the proposed preconditioned algorithm will be ensured.

Application

A typical application of this algorithm is in the image denoising framework, based on total variation. It operates on the concept that signals containing excessive and potentially erroneous details exhibit a high total variation, which represents the integral of the absolute value gradient of the image. By adhering to this principle, the process aims to decrease the total variation of the signal while maintaining its similarity to the original signal, effectively eliminating unwanted details while preserving crucial features like edges. In the classical bi-dimensional discrete setting, consider $X = R^{N M}$ , where an element $u \in X$ represents an image with the pixels values collocated in a Cartesian grid $N \times M$ .

Define the inner product on $X$ as

⟨ u, v ⟩_{X} = \sum_{i, j} u_{i, j} v_{i, j}, u, v \in X

that induces an $L^{2}$ norm on $X$ , denoted as $‖ \cdot ‖_{2}$ .

Hence, the gradient of $u$ is computed with the standard finite differences,

{(\nabla u)}_{i, j} = (\begin{matrix} {(\nabla u)}_{i, j}^{1} \\ {(\nabla u)}_{i, j}^{2} \end{matrix})

which is an element of the space $Y = X \times X$ , where

\begin{matrix} {(\nabla u)}_{i, j}^{1} = {\begin{matrix} \frac{u_{i + 1, j} - u_{i, j}}{h} & if i < M \\ 0 & if i = M \end{matrix}, \\ {(\nabla u)}_{i, j}^{2} = {\begin{matrix} \frac{u_{i, j + 1} - u_{i, j}}{h} & if j < N \\ 0 & if j = N \end{matrix} \end{matrix}

On $Y$ is defined an $L^{1} -$ based norm as

‖ p ‖_{1} = \sum_{i, j} \sqrt{{(p_{i, j}^{1})}^{2} + {(p_{i, j}^{2})}^{2}}, p \in Y .

Then, the primal problem of the ROF model, proposed by Rudin, Osher, and Fatemi, is given by

h^{2} min_{u \in X} ‖ \nabla u ‖_{1} + \frac{λ}{2} ‖ u - g ‖_{2}^{2}

where $u \in X$ is the unknown solution and $g \in X$ the given noisy data, instead $λ$ describes the trade-off between regularization and data fitting.

The primal-dual formulation of the ROF problem is formulated as follow

min_{u \in X} max_{p \in Y} - ⟨ u, div p ⟩_{X} + \frac{λ}{2} ‖ u - g ‖_{2}^{2} - δ_{P} (p)

where the indicator function is defined as

δ_{P} (p) = {\begin{matrix} 0, & if p \in P \\ + \infty, & if p \notin P \end{matrix}

on the convex set $P = {p \in Y : max_{i, j} \sqrt{{(p_{i, j}^{1})}^{2} + {(p_{i, j}^{2})}^{2}} \leq 1},$ which can be seen as $L^{\infty}$ unitary balls with respect to the defined norm on $Y$ .

Observe that the functions involved in the stated primal-dual formulation are simple, since their proximal operator can be easily computed $\begin{matrix} p & = {prox}_{σ F^{*}} (\tilde{p}) & ⟺ p_{i, j} & = \frac{{\tilde{p}}_{i, j}}{max {1, | {\tilde{p}}_{i, j} |}} \\ u & = {prox}_{τ G} (\tilde{u}) & ⟺ u_{i, j} & = \frac{{\tilde{u}}_{i, j} + τ λ g_{i, j}}{1 + τ λ} \end{matrix}$ The image total-variation denoising problem can be also treated with other algorithms such as the alternating direction method of multipliers (ADMM), projected (sub)-gradient or fast iterative shrinkage thresholding.

Implementation

Muhammad Maaz implements the algorithm in PyTorch for GPU-accelerated linear programming in his Primal-Dual Algorithm for Linear Programming GitHub Repository

The Manopt.jl package implements the algorithm in Julia generalised to Riemannian manifolds
Gabriel Peyré implements the algorithm in MATLAB, Julia, R and Python
In the Operator Discretization Library (ODL), a Python library for inverse problems, chambolle_pock_solver implements the method.