Farkas' lemma

Example

Let m, n = 2, ${\displaystyle \mathbf{A}=\left[\begin{array}{cc}6& 4\\ 3& 0\end{array}\right],}$ and ${\displaystyle \mathbf{b}=\left[\begin{array}{c}{b}_1\\ b_2\end{array}\right].}$ The lemma says that exactly one of the following two statements must be true (depending on b₁ and b₂):

1. There exist x₁ ≥ 0, x₂ ≥ 0 such that 6x₁ + 4x₂ = b₁ and 3x₁ = b₂, or
2. There exist y₁, y₂ such that 6y₁ + 3y₂ ≥ 0, 4y₁ ≥ 0, and b₁ y₁ + b₂ y₂ < 0.

Here is a proof of the lemma in this special case:

• If b₂ ≥ 0 and b₁ − 2b₂ ≥ 0, then option 1 is true, since the solution of the linear equations is ${\displaystyle x_1=\frac{b_2}{3}}$ and ${\displaystyle x_2=\frac{b_1-2b_2}{4}.}$ Option 2 is false, since $${\displaystyle b_1{y}_1+b_2{y}_2\ge b_2(2y_1+y_2)=b_2\frac{6y_1+3y_2}{3},}$$ so if the right-hand side is positive, the left-hand side must be positive too.
• Otherwise, option 1 is false, since the unique solution of the linear equations is not weakly positive. But in this case, option 2 is true:
  • If b₂ < 0, then we can take e.g. y₁ = 0 and y₂ = 1.
  • If b₁ − 2b₂ < 0, then, for some number B > 0, b₁ = 2b₂ − B, so: $${\displaystyle b_1{y}_1+b_2{y}_2=2b_2{y}_1+b_2{y}_2-B{y}_1=b_2\frac{6y_1+3y_2}{3}-B{y}_1.}$$ Thus we can take, for example, y₁ = 1, y₂ = −2.

Geometric interpretation

Consider the closed convex cone ${\displaystyle C(\mathbf{A})}$ spanned by the columns of A; that is,

${\displaystyle C(\mathbf{A})=\{\mathbf{A}\mathbf{x}\mid \mathbf{x}\ge 0\}.}$

Observe that ${\displaystyle C(\mathbf{A})}$ is the set of the vectors b for which the first assertion in the statement of Farkas' lemma holds. On the other hand, the vector y in the second assertion is orthogonal to a hyperplane that separates b and ${\displaystyle C(\mathbf{A}).}$ The lemma follows from the observation that b belongs to ${\displaystyle C(\mathbf{A})}$ if and only if there is no hyperplane that separates it from ${\displaystyle C(\mathbf{A}).}$

More precisely, let ${\displaystyle {\mathbf{a}}_1,\dots ,{\mathbf{a}}_n\in {\mathbb{R}}^m}$ denote the columns of A. In terms of these vectors, Farkas' lemma states that exactly one of the following two statements is true:

1. There exist non-negative coefficients ${\displaystyle x_1,\dots ,x_n\in \mathbb{R}}$ such that ${\displaystyle \mathbf{b}=x_1{\mathbf{a}}_1+\cdots +x_n{\mathbf{a}}_n.}$
2. There exists a vector ${\displaystyle \mathbf{y}\in {\mathbb{R}}^m}$ such that ${\displaystyle {\mathbf{a}}_i^{\mathrm{\top }}\mathbf{y}\ge 0}$ for ${\displaystyle i=1,\dots ,n,}$ and ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}<0.}$

The sums ${\displaystyle x_1{\mathbf{a}}_1+\cdots +x_n{\mathbf{a}}_n}$ with nonnegative coefficients ${\displaystyle x_1,\dots ,x_n}$ form the cone spanned by the columns of A. Therefore, the first statement tells that b belongs to ${\displaystyle C(\mathbf{A}).}$

The second statement tells that there exists a vector y such that the angle of y with the vectors a_i is at most 90°, while the angle of y with the vector b is more than 90°. The hyperplane normal to this vector has the vectors a_i on one side and the vector b on the other side. Hence, this hyperplane separates the cone spanned by ${\displaystyle {\mathbf{a}}_1,\dots ,{\mathbf{a}}_n}$ from the vector b.

For example, let n, m = 2, a₁ = (1, 0)^T, and a₂ = (1, 1)^T. The convex cone spanned by a₁ and a₂ can be seen as a wedge-shaped slice of the first quadrant in the xy plane. Now, suppose b = (0, 1). Certainly, b is not in the convex cone a₁x₁ + a₂x₂. Hence, there must be a separating hyperplane. Let y = (1, −1)^T. We can see that a₁ · y = 1, a₂ · y = 0, and b · y = −1. Hence, the hyperplane with normal y indeed separates the convex cone a₁x₁ + a₂x₂ from b.

Logic interpretation

A particularly suggestive and easy-to-remember version is the following: if a set of linear inequalities has no solution, then a contradiction can be produced from it by linear combination with nonnegative coefficients. In formulas: if ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ is unsolvable then ${\displaystyle {\mathbf{y}}^{\mathrm{\top }}\mathbf{A}=0,}$ ${\displaystyle {\mathbf{y}}^{\mathrm{\top }}\mathbf{b}=-1,}$ ${\displaystyle \mathbf{y}\ge 0}$ has a solution. Note that ${\displaystyle {\mathbf{y}}^{\mathrm{\top }}\mathbf{A}}$ is a combination of the left-hand sides, ${\displaystyle {\mathbf{y}}^{\mathrm{\top }}\mathbf{b}}$ a combination of the right-hand side of the inequalities. Since the positive combination produces a zero vector on the left and a −1 on the right, the contradiction is apparent.

Thus, Farkas' lemma can be viewed as a theorem of logical completeness: ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ is a set of "axioms", the linear combinations are the "derivation rules", and the lemma says that, if the set of axioms is inconsistent, then it can be refuted using the derivation rules.

Implications in complexity theory

Farkas' lemma implies that the decision problem "Given a system of linear equations, does it have a non-negative solution?" is in the intersection of NP and co-NP. This is because, according to the lemma, both a "yes" answer and a "no" answer have a proof that can be verified in polynomial time. The problems in the intersection ${\displaystyle NP\cap coNP}$ are also called well-characterized problems. It is a long-standing open question whether ${\displaystyle NP\cap coNP}$ is equal to P. In particular, the question of whether a system of linear equations has a non-negative solution was not known to be in P, until it was proved using the ellipsoid method.

Variants

The Farkas Lemma has several variants with different sign constraints (the first one is the original version):

• Either ${\displaystyle \mathbf{A}\mathbf{x}=\mathbf{b}}$ has a solution ${\displaystyle \mathbf{x}\ge 0,}$ or ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}\ge 0}$ has a solution ${\displaystyle \mathbf{y}\in {\mathbb{R}}^m}$ with ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}<0.}$
• Either ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ has a solution ${\displaystyle \mathbf{x}\ge 0,}$ or ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}\ge 0}$ has a solution ${\displaystyle \mathbf{y}\ge 0}$ with ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}<0}$
• Either ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ has a solution ${\displaystyle \mathbf{x}\in {\mathbb{R}}^n,}$ or ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}=0}$ has a solution ${\displaystyle \mathbf{y}\ge 0}$ with ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}<0}$.
• Either ${\displaystyle \mathbf{A}\mathbf{x}=\mathbf{b}}$ has a solution ${\displaystyle \mathbf{x}\in {\mathbb{R}}^n,}$ or ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}=0}$ has a solution ${\displaystyle \mathbf{y}\in {\mathbb{R}}^m}$ with ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}\ne 0.}$

The latter variant is mentioned for completeness; it is not actually a "Farkas lemma" since it contains only equalities. Its proof is an exercise in linear algebra.

There are also Farkas-like lemmas for integer programs. For systems of equations, the lemma is simple:

• Assume that A and b have rational coefficients. Then either ${\displaystyle \mathbf{A}\mathbf{x}=\mathbf{b}}$ has an integral solution ${\displaystyle \mathbf{x}\in {\mathbb{Z}}^n,}$ or there exists ${\displaystyle \mathbf{y}\in {\mathbb{R}}^n}$such that ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}}$ is integral and ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}}$ is not integral.

For system of inequalities, the lemma is much more complicated. It is based on the following two rules of inference:

1. Given inequalities ${\displaystyle a_1^{T}x\le b_1,\dots ,a_m^{T}x\le b_m}$ and coefficients ${\displaystyle w_1,\dots ,w_m}$, infer the inequality ${\displaystyle \left(\sum _{i=1}^m{w}_i{a}_i^T\right)x\le \sum _{i=1}^m{w}_i{b}_i}$.
2. Given an inequality ${\displaystyle a_1{x}_1+\cdots +a_m{x}_m\le b}$, infer the inequality ${\displaystyle \lfloor a_1\rfloor x_1+\cdots +\lfloor a_m\rfloor x_m\le \lfloor b\rfloor }$.

The lemma says that:

• Assume that A and b have rational coefficients. Then either ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ has an integral solution ${\displaystyle \mathbf{x}\in {\mathbb{Z}}^n,x\ge 0}$, or it is possible to infer from ${\displaystyle \mathbf{A}\mathbf{x}\le \mathbf{b}}$ using finitely many applications of inference rules 1,2 the inequality ${\displaystyle 0^{T}x\le -1}$.

The variants are summarized in the table below.

Generalizations

Generalized Farkas' lemma— Let ${\displaystyle \mathbf{A}\in {\mathbb{R}}^{m\times n},}$ ${\displaystyle \mathbf{b}\in {\mathbb{R}}^m,}$ ${\displaystyle \mathbf{S}}$ is a closed convex cone in ${\displaystyle {\mathbb{R}}^n,}$ and the dual cone of ${\displaystyle \mathbf{S}}$ is ${\displaystyle {\mathbf{S}}^{\ast }=\{\mathbf{z}\in {\mathbb{R}}^n\mid {\mathbf{z}}^{\mathrm{\top }}\mathbf{x}\ge 0,\mathrm{\forall }\mathbf{x}\in \mathbf{S}\}.}$ If convex cone ${\displaystyle C(\mathbf{A})=\{\mathbf{A}\mathbf{x}\mid \mathbf{x}\in \mathbf{S}\}}$ is closed, then exactly one of the following two statements is true:

1. There exists an ${\displaystyle \mathbf{x}\in {\mathbb{R}}^n}$ such that ${\displaystyle \mathbf{A}\mathbf{x}=\mathbf{b}}$ and ${\displaystyle \mathbf{x}\in \mathbf{S}.}$
2. There exists a ${\displaystyle \mathbf{y}\in {\mathbb{R}}^m}$ such that ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}\in {\mathbf{S}}^{\ast }}$ and ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}<0.}$

Generalized Farkas' lemma can be interpreted geometrically as follows: either a vector is in a given closed convex cone, or there exists a hyperplane separating the vector from the cone; there are no other possibilities. The closedness condition is necessary, see Separation theorem I in Hyperplane separation theorem. For original Farkas' lemma, ${\displaystyle \mathbf{S}}$ is the nonnegative orthant ${\displaystyle {\mathbb{R}}_{+}^n,}$ hence the closedness condition holds automatically. Indeed, for polyhedral convex cone, i.e., there exists a ${\displaystyle \mathbf{B}\in {\mathbb{R}}^{n\times k}}$ such that ${\displaystyle \mathbf{S}=\{\mathbf{B}\mathbf{x}\mid \mathbf{x}\in {\mathbb{R}}_{+}^k\},}$ the closedness condition holds automatically. In convex optimization, various kinds of constraint qualification, e.g. Slater's condition, are responsible for closedness of the underlying convex cone ${\displaystyle C(\mathbf{A}).}$

By setting ${\displaystyle \mathbf{S}={\mathbb{R}}^n}$ and ${\displaystyle {\mathbf{S}}^{\ast }=\{0\}}$ in generalized Farkas' lemma, we obtain the following corollary about the solvability for a finite system of linear equalities:

Corollary (Fredholm alternative)— Let ${\displaystyle \mathbf{A}\in {\mathbb{R}}^{m\times n}}$ and ${\displaystyle \mathbf{b}\in {\mathbb{R}}^m.}$ Then exactly one of the following two statements is true:

1. There exists an ${\displaystyle \mathbf{x}\in {\mathbb{R}}^n}$ such that ${\displaystyle \mathbf{A}\mathbf{x}=\mathbf{b}.}$
2. There exists a ${\displaystyle \mathbf{y}\in {\mathbb{R}}^m}$ such that ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}=0}$ and ${\displaystyle {\mathbf{b}}^{\mathrm{\top }}\mathbf{y}\ne 0.}$

Further implications

Farkas' lemma can be varied to many further theorems of alternative by simple modifications, such as Gordan's theorem: Either ${\displaystyle \mathbf{A}\mathbf{x}<0}$ has a solution x, or ${\displaystyle {\mathbf{A}}^{\mathrm{\top }}\mathbf{y}=0}$ has a nonzero solution y with y ≥ 0.

Common applications of Farkas' lemma include proving the strong duality theorem associated with linear programming and the Karush–Kuhn–Tucker conditions. An extension of Farkas' lemma can be used to analyze the strong duality conditions for and construct the dual of a semidefinite program. It is sufficient to prove the existence of the Karush–Kuhn–Tucker conditions using the Fredholm alternative but for the condition to be necessary, one must apply von Neumann's minimax theorem to show the equations derived by Cauchy are not violated.

This is used for Dill's Reluplex method for verifying deep neural networks.