Lably
Skip to content

LU decomposition

Type of matrix factorization

  • 55 min read
  • 7 chapters

Related on Lably

Through Numerical analysis, Linear algebra and Matrix (mathematics)

Definitions

Let A be a square matrix. An LU factorization refers to expression of A into product of two factors – a lower triangular matrix L and an upper triangular matrix U such that A = LU. Sometimes factorization is impossible without prior reordering of A to prevent division by zero or uncontrolled growth of rounding errors. Hence alternative expression becomes PAQ = LU, where in formal notation permutation matrix factors P and Q indicate permutation of rows (or columns) of A. In theory P (or Q) is obtained by permutations of rows (or columns) of the identity matrix; in practice the corresponding permutations are applied directly to rows (or columns) of A.

Matrix A of side n has n2 coefficients while two triangle matrices combined contain n(n + 1) coefficients, and therefore n coefficients of matrices LU are not independent. The convention is to set L unitriangular, i.e. with all n main diagonal elements equal to one. However, setting instead U matrix unitriangular reduces to the same procedure after transpose of matrix product (cf. properties of matrix transposition): B=AT=(LU)T=UTLT. After transposition, UT is lower triangle while LT is upper unitriangular factor of B. This demonstrates also, that operations on rows (e.g. pivoting) are equivalent to those on columns of a transposed matrix, and in general choice of row or column algorithm offers no advantage.

In the lower triangular matrix all elements above the main diagonal are zero, in the upper triangular matrix, all the elements below the diagonal are zero. For example, for a 3 × 3 matrix A, its LU decomposition looks like this: [a11a12a13a21a22a23a31a32a33]=[ℓ1100ℓ21ℓ220ℓ31ℓ32ℓ33][u11u12u130u22u2300u33].

Without a proper ordering or permutations in the matrix, the factorization may fail to materialize. For example, it is easy to verify (by expanding the matrix multiplication) that a11=ℓ11u11. If a11=0, then at least one of ℓ11 and u11 has to be zero, which implies that either L or U is singular. This is impossible if A is nonsingular (invertible). In terms of operations, zeroing/elimination of remaining elements of first column of A involves division of a21,a31 with a11, impossible if it is 0. This is a procedural problem. It can be removed by simply reordering the rows of A so that the first element of the permuted matrix is nonzero. The same problem in subsequent factorization steps can be removed the same way. For numerical stability against rounding errors/division by small numbers it is important to select a11 of large absolute value (cf. pivoting).

LU through recursion

The above example of 3 × 3 matrices demonstrates that matrix product of top row and leftmost columns of involved matrices plays special role for LU to succeed. Let us mark consecutive versions of matrices with (0),(1),… and then let us write matrix product A≡A(0)=L(0)U(0) in such way that these rows and columns are separated from the rest. In doing so we shall use block matrix notation, such that e.g. a≡a11 is an ordinary number, wT≡(a12,a13)T is a row vector and v=(a21,a31) is a column vector and A′ is sub-matrix of matrix A(0) without top row and leftmost column. Then we can replace A(0)=L(0)U(0) with a block matrix product. Namely it turns out that one can multiply matrix blocks in such way as if they were ordinary numbers, i.e. row times column, except that now their components are sub-matrices, sometimes reduced to scalars or vectors. Thus ul denotes a vector obtained from l after multiplication of each component by a number u, luT is an outer product of vectors l,u, i.e. a matrix which first column is u12l, next is u13l and so on for all components of u and L(1)U(1) is a product of sub-matrices of L(0),U(0) (awTvA′)=(10TlL(1))(uuT0U(1))=(uuTulluT+L(1)U(1))

From equality of first and last matrices follow final u=a, u=w, l=(1/a)v while matrix A′ becomes updated/replaced with A(1)≡L(1)U(1)= A′−luT. Now comes the crucial observation: nothing prevents us to treat A(1) the same way as we did with A(0), repeatedly. If dimension of A is n × n, after n− 1 such steps all columns v form sub-diagonal part of triangle matrix L and all pivots a combined with rows wT form upper triangle matrix U, as required. In the above example n = 3 so only two steps suffice.

The above procedure demonstrates that at no step the top diagonal pivot element a of consecutive sub-matrices can be zero. To avoid it columns or rows may be swapped so that a becomes nonzero. Such procedure involving permutation is called LUP, decomposition with pivoting.

Permutation of columns corresponds to matrix product AQ(0) where Q(0) is a permutation matrix, i.e. the identity matrix I after the same column permutation. After all steps such LUP decomposition applies to AQ(0)⋯Q(n−1)≡AQ=LU. Present computation scheme and similar in Cormen et al. are examples of recurrence algorithms. They demonstrate two general properties of LU factorization:

  1. the need for pivoting at each step; and
  2. final values of L and U matrices are obtained gradually, one row or column per step.

Recurrence algorithms are not overly costly in terms of algebraic operations yet they suffer from practical disadvantage due to need to update and store most elements of A at each step. It will be seen that by reordering calculations it is possible to dispose with storage of intermediate values.

LU factorization with partial pivoting

It turns out that a proper permutation of rows (or columns) to select column (or row) absolute maximal pivot a11 is sufficient for numerically stable LU factorization, except for known pathological cases. It is called 'LU factorization with partial pivoting' (LUP): PA=LU,(AQ=LU), where L and U are again lower and upper triangular matrices, and P and Q are corresponding permutation matrices, which, when correspondingly left- and right-multiplied to A, reorder the rows and columns of A. It turns out that all square matrices can be factorized in this form, and the factorization is numerically stable in practice. This makes LUP decomposition a useful technique in practice.

A variant called 'rook pivoting' at each step involves search of maximum element the way rook moves on a chessboard, along column, row, column again and so on till reaching a pivot maximal in both its row and column. It can be proven that for large matrices of random elements its cost of operations at each step is similarly to partial pivoting proportional to the length of matrix side unlike its square for full pivoting.

LU factorization with full pivoting

An 'LU factorization with full pivoting' involves both row and column permutations to find absolute maximum element in the whole submatrix: PAQ=LU, where L, U, and P are defined as before, and Q is a permutation matrix that reorders the columns of A.

Lower-diagonal-upper (LDU) decomposition

A 'lower-diagonal-upper decomposition' (LDU) is a decomposition of the form A=LDU, where D is a diagonal matrix, and L and U are unitriangular matrices, meaning that all the entries on the diagonals of L and U are one.

Rectangular matrices

Above we required that A be a square matrix, but these decompositions can all be generalized to rectangular matrices as well. In that case, L and D are square matrices both of which have the same number of rows as A, and U has exactly the same dimensions as A. 'Upper triangular' should be interpreted as having only zero entries below the main diagonal, which starts at the upper left corner. Similarly, the more precise term for U is that it is the row echelon form of the matrix A.

Sign in to keep reading

6 more chapters are free with a Lably account.

More in Algebra