Analysis of parallel algorithms

Definitions

Suppose computations are executed on a machine that has p processors. Let T_p denote the time that expires between the start of the computation and its end. Analysis of the computation's running time focuses on the following notions:

• The work of a computation executed by p processors is the total number of primitive operations that the processors perform. Ignoring communication overhead from synchronizing the processors, this is equal to the time used to run the computation on a single processor, denoted T₁.
• The depth or span is the length of the longest series of operations that have to be performed sequentially due to data dependencies (the critical path). The depth may also be called the critical path length of the computation. Minimizing the depth/span is important in designing parallel algorithms, because the depth/span determines the shortest possible execution time. Alternatively, the span can be defined as the time T_∞ spent computing using an idealized machine with an infinite number of processors.
• The cost of the computation is the quantity pT_p. This expresses the total time spent, by all processors, in both computing and waiting.

Several useful results follow from the definitions of work, span and cost:

• Work law. The cost is always at least the work: pT_p ≥ T₁. This follows from the fact that p processors can perform at most p operations in parallel.
• Span law. A finite number p of processors cannot outperform an infinite number, so that T_p ≥ T_∞.

Using these definitions and laws, the following measures of performance can be given:

• Speedup is the gain in speed made by parallel execution compared to sequential execution: S_p = T₁ / T_p. When the speedup is Ω(p) for p processors (using big O notation), the speedup is linear, which is optimal in simple models of computation because the work law implies that T₁ / T_p ≤ p (super-linear speedup can occur in practice due to memory hierarchy effects). The situation T₁ / T_p = p is called perfect linear speedup. An algorithm that exhibits linear speedup is said to be scalable. Analytical expressions for the speedup of many important parallel algorithms are presented in this book.
• Efficiency is the speedup per processor, S_p / p.
• Parallelism is the ratio T₁ / T_∞. It represents the maximum possible speedup on any number of processors. By the span law, the parallelism bounds the speedup: if p > T₁ / T_∞, then: $${\displaystyle \frac{T_1}{T_p}\le \frac{T_1}{T_{\mathrm{\infty }}}<p.}$$
• The slackness is T₁ / (pT_∞). A slackness less than one implies (by the span law) that perfect linear speedup is impossible on p processors.

Execution on a limited number of processors

Analysis of parallel algorithms is usually carried out under the assumption that an unbounded number of processors is available. This is unrealistic, but not a problem, since any computation that can run in parallel on N processors can be executed on p < N processors by letting each processor execute multiple units of work. A result called Brent's law states that one can perform such a "simulation" in time T_p, bounded by

${\displaystyle T_p\le T_N+\frac{T_1-T_N}{p},}$

or, less precisely,

${\displaystyle T_p=O\left(T_N+\frac{T_1}{p}\right).}$

An alternative statement of the law bounds T_p above and below by

${\displaystyle \frac{T_1}{p}\le T_p\le \frac{T_1}{p}+T_{\mathrm{\infty }}}$.

showing that the span (depth) T_∞ and the work T₁ together provide reasonable bounds on the computation time.