Loss function

Constructing loss and objective functions

In many applications, objective functions, including loss functions as a particular case, are determined by the problem formulation. In other situations, the decision maker’s preference must be elicited and represented by a scalar-valued function (called also utility function) in a form suitable for optimization — the problem that Ragnar Frisch has highlighted in his Nobel Prize lecture. The existing methods for constructing objective functions are collected in the proceedings of two dedicated conferences. In particular, Andranik Tangian showed that the most usable objective functions — quadratic and additive — are determined by a few indifference points. He used this property in the models for constructing these objective functions from either ordinal or cardinal data that were elicited through computer-assisted interviews with decision makers. Among other things, he constructed objective functions to optimally distribute budgets for 16 Westfalian universities and the European subsidies for equalizing unemployment rates among 271 German regions.

Expected loss

In some contexts, the value of the loss function itself is a random quantity because it depends on the outcome of a random variable ⁠${\displaystyle X}$⁠.

Statistics

Both frequentist and Bayesian statistical theory involve making a decision based on the expected value of the loss function; however, this quantity is defined differently under the two paradigms.

Frequentist expected loss

We first define the expected loss in the frequentist context. It is obtained by taking the expected value with respect to the probability distribution, ${\displaystyle P_{\theta }}$, of the observed data, ${\displaystyle X}$. This is also referred to as the risk function of the decision rule ${\displaystyle \delta }$ and the parameter ${\displaystyle \theta }$. Here the decision rule depends on the outcome of ${\displaystyle X}$. The risk function is given by:

${\displaystyle R(\theta ,\delta )={\mathrm{E}}_{\theta }L(\theta ,\delta (X))={\int }_{X}L(\theta ,\delta (x))\mathrm{d}{P}_{\theta }(x).}$

Here, ${\displaystyle \theta }$ is a fixed but possibly unknown state of nature, ${\displaystyle X}$ is a vector of observations stochastically drawn from a population, ${\displaystyle {\mathrm{E}}_{\theta }}$ is the expectation over all population values of ${\displaystyle X}$, ${\displaystyle \mathrm{d}{P}_{\theta }}$ is a probability measure over the event space of ${\displaystyle X}$ (parametrized by ${\displaystyle \theta }$) and the integral is evaluated over the entire support of ${\displaystyle X}$.

Bayes Risk

In a Bayesian approach, the expectation is calculated using the prior distribution ${\displaystyle {\pi }^{\ast }}$ of the parameter ${\displaystyle \theta }$:

${\displaystyle \rho ({\pi }^{\ast },a)={\int }_{\mathrm{\Theta }}{\int }_{\mathbf{X}}L(\theta ,a(\mathbf{x}))\mathrm{d}P(\mathbf{x}|\theta )\mathrm{d}{\pi }^{\ast }(\theta )={\int }_{\mathbf{X}}{\int }_{\mathrm{\Theta }}L(\theta ,a(\mathbf{x}))\mathrm{d}{\pi }^{\ast }(\theta |\mathbf{x})\mathrm{d}M(\mathbf{x})}$

where ${\displaystyle M(\mathbf{x})}$ is known as the predictive likelihood wherein ${\displaystyle \theta }$ has been "integrated out," ${\displaystyle {\pi }^{\ast }(\theta |\mathbf{x})}$ is the posterior distribution, and the order of integration has been changed. One then should choose the action ${\displaystyle a^{\ast }}$ which minimises this expected loss, which is referred to as Bayes Risk. In the latter equation, the integrand inside ${\displaystyle \mathrm{d}x}$ is known as the Posterior Risk, and minimising it with respect to decision ${\displaystyle a}$ also minimizes the overall Bayes Risk. This optimal decision, ${\displaystyle a^{\ast }}$ is known as the Bayes (decision) Rule - it minimises the average loss over all possible states of nature ${\displaystyle \theta }$, over all possible (probability-weighted) data outcomes. One advantage of the Bayesian approach is to that one need only choose the optimal action under the actual observed data to obtain a uniformly optimal one, whereas choosing the actual frequentist optimal decision rule as a function of all possible observations, is a much more difficult problem. Of equal importance though, the Bayes Rule reflects consideration of loss outcomes under different states of nature, ${\displaystyle \theta }$.

Examples in statistics

• For a scalar parameter ${\displaystyle \theta }$, a decision function whose output ${\displaystyle \hat{\theta }}$ is an estimate of ${\displaystyle \theta }$, and a quadratic loss function (squared error loss) $${\displaystyle L(\theta ,\hat{\theta })=(\theta -\hat{\theta }{)}^2,}$$ the risk function becomes the mean squared error of the estimate, $${\displaystyle R(\theta ,\hat{\theta })={\mathrm{E}}_{\theta }\left[(\theta -\hat{\theta }{)}^2\right].}$$An estimator found by minimizing the mean squared error estimates the posterior distribution's mean.
• In density estimation, the unknown parameter is probability density itself. The loss function is typically chosen to be a norm in an appropriate function space. For example, for ${\displaystyle L^2}$ norm, $${\displaystyle L(f,\hat{f})=\Vert f-\hat{f}{\Vert }_2^2,}$$ the risk function becomes the mean integrated squared error $${\displaystyle R(f,\hat{f})=\mathrm{E}\left(\Vert f-\hat{f}{\Vert }^2\right).}$$

Economic choice under uncertainty

In economics, decision-making under uncertainty is often modelled using the von Neumann–Morgenstern utility function of the uncertain variable of interest, such as end-of-period wealth. Since the value of this variable is uncertain, so is the value of the utility function; it is the expected value of utility that is maximized.

Decision rules

A decision rule makes a choice using an optimality criterion. Some commonly used criteria are:

• Minimax: Choose the decision rule with the lowest worst loss — that is, minimize the worst-case (maximum possible) loss: $${\displaystyle \underset{\delta }{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\underset{\theta \in \mathrm{\Theta }}{max}R(\theta ,\delta ).}$$
• Invariance: Choose the decision rule which satisfies an invariance requirement.
• Choose the decision rule with the lowest average loss (i.e., minimize the expected value of the loss function): $${\displaystyle \underset{\delta }{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}{\mathrm{E}}_{\theta \in \mathrm{\Theta }}[R(\theta ,\delta )]=\underset{\delta }{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}{\int }_{\theta \in \mathrm{\Theta }}R(\theta ,\delta )p(\theta )d\theta .}$$

Selecting a loss function

Sound statistical practice requires selecting an estimator consistent with the actual acceptable variation experienced in the context of a particular applied problem. Thus, in the applied use of loss functions, selecting which statistical method to use to model an applied problem depends on knowing the losses that will be experienced from being wrong under the problem's particular circumstances.

A common example involves estimating "location". Under typical statistical assumptions, the mean or average is the statistic for estimating location that minimizes the expected loss experienced under the squared-error loss function, while the median is the estimator that minimizes expected loss experienced under the absolute-difference loss function. Still different estimators would be optimal under other, less common circumstances.

In economics, when an agent is risk neutral, the objective function is simply expressed as the expected value of a monetary quantity, such as profit, income, or end-of-period wealth. For risk-averse or risk-loving agents, loss is measured as the negative of a utility function, and the objective function to be optimized is the expected value of utility.

Other measures of cost are possible, for example mortality or morbidity in the field of public health or safety engineering.

For most optimization algorithms, it is desirable to have a loss function that is globally continuous and differentiable.

Two very commonly used loss functions are the squared loss, ${\displaystyle L(a)=a^2}$, and the absolute loss, ${\displaystyle L(a)=|a|}$. However the absolute loss has the disadvantage that it is not differentiable at ${\displaystyle a=0}$. The squared loss has the disadvantage that it has the tendency to be dominated by outliers—when summing over a set of ${\displaystyle a}$'s (as in $\sum _{i=1}^{n}L(a_i)$), the final sum tends to be the result of a few particularly large ${\displaystyle a}$-values, rather than an expression of the average ${\displaystyle a}$-value.

The choice of a loss function is not arbitrary. It is very restrictive and sometimes the loss function may be characterized by its desirable properties. Among the choice principles are, for example, the requirement of completeness of the class of symmetric statistics in the case of i.i.d. observations, the principle of complete information, and some others.

W. Edwards Deming and Nassim Nicholas Taleb argue that empirical reality, not nice mathematical properties, should be the sole basis for selecting loss functions, and real losses often are not mathematically nice and are not differentiable, continuous, symmetric, etc. For example, a person who arrives before a plane gate closure can still make the plane, but a person who arrives after cannot, a discontinuity and asymmetry which makes arriving slightly late much more costly than arriving slightly early. In drug dosing, the cost of too little drug may be lack of efficacy, while the cost of too much may be tolerable toxicity, another example of asymmetry. Traffic, pipes, beams, ecologies, climates, etc. may tolerate increased load or stress with little noticeable change up to a point, then become backed up or break catastrophically. These situations, Deming and Taleb argue, are common in real-life problems, perhaps more common than classical smooth, continuous, symmetric, differentials cases.