Huber loss

Motivation

Two very commonly used loss functions are the squared loss, ${\displaystyle L(a)=a^2}$, and the absolute loss, ${\displaystyle L(a)=|a|}$. The squared loss function results in an arithmetic mean-unbiased estimator, and the absolute-value loss function results in a median-unbiased estimator (in the one-dimensional case, and a geometric median-unbiased estimator for the multi-dimensional case). The squared loss has the disadvantage that it has the tendency to be dominated by outliers—when summing over a set of ${\displaystyle a}$'s (as in $\sum _{i=1}^{n}L(a_i)$), the sample mean is influenced too much by a few particularly large ${\displaystyle a}$-values when the distribution is heavy tailed: in terms of estimation theory, the asymptotic relative efficiency of the mean is poor for heavy-tailed distributions.

As defined above, the Huber loss function is strongly convex in a uniform neighborhood of its minimum ${\displaystyle a=0}$; at the boundary of this uniform neighborhood, the Huber loss function has a differentiable extension to an affine function at points ${\displaystyle a=-\delta }$ and ${\displaystyle a=\delta }$. These properties allow it to combine much of the sensitivity of the mean-unbiased, minimum-variance estimator of the mean (using the quadratic loss function) and the robustness of the median-unbiased estimator (using the absolute value function).

Pseudo-Huber loss function

The Pseudo-Huber loss function can be used as a smooth approximation of the Huber loss function. It combines the best properties of L2 squared loss and L1 absolute loss by being strongly convex when close to the target/minimum and less steep for extreme values. The scale at which the Pseudo-Huber loss function transitions from L2 loss for values close to the minimum to L1 loss for extreme values and the steepness at extreme values can be controlled by the ${\displaystyle \delta }$ value. The Pseudo-Huber loss function ensures that derivatives are continuous for all degrees. It is defined as

$${\displaystyle L_{\delta }(a)={\delta }^2\left(\sqrt{1+(a/\delta {)}^2}-1\right).}$$

As such, this function approximates ${\displaystyle a^2/2}$ for small values of ${\displaystyle a}$, and approximates a straight line with slope ${\displaystyle \delta }$ for large values of ${\displaystyle a}$.

While the above is the most common form, other smooth approximations of the Huber loss function also exist.

Variant for classification

For classification purposes, a variant of the Huber loss called modified Huber is sometimes used. Given a prediction ${\displaystyle f(x)}$ (a real-valued classifier score) and a true binary class label ${\displaystyle y\in \{+1,-1\}}$, the modified Huber loss is defined as

$${\displaystyle L(y,f(x))=\{\begin{array}{ll}max(0,1-yf(x){)}^2& \text{for }yf(x)>-1,\\ -4yf(x)& \text{otherwise.}\end{array}}$$

The term ${\displaystyle max(0,1-yf(x))}$ is the hinge loss used by support vector machines; the quadratically smoothed hinge loss is a generalization of ${\displaystyle L}$.

Applications

The Huber loss function is used in robust statistics, M-estimation and additive modelling.