Ordinal regression

Linear models for ordinal regression

Ordinal regression can be performed using a generalized linear model (GLM) that fits both a coefficient vector and a set of thresholds to a dataset. Suppose one has a set of observations, represented by length-p vectors x₁ through x_n, with associated responses y₁ through y_n, where each y_i is an ordinal variable on a scale 1, ..., K. For simplicity, and without loss of generality, we assume y is a non-decreasing vector, that is, y_i ${\displaystyle \le }$ y_i+1. To this data, one fits a length-p coefficient vector w and a set of thresholds θ₁, ..., θ_K−1 with the property that θ₁ < θ₂ < ... < θ_K−1. This set of thresholds divides the real number line into K disjoint segments, corresponding to the K response levels.

The model can now be formulated as

${\displaystyle Pr(y\le i\mid \mathbf{x})=\sigma ({\theta }_i-\mathbf{w}\cdot \mathbf{x})}$

or, the cumulative probability of the response y being at most i is given by a function σ (the inverse link function) applied to a linear function of x. Several choices exist for σ; the logistic function

${\displaystyle \sigma ({\theta }_i-\mathbf{w}\cdot \mathbf{x})=\frac{1}{1+e^{-({\theta }_i-\mathbf{w}\cdot \mathbf{x})}}}$

gives the ordered logit model, while using the CDF of the standard normal distribution gives the ordered probit model. A third option is to use an exponential function

${\displaystyle \sigma ({\theta }_i-\mathbf{w}\cdot \mathbf{x})=1-\exp (-\exp ({\theta }_i-\mathbf{w}\cdot \mathbf{x}))}$

which gives the proportional hazards model.

Latent variable model

The probit version of the above model can be justified by assuming the existence of a real-valued latent variable (unobserved quantity) y*, determined by

${\displaystyle y^{\ast }=\mathbf{w}\cdot \mathbf{x}+\epsilon }$

where ε is normally distributed with zero mean and unit variance, conditioned on x. The response variable y results from an "incomplete measurement" of y*, where one only determines the interval into which y* falls:

${\displaystyle y=\{\begin{array}{ll}1& \text{if}{y}^{\ast }\le {\theta }_1,\\ 2& \text{if}{\theta }_1<y^{\ast }\le {\theta }_2,\\ 3& \text{if}{\theta }_2<y^{\ast }\le {\theta }_3\\ ⋮\\ K& \text{if}{\theta }_{K-1}<y^{\ast }.\end{array}}$

Defining θ₀ = -∞ and θ_K = ∞, the above can be summarized as y = k if and only if θ_k−1 < y* ≤ θ_k.

From these assumptions, one can derive the conditional distribution of y as

${\displaystyle \begin{array}{cc}P(y=k\mid \mathbf{x})& =P({\theta }_{k-1}<y^{\ast }\le {\theta }_k\mid \mathbf{x})\\ & =P({\theta }_{k-1}<\mathbf{w}\cdot \mathbf{x}+\epsilon \le {\theta }_k)\\ & =\mathrm{\Phi }({\theta }_k-\mathbf{w}\cdot \mathbf{x})-\mathrm{\Phi }({\theta }_{k-1}-\mathbf{w}\cdot \mathbf{x})\end{array}}$

where Φ is the cumulative distribution function of the standard normal distribution, and takes on the role of the inverse link function σ. The log-likelihood of the model for a single training example x_i, y_i can now be stated as

${\displaystyle \log \mathcal{L}(\mathbf{w},\theta \mid {\mathbf{x}}_i,y_i)=\sum _{k=1}^K[y_i=k]\log [\mathrm{\Phi }({\theta }_k-\mathbf{w}\cdot {\mathbf{x}}_i)-\mathrm{\Phi }({\theta }_{k-1}-\mathbf{w}\cdot {\mathbf{x}}_i)]}$

(using the Iverson bracket [y_i = k].) The log-likelihood of the ordered logit model is analogous, using the logistic function instead of Φ.