Censoring (statistics)

Analysis

Special techniques may be used to handle censored data. Tests with specific failure times are coded as actual failures; censored data are coded for the type of censoring and the known interval or limit. Special software programs (often reliability oriented) can conduct a maximum likelihood estimation for summary statistics, confidence intervals, etc.

Epidemiology

One of the earliest attempts to analyse a statistical problem involving censored data was Daniel Bernoulli's 1766 analysis of smallpox morbidity and mortality data to demonstrate the efficacy of vaccination. An early paper to use the Kaplan–Meier estimator for estimating censored costs was Quesenberry et al. (1989), however this approach was found to be invalid by Lin et al. unless all patients accumulated costs with a common deterministic rate function over time, they proposed an alternative estimation technique known as the Lin estimator.

Operating life testing

Reliability testing often consists of conducting a test on an item (under specified conditions) to determine the time it takes for a failure to occur.

• Sometimes a failure is planned and expected but does not occur: operator error, equipment malfunction, test anomaly, etc. The test result was not the desired time-to-failure but can be (and should be) used as a time-to-termination. The use of censored data is unintentional but necessary.
• Sometimes engineers plan a test program so that, after a certain time limit or number of failures, all other tests will be terminated. These suspended times are treated as right-censored data. The use of censored data is intentional.

An analysis of the data from replicate tests includes both the times-to-failure for the items that failed and the time-of-test-termination for those that did not fail.

Censored regression

An earlier model for censored regression, the tobit model, was proposed by James Tobin in 1958.

Likelihood

The likelihood is the probability or probability density of what was observed, viewed as a function of parameters in an assumed model. To incorporate censored data points in the likelihood the censored data points are represented by the probability of the censored data points as a function of the model parameters given a model, i.e. a function of CDF(s) instead of the density or probability mass.

The most general censoring case is interval censoring: ${\displaystyle Pr(a<x⩽b)=F(b)-F(a)}$, where ${\displaystyle F(x)}$ is the CDF of the probability distribution, and the two special cases are:

• left censoring: ${\displaystyle Pr(-\mathrm{\infty }<x\le b)=F(b)-F(-\mathrm{\infty })=F(b)-0=F(b)=Pr(x\le b)}$
• right censoring: ${\displaystyle Pr(a<x\le \mathrm{\infty })=F(\mathrm{\infty })-F(a)=1-F(a)=1-Pr(x\le a)=Pr(x>a)}$

For continuous probability distributions: ${\displaystyle Pr(a<x\le b)=Pr(a<x<b)}$

Example

Suppose we are interested in survival times, ${\displaystyle T_1,T_2,\dots ,T_n}$, but we don't observe ${\displaystyle T_i}$ for all ${\displaystyle i}$. Instead, we observe

• ${\displaystyle (U_i,{\delta }_i)}$, with ${\displaystyle U_i=T_i}$ and ${\displaystyle {\delta }_i=1}$ if ${\displaystyle T_i}$ is actually observed, and
• ${\displaystyle (U_i,{\delta }_i)}$, with ${\displaystyle U_i<T_i}$ and ${\displaystyle {\delta }_i=0}$ if all we know is that ${\displaystyle T_i}$ is longer than ${\displaystyle U_i}$.

When ${\displaystyle T_i>U_i,U_i}$ is called the censoring time.

If the censoring times are all known constants, then the likelihood is

$${\displaystyle L=\prod _{i,{\delta }_i=1}f(u_i)\prod _{i,{\delta }_i=0}S(u_i)}$$

where ${\displaystyle f(u_i)}$ is the probability density function evaluated at ${\displaystyle u_i}$, and ${\displaystyle S(u_i)}$ is the probability that ${\displaystyle T_i}$ is greater than ${\displaystyle u_i}$, called the survival function.

This can be simplified by defining the hazard function, the instantaneous force of mortality, as

$${\displaystyle \lambda (u)=\frac{f(u)}{S(u)}}$$

so

$${\displaystyle f(u)=\lambda (u)S(u).}$$

Then

$${\displaystyle L=\prod _i\lambda (u_i{)}^{{\delta }_i}S(u_i).}$$

For the exponential distribution, this becomes even simpler, because the hazard rate, ${\displaystyle \lambda }$, is constant, and ${\displaystyle S(u)=\exp (-\lambda u)}$. Then:

$${\displaystyle L(\lambda )={\lambda }^k\exp \left(-\lambda \sum _i{u}_i\right),}$$

where $k=\sum _i{\delta }_i$.

From this we easily compute ${\displaystyle \hat{\lambda }}$, the maximum likelihood estimate (MLE) of ${\displaystyle \lambda }$, as follows:

$${\displaystyle \ell (\lambda )=\log (L(\lambda ))=k\log (\lambda )-\lambda \sum _i{u}_i.}$$

Then

$${\displaystyle \frac{d\ell }{d\lambda }=\frac{k}{\lambda }-\sum _i{u}_i.}$$

We set this to 0 and solve for ${\displaystyle \lambda }$ to get:

$${\displaystyle \hat{\lambda }=\frac{k}{\sum _i{u}_i}.}$$

Equivalently, the mean time to failure is:

$${\displaystyle \frac{1}{\hat{\lambda }}=\frac{\sum _i{u}_i}{k}.}$$

This differs from the standard MLE for the exponential distribution in that the censored observations are considered only in the numerator.