Pattern recognition

Problem statement

The problem of pattern recognition can be stated as follows: Given an unknown function ${\displaystyle g:\mathcal{X}\to \mathcal{Y}}$ (the ground truth) that maps input instances ${\displaystyle \mathit{x}\in \mathcal{X}}$ to output labels ${\displaystyle y\in \mathcal{Y}}$, along with training data ${\displaystyle \mathbf{D}=\{({\mathit{x}}_1,y_1),\dots ,({\mathit{x}}_n,y_n)\}}$ assumed to represent accurate examples of the mapping, produce a function ${\displaystyle h:\mathcal{X}\to \mathcal{Y}}$ that approximates as closely as possible the correct mapping ${\displaystyle g}$. (For example, if the problem is filtering spam, then ${\displaystyle {\mathit{x}}_i}$ is some representation of an email and ${\displaystyle y}$ is either "spam" or "non-spam"). In order for this to be a well-defined problem, "approximates as closely as possible" needs to be defined rigorously. In decision theory, this is defined by specifying a loss function or cost function that assigns a specific value to "loss" resulting from producing an incorrect label. The goal then is to minimize the expected loss, with the expectation taken over the probability distribution of ${\displaystyle \mathcal{X}}$. In practice, neither the distribution of ${\displaystyle \mathcal{X}}$ nor the ground truth function ${\displaystyle g:\mathcal{X}\to \mathcal{Y}}$ are known exactly, but can be computed only empirically by collecting a large number of samples of ${\displaystyle \mathcal{X}}$ and hand-labeling them using the correct value of ${\displaystyle \mathcal{Y}}$ (a time-consuming process, which is typically the limiting factor in the amount of data of this sort that can be collected). The particular loss function depends on the type of label being predicted. For example, in the case of classification, the simple zero-one loss function is often sufficient. This corresponds simply to assigning a loss of 1 to any incorrect labeling and implies that the optimal classifier minimizes the error rate on independent test data (i.e. counting up the fraction of instances that the learned function ${\displaystyle h:\mathcal{X}\to \mathcal{Y}}$ labels wrongly, which is equivalent to maximizing the number of correctly classified instances). The goal of the learning procedure is then to minimize the error rate (maximize the correctness) on a "typical" test set.

For a probabilistic pattern recognizer, the problem is instead to estimate the probability of each possible output label given a particular input instance, i.e., to estimate a function of the form

${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{x},\mathit{\theta })=f\left(\mathit{x};\mathit{\theta }\right)}$

where the feature vector input is ${\displaystyle \mathit{x}}$, and the function f is typically parameterized by some parameters ${\displaystyle \mathit{\theta }}$. In a discriminative approach to the problem, f is estimated directly. In a generative approach, however, the inverse probability ${\displaystyle p(\mathit{x}|\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l})}$ is instead estimated and combined with the prior probability ${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{\theta })}$ using Bayes' rule, as follows:

${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{x},\mathit{\theta })=\frac{p(\mathit{x}|\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l},\mathit{\theta })p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{\theta })}{\sum _{L\in \text{all labels}}p(\mathit{x}|L)p(L|\mathit{\theta })}.}$

When the labels are continuously distributed (e.g., in regression analysis), the denominator involves integration rather than summation:

${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{x},\mathit{\theta })=\frac{p(\mathit{x}|\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l},\mathit{\theta })p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{\theta })}{{\int }_{L\in \text{all labels}}p(\mathit{x}|L)p(L|\mathit{\theta })\mathrm{d}L}.}$

The value of ${\displaystyle \mathit{\theta }}$ is typically learned using maximum a posteriori (MAP) estimation. This finds the best value that simultaneously meets two conflicting objects: To perform as well as possible on the training data (smallest error-rate) and to find the simplest possible model. Essentially, this combines maximum likelihood estimation with a regularization procedure that favors simpler models over more complex models. In a Bayesian context, the regularization procedure can be viewed as placing a prior probability ${\displaystyle p(\mathit{\theta })}$ on different values of ${\displaystyle \mathit{\theta }}$. Mathematically:

${\displaystyle {\mathit{\theta }}^{\ast }=\arg \underset{\mathit{\theta }}{max}p(\mathit{\theta }|\mathbf{D})}$

where ${\displaystyle {\mathit{\theta }}^{\ast }}$ is a point estimate (such as the MAP estimate) used for ${\displaystyle \mathit{\theta }}$ in point-estimate evaluation, and ${\displaystyle p(\mathit{\theta }|\mathbf{D})}$, the posterior probability of ${\displaystyle \mathit{\theta }}$, is given by Bayes' theorem:

${\displaystyle p(\mathit{\theta }|\mathbf{D})=\frac{\left[\prod _{i=1}^{n}p(y_i|{\mathit{x}}_i,\mathit{\theta })\right]p(\mathit{\theta })}{p(\mathbf{D})}\propto \left[\prod _{i=1}^{n}p(y_i|{\mathit{x}}_i,\mathit{\theta })\right]p(\mathit{\theta }),}$

where ${\displaystyle \mathbf{D}=\{({\mathit{x}}_i,y_i){\}}_{i=1}^n}$ represents the training dataset, and

${\displaystyle p(\mathbf{D})=\int \left[\prod _{i=1}^{n}p(y_i|{\mathit{x}}_i,\mathit{\theta })\right]p(\mathit{\theta })d\mathit{\theta }}$

is the marginal likelihood (or Bayesian evidence).

In the full Bayesian approach, instead of selecting a single parameter vector ${\displaystyle {\mathit{\theta }}^{\ast }}$, the probability of a label ${\displaystyle y^{\ast }}$ for a new query instance ${\displaystyle {\mathit{x}}^{\ast }}$ given the dataset ${\displaystyle \mathbf{D}}$ is computed by integrating over all possible values of ${\displaystyle \mathit{\theta }}$, weighted according to the posterior probability:

${\displaystyle p(y^{\ast }|{\mathit{x}}^{\ast },\mathbf{D})=\int p(y^{\ast }|{\mathit{x}}^{\ast },\mathit{\theta })p(\mathit{\theta }|\mathbf{D})\mathrm{d}\mathit{\theta }=\frac{1}{p(\mathbf{D})}\int p(y^{\ast }|{\mathit{x}}^{\ast },\mathit{\theta })\left[\prod _{i=1}^{n}p(y_i|{\mathit{x}}_i,\mathit{\theta })\right]p(\mathit{\theta })\mathrm{d}\mathit{\theta }.}$

Frequentist or Bayesian approach to pattern recognition

The first pattern classifier – the linear discriminant presented by Fisher – was developed in the frequentist tradition. The frequentist approach entails that the model parameters are considered unknown, but objective. The parameters are then computed (estimated) from the collected data. For the linear discriminant, these parameters are precisely the mean vectors and the covariance matrix. Also the probability of each class ${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{\theta })}$ is estimated from the collected dataset. Note that the usage of 'Bayes' rule' in a pattern classifier does not make the classification approach Bayesian.

Bayesian statistics has its origin in Greek philosophy where a distinction was already made between the 'a priori' and the 'a posteriori' knowledge. Later Kant defined his distinction between what is a priori known – before observation – and the empirical knowledge gained from observations. In a Bayesian pattern classifier, the class probabilities ${\displaystyle p(\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}|\mathit{\theta })}$ can be chosen by the user, which are then a priori. Moreover, experience quantified as a priori parameter values can be weighted with empirical observations – using e.g., the Beta- (conjugate prior) and Dirichlet-distributions. The Bayesian approach facilitates a seamless intermixing between expert knowledge in the form of subjective probabilities, and objective observations.

Probabilistic pattern classifiers can be used according to a frequentist or a Bayesian approach.

Uses

Within medical science, pattern recognition is the basis for computer-aided diagnosis (CAD) systems. CAD describes a procedure that supports the doctor's interpretations and findings. Other typical applications of pattern recognition techniques are automatic speech recognition, speaker identification, classification of text into several categories (e.g., spam or non-spam email messages), the automatic recognition of handwriting on postal envelopes, automatic recognition of images of human faces, or handwriting image extraction from medical forms. The last two examples form the subtopic image analysis of pattern recognition that deals with digital images as input to pattern recognition systems.

Optical character recognition is an example of the application of a pattern classifier. The method of signing one's name was captured with stylus and overlay starting in 1990. The strokes, speed, relative min, relative max, acceleration and pressure is used to uniquely identify and confirm identity. Banks were first offered this technology, but were content to collect from the FDIC for any bank fraud and did not want to inconvenience customers.

Pattern recognition has many real-world applications in image processing. Some examples include:

• identification and authentication: e.g., license plate recognition, fingerprint analysis, face detection/verification, and voice-based authentication.
• medical diagnosis: e.g., screening for cervical cancer (Papnet), breast tumors or heart sounds;
• defense: various navigation and guidance systems, target recognition systems, shape recognition technology etc.
• mobility: advanced driver assistance systems, autonomous vehicle technology, etc.

In psychology, pattern recognition is used to make sense of and identify objects, and is closely related to perception. This explains how the sensory inputs humans receive are made meaningful. Pattern recognition can be thought of in two different ways. The first concerns template matching and the second concerns feature detection. A template is a pattern used to produce items of the same proportions. The template-matching hypothesis suggests that incoming stimuli are compared with templates in the long-term memory. If there is a match, the stimulus is identified. Feature detection models, such as the Pandemonium system for classifying letters (Selfridge, 1959), suggest that the stimuli are broken down into their component parts for identification. One observation is a capital E having three horizontal lines and one vertical line.

Algorithms

Algorithms for pattern recognition depend on the type of label output, on whether learning is supervised or unsupervised, and on whether the algorithm is statistical or non-statistical in nature. Statistical algorithms can further be categorized as generative or discriminative.

Classification methods (methods predicting categorical labels)

Parametric:

• Linear discriminant analysis
• Quadratic discriminant analysis
• Maximum entropy classifier (aka logistic regression, multinomial logistic regression): Note that logistic regression is an algorithm for classification, despite its name. (The name comes from the fact that logistic regression uses an extension of a linear regression model to model the probability of an input being in a particular class.)

Nonparametric:

• Decision trees, decision lists
• Kernel estimation and K-nearest-neighbor algorithms
• Naive Bayes classifier
• Neural networks (multi-layer perceptrons)
• Perceptrons
• Support vector machines
• Gene expression programming

Clustering methods (methods for classifying and predicting categorical labels)

• Categorical mixture models
• Hierarchical clustering (agglomerative or divisive)
• K-means clustering
• Correlation clustering
• Kernel principal component analysis (Kernel PCA)

Ensemble learning algorithms (supervised meta-algorithms for combining multiple learning algorithms together)

• Boosting (meta-algorithm)
• Bootstrap aggregating ("bagging")
• Ensemble averaging
• Mixture of experts, hierarchical mixture of experts

General methods for predicting arbitrarily-structured (sets of) labels

• Bayesian networks
• Markov random fields

Multilinear subspace learning algorithms (predicting labels of multidimensional data using tensor representations)

Unsupervised:

• Multilinear principal component analysis (MPCA)

Real-valued sequence labeling methods (predicting sequences of real-valued labels)

• Kalman filters
• Particle filters

Regression methods (predicting real-valued labels)

• Gaussian process regression (kriging)
• Linear regression and extensions
• Independent component analysis (ICA)
• Principal components analysis (PCA)

Sequence labeling methods (predicting sequences of categorical labels)

• Conditional random fields (CRFs)
• Hidden Markov models (HMMs)
• Maximum entropy Markov models (MEMMs)
• Recurrent neural networks (RNNs)
• Dynamic time warping (DTW)