Lably
Skip to content

Anomaly detection

Approach in data analysis

  • 11 min read
  • 8 chapters

Related on Lably

Through Data analysis, Computer security and Medicine

Lab · Class 8–12MicroorganismsLab · Class 8Wastewater StoryLab · Class 11NodeMCU & IoT

Anomaly detection finds application in many domains including cybersecurity, medicine, machine vision, statistics, neuroscience, law enforcement and financial fraud to name only a few. Anomalies were initially searched for clear rejection or omission from the data to aid statistical analysis, for example to compute the mean or standard deviation. They were also removed to better predictions from models such as linear regression, and more recently their removal aids the performance of machine learning algorithms. However, in many applications anomalies themselves are of interest and are the observations most desirous in the entire data set, which need to be identified and separated from noise or irrelevant outliers.

Three broad categories of anomaly detection techniques exist:

  • Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier. However, this approach is rarely used in anomaly detection due to the general unavailability of labelled data and the inherent unbalanced nature of the classes.
  • Semi-supervised anomaly detection techniques assume that some portion of the data is labelled. This may be any combination of the normal or anomalous data, but more often than not, the techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance to be generated by the model.
  • Unsupervised anomaly detection techniques assume the data is unlabelled and are by far the most commonly used due to their wider and relevant application.

Definition and terminology

Anomaly detection is the identification of observations, events, or patterns that depart substantially from normal behavior expected in a particular setting. There is no single operational definition for anomaly detection that applies to every domain, because the notion of normality depends on the data, the representation of the observations, the context in which they occur, and the assumptions made by the detection method.

An anomaly is not necessarily a data error. Depending on the application, it may represent a measurement or recording error, a change in the process that generated the data, an unusual but valid observation, or an event of interest such as fraud, equipment failure, or a security intrusion. Anomalies are also not defined solely by rarity or by distance from other observations. Different methods characterize normality using probability distributions, distances or local densities, cluster membership, boundaries around normal data, prediction errors, reconstruction errors, or other model-specific criteria.

Many anomaly-detection methods return a numerical anomaly score rather than an immediate categorical label. The score represents the degree to which an observation departs from the fitted model of normality and is converted into a binary decision by applying a decision threshold. Consequently, whether an observation is labelled anomalous depends on the model of normality, the available context, and the decision rule used.

Types of anomalies

A widely used classification distinguishes three types of anomalies:

  • A point anomaly is an individual observation that is considered anomalous relative to the rest of the data e.g., an unusually large transaction may be anomalous when compared with the other transactions in a dataset.
  • A contextual anomaly, also called a conditional anomaly, is anomalous only within a particular context. An observation may therefore be normal under one set of circumstances but anomalous under another e.g., a particular temperature may be normal in summer but unusual in winter. Context may be temporal, spatial, demographic, or otherwise defined by the application.
  • A collective anomaly is a related collection of observations that is anomalous as a whole, even when its individual observations are not anomalous by themselves e.g., an unusual subsequence in a time series or an unexpected pattern of activity in a computer network.

The appropriate category depends partly on the structure of the data. Point-anomaly methods may be sufficient for independent observations, whereas time series, spatial data, sequences, and graphs often require contextual or collective relationships to be modelled explicitly.

The terms anomaly, outlier, and novelty are sometimes used interchangeably, but their usage varies among statistics, machine learning, signal processing, and individual application domains. An outlier commonly denotes an observation that appears inconsistent with the other observations in a dataset. Outlier-detection methods often allow such observations to be present in the data used to fit the model.

In novelty detection, the training data are generally assumed to represent normal behavior, and previously unseen test observations are assessed against the resulting model of normality. Novelty detection is consequently often formulated as a form of one-class classification. These distinctions are not applied consistently throughout the literature, and the broader term anomaly detection is frequently used to include all of these settings.

Sign in to keep reading

7 more chapters are free with a Lably account.

More in Statistics

Mortality rateDeaths per 1,000 individuals per yearSystematic reviewComprehensive review of research literature using systematic methodsSensitivity and specificityStatistical measure of a binary classificationLongitudinal studyStudy with repeated observations over timePrevalenceNumber of disease cases in a given population at a specific timeStochasticRandomly determined process