Feature scaling

Methods

Rescaling (min-max normalization)

Also known as min-max scaling or min-max normalization, rescaling is the simplest method and consists in rescaling the range of features to scale the range in [0, 1] or [−1, 1]. Selecting the target range depends on the nature of the data. The general formula for a min-max of [0, 1] is given as:

${\displaystyle x^{\prime }=\frac{x-\text{min}(x)}{\text{max}(x)-\text{min}(x)}}$

where ${\displaystyle x}$ is an original value, ${\displaystyle x^{\prime }}$ is the normalized value. For example, suppose that we have the students' weight data, and the students' weights span [160 pounds, 200 pounds]. To rescale this data, we first subtract 160 from each student's weight and divide the result by 40 (the difference between the maximum and minimum weights).

To rescale a range between an arbitrary set of values [a, b], the formula becomes:

${\displaystyle x^{\prime }=a+\frac{(x-\text{min}(x))(b-a)}{\text{max}(x)-\text{min}(x)}}$

where ${\displaystyle a,b}$ are the min-max values.

Mean normalization

${\displaystyle x^{\prime }=\frac{x-\overline{x}}{\text{max}(x)-\text{min}(x)}}$

where ${\displaystyle x}$ is an original value, ${\displaystyle x^{\prime }}$ is the normalized value, ${\displaystyle \overline{x}=\text{average}(x)}$ is the mean of that feature vector. There is another form of the means normalization which divides by the standard deviation which is also called standardization.

Standardization (Z-score Normalization)

In machine learning, we can handle various types of data, e.g. audio signals and pixel values for image data, and this data can include multiple dimensions. Feature standardization makes the values of each feature in the data have zero-mean (when subtracting the mean in the numerator) and unit-variance. This method is widely used for normalization in many machine learning algorithms (e.g., support vector machines, logistic regression, and artificial neural networks). The general method of calculation is to determine the distribution mean and standard deviation for each feature. Next we subtract the mean from each feature. Then we divide the values (mean is already subtracted) of each feature by its standard deviation.

${\displaystyle x^{\prime }=\frac{x-\overline{x}}{\sigma }}$

Where ${\displaystyle x}$ is the original feature vector, ${\displaystyle \overline{x}=\text{average}(x)}$ is the mean of that feature vector, and ${\displaystyle \sigma }$ is its standard deviation.

Robust Scaling

Robust scaling, also known as standardization using median and interquartile range (IQR), is designed to be robust to outliers. It scales features using the median and IQR as reference points instead of the mean and standard deviation:$${\displaystyle x^{\prime }=\frac{x-Q_2(x)}{Q_3(x)-Q_1(x)}}$$where ${\displaystyle Q_1(x),Q_2(x),Q_3(x)}$ are the three quartiles (25th, 50th, 75th percentile) of the feature.

Unit vector normalization

Unit vector normalization regards each individual data point as a vector, and divide each by its vector norm, to obtain ${\displaystyle x^{\prime }=x/\Vert x\Vert }$. Any vector norm can be used, but the most common ones are the L1 norm and the L2 norm.

For example, if ${\displaystyle x=(v_1,v_2,v_3)}$, then its Lp-normalized version is:$${\displaystyle \left(\frac{v_1}{(|v_1{|}^p+|v_2{|}^p+|v_3{|}^p{)}^{1/p}},\frac{v_2}{(|v_1{|}^p+|v_2{|}^p+|v_3{|}^p{)}^{1/p}},\frac{v_3}{(|v_1{|}^p+|v_2{|}^p+|v_3{|}^p{)}^{1/p}}\right)}$$