preprocessing, weight initialization, batch normalization, regularization (L2/dropout), loss functions

Data Preprocessing

We will assume matrix X is of size [N x D] (N is the number of data, D is their dimensionality)

Mean subtraction

most common form of preprocessing

• Subtracting the mean across every individual feature in the data
• It has the geometric interpretation of centering the cloud of data around the origin along every dimension.
• X -= np.mean(X, axis = 0)

Nomalization

normalizing the data dimensions so that they are of approximately the same scale.

• Divide each dimension by its standard deviation

  → zero-centered: (X /= np.std(X, axis = 0))

• Another form of preprocessing normalizes each dimension so that the min and max along the dimension is -1 and 1 respectively.

Common data preprocessing pipeline. Left: Original toy, 2-dimensional input data. Middle: The data is zero-centered by subtracting the mean in each dimension. The data cloud is now centered around the origin. Right: Each dimension is additionally scaled by its standard deviation. The red lines indicate the extent of the data - they are of unequal length in the middle, but of equal length on the right.

PCA