Support Vector Machine, Softmax

parameteric approah, bias trick, hinge loss, cross-entropy loss, L2 regularization

Intro of Linear Classification

• In the last section we introduced the problem of Image Classification.

• K-Nearest Neighbor (kNN) classifier - it has a number of disadvantages.

  → The classifier must remember all of the training data and store it for future comparisons with the test data.

  → Classifying a test image is expensive since it requires a comparison to all training images.

• Develop a more powerful approach to image classification

  1. Score Function

  → maps the raw data to class scores

  2. Loss Function

  → quantifies the agreement between the predicted scores and the ground truth labels

  3. Optimization

  → minimize the loss function

Linear Score Function

Parameterized mapping from images to label scores

• Define the score function that maps the pixel values of an image to confidence scores for each class.

• Example

• The parameters in W are often called the weights, and b is called the bias vector because it influences the output scores, but without interacting with the actual data x
• Notice

  1. The single matrix multiplication Wxi is effectively evaluating 10 separate classifiers in parallel (one for each class), where each classifier is a row of W.

  2. We think of the input data (xi,yi) as given and fixed, but we have control over the setting of the parameters W,b.

  3. Our goal will be to set these in such way that the computed scores match the ground truth labels across the whole training set.

  4. Training data is used to learn the parameters W,b, but once the learning is complete we can discard the entire training set

     → New test image can be simply forwarded through the function and classified based on the computed scores.

     → faster than comparing a test image to all training images

Interpreting a linear classifier

• A linear classifier computes the score of a class as a weighted sum of all of its pixel values across all 3 of its color channels(RGB).

An example of mapping an image to class scores. For the sake of visualization, we assume the image only has 4 pixels (4 monochrome pixels, we are not considering color channels in this example for brevity), and that we have 3 classes (red (cat), green (dog), blue (ship) class). (Clarification: in particular, the colors here simply indicate 3 classes and are not related to the RGB channels.) We stretch the image pixels into a column and perform matrix multiplication to get the scores for each class. Note that this particular set of weights W is not good at all: the weights assign our cat image a very low cat score. In particular, this set of weights seems convinced that it's looking at a dog.

Analogy of images as high-dimensional points