layers, spatial arrangement, layer patterns, layer sizing patterns, AlexNet/ZFnet/VGGNet case studies, computational considerations

Convolutional Neural Networks (CNNs / ConvNets)

• CNNs are very similar to ordinary NN.

  → made up of neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity. still have scores and loss function(SVM/Softmax) on the last (fully-connected) layer

• ConvNet architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties.

  → These make the forward function more efficient to implement and vastly reduce the amount of parameters in the network

Architecture Overview

• Regular Neural Nets don't scale well to full images.

  → In CIFAR-10, images are only of size 32x32x3, so a single FC neuron in a first hidden layer wolud have 32323=3072 weights.

  → If image size and neurons increase, the parameters would add up quickly. This full connectivity is wasteful and the huge number of parameters would quickly lead to overfitting.

• The layers of a ConvNet have neurons arranged in 3 dimensions: width, height, depth.

Left: A regular 3-layer Neural Network.

Right: A ConvNet arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a ConvNet transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels)

Layers used to build ConvNets

A simple ConvNet is a sequence of layers, and every layer of a ConvNet transforms one volume of activations to another through a differentiable function.

• Three main types of layers to build ConvNet Architectures

  1. Convolutional Layer
  2. Pooling Layer
  3. Fully-Connected Layer(exactuly as seen in regular NNs)

• Example : CIFAR-10 classification [INPUT - CONV - RELU - POOL - FC]

  1. INPUT[32x32x3] will hold the raw pixel values of the image
  2. CONV layer will compute the output of neurons that are connecte to local regions in the input. This may result in volume such as [32x32x12] if we decided to use 12 filters
  3. RELU layer will apply an elementwise activation function, such as the max(0,x) thresholding at zero. This leaves the size of the volume unchanged [32x32x12]
  4. POOL layer will perfom a downsampling operation along the spatial dimensions(width, height), resulting in volume such as [16x16x12]
  5. FC layer will compute the class scores, resulting in volume of size [1x1x10], where each of the 10 numbers correspond to a class score. Each neuron in this layer will be connected to all the numbers in the previous volume.

  In this way, ConvNets transform the original image layer by layer from the original pixel values to the final class scores.

• The RELU/POOL layers will implement a fixed function(don't have updated parameters). The parameters in the CONV/FC layers will be trained with gradient descent so that the class scores that the ConvNet computes are consistent with the labels in the training set for each image.