chain rule interpretation, real-valued circuits, patterns in gradient flow

Introduction

Intuitive understanding of backpropagation

• A way of computing gradients of expressions through recursive application of chain rule.

• Understanding of this process is critical to understand, and effectively develop, design and debug neural networks.

• Think of the training data as given and fixed, and of the weights as variables we have control over.

  → We usually only compute the gradient for the parameters (e.g. W,b) so that we can use it to perform a parameter update.

• The gradient on x_i can still be useful sometimes, for example for purposes of visualization and interpreting what the Neural Network might be doing.

Simple expressions & interpretation of the gradient

• Multiplication function of two numbers f(x,y)=xy (partial derivative for either input)

• Derivatives indicate the rate of change of a function with respect to that variable surrounding an infinitesimally small region near a particular point

• When h is very small, then the function is well-approximated by a straight line, and the derivative is its slope.

  → Derivative on each variable tells the sensitivity of the whole expression on its value.