gradient checks, sanity checks, babysitting the learning process, momentum(+nesterov), second-order methods, Adagrad/RMSprop, hyperparameter optimization, model ensembles

Learning

This section is devoted to the dynamics, the process of learning the parameters and finding good hyperparameters

Gradient Checks

• Use the centered formula

  finite difference approximation when evaluating the numerical gradient looks as follows:

In practice, it turns out that it is much better to use the centered difference formula of the form:

• Use relative error for the comparison

  What are the details of comparing the numerical gradient f′n and analytic gradient f′a?

→ considers their ratio of the differences to the ratio of the absolute values of both gradients.

• relative error > 1e-2 usually means the gradient is probably wrong
• 1e-2 > relative error > 1e-4 should make you feel uncomfortable
• 1e-4 > relative error is usually okay for objectives with kinks. But if there are no kinks (e.g. use of tanh nonlinearities and softmax), then 1e-4 is too high.
• 1e-7 and less you should be happy.

the deeper the network, the higher the relative errors will be.

→ So if you are gradient checking the input data for a 10-layer network, a relative error of 1e-2 might be okay because the errors build up on the way.

Before learning : sanity checks Tips/Tricks

a few sanity checks you might consider running before you plunge into expensive optimization:

• Look for correct loss at chance performance

  Make sure you’re getting the loss you expect when you initialize with small parameters.

  first check the data loss alone (so set regularization strength to zero)