model of a biological neuron, activation functions, neural net architecture, representational power

Biological motivation and connections

basic computational unit of the brain is a neuron

A cartoon drawing of a biological neuron (left) and its mathematical model (right).

• Model the firing rate of the neuron with an activation function f, which represents the frequency of the spikes along the axon.

• Historically, a common choice of activation function is the sigmoid function σ, since it takes a real-valued input (the signal strength after the sum) and squashes it to range between 0 and 1

• Example code for forward-propagating a single neuron

class Neuron(object):
  # ... 
  def forward(self, inputs):
    """ assume inputs and weights are 1-D numpy arrays and bias is a number """
    cell_body_sum = np.sum(inputs * self.weights) + self.bias
    firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation function
    return firing_rate

• Each neuron performs a dot product with the input and its weights, adds the bias and applies the non-linearity (or activation function)

Single neuron as a linear classifier

As we saw with linear classifiers, a neuron has the capacity to “like” (activation near one) or “dislike” (activation near zero) certain linear regions of its input space.

• With an appropriate loss function on the neuron’s output, we can turn a single neuron into a linear classifier:

Binary Softmax classifier (a.k.a. Logistic Regression)

We can interpret σ(∑_i(w_i*x_i+b)) to be the probability of one of the classes P(y_i=1∣ x_i;w)