Lecture 10 CSCI 784 - Back Propagation

• Background:
  • Correlation Matrix Computation Example
    • On HW problem you were given orthonormal vectors
    • If this is not the case then the Correlation matrix will have some errors
    • Gram-Schmidt can be used to convert orthonormal
    • normalize by dividing by the norm of the vector
  • Chain rule
  • Example: Logistic function; phi(v) = 1/(1 + exp(-av))
  • simplify to show phi'(v) = a(1-phi(v))phi(v)
• Multilayer Perceptrons
• Notational conventions (page 142)
  1. neuron i -> neuron j -> neuron k
  2. n = the nth training pattern
  3. e_j(n) = d_j(n) - y_j(n) if neuron j is in the output layer
  4. E(n) - instantaneous sum square errors = ...
  5. E_AV average sum square error = ...
  6. y_i = phi(v_i)
  7. v_i = ...
     Note the y_i where you would normally have x_i, but remember the input to layer with neuron j is the output of the layer with neuron i.
• Recall
  • w_j0 = theata_j
  • x_j0 = -1
  • phi the same throughout the network
• Delta Rule
  • delta-w_ji = - eta * gradient
• Use chain rule to calculate the Gradient of the instantaneous error sum wrt the weights
• local gradient: delta_j
• Updating weights
  • Case 1 Neuron j is in the output layer
    • use definition to calculate e_j(n)
    • simple to update
  • Case 2 Neuron j is a hidden neuron
    • e_j(n) does not exist because d_j(n) does not exist
    • Calculate local gradient