Lecture 9 CSCI 784 - Analysis LMS Algorithm

• Pragmatics:
  • One misterm Feb 26 vs Two midterms Feb 12 and Mar 31
  • Question from last time? ?
• Background:
  • chain rule
  • eigenvalues
  • Trace[M] = sum of eigenvalues of M
• Review
  • HW set #3
  • HW set #4
  • Spatial Filter Equations (Wiener-Hopf)
    • Expansion of mean-square-error
    • mean-square-value
    • cross-correlation
    • auto-correlation of inputs
  • Method of Steepest Descent
    • Use -1 * gradient as direction
  • Least-Mean-Square Algorithm
    uses instantaneous estimates
    • r_dx(k,n) = x_k(n) d(n)
    • r(j, k, n) = x_j(n) x_k(n)
  • Summary of LMS (Table 5.1)
• Convergence of LMS
  • Convergent in mean
    E[w(n)] -> w₀ as n goes to infinity
  • Convergent in mean square
    E[e²(n)] -> constant as n goes to infinity
  • "Convergence in mean square" implies "Convergence in mean"
  • Convergence in mean result
    
  • Convergence in mean square result
    
  • Example: problem 5.2
• Learning Rate Annealing Schedule
  • Choices for eta(n)
  • eta(n) = Constant
  • eta(n) = c/n (stochastic approximation)
  • eta(n) = eta₀ / (1 + n/tau) (search-then-converge)
  • figure 5.5
• Adaline ADAptive LINear Element fig 5.6
  • phi(v) = sign(v)
  • McCullough-Pitts + learning rule
  • 14 of the 16 two input boolean functions are implementable
    exceptions: XOR, NOR
  • Madalines
• LMS applied in time Delay Situations
  • x(n) = [x(n), x(n-1), ... x(n-p+1)]
  • figure 5.7
  • Widrow-Hopf 1960
• Multi-Layer Perceptrons (MLPs) fig 6.1
  • Feedforward network with neurons arranged in layers with only connections from one layer to the next layer. The first layer is input the last is the output layer.
  • phi must be differentiable; default logistic function
  • credit assignment problem in MLPs
  • Notation for derivation of backpropagation
  • figures 6.3, 6.4
• Backpropagation History
  • term "back propagation" from Werbos PhD thesis (1985)
  • Rumelhart, Hinton and Williams in "Parallel Distributed Processing"(1986)
• Gradient of the instantaneous error sum wrt the weights

[Handouts:]

Correlation Matrix, HW3/HW4

[Readings:]

5.5-6.4

[Assignment:]