What is Delta Rule?

Delta Rule is a learning rule which is used in Gradient Descent for updating the weight of a neuron in a single layer network.

It is also called as Widrow & Hoff Learning Rule or Least Mean Square (LMS)

The difference between the desired value and the predicted value is measured, which is also called the error function.




w = weight
x = input
j = input to the neuron “ i ” from neuron “ j ” in previous layer

Mean Square Error is :





By taking partial derivative with respect to Wij of Mean Square Error and using Chain Rule we will get:



So, where


This is known as the Generalised Delta Rule.
Hence f’ is the main reason, we always need a continuous transfer function.