Back Propagation in ML

Back-Propagation

What is Backpropagation?

Back-propagation is the essence of neural net training. It is the method of fine-tuning the weights of a neural net based on the error rate obtained in the previous epoch (i.e., iteration). Proper tuning of the weights allows you to reduce error rates and to make the model reliable by increasing its generalization.

Backpropagation is a short form for "backward propagation of errors." It is a standard method of training artificial neural networks. This method helps to calculate the gradient of a loss function with respects to all the weights in the network.

Why We Need Backpropagation?

• Backpropagation is fast, simple and easy to program
• It has no parameters to tune apart from the numbers of input
• It is a flexible method as it does not require prior knowledge about the network
• It is a standard method that generally works well
• It does not need any special mention of the features of the function to be learned.

Types of Backpropagation Networks

Two Types of Backpropagation Networks are:

• Static Back-propagation
• Recurrent Backpropagation

Static back-propagation:

It is one kind of backpropagation network which produces a mapping of a static input for static output. It is useful to solve static classification issues like optical character recognition.

Recurrent Backpropagation:

Recurrent backpropagation is fed forward until a fixed value is achieved. After that, the error is computed and propagated backward.

The main difference between both of these methods is: that the mapping is rapid in static back-propagation while it is nonstatic in recurrent backpropagation.

Backpropagation Key Points

• Simplifies the network structure by elements weighted links that have the least effect on the trained network
• You need to study a group of input and activation values to develop the relationship between the input and hidden unit layers.
• It helps to assess the impact that a given input variable has on a network output. The knowledge gained from this analysis should be represented in rules.
• Backpropagation is especially useful for deep neural networks working on error-prone projects, such as image or speech recognition.
• Backpropagation takes advantage of the chain and power rules allows backpropagation to function with any number of outputs.

Best practice Backpropagation

• Backpropagation can be explained with the help of "Shoe Lace" analogy

  Too little tension =

  • Not enough constraining and very loose

  Too much tension =

  • Too much constraint (overtraining)
  • Taking too much time (relatively slow process)
  • Higher likelihood of breaking

  Pulling one lace more than other =

  • Discomfort (bias)

  Disadvantages of using Backpropagation

  • The actual performance of backpropagation on a specific problem is dependent on the input data.
  • Backpropagation can be quite sensitive to noisy data
  • You need to use the matrix-based approach for backpropagation instead of mini-batch.

  Summary

  • A neural network is a group of connected it I/O units where each connection has a weight associated with its computer programs.
  • Backpropagation is a short form for "backward propagation of errors." It is a standard method of training artificial neural networks
  • Backpropagation is fast, simple and easy to program
  • A feedforward neural network is an artificial neural network.
  • Two Types of Backpropagation Networks are 1)Static Back-propagation 2) Recurrent Backpropagation
  • In 1961, the basics concept of continuous backpropagation were derived in the context of control theory by J. Kelly, Henry Arthur, and E. Bryson.
  • Backpropagation simplifies the network structure by removing weighted links that have a minimal effect on the trained network.
  • It is especially useful for deep neural networks working on error-prone projects, such as image or speech recognition.
  • The biggest drawback of the Backpropagation is that it can be sensitive for noisy data.