017/DeepLearning

/DeepLearning/DeepLearning/Week5/Week5-DeepLearning-Projects/1-Deep-Neural-Networks-and-optimization/1-Deep-Neural-Networks-and-optimization/1.3-Deep-Neural-Networks-and-optimization.ipynb
#1-Deep-Neural-Networks-and-optimization


  • Deep Neural Networks

  • Optimization of Deep Neural Networks

  • Regularization

![](1-Deep-Neural-Networks-and-optimization.png)

1-Deep-Neural-Networks-and-optimization

  • Deep Neural Networks
  • Optimization of Deep Neural Networks
  • Regularization

1.1-Deep-Neural-Networks

  • Deep Neural Networks
  • Neural Network
  • Layers
  • Input Layer
  • Hidden Layers
  • Output Layer
  • Activation Function
  • Forward Propagation
  • Backward Propagation
  • Cost Function
  • Gradient Descent

![](1-Deep-Neural-Networks-and-optimization-1.png)

  • A deep neural network is a neural network with multiple layers.
  • A neural network is a machine learning model that is modeled after the human brain.
  • The input layer is the layer that takes in the data.
  • The hidden layers are the layers in between the input and output layers.
  • The output layer is the layer that produces the final output.
  • An activation function is a function that is used to introduce nonlinearity into the network.
  • Forward propagation is the process of passing data through the network.
  • Backward propagation is the process of adjusting the weights of the network based on the error.
  • The cost function is a function that measures the error between the predicted output and the actual output.
  • Gradient descent is an optimization algorithm that is used to minimize the cost function.

1.2-Optimization-of-Deep-Neural-Networks

  • Optimization of Deep Neural Networks
  • Gradient Descent
  • Stochastic Gradient Descent
  • Mini-Batch Gradient Descent
  • Momentum
  • Nesterov Accelerated Gradient
  • Adaptive Moment Estimation (Adam)
  • Learning Rate Decay

![](1-Deep-Neural-Networks-and-optimization-2.png)

  • Optimization of Deep Neural Networks refers to the process of finding the best set of weights for the network.
  • Gradient Descent is an optimization algorithm that is used to minimize the cost function.
  • Stochastic Gradient Descent is a variant of gradient descent that uses a single data point to update the weights.
  • Mini-Batch Gradient Descent is a variant of gradient descent that uses a small batch of data points to update the weights.
  • Momentum is an optimization algorithm that uses the past gradients to accelerate the convergence of the weights.
  • Nesterov Accelerated Gradient is an optimization algorithm that uses the past gradients to accelerate the convergence of the weights.
  • Adaptive Moment Estimation (Adam) is an optimization algorithm that adapts the learning rate for each weight.
  • Learning Rate Decay is a technique that decreases the learning rate over time to prevent the weights from converging too quickly.

1.3-Regularization

  • Regularization
  • L1 Regularization
  • L2 Regularization
  • Dropout
  • Weight Initialization

![](1-Deep-Neural-Networks-and-optimization-3.png)

  • Regularization is a technique that is used to prevent overfitting in deep neural networks.
  • L1 regularization is a technique that adds a penalty to the cost function based on the absolute value of the weights.
  • L2 regularization is a technique that adds a penalty to the cost function based on the square of the weights.
  • Dropout is a technique that randomly sets a fraction of the activations to zero during training to prevent overfitting.
  • Weight Initialization is the process of initializing the weights in the network to prevent overfitting.

1.3-Deep-Neural-Networks-and-optimization

  • Deep Neural Networks
  • Optimization of Deep Neural Networks
  • Regularization

1-Deep-Neural-Networks-and-optimization

  • Deep Neural Networks
  • Neural Network
  • Layers
  • Input Layer
  • Hidden Layers
  • Output Layer
  • Activation Function
  • Forward Propagation
  • Backward Propagation
  • Cost Function
  • Gradient Descent

![](1-Deep-Neural-Networks-and-optimization-1.png)

  • A deep neural network is a neural network with multiple layers.
  • A neural network is a machine learning model that is modeled after the human brain.
  • The input layer is the layer that takes in the data.
  • The hidden layers are the layers in between the input and output layers.
  • The output layer is the layer that produces the final output.
  • An activation function is a function that is used to introduce nonlinearity into the network.
  • Forward propagation is the process of passing data through the network.
  • Backward propagation is the process of adjusting the weights of the network based on the error.
  • The cost function is a function that measures the error between the predicted output and the actual output.
  • Gradient descent is an optimization algorithm that is used to minimize the cost function.

1.2-Optimization-of-Deep-Neural-Networks

  • Optimization of Deep Neural Networks
  • Gradient Descent
  • Stochastic Gradient Descent
  • Mini-Batch Gradient Descent
  • Momentum
  • Nesterov Accelerated Gradient
  • Adaptive Moment Estimation (Adam)
  • Learning Rate Decay

![](1-Deep-Neural-Networks-and-optimization-2.png)

  • Optimization of Deep Neural Networks refers to the process of finding the best set of weights for the network.
  • Gradient Des