Although it has been proven that a three-layer perceptron can approximate any function by increasing the number of neurons in its hidden layer, training such networks is often difficult. Deep learning (DL) addresses this challenge by increasing the number of layers, allowing each layer to learn progressively more complex features and representations.

Deep learning builds hierarchical representations, where lower layers learn simple features and higher layers learn more abstract concepts.

1. Adjusting step size

• If the gradient consistently points in the same direction, the learning rate can be increased; If the gradient direction frequently changes, the learning rate should be reduced.
• Weight updates should follow the gradient direction, with the magnitude determined by current and historical gradient values.

2. Initialization

Proper initialization helps neural networks learn efficiently and improves convergence.

• Subtract the mean from each feature.
• Divide by the standard deviation.
• The resulting features have a mean of 0 and a variance close to 1.

Categorical inputs are commonly transformed using one-hot encoding, where each possible category is represented by a Boolean indicator variable.

• Hidden-layer weights should not all be initialized with the same value.
• Output-layer biases can be initialized to values that provide reasonable predictions when other inputs are zero.

3. Fitting the Training Data

A well-designed model should outperform a naive baseline on the training set.

• If performance remains poor, adjustments may be needed in:
  • Network architecture
  • Optimization algorithm
  • Learning rate
  • Initialization method
• Poor training performance usually indicates underfitting, meaning the model is too simple to represent the target function.

4. Validation

Once the model successfully fits the training data, its performance should be evaluated on a validation set.

• The validation set serves as a proxy for the unseen test set.
• It helps determine optimal hyperparameters, such as learning rate, number of layers, and batch size.
• After selecting hyperparameters, training and validation data can be combined to build the final model.
• The final model is then evaluated using the test set.

A common validation technique is k-fold cross validation.

5. Dropout

Dropout is a regularization technique that reduces overfitting by randomly disabling some neurons during training.

• Selected neurons temporarily produce an output of zero.
• Different subsets of neurons are ignored during different training iterations.
• Dropout is applied only during training and is disabled during prediction.

6. Charllanges of DL

• High requirements for both training data and training time.
• Catastrophic forgetting: Neural networks may forget previously learned information when trained on new tasks or datasets.
• Adversarial examples: Carefully crafted input modifications can cause a neural network to make incorrect predictions, even when changes are nearly imperceptible to humans.
• Explainability: Deep neural networks are often difficult to interpret, making it challenging to explain how specific decisions are made.