Definition
ReLU is widely used in deep learning neural networks due to its computational efficiency and ability to mitigate the vanishing gradient problem. Mathematically defined as f(x) = max(0, x), it introduces non-linearity into the model without saturating neurons for positive inputs. Despite potential issues like dying ReLUs, it remains a standard choice for hidden layers in convolutional and fully connected networks.
Summary
Rectified Linear Unit is an activation function that outputs the input directly if positive, otherwise zero.
Key Concepts
- Non-linearity
- Activation Function
- Vanishing Gradient
- Piecewise Linear
Use Cases
- Hidden layers in CNNs
- Deep Feedforward Networks
- Image Recognition Models
Code Example
| |