AI Terms Dictionary

Definition

An activation function introduces non-linearity into a neural network, allowing it to learn complex patterns and relationships within data. Without these functions, a multi-layered network would behave like a single linear regression model, severely limiting its expressive power. Common examples include ReLU, Sigmoid, and Tanh. They decide whether a neuron should be activated or not by calculating a weighted sum and possibly adding a bias, effectively filtering signals to propagate only significant information through the network layers during forward propagation.

Summary

A mathematical equation that determines the output of a neural network node based on its input signal.

Key Concepts

• Non-linearity
• Gradient Descent
• Neuron Activation
• Backpropagation

Use Cases

• Enabling deep neural networks to classify images
• Facilitating natural language processing tasks
• Improving convergence speed in training generative models

Code Example

1
2
3

import torch.nn as nn
relu = nn.ReLU()
output = relu(input_tensor)

Related Terms

• ReLU
• Sigmoid
• Tanh
• Softmax