wiki/machine-learning/neural-networks/artificial-neural-networks.md

Learning Rate

#Machine Learning #Deep Learning #Basics #Learning Rate

Find a good learning rate

Initialize Artificial Neural Networks

#Artificial Neuron #Neural Network #Basics

Initialize a neural network is important for the training and performance. Some initializations simply don't work, some will degrade the performance of the model. We should choose wisely.

The log-sum-exp Trick

#Numerical #Neural Network #Basics

For numerical stability we can use the log-sum-exp trick to calculate some loss such as cross entropy

McCulloch-Pitts Model

#Artificial Neuron #Neural Network #Basics

Artificial neuron that separates the state space

Rosenblatt's Perceptron

#Artificial Neuron #Neural Network #Perceptron #Basics

Connected perceptrons

BiPolar Sigmoid

#Artificial Neuron #Neural Network #Basics #Activation Function

BiPolar sigmoid function and its properties

Conic Section Function

#Artificial Neuron #Neural Network #Basics #Activation Function

Conic Section Function and its properties

ELU

#Artificial Neuron #Neural Network #Basics #Activation Function

ELU and its properties

Hyperbolic Tanh

#Artificial Neuron #Neural Network #Basics #Activation Function

Tanh function and its properties

Leaky ReLu

#Artificial Neuron #Neural Network #Basics #Activation Function

Leaky ReLu and its properties

Radial Basis Function

#Artificial Neuron #Neural Network #Basics #Activation Function

Radial Basis Function function and its properties

ReLu

#Artificial Neuron #Neural Network #Basics #Activation Function

Rectified Linear Unit, aka ReLu, and its properties

Swish

#Artificial Neuron #Neural Network #Basics #Activation Function

Swish and its properties

Uni-Polar Sigmoid

#Artificial Neuron #Neural Network #Basics #Activation Function

Uni-polar sigmoid function and its properties