Artificial Neural Networks (ANNs)
Despite its recent success in many applications, deep learning is not new and according to Ian Goodfellow, Yoshua Bengio, and Aaron Courville, there have been three eras:
- Cybernetics between the 1940s and the 1960s
- Connectionism between the 1980s and the 1990s
- The current deep learning renaissance beginning in 2006
Mathematically speaking, a neural network is a graph consisting of non-linear equations whose parameters can be estimated using methods such as stochastic gradient descent and backpropagation. We will introduce ANNs step by step, starting with linear and logistic regression.
Linear regression is used to estimate the parameters of a model to describe the relationship between an output variable and the given input variables. It can be mathematically described as a weighted sum of input variables:
Here, the weight, 
, and inputs, 
, are vectors in 
; in other words, they are real-valued vectors with 
dimensions, 
as a scalar bias term, and 
as a scalar term that represents the valuation of the 
function at the input 
. In ANNs, the output of a single neuron without non-linearities is similar to the output of the linear model described in the preceding linear regression equation and the following diagram:
Logistic regression is a special version of regression where a specific non-linear function, the sigmoid function, is applied to the output of the linear model in the earlier linear regression equation:
The In ANNs, the non-linear model described in the logistic regression equation is similar to the output of a single neuron with a sigmoid non-linearity in the following diagram:
A combination of such neurons defines a hidden layer in a neural network, and the neural networks are organized as a chain of layers. The output of a hidden layer is described by the following equation and diagram:
Here, the weight, 
, and the input, 
, are vectors in 
; 
is a scalar bias term, 
is a vector, and 
is a non-linearity:
The preceding diagram depicts a fully connected neural network with two inputs, two hidden layers with three nodes each, and one output node.
In general, neural networks have a chain-like structure that is easy to visualize in equation form or as a graph, as the previous diagram confirms. For example, consider the 
and 
functions that are used in the 
model. In this simple model of a neural network, the input 
is used to produce the output 
; the output of 
is used as the input of 
that finally produces 
.
In this simple model, the function 
is considered to be the first hidden layer and the function 
is considered to be the second hidden layer. These layers are called hidden because, unlike the input and the output values of the model that are known a priori, their values are not known.
In each layer, the network is learning features or projections of the data that are useful for the task at hand. For example, in computer vision, there is evidence that the layers of the network closer to the input can learn filters that are associated with basic shapes, whereas in the layers closer to the output the network might learn filters that are closer to images.
The following figure taken from the paper Visualizing and Understanding Convolutional Networks by Zeiler et Fergus, provides a visualization of the filters on the first convolution layer of a trained AlexNet:
