Networks

Neural networks #

written and developed by Thomas Pietraho

A neural network assembles neurons to represent a more complex function. The construction itself relies on a number of design parameters:

the number of input and output neurons,
the number of hidden neurons and how they are connected,
weights, biases, and activation functions for each.

Together, the first two are often called the architecture of the neural network. We will try to get a sense of the type of functions that can be encoded by a neural network. We will focus on single-input single-output networks, that is, neural networks \(\mathcal{N} : \mathbb{R} \rightarrow \mathbb{R}.\)

Single hidden layer #

The following is an example of a one-input one-output neural network with a single hidden layer consisting of two neurons:

The graph of its output function is as follows:

Figure 2: Output function of neural network in Figure 1.

Exercise: Compute the exact value \(\mathcal{N}(x)\) of the neural network in Figure 1 when \(x = -1\) and the activation function used by all neurons is \(\sigma(x) = \text{ReLU}(x)\).

Exercise: A common, but perhaps boring, activation function is \(\sigma(x) = x\). If this is the the activation function used by all of the neurons in Figure 1, write down a formula for \(\mathcal{N}(x)\).

Exercise: Extend your work from the above to any neural network \(\mathcal{N}: \mathbb{R} \rightarrow \mathbb{R}\) that only uses \(\sigma(x) = x\) as an activation function. What general form must the formula for \(\mathcal{N}(x)\) take?

Exercise:

Consider the function: \(\mathcal{f}(x) = \text{ReLU}(x-1)+ \text{ReLU}(-x-1) + 2.\) Its output function has the following graph:

Design a neural network that represents this function. Use the Mathematica notebook to graph and check your work. Your answer should be a neural network diagram specifying all weights, biases, and activation functions.

Exercise: Neural networks \(\mathcal{N}: \mathbb{R} \rightarrow \mathbb{R}\) designed with the ReLU activation function are piecewise linear: that is, their graphs are made up of line segments. Design a neural network whose graph is made up of four line segments. Use the Mathematica notebook to graph and check your work. Your answer should be a neural network diagram specifying all weights, biases, and activation functions together with its graph.

Two hidden layers #

The following is an example of a one-input one-output neural network with two hidden layers, each consisting of two neurons:

Figure 3: Diagram of a two hidden layer neural network.

The output function of this neural network then looks like:

Figure 4: Output function of the neural network in Figure 3.

In general, the number of hidden layers and the number of neurons in each layer influence the complexity of the output function. It is an established theorem that any continuous function can be approximated by a neural network, albeit many neurons may be required to do so.

Exercise: Consider a neural network with a single input neuron and a single neuron. It may have several hidden layers, each with several neurons. In this network, we will only use the activation function \(\sigma(x) = x \). Explain why the graph of the output function of this neural network is just a line.