Mathematical neurons #

Our goal is to get a sense of how the design parameters of a neuron and the output function are related. We will do this explicitly. First of all, the design parameters for a mathematical neuron are:

• the number of its inputs,
• the weight given to each input,
• the bias of the neuron, and
• its activation function.

Let us first consider neurons with a single input.

1. Single input #

In class, we encoded a neuron with a single input as a diagram of the following form:

The neuron in the image above has one input of weight \(w = 0.2\) and bias \(b = -0.4\). If we take the activation function \(\sigma\) to be the rectified linear unit, then it encodes the function \[\mathcal{N}(x) = \text{ReLU}(0.2 x - 0.4).\] Its output function is represented in the following graph:

Exercises #

1. Reconstruct the graph of the neuron from Figure 1 by graphing the following sequence of increasingly sophisticated functions: \[\text{ReLU}(x), \; \; \; \text{ReLU}(0.2 x), \text{ and} \; \; \; \text{ReLU}(0.2 x -0.4).\] Hint: Note that \(0.2x - 0.4 = 0.2(x -2)\)

2. Graph the output function of a neuron that has one input of weight \(w = 2\), bias \(b = 1\), and uses the ReLU activation function. To check your answer, you can use the following Mathematica notebook where you can design and graph your very own neuron.

3. Is it possible to construct a single neuron whose output function has the following graph? Explain why or why not.

2. Multiple inputs #

Consider the neuron encoded by the following diagram:

Its inputs \(x\) and \(y\) have weights \(w_1 = -1.0\) and \(w_2 = 1.0\) respectively and the neuron has bias \(b = 1.0\). Let us take ReLU as the activation function. As there are two inputs and one output, the neuron represents a function of two variables:

\[\mathcal{N}(x,y) = \text{ReLU}(-x+y+1).\]

We can hope to graph it. In fact, this Mathematica notebook can be used to do so. The result is this surface:

Two-dimensional images of surfaces in three dimensions leave something to be desired. In the exercises, we ask you to clarify what is going on in Figure 3.

Exercises #

4. What is the surface actually represented in Figure 3? Graph the following functions in turn. Each surface has a crease; to help visualize this graph, describe the set of points that lie on this crease.

   • \(\mathcal{N}(x,y) = \text{ReLU}(x),\)
   • \(\mathcal{N}(x,y) =\text{ReLU}(x+1), \text{ and}\)
   • \(\mathcal{N}(x,y) = \text{ReLU}(-x+y+1).\)

   Hint: Remember the notion of a cylinder, that is, the graph of a function with a missing variable.

5. Having worked out the details of the example above, explain what the output function of a general two-input ReLU neuron looks like. Test your hypothesis by generating graphs for a variety of two-input neurons using the Mathematica notebook linked above.