Can neural networks multiply? #
Fairly early in the modern era, I tried to use neural network models to solve an old problem about the complexity of matrix multiplication. TLDR; there is an algorithm for multiplying matrices which is simpler than the one you have been taught. First found by Volker Strassen, there have been numerous attempts to improve it. And except for a few specific scenarios, the optimal algorithm is not known. The details of my work are summarized in this talk, which suggests that substantially simpler algorithms exist, but does little to help us find them.
As part of this work, it was necessary to find a small neural network that simply multiplies its two inputs together. The purpose of this exercise is to give you an opportunity to rediscover it.
Design a neural network \(\mathcal{N} : \mathbb{R}^2 \rightarrow \mathbb{R}\) with one hidden layer so that \(\mathcal{N}(x,y) = x \cdot y\).
- First, do so using the activation function \(\sigma(x) = x^2\).
- Then, do so using the much more standard activation function \(\sigma(x) = \text{ReLU(x)}\).
To describe these, feel free to simply draw the neural network graphs. But you should also convince me that what you suggest actually works!