The neural tangent kernel #

Neural networks are complex and answers to many basic questions about their training are not understood:

• Why do neural network models generalize well to new data?
• How should one model gradient descent on complex loss functions?

The neural tangent kernel is a recent innovation that holds a lot of promise as a way of understanding neural networks. Here is the basic inspiration: suppose we have a potentially complicated function \(\mathcal{N} : \mathbb{R}^n \rightarrow \mathbb{R}^m \) whose behavior we would like to understand. And suppose further that by some miracle we have also found a much simpler function \(\mathcal{K} : \mathbb{R}^n \rightarrow \mathbb{R}^m \) such that \(\mathcal{N}(\vec{x}) \approx \mathcal{K}(\vec{x})\) for relevant values of \(\vec{x}\). The hope is that some questions about \(\mathcal{N}\) can be answered by studying \(\mathcal{K}\).

Project outline #

There is a bit of theoretical background in this project. While there are a number of possible pathways, the following is a gentle introduction to the subject:

1. Start by learning about kernel methods in machine learning. This is a classical tool and a good one to have in your toolbox regardless of the rest of the project. One possible reference is this lecture; another is Chapter 14 of this book.

2. Two primary sources about neural tangent kernels are the original article that introduced them, and a very well-written blog post. Start with the latter and feel free to continue with its references or the many other online resources which are available.

3. Train a neural tangent kernel classifier on your favorite data set and compare your results with those obtained by a neural network. Evaluate the performance of both and comment on whether NKTs approximate the behavior of NNs sufficiently well to justify their use.

Your write up should consist of a detailed description of neural tangent kernels and any relevant mathematical results as well as a detailed description of any experiments you performed. You may also include a literature survey of current research results to contextualize your work. And as with any other project, you may use this as a starting point and see where your work takes you. Serendipity is a crucial part of research and more often than not and a crucial skill in life is identifying interesting pivots.