MATH Seminar

Title: Discretize-Optimize Methods for Residual Neural Networks
Seminar: Numerical Analysis and Scientific Computing
Speaker: Derek Onken of Emory University
Contact: Lars Ruthotto, lruthotto@emory.edu
Date: 2019-04-26 at 2:00PM
Venue: MSC W301
Download Flyer
Abstract: Neural networks (discrete universal approximators) demonstrate impressive performance in myriad tasks. Specifically, Residual Neural Networks (ResNets) have won numerous image classification contests since they were introduced a few years ago. Deep learning centers around the addition of more and more layers (and thus parameters) to these networks in efforts to improve performance. In this talk, we interpret ResNets as a discretization of an ordinary differential equation (ODE). This viewpoint exposes the similarity between the learning problem and problems of optimal control of the ODE. We use a discretize-optimize approach for training the weights of the ResNet and study the impact of the particular discretization strategy on the network performance. Varying the discretization of the features and parameters allows us to determine if the improved accuracy from deeper architectures stems from the larger number of parameters or more layers.