Interplay between depth and width for interpolation in neural ODEs

Álvarez-López A, Slimane AH, Zuazua E (2024)

Publication Type: Journal article

Publication year: 2024

Journal

Neural Networks Elsevier

Book Volume: 180

Article Number: 106640

DOI: 10.1016/j.neunet.2024.106640

Abstract

Neural ordinary differential equations have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of the role played by their architecture remains elusive. In this work, we examine the interplay between the width p and the number of transitions between layers L (corresponding to a depth of L+1). Specifically, we construct explicit controls interpolating either a finite dataset D, comprising N pairs of points in R^d, or two probability measures within a Wasserstein error margin ɛ>0. Our findings reveal a balancing trade-off between p and L, with L scaling as 1+O(N/p) for data interpolation, and as 1+Op⁻¹+(1+p)⁻¹ɛ^−d for measures. In the high-dimensional and wide setting where d,p>N, our result can be refined to achieve L=0. This naturally raises the problem of data interpolation in the autonomous regime, characterized by L=0. We adopt two alternative approaches: either controlling in a probabilistic sense, or by relaxing the target condition. In the first case, when p=N we develop an inductive control strategy based on a separability assumption whose probability increases with d. In the second one, we establish an explicit error decay rate with respect to p which results from applying a universal approximation theorem to a custom-built Lipschitz vector field interpolating D.

Authors with CRIS profile

Enrique Zuazua Iriondo

Involved external institutions

École Normale Supérieure de Cachan

France (FR) Universidad Autónoma de Madrid (UAM)

Spain (ES)

How to cite

APA:

Álvarez-López, A., Slimane, A.H., & Zuazua, E. (2024). Interplay between depth and width for interpolation in neural ODEs. Neural Networks, 180. https://doi.org/10.1016/j.neunet.2024.106640

MLA:

Álvarez-López, Antonio, Arselane Hadj Slimane, and Enrique Zuazua. "Interplay between depth and width for interpolation in neural ODEs." Neural Networks 180 (2024).

BibTeX: Download