CLIP: Cheap Lipschitz Training of Neural Networks

Bungert L, Raab R, Roith T, Schwinn L, Tenbrinck D (2021)

Publication Type: Conference contribution, Conference Contribution

Publication year: 2021

Publisher: Springer

Series: Lecture Notes in Computer Science

City/Town: Cham

Book Volume: 12679

Conference Proceedings Title: SSVM 2021: Scale Space and Variational Methods in Computer Vision

URI: https://arxiv.org/abs/2103.12531

DOI: 10.1007/978-3-030-75549-2_25

Abstract

Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of each layer which gives a poor bound for the actual Lipschitz constant.

In this paper we investigate a variational regularization method named CLIP for controlling the Lipschitz constant of a neural network, which can easily be integrated into the training procedure. We mathematically analyze the proposed model, in particular discussing the impact of the chosen regularization parameter on the output of the network. Finally, we numerically evaluate our method on both a nonlinear regression problem and the MNIST and Fashion-MNIST classification databases, and compare our results with a weight regularization approach.

Authors with CRIS profile

Leon Bungert
René Raab Machine Learning and Data Analytics Lab Tim Roith Lehrstuhl für Angewandte Mathematik (Modellierung und Numerik) Leo Schwinn Machine Learning and Data Analytics Lab Daniel Tenbrinck Lehrstuhl für Angewandte Mathematik (Modellierung und Numerik)

How to cite

APA:

Bungert, L., Raab, R., Roith, T., Schwinn, L., & Tenbrinck, D. (2021). CLIP: Cheap Lipschitz Training of Neural Networks. In Abderrahim Elmoataz, Jalal Fadili, Yvain Quéau, Julien Rabin, Loïc Simon (Eds.), SSVM 2021: Scale Space and Variational Methods in Computer Vision. Cham: Springer.

MLA:

Bungert, Leon, et al. "CLIP: Cheap Lipschitz Training of Neural Networks." Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision Ed. Abderrahim Elmoataz, Jalal Fadili, Yvain Quéau, Julien Rabin, Loïc Simon, Cham: Springer, 2021.

BibTeX: Download