Analyzing and exploiting NARX recurrent neural networks for long-term dependencies
DiPietro R, Rupprecht C, Navab N, Hager GD (2018)
Publication Type: Conference contribution
Publication year: 2018
Publisher: International Conference on Learning Representations, ICLR
Conference Proceedings Title: 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings
Event location: Vancouver, BC, CAN
Abstract
Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies.
Involved external institutions
Johns Hopkins University (JHU)
United States (USA) (US)
Technische Universität München (TUM)
Germany (DE)
United States (USA) (US)
Technische Universität München (TUM)
Germany (DE)
How to cite
APA:
DiPietro, R., Rupprecht, C., Navab, N., & Hager, G.D. (2018). Analyzing and exploiting NARX recurrent neural networks for long-term dependencies. In 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings. Vancouver, BC, CAN: International Conference on Learning Representations, ICLR.
MLA:
DiPietro, Robert, et al. "Analyzing and exploiting NARX recurrent neural networks for long-term dependencies." Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, CAN International Conference on Learning Representations, ICLR, 2018.
BibTeX: Download