Dynamically Adaptable Ensemble Proxies for Training-Free Neural Architecture Search

Heidorn C, Herderich T, Teich J (2026)

Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2026

Publisher: ACM

Event location: Edinburgh GB

ISBN: 979-8-4007-2605-7/26/04

DOI: 10.1145/3805621.3807635

Abstract

In neural architecture search (NAS), evaluating a candidate network’s accuracy is considered the most resource- and time-consuming step, as this usually requires full training and validation, which may take hours or even days. Recently, training-free, also called zero-shot NAS, has attracted increasing attention. In zero-shot NAS, a so-called zero-shot proxy (ZSP) is used to estimate the network’s accuracy with-out training, using, e.g., architectural or gradient-based features. However, the predictive performance and computational cost of proposed ZSPs can vary significantly across different datasets and search spaces. To address this issue, we introduce the notion of dynamically adaptable ensemble proxies (DAEP), a new class of zero-shot proxies, defined as a weighted sum of existing proxies, and customizable to a specific NAS design space by dynamic adaptation. Here, a weighting strategy is proposed that, in the first phase of NAS, adapts the weights of accuracy-estimating ZSPs dynamically, by amplifying or reducing each weight according to the correlation of their zero-proxy estimate with the accuracy obtained by training each sampled model. This process is repeated iteratively until no improvement in the mean accuracy estimation can be observed over a given number of evaluated models. For the proposed class of ensemble proxies, we demonstrate a) that DAEP can deliver highly correlated accuracy estimators b) already after very few model evaluations. Moreover, we show that c) different NAS problems require different weight vectors to achieve a high correlation with network accuracy. Finally, it is illustrated for multiple NAS search spaces (i.e., NATS-Bench and NDS) that d) DAEPs can drastically reduce the total NAS exploration time while accounting for the quality of accuracy estimation.

Authors with CRIS profile

Christian Heidorn Lehrstuhl für Informatik 12 (Hardware-Software-Co-Design) Jürgen Teich Lehrstuhl für Informatik 12 (Hardware-Software-Co-Design)

Related research project(s)

Neural Approximate Accelerator Architecture Optimization for DNN Inference on Lightweight FPGAs (NA³Os) May 1, 2024 - April 30, 2027 Optimization and Toolchain for Embedding AI (OpTC) March 1, 2023 - Feb. 28, 2026

How to cite

APA:

Heidorn, C., Herderich, T., & Teich, J. (2026). Dynamically Adaptable Ensemble Proxies for Training-Free Neural Architecture Search. In Proceedings of the The 6th Workshop on Machine Learning and Systems. Edinburgh, GB: ACM.

MLA:

Heidorn, Christian, Tina Herderich, and Jürgen Teich. "Dynamically Adaptable Ensemble Proxies for Training-Free Neural Architecture Search." Proceedings of the The 6th Workshop on Machine Learning and Systems, Edinburgh ACM, 2026.

BibTeX: Download