The demand for efficient natural language processing (NLP) systems has led to
the development of lightweight language models. Previous work in this area has
primarily focused on manual design or training-based neural architecture search
(NAS) methods. Recently, zero-shot NAS methods have been proposed for
evaluating language models without the need for training. Cependant, prevailing
approaches to zero-shot NAS often face challenges such as biased evaluation
metrics and computational inefficiencies. In this paper, we introduce
weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored
for lightweight language models. Our approach utilizes two evaluation proxies:
the parameter count and the number of principal components with cumulative
contribution exceeding $\eta$ in the feed-forward neural (FFN) layer.
Additionally, by eliminating the need for gradient computations, we optimize
the evaluation time, thus enhancing the efficiency of designing and evaluating
lightweight language models. We conduct a comparative analysis on the GLUE and
SQuAD datasets to evaluate our approach. The results demonstrate that our
method significantly reduces training time compared to one-shot NAS methods and
achieves higher scores in the testing phase compared to previous
state-of-the-art training-based methods. Furthermore, we perform ranking
evaluations on a dataset sampled from the FlexiBERT search space. Our approach
exhibits superior ranking correlation and further reduces solving time compared
to other zero-shot NAS methods that require gradient computation.

Cet article explore les excursions dans le temps et leurs implications.

Télécharger PDF: