BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning
Current pre-trained language models rely on large datasets for achieving state-of-the-art performance. However, past research has shown that not all examples in a dataset are equally important during training. In fact, it is sometimes possible to prune a considerable fraction of the training set whi...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Current pre-trained language models rely on large datasets for achieving
state-of-the-art performance. However, past research has shown that not all
examples in a dataset are equally important during training. In fact, it is
sometimes possible to prune a considerable fraction of the training set while
maintaining the test performance. Established on standard vision benchmarks,
two gradient-based scoring metrics for finding important examples are GraNd and
its estimated version, EL2N. In this work, we employ these two metrics for the
first time in NLP. We demonstrate that these metrics need to be computed after
at least one epoch of fine-tuning and they are not reliable in early steps.
Furthermore, we show that by pruning a small portion of the examples with the
highest GraNd/EL2N scores, we can not only preserve the test accuracy, but also
surpass it. This paper details adjustments and implementation choices which
enable GraNd and EL2N to be applied to NLP. |
---|---|
DOI: | 10.48550/arxiv.2211.05610 |