RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models
To audit the robustness of named entity recognition (NER) models, we propose RockNER, a simple yet effective method to create natural adversarial examples. Specifically, at the entity level, we replace target entities with other entities of the same semantic class in Wikidata; at the context level,...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | To audit the robustness of named entity recognition (NER) models, we propose
RockNER, a simple yet effective method to create natural adversarial examples.
Specifically, at the entity level, we replace target entities with other
entities of the same semantic class in Wikidata; at the context level, we use
pre-trained language models (e.g., BERT) to generate word substitutions.
Together, the two levels of attack produce natural adversarial examples that
result in a shifted distribution from the training data on which our target
models have been trained. We apply the proposed method to the OntoNotes dataset
and create a new benchmark named OntoRock for evaluating the robustness of
existing NER models via a systematic evaluation protocol. Our experiments and
analysis reveal that even the best model has a significant performance drop,
and these models seem to memorize in-domain entity patterns instead of
reasoning from the context. Our work also studies the effects of a few simple
data augmentation methods to improve the robustness of NER models. |
---|---|
DOI: | 10.48550/arxiv.2109.05620 |