Systematic Study of Long Tail Phenomena in Entity Linking

State-of-the-art entity linkers achieve high accuracy scores with probabilistic methods. However,<br/>these scores should be considered in relation to the properties of the datasets they are evaluated<br/>on. Until now, there has not been a systematic investigation of the properties of entity linking<br/>datasets and their impact on system performance. In this paper we report on a series of hypotheses<br/>regarding the long tail phenomena in entity linking datasets, their interaction, and their impact<br/>on system performance. Our systematic study of these hypotheses shows that evaluation datasets<br/>mainly capture head entities and only incidentally cover data from the tail, thus encouraging<br/>systems to overfit to popular/frequent and non-ambiguous cases. We find the most difficult cases<br/>of entity linking among the infrequent candidates of ambiguous forms. With our findings, we<br/>hope to inspire future designs of both entity linking systems and evaluation datasets. To support<br/>this goal, we provide a list of recommended actions for better inclusion of tail cases.