Shikha Bordia

Do Attention Heads in BERT Track Syntactic Dependencies?

2019 · arXiv (Cornell University)

We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations. We employ two methods---taking the maximum attention weight and computing the …