InfoBERT: Improving Robustness of Language Models from An Information\n Theoretic Perspective

Large-scale language models such as BERT have achieved state-of-the-art\nperformance across a wide range of NLP tasks. Recent studies, however, show\nthat such BERT-based models are vulnerable facing the threats of textual\nadversarial attacks. We aim to address this problem from an\ninformation-theoretic perspective, and propose InfoBERT, a novel learning\nframework for robust fine-tuning of pre-trained language models. InfoBERT\ncontains two mutual-information-based regularizers for model training: (i) an\nInformation Bottleneck regularizer, which suppresses noisy mutual information\nbetween the input and the feature representation; and (ii) a Robust Feature\nregularizer, which increases the mutual information between local robust\nfeatures and global features. We provide a principled way to theoretically\nanalyze and improve the robustness of representation learning for language\nmodels in both standard and adversarial training. Extensive experiments\ndemonstrate that InfoBERT achieves state-of-the-art robust accuracy over\nseveral adversarial datasets on Natural Language Inference (NLI) and Question\nAnswering (QA) tasks. Our code is available at\nhttps://github.com/AI-secure/InfoBERT.\n