What you can cram into a single vector: Probing sentence embeddings for\n linguistic properties

Although much effort has recently been devoted to training high-quality\nsentence embeddings, we still have a poor understanding of what they are\ncapturing. "Downstream" tasks, often based on sentence classification, are\ncommonly used to evaluate the quality of sentence representations. The\ncomplexity of the tasks makes it however difficult to infer what kind of\ninformation is present in the representations. We introduce here 10 probing\ntasks designed to capture simple linguistic features of sentences, and we use\nthem to study embeddings generated by three different encoders trained in eight\ndistinct ways, uncovering intriguing properties of both encoders and training\nmethods.\n