On Dependence of Bert Embeddings on the Syntactic Structure of a Sentence

Experiments demonstrate that Bert embeddings are reflecting syntactic structure of a sentence. In this paper we conduct some experiments demonstrating dependency between cosine similarity of Bert embeddings and position of neighbouring words with consideration of a consistency structure of a sentence. Our experiments demonstrated that Bert embeddings are dependent on the number of started and finished consistencies between those words. The more the number of borders of consistencies, the less is the cosine similarity; finished borders take more influence than started ones. Increasing the distance between words up to 3 cosine similarity decreases. Moreover, we have found that Bert model assigns very close vectors which cosine similarity does not fall less than 0.6. The Word2Vec model, which takes not a context of a word into consideration, does not demonstrate such a behaviour.