Erik Ylipää Hellqvist

Semantic Re-tuning with Contrastive Tension

2021 · KTH Publication Database DiVA (KTH Royal Institute of Technology)

Extracting semantically useful natural language sentence representations frompre-trained deep neural networks such as Transformers remains a challenge. Wefirst demonstrate that pre-training objectives impose a significant task bias ontothe final layers of models, with a layer-wise …