Weize Chen

Exploring Mode Connectivity for Pre-trained Language Models

2022

Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found. Although plenty of …

Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communication

2024 · arXiv (Cornell University)

Natural language (NL) has long been the predominant format for human cognition and communication, and by extension, has been similarly pivotal in the development and application of Large Language Models (LLMs). Yet, besides NL, LLMs …

Parameter-efficient fine-tuning of large-scale pre-trained language models

2023 · Nature Machine Intelligence

Abstract With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all …