Pingyi Zhou

Extending Context Window of Large Language Models via Semantic Compression

2023 · arXiv (Cornell University)

Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We …

SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation

2021 · arXiv (Cornell University)

Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. Recently, many pre-trained language models for source code (e.g., …