Iz Beltagy

SciBERT: Pretrained Contextualized Embeddings for Scientific Text

2019

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, …

Cross-Document Language Modeling.

2021 · arXiv (Cornell University)

We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our cross-document language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we …

PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization.

2021 · arXiv (Cornell University)

Recently proposed pre-trained generation models achieve strong performance on single-document summarization benchmarks. However, most of them are pre-trained with general-purpose objectives and mainly aim to process single document inputs. In this paper, we propose PRIMER, …

Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

2023 · arXiv (Cornell University)

Since the release of TÜLU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into …

Construction of the Literature Graph in Semantic Scholar

2018

Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam …

ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing

2019

Despite recent advances in natural language processing, many statistical models for processing text perform extremely poorly under domain shift. Processing biomedical and clinical text is a critically important application area of natural language processing, for …

SciBERT: A Pretrained Language Model for Scientific Text

2019

Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Longformer: The Long-Document Transformer

2020 · arXiv (Cornell University)

The quadratic complexity of standard attention (O(N²)) remains the dominant bottleneck for training and deploying large language models on long sequences. We introduce Murmurative Attention, a novel attention mechanism that replaces pairwise token-token interactions with …

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

2022 · arXiv (Cornell University)

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed …