Percy Liang

Unsupervised Risk Estimation Using Only Conditional Independence Structure

2016 · arXiv (Cornell University)

We show how to estimate a model's test error from unlabeled data, on distributions very different from the training distribution, while assuming only that certain conditional independencies are preserved between train and test. We do …

Macro Grammars and Holistic Triggering for Efficient Semantic Parsing

2017 · arXiv (Cornell University)

To learn a semantic parser from denotations, a learning algorithm must search over a combinatorially large space of logical forms for ones consistent with the annotated denotations. We propose a new online learning algorithm that …

Semidefinite relaxations for certifying robustness to adversarial examples

2018 · arXiv (Cornell University)

Despite their impressive performance on diverse tasks, neural networks fail catastrophically in the presence of adversarial inputs---imperceptibly but adversarially perturbed versions of natural inputs. We have witnessed an arms race between defenders who attempt to …

Designing and Interpreting Probes with Control Tasks

2019 · arXiv (Cornell University)

Probes, supervised models trained to predict properties (like parts-of-speech) from representations (like ELMo), have achieved high accuracy on a range of linguistic tasks. But does this mean that the representations encode linguistic structure or just …

A Tight Analysis of Greedy Yields Subexponential Time Approximation for Uniform Decision Tree

2019 · Society for Industrial and Applied Mathematics eBooks

Decision Tree is a classic formulation of active learning: given n hypotheses with nonnegative weights summing to 1 and a set of tests that each partition the hypotheses, output a decision tree using the provided …

Learning Adaptive Language Interfaces through Decomposition

2020 · arXiv (Cornell University)

Our goal is to create an interactive natural language interface that efficiently and reliably learns from users to complete tasks in simulated robotics settings. We introduce a neural semantic parsing system that learns new high-level …

Diffusion-LM Improves Controllable Text Generation

2022 · arXiv (Cornell University)

Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), there has been little …

Emergent Abilities of Large Language Models

2022 · arXiv (Cornell University)

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities …

Compositional Semantic Parsing on Semi-Structured Tables

2015 · arXiv (Cornell University)

Two important aspects of semantic parsing for question answering are the breadth of the knowledge source and the depth of logical compositionality. While existing work trades off one aspect for another, this paper simultaneously makes …

Building a Semantic Parser Overnight

2015

How do we build a semantic parser in a new domain starting with zero training ex-amples? We introduce a new methodol-ogy for this setting: First, we use a simple grammar to generate logical forms paired …

Imitation Learning of Agenda-based Semantic Parsers

2015 · Transactions of the Association for Computational Linguistics

Semantic parsers conventionally construct logical forms bottom-up in a fixed order, resulting in the generation of many extraneous partial logical forms. In this paper, we combine ideas from imitation learning and agenda-based parsing to train …

SQuAD: 100,000+ Questions for Machine Comprehension of Text

2016 · arXiv (Cornell University)

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment …

Adversarial Examples for Evaluating Reading Comprehension Systems

2017 · arXiv (Cornell University)

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an …

Know What You Don't Know: Unanswerable Questions for SQuAD

2018 · arXiv (Cornell University)

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated …

QuAC: Question Answering in Context

2018

We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform …

Transforming Question Answering Datasets Into Natural Language Inference Datasets

2018 · arXiv (Cornell University)

Existing datasets for natural language inference (NLI) have propelled research on language understanding. We propose a new method for automatically deriving NLI datasets from the growing abundance of large-scale question answering datasets. Our approach hinges …

Generating Sentences by Editing Prototypes

2018 · Transactions of the Association for Computational Linguistics

We propose a new generative language model for sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional language models that generate from …

Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings

2017

We study a symmetric collaborative dialogue setting in which two agents, each with private knowledge, must strategically communicate to achieve a common goal. The open-ended dialogue state in this setting poses new challenges for existing …

Unifying Human and Statistical Evaluation for Natural Language Generation

2019

Tatsunori B. Hashimoto, Hugh Zhang, Percy Liang. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

Data Recombination for Neural Semantic Parsing

2016

Modeling crisp logical regularities is crucial in semantic parsing, making it difficult for neural models with no task-specific prior knowledge to achieve good results. In this paper, we introduce data recombination, a novel framework for …

Inferring Logical Forms From Denotations

2016

A core problem in learning semantic parsers from denotations is picking out consistent logical forms-those that yield the correct denotation-from a combinatorially large space. To control the search space, previous work relied on restricted set …

From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood

2017

Our goal is to learn a semantic parser that maps natural language utterances into executable programs when only indirect supervision is available: examples are labeled with the correct execution result, but not the program itself. …

Certified Robustness to Adversarial Word Substitutions

2019

Robin Jia, Aditi Raghunathan, Kerem Göksel, Percy Liang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Distributionally Robust Language Modeling

2019

Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, Percy Liang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

2021

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Prefix-Tuning: Optimizing Continuous Prompts for Generation

2021

Xiang Lisa Li, Percy Liang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

LinkBERT: Pretraining Language Models with Document Links

2022 · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Language model (LM) pretraining captures various knowledge from text corpora, helping downstream NLP tasks. However, existing methods such as BERT model a single document, failing to capture document dependencies and knowledge that spans across documents. …

Holistic Evaluation of Language Models

2023 · Annals of the New York Academy of Sciences

Language models (LMs) like GPT-3, PaLM, and ChatGPT are the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) …

Benchmarking Large Language Models for News Summarization

2024 · Transactions of the Association for Computational Linguistics

Abstract Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model …

Lost in the Middle: How Language Models Use Long Contexts

2024 · Transactions of the Association for Computational Linguistics

Abstract While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks …