Tobias Schnabel

Unbiased Comparative Evaluation of Ranking Functions

2016 · arXiv (Cornell University)

Eliciting relevance judgments for ranking evaluation is labor-intensive and costly, motivating careful selection of which documents to judge. Unlike traditional approaches that make this selection deterministically, probabilistic sampling has shown intriguing promise since it enables …

Evaluation methods for unsupervised word embeddings

2015

We present a comprehensive study of evaluation methods for unsupervised embedding techniques that obtain meaningful representations of words from text. Different evaluations result in different orderings of embedding methods, calling into question the common assumption …

Recommendations as Treatments: Debiasing Learning and Evaluation

2016 · arXiv (Cornell University)

Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself. In this paper, we provide a principled …

Unbiased Learning-to-Rank with Biased Feedback

2017

Implicit feedback (e.g., clicks, dwell times, etc.) is an abundant source of data in human-interactive systems. While implicit feedback has many advantages (e.g., it is inexpensive to collect, user centric, and timely), its inherent biases …

<scp>SummaC</scp>: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

2022 · Transactions of the Association for Computational Linguistics

Abstract In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied …