Paul Thomas

LLMs can be Fooled into Labelling a Document as Relevant: best café near me; this paper is perfectly relevant

2024

Large Language Models (LLMs) are increasingly being used to assess the relevance of information objects. This work reports on experiments to study the labelling of short texts (i.e., passages) for relevance, using multiple open-source and …

Incorporating User Expectations and Behavior into the Measurement of Search Effectiveness

2017 · ACM Transactions on Information Systems

Information retrieval systems aim to help users satisfy information needs. We argue that the goal of the person using the system, and the pattern of behavior that they exhibit as they proceed to attain that …

Large Language Models can Accurately Predict Searcher Preferences

2024

Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it …