Users, Adaptivity, and Bad Abandonment

We consider two recent proposals for effectiveness metrics that have been argued to be adaptive, those of Moffat et al. (ACM TOIS, 2017) and Jiang and Allan (CIKM, 2017), and consider the user interaction models that they give rise to. By categorizing non-relevant documents into those that are plausibly non-relevant and those that are egregiously non-relevant, we capture all of the attributes incorporated into the two proposals, and hence develop an effectiveness metric that better reflects user behavior when viewing the SERP, including bad abandonment.