Inter-sentential Processing of Language Models: The Case of Felicity Conditions

This study used allosentences to investigate the extent to which language models comprehend the nonliteral and pragmatic elements of natural language. Allosentences are sentences that share the same truth conditions but differ in felicity conditions and are thus pragmatically differentiated. We constructed a dataset of allosentences to assess whether current language models could capture pragmatic distinctions rooted in felicity conditions, which are crucial for human-like communication. Using the BERT-base/large and GPT-3.5/ GPT-4 models, we conducted the following two experiments: Experiment 1 (next sentence prediction (NSP) task) and Experiment 2 (text generation task). Both aim to determine whether language models can select felicitous answers to questions in a human-like manner. The findings reveal that, although not flawless, language models exhibit the capacity to generate more felicitous responses in specific contexts, shedding light on their ability for human-like communication. We suggest that inter-sentential processing is required in addition to intra-sentential processing to engage in a genuinely human-like conversation. At the same time, we report that language models share with humans a bias related to subject-object asymmetry, showing better performance on object questions.