Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction

Feature importance explanation, which highlights input features that are most influential to the output, is a popular explainable AI (XAI) technique to help users understand machine learning model predictions.However, features deemed predictive by machines can still be puzzling or even appear unintuitive to end-users.Explaining why a feature is predictive is an underexplored area in current XAI research.In this paper, we used deception detection as a case study.We leveraged a large language model (LLM) to explain why a word is predictive of genuine or deceptive reviews.We first validated the LLM-generated explanations to be non-hallucinated through an algorithmic evaluation.Then, we conducted a crowdsourced study ( = 220) to investigate how unintuitive words and LLM-generated explanations influence participants in a deception detection task.Our study results found that showing unintuitive features without explaining why they are predictive was no better than not showing them at all, while explaining why these features are predictive significantly enhanced participants' learning of the task, appropriate reliance on AI assistance, and perceptions of the AI system.