Scanpath Prediction Via Semantic Representation of The Scene

Aiming at the problem that the current scanpath prediction methods have insufficient representation of object association, we propose a scanpath prediction model based on semantic representation of the scene. Our model uses a panoramic segmentation network to separate object instances and backgrounds in scenes, and uses the attention mechanism to learn the semantic correlation between objects, which effectively extracts the deep image information related to the current task. We also propose a dual-branch structure predicting the fixation position and duration simultaneously, to fully simulate the temporal and spatial distribution of the human eye’s attention in visual search. Experimental results show that our model has obvious advantages over the existing scanpath prediction methods in search efficiency and scanpath similarity, and can accurately predict the fixation duration.