Jindong Chen

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

2020 · arXiv (Cornell University)

As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is a common aspect of daily life for many people. To improve the accessibility of these devices and to enable …

UIBert: Learning Generic Multimodal Representations for UI Understanding

2021 · arXiv (Cornell University)

To improve the accessibility of smart devices and to simplify their usage, building models which understand user interfaces (UIs) and assist users to complete their tasks is critical. However, unique challenges are proposed by UI-specific …