The Future is Here: Verbal Interaction with NPCs on Mobile.

Applying large Language Models (LM) to unlock natural language based interactions with game NPCs is an emerging frontier. But the LMs of today are prohibitively expensive to run locally, especially on mobile platforms due to limited runtime resources, thermal issues and strong FPS requirements. In this work, we identified a series of mobile-friendly small LMs called TinyStories[Eldan and Li 2023] that can generate simple but coherent English. We repurposed this model to generate NPC dialogues. We then combined it with Unity ML-Agents, creating an NPC that can perform complex actions and interact verbally with the player. Our implementation runs on current-generation mobile platforms, at the level of performance required for real-time gaming. With this work, we offer a glimpse into the future of gaming: interacting with complex NPCs using natural speech. We also demonstrate it is possible to run LMs alongside Unity ML-Agents on mobile with great results. This will significantly enhance the accessibility and interactivity in games. Lastly, we want to attract academia and industry attention to the area of mobile / edge friendly LMs, as we believe this will be a fertile ground for exploration.