Robot Behavior Generation Based on “Animal Behaviors Inspired gMLP” with Environmental Event Information

This study aims to create appealing robots capable of diverse motions by proposing a foundational framework for robot motion, upon which motion generation was verified. The two-phase approach involves training a natural language processing model and then using it to control robot motions. During the learning phase, the dataset was generated by extracting 2D keypoints from cat videos, converting these 2D keypoints into 3D keypoints, and calculating joint angles using inverse kinematics based on joint positions. Additionally, environmental audio and object information were extracted from the videos and added to the dataset as event information. This data, consisting of joints and their positions and event information, was then transformed into a language-like format, referred to as “motion language.” This converted data was then used to train the gMLP, an large language model. The gMLP showed its lowest validation loss at 3 epoch, leading to the use of the model trained up to this point for generating motion language. We investigated how event information impacts gMLP motion generation. In the experiment, sequences were generated in the same motion context both with and without event information to examine differences in the subsequent sequence generation. First, the proportion of event information within the generated data was calculated. Additionally, conditional probabilities were computed and compared to assess the impact on generation. Variations were observed in the likelihood of the next sequence depending on the context. Furthermore, for contexts where only the event information differed, we calculated the extent to which identical sequences appeared within each context. Spearman's rank correlation coefficient was used to analyze ranking tendencies. As a result, different sequences were generated based on event information. This information also influenced the ranking, indicating a tendency for variations in generation based on specific events.