A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream

While we know that large language models (LLMs) can solve some planning problems, we do not understand the extent of these capabilities for robotics. One promising direction is to integrate the semantic knowledge of LLMs with the formal reasoning of task and motion planning (TAMP). However, designing such systems is complicated by the myriad of choices for how to integrate LLMs within TAMP. We develop 16 algorithms that use LLMs to substitute key TAMP components. Our zero-shot experiments across 13750 evaluations and three domains reveal that LLM-based planners exhibit lower success rates and higher planning times than engineered systems. Providing geometric details increases the number of task-planning errors compared to pure PDDL descriptions, and (faster) direct LLM variants outperform (slower) reasoning variants in most cases. Code and results are available at https://github.com/jorge-a-mendez/llm-pddlstream.