Raghav Arora*1 Shivam Singh∗1 Karthik Swaminathan∗1 Ahana Dutta1 Snehasis Banerjee2 Brojeshwar Bhowmick2 Krishna Murthy Jatavallabhula3 Mohan Sridharan4 Madhava Krishna1
1 Robotics Research Center, IIIT Hyderabad, India 2 TCS Research, Tata Consultancy Services, India 3 CSAIL, Massachusetts Institute of Technology, USA 4 School of Informatics, University of Edinburgh, UK
Assistive agents performing household tasks such as making the bed, preparing coffee, or cooking breakfast, often consider one task at a time by computing a plan of actions that accomplishes this task. The agents can be more efficient by anticipating upcoming tasks, and computing and executing an action sequence that jointly achieves these tasks. State of the art methods for task anticipating use data-driven deep network architectures and Large Language Models (LLMs) for task estimation but they do so at the level of high-level tasks and/or require a large number of training examples. Our framework leverages the generic knowledge of LLMs through a small number of prompts to perform high-level task anticipation, using the anticipated tasks as joint goals in a classical planning system to compute a sequence of finer-granularity actions that jointly achieve these goals. We ground and evaluate our frameworks capabilities in realistic simulated scenarios in the VirtualHome environment and demonstrate a 31% reduction in the execution time in comparison with a system that does not consider upcoming tasks