CLIPGraphs: Multimodal Graph Networks to Infer Object-Room Affinities

Ayush Agrawal∗¹    Raghav Arora∗¹    Ahana Datta¹    Snehasis Banerjee^{1, 2}    Brojeshwar Bhowmick²    Krishna Murthy Jatavallabhula³    Mohan Sridharan⁴    Madhava Krishna¹   

¹ IIIT Hyderabad, India    ² TCS Research, Tata Consultancy Services, India    ³ CSAIL, Massachusetts Institute of Technology, USA    ⁴ Intelligent Robotics Lab, University of Birmingham, UK   

This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines.