Soft object manipulation tasks in domestic scenes pose a significant
challenge for existing robotic skill learning techniques due to their complex
dynamics and variable shape characteristics. Since learning new manipulation
skills from human demonstration is an effective way for robot applications,
developing prior knowledge of the representation and dynamics of soft objects
is necessary. In this regard, we propose a pre-trained soft object manipulation
skill learning model, namely SoftGPT, that is trained using large amounts of
exploration data, consisting of a three-dimensional heterogeneous graph
representation and a GPT-based dynamics model. For each downstream task, a
goal-oriented policy agent is trained to predict the subsequent actions, and
SoftGPT generates the consequences of these actions. Integrating these two
approaches establishes a thinking process in the robot’s mind that provides
rollout for facilitating policy learning. Our results demonstrate that
leveraging prior knowledge through this thinking process can efficiently learn
various soft object manipulation skills, with the potential for direct learning
from human demonstrations.