Robotic learning often occurs in dynamic environments with large behavior spaces. Traditional terminal success signals only indicate whether a task is completed, without revealing if the current behavior is making progress, remaining unchanged, or undoing earlier progress. This limitation has motivated recent research into progress reward modeling.
Progress reward models provide intermediate feedback that guides robots during task execution. They evaluate the current state relative to the goal, offering dense rewards that can accelerate learning and improve sample efficiency. These models are particularly useful in long-horizon tasks where terminal rewards are sparse.
Various approaches have been proposed for progress reward modeling, including learning from demonstrations, using temporal distance metrics, and leveraging hierarchical structures. Some methods learn a value function that predicts progress, while others use contrastive learning to compare states.
The survey categorizes these methods and discusses their applications in manipulation, navigation, and other robotic domains. It also highlights challenges such as reward hacking, generalization across tasks, and the need for efficient computation.
Future directions include integrating progress rewards with large language models for task understanding, and developing more robust reward models that can handle uncertainty and partial observability.
Source: arXiv (2607.21655).