Robotic Perception of Products in Retail Environments A literature survey

by Stefan Bonhof/

Stefan Bonhof


Deploying robots in highly dynamic environments poses a lot of challenges. This literature survey aims to identify the specific challenges for robot perception in a retail store environment regarding the recognition and pose estimation of supermarket products. The main challenge for object recognition and pose estimation in academic literature is the recognition of objects that belong to a very large set, with reasonable accuracy and speed. Within a retail store there are a lot of different products, many of which look very similar to each other, and many of which rarely look exactly the same (e.g. fruits or deformable products). This makes for a very large and difficult set of object classes. Furthermore, since retail stores are prone to changes in a class set, the perception algorithm should be able to adapt to a new class set quickly. In the past years, deep learning methods have surpassed classical computer vision methods by a rather long shot. However, deep neural networks designed to classify a large number of classes need a lot of parameters to get a strong model, which requires a lot of training data and time to train, which makes them unable to adapt quickly. Other approaches focused on using knowledge representation and reasoning to improve perception, but the current state of the art is not directly extendable to the retail environment case. Nevertheless, the use of knowledge on a retail store’s layout could bring several advantages. In this survey, then, we identify the requirements for the perception system in a retail scenario and we draft the idea of using knowledge representation and reasoning together with one or more perception methods, to identify products that are misplaced quickly and reliably. In particular, instead of identifying each object individually to solve this problem, a bigger world belief state can be generated for comparison with the real world, to provide a similarity measure and speed up the recognition process. The author of this survey will explore and implement this idea in his MSc thesis.

  • Student projects
  • TU Delft