Scene Understanding

Every day scenes pose a huge variety of problems like differences in object scale and level of detail, clutter and the possibility to decompose objects into parts. This level of complexity makes scene understanding still a challenging problem even though some sub-tasks like semantic segmentation or object detection have seen great progress through the application of deep neural networks. We try to further progress these abilities by addressing the problems of visual search and clutter.
Overlay of a street scene with it's segmentation mask. (Image credit: Mapillary Vistas Dataset).

One-Shot Instance Segmentation

One major problem of computer vision models is their hunger for data and their limitation to a pre-defined number of object categories. In contrast humans have no problem to acquire new concepts from very few examples. For example when shown a new object humans can point at similar objects and draw their outlines. We defined a number of One-Shot Segmentation tasks which we hope will help to teach computer vision models similar abilities.
one shot instance segmentation
One-shot visual search. Given a query image and a reference image showing an object of a novel category, we seek to detect and segment all instances of the corresponding category (‘person’ on the left, ‘car’ on the right). Note that no ground truth annotations of reference categories are used during training.


Current computer vision models are heavily influenced by clutter. We use visual search tasks to measure these effects and design new models, which can cope even with highly cluttered scenes.
cluttered omniglot
Examples from the Cluttered Omniglot Dataset .

Key Papers

C. Michaelis, I. Ustyuzhaninov, M. Bethge, and A. S. Ecker
One-Shot Instance Segmentation
arXiv, 2018
Code, URL, BibTex

I. Ustyuzhaninov, C. Michaelis, W. Brendel, and M. Bethge
One-shot Texture Segmentation
arXiv, 2018
Code, URL, BibTex

C. Michaelis, M. Bethge, and A. S. Ecker
One-Shot Segmentation in Clutter
ICML, 2018
#segmentation, #one-shot learning, #clutter
Code, URL, BibTex

University of Tuebingen BCCN CIN MPI