Intra-Batch Supervision for Panoptic Segmentation
on High-Resolution Images
Daan de Geus
Gijs Dubbelman


Unified panoptic segmentation methods are achieving state-of-the-art results on several datasets. To achieve these results on large-resolution datasets, these methods apply crop-based training. In this work, we find that, although crop-based training is advantageous in general, it also has a harmful side-effect. Specifically, it limits the ability of unified networks to discriminate between large object instances, causing them to make predictions that are confused between multiple instances. To solve this, we propose Intra-Batch Supervision (IBS), which improves a network's ability to discriminate between instances by introducing additional supervision using multiple images from the same batch. We show that, with our IBS, we successfully address the confusion problem and consistently improve the performance of unified networks. For the high-resolution Cityscapes and Mapillary Vistas datasets, we achieve improvements of up to +2.5 on the Panoptic Quality for thing classes, and even more considerable gains of up to +5.8 on both the pixel accuracy and pixel precision, which we identify as better metrics to capture the confusion problem.

Intra-Batch Supervision

We take the product of thing embeddings E from one image crop and features F from another image crop in the batch (indicated in red), and as the thing instances belonging to the embeddings are not present in the other image crop, the result should be 'empty' predicted masks. By applying a loss to these predictions, we teach the network to generate more discriminative embeddings and features, leading to predicted masks with less confusion.


We apply Intra-Batch Supervision to two unified panoptic segmentation networks: Panoptic FCN and Mask2Former. The code for both Panoptic FCN + IBS and Mask2former + IBS is publicly released and can be found through the links below.

 [Panoptic FCN + IBS]
 [Mask2Former + IBS]

Paper and Supplementary Material

Daan de Geus, Gijs Dubbelman
Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images
In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023


This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.