Updated README and new set of converters with additional descriptions

CONVERTERS.md

@@ -0,0 +1,70 @@
+# Converters for COCO panoptic segmentation format
+
+Different COCO formats are described [here](http://cocodataset.org/#format-data).
+
+## From COCO detection format to COCO panoptic format
+
+COCO detection format is used to store both COCO instance segmentation and COCO stuff annotations.
+The script `converters/detection2panoptic_coco_format.py` converts it to COCO panoptic format. Note that panoptic segmentation does not allow different segments to overlap, therefore, only dataset without overlaps can be converted.
+
+Conversion example:
+``` bash
+python converters/detection2panoptic_coco_format.py \
+  --input_json_file sample_data/panoptic_coco_detection_format.json \
+  --output_json_file converted_data/panoptic_coco_panoptic_format.json
+```
+
+## From COCO panoptic format to COCO detection formats
+
+The script `converts/panoptic2detection_coco_format.py` converts COCO panoptic format to COCO detection format. Each segmentation is stored as RLE. Note, some frameworks (for example [Detectron](https://github.com/facebookresearch/Detectron)) cannot work with segments stored as RLEs. There are, however, several ways ([1](https://github.com/facebookresearch/Detectron/issues/100), [2](https://github.com/facebookresearch/Detectron/pull/458)) to overcome this issue.
+
+To convert all data to COCO detection format:
+``` bash
+python converters/panoptic2detection_coco_format.py \
+  --input_json_file sample_data/panoptic_examples.json \
+  --output_json_file converted_data/panoptic_coco_detection_format.json
+```
+
+To convert only segments of *things* classes to COCO detection format:
+``` bash
+python converters/panoptic2detection_coco_format.py \
+  --input_json_file sample_data/panoptic_examples.json \
+  --output_json_file converted_data/panoptic_coco_detection_format_things_only.json \
+  --things_only
+```
+
+## Extract semantic segmentation from data in COCO panoptic format
+
+The script `converters/panoptic2semantic_segmentation.py` merges all segments of the same category on an image into one segment.
+
+It can be used to get semantic segmentation in COCO detection format:
+``` bash
+python converters/panoptic2semantic_segmentation.py \
+  --input_json_file sample_data/panoptic_examples.json \
+  --output_json_file converted_data/semantic_segmentation_coco_format.json
+```
+
+or to save semantic segmentation in a PNG format (pixel values corresponds to semantic categories):
+``` bash
+python converters/panoptic2semantic_segmentation.py \
+  --input_json_file sample_data/panoptic_examples.json \
+  --semantic_seg_folder converted_data/semantic_segmentation_pngs
+```
+
+In COCO stuff segmentation challenge 2017 all thing classes were merged into one *other* semantic category (`category_id=183`). Option `--things_other` in this script will do the same merging.
+
+## Convert panoptic segmentation from 2 channels format to COCO panoptic format.
+
+In the panoptic segmentation [paper](https://arxiv.org/abs/1801.00868) naive format to store panoptic segmentation is proposed. We call the format *2 channel format*. Each segment is defined by two labels:
+(1) semantic category label and (2) instance ID label. Together this two labels form a unique pair that distinguishes one segment from another. These two labels are stored as first two channels of a PNG file correspondingly. Example of panoptic data saved in the 2 channel format can be found in [sample_data/panoptic_examples_2ch_format](https://github.com/cocodataset/panopticapi/blob/master/sample_data/panoptic_examples_2ch_format) folder.
+
+The script `converters/2channels2panoptic_coco_format.py` converts panoptic segmentation prediction from 2 channels format to COCO panoptic format:
+
+``` bash
+python converters/2channels2panoptic_coco_format.py \
+  --source_folder sample_data/panoptic_examples_2ch_format \
+  --images_json_file sample_data/images_info_examples.json \
+  --prediction_json_file converted_data/panoptic_coco_from_2ch.json
+```
+
+In this script `--images_json_file` json file is a file that contains information (in COCO [format](http://cocodataset.org/#format-data)) about all images that will be converted. An example is [images_info_examples.json](https://github.com/cocodataset/panopticapi/blob/master/sample_data/images_info_examples.json). Note, that conversion script assumes that PNGs with 2 channel formatted panoptic segmentations have the following name structure `image['file_name'].replace('.jpg', '.png')`.

README.md

@@ -2,15 +2,29 @@
 This API is an experimental version of [COCO 2018 Panoptic Segmentation Task API](http://cocodataset.org/#panoptic-2018).
 
 ## Summary
-- **evaluation.py** script calculates [PQ metrics](http://cocodataset.org/#panoptic-eval).
+**Evaluation script**
 
-- **format_converter.py** script converts *2 channels PNG panoptic format* (for each pixel first 2 channels of the PNG encode semantic label and instance id respectively) to [*COCO format*](http://cocodataset.org/#format-results).
+*evaluation.py* calculates [PQ metrics](http://cocodataset.org/#panoptic-eval). For more information about the script usage: `python evaluation.py --help`
 
-- **instance_data.py** script extracts things annotations from panoptic ground truth and saves it in [COCO instance segmentation format](http://cocodataset.org/#format-data).
+**Format converters**
 
-- **semantic_data.py** script extracts semantic segmentation annotation for stuff and things categories from panoptic ground truth. It saves the semantic segmentation as a single channel PNG.
+COCO panoptic segmentation is stored in a new [format](http://cocodataset.org/#format-data). Unlike COCO detection format that stores each segment independently, COCO panoptic format stores all segmentations for an image in a single PNG file. This compact representation naturally maintains non-overlapping property of the panoptic segmentation.
 
-- **visualization.py** script provides an example of visualization for panoptic segmentation data.
+We provide several converters for COCO panoptic format. Full description and usage examples are available [here](https://github.com/cocodataset/panopticapi/blob/master/CONVERTERS.md).
+
+**Semantic and instance segmentation heuristic combination**
+
+We provide simple script that heuristically combines semantic and instance segmentation predictions into panoptic segmentation prediction.
+
+The merging logic of the script is described in the panoptic segmentation [paper](https://arxiv.org/abs/1801.00868). In addition, this script is able to filter out stuff predicted segments that have their area below the threshold defined by `--stuff_area_limit` parameter. For more information about the script logic and usage: `python combine_semantic_and_instance_predictions.py --help`
+
+**COCO panoptic segmentation challenge categories**
+
+Json file [panoptic_coco_categories.json](https://github.com/cocodataset/panopticapi/blob/master/panoptic_coco_categories.json) contains the list of all categories used in COCO panoptic segmentation challenge 2018.
+
+**Visualization**
+
+*visualization.py* provides an example of generating visually appealing representation of the panoptic segmentation data.
 
 ## Contact
 If you have any questions regarding this API, please contact us at alexander.n.kirillov-at-gmail.com.

combine_semantic_and_instance_predictions.py

@@ -0,0 +1,223 @@
+#!/usr/bin/env python2
+'''
+The script uses a simple procedure to combine semantic segmentation and instance
+segmentation predictions. The procedure is described in section 7 of the
+panoptic segmentation paper https://arxiv.org/pdf/1801.00868.pdf.
+
+On top of the procedure described in the paper. This script remove from
+prediction small segments of stuff semantic classes. This addition allows to
+decrease number of false positives.
+'''
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+import os, sys
+import argparse
+import numpy as np
+from collections import defaultdict
+import json
+import time
+import multiprocessing
+import copy
+
+from utils import IdGenerator, id2rgb
+
+try:
+    import PIL.Image     as Image
+except:
+    print("Failed to import the image processing packages.")
+    sys.exit(-1)
+
+try:
+    # set up path for pycocotools
+    # sys.path.append('./cocoapi-master/PythonAPI/')
+    from pycocotools import mask as COCOmask
+except:
+    raise Exception("Please install pycocotools module from https://github.com/cocodataset/cocoapi")
+
+
+def combine_to_panoptic(proc_id, img_ids, img_id2img, inst_by_image,
+                        sem_by_image, segmentations_folder, overlap_thr,
+                        stuff_area_limit, categories):
+    panoptic_json = []
+    id_generator = IdGenerator(categories)
+
+    for idx, img_id in enumerate(img_ids):
+        img = img_id2img[img_id]
+
+        if idx % 100 == 0:
+            print('Core: {}, {} from {} images processed.'.format(proc_id, idx,
+                                                                  len(img_ids)))
+
+        pan_segm_id = np.zeros((img['height'],
+                                img['width']), dtype=np.uint32)
+        used = None
+        annotation = {}
+        annotation['image_id'] = img_id
+        annotation['file_name'] = img['file_name'].replace('.jpg', '.png')
+
+        segments_info = []
+        for ann in inst_by_image[img_id]:
+            area = COCOmask.area(ann['segmentation'])
+            if area == 0:
+                continue
+            if used is None:
+                intersect = 0
+                used = copy.deepcopy(ann['segmentation'])
+            else:
+                intersect = COCOmask.area(
+                    COCOmask.merge([used, ann['segmentation']], intersect=True)
+                )
+            if intersect / area > overlap_thr:
+                continue
+            used = COCOmask.merge([used, ann['segmentation']], intersect=False)
+
+            mask = COCOmask.decode(ann['segmentation']) == 1
+            if intersect != 0:
+                mask = np.logical_and(pan_segm_id == 0, mask)
+            segment_id = id_generator.get_id(ann['category_id'])
+            panoptic_ann = {}
+            panoptic_ann['id'] = segment_id
+            panoptic_ann['category_id'] = ann['category_id']
+            pan_segm_id[mask] = segment_id
+            segments_info.append(panoptic_ann)
+
+        for ann in sem_by_image[img_id]:
+            mask = COCOmask.decode(ann['segmentation']) == 1
+            mask_left = np.logical_and(pan_segm_id == 0, mask)
+            if mask_left.sum() < stuff_area_limit:
+                continue
+            segment_id = id_generator.get_id(ann['category_id'])
+            panoptic_ann = {}
+            panoptic_ann['id'] = segment_id
+            panoptic_ann['category_id'] = ann['category_id']
+            pan_segm_id[mask_left] = segment_id
+            segments_info.append(panoptic_ann)
+
+        annotation['segments_info'] = segments_info
+        panoptic_json.append(annotation)
+
+        Image.fromarray(id2rgb(pan_segm_id)).save(
+            os.path.join(segmentations_folder, annotation['file_name'])
+        )
+
+    return panoptic_json
+
+def combine_predictions(semseg_json_file, instseg_json_file, images_json_file,
+                        categories_json_file, segmentations_folder,
+                        panoptic_json_file, confidence_thr, overlap_thr,
+                        stuff_area_limit):
+    start_time = time.time()
+
+    with open(semseg_json_file, 'r') as f:
+        sem_results = json.load(f)
+    with open(instseg_json_file, 'r') as f:
+        inst_results = json.load(f)
+    with open(images_json_file, 'r') as f:
+        images_d = json.load(f)
+    img_id2img = {img['id']: img for img in images_d['images']}
+
+    with open(categories_json_file, 'r') as f:
+        categories_list = json.load(f)
+    categories = {el['id']: el for el in categories_list}
+
+    if segmentations_folder is None:
+        segmentations_folder = panoptic_json_file.rsplit('.', 1)[0]
+    if not os.path.isdir(segmentations_folder):
+        print("Creating folder {} for panoptic segmentation PNGs".format(segmentations_folder))
+        os.mkdir(segmentations_folder)
+
+    print("Combining:")
+    print("Semantic segmentation:")
+    print("\tJSON file: {}".format(semseg_json_file))
+    print("and")
+    print("Instance segmentations:")
+    print("\tJSON file: {}".format(instseg_json_file))
+    print("into")
+    print("Panoptic segmentations:")
+    print("\tSegmentation folder: {}".format(segmentations_folder))
+    print("\tJSON file: {}".format(panoptic_json_file))
+    print("List of images to combine is takes from {}".format(images_json_file))
+    print('\n')
+
+    inst_by_image = defaultdict(list)
+    for inst in inst_results:
+        if inst['score'] < confidence_thr:
+            continue
+        inst_by_image[inst['image_id']].append(inst)
+    for img_id in inst_by_image.keys():
+        inst_by_image[img_id] = sorted(inst_by_image[img_id], key=lambda el: -el['score'])
+
+    sem_by_image = defaultdict(list)
+    for sem in sem_results:
+        if categories[sem['category_id']]['isthing'] == 1:
+            continue
+        sem_by_image[sem['image_id']].append(sem)
+
+    imgs_ids_all = img_id2img.keys()
+    cpu_num = multiprocessing.cpu_count()
+    img_ids_split = np.array_split(imgs_ids_all, cpu_num)
+    print("Number of cores: {}, images per core: {}".format(cpu_num, len(img_ids_split[0])))
+    workers = multiprocessing.Pool(processes=cpu_num)
+    processes = []
+    for proc_id, img_ids in enumerate(img_ids_split):
+        p = workers.apply_async(combine_to_panoptic,
+                                (proc_id, img_ids, img_id2img, inst_by_image,
+                                 sem_by_image, segmentations_folder, overlap_thr,
+                                 stuff_area_limit, categories))
+        processes.append(p)
+    panoptic_json = []
+    for p in processes:
+        panoptic_json.extend(p.get())
+
+    with open(images_json_file, 'r') as f:
+        coco_d = json.load(f)
+    coco_d['annotations'] = panoptic_json
+    coco_d['categories'] = categories.values()
+    with open(panoptic_json_file, 'w') as f:
+        json.dump(coco_d, f)
+
+    t_delta = time.time() - start_time
+    print("Time elapsed: {:0.2f} seconds".format(t_delta))
+
+
+if __name__ == "__main__":
+        parser = argparse.ArgumentParser(
+            description="This script uses a simple procedure to combine semantic \
+            segmentation and instance segmentation predictions. See this \
+            file's head for more information."
+        )
+        parser.add_argument('--semseg_json_file', type=str,
+                            help="JSON file with semantic segmentation predictions")
+        parser.add_argument('--instseg_json_file', type=str,
+                            help="JSON file with instance segmentation predictions")
+        parser.add_argument('--images_json_file', type=str,
+                            help="JSON file with correponding image set information")
+        parser.add_argument('--categories_json_file', type=str,
+                            help="JSON file with Panoptic COCO categories information",
+                            default='./panoptic_coco_categories.json')
+        parser.add_argument('--panoptic_json_file', type=str,
+                            help="JSON file with resulting COCO panoptic format prediction")
+        parser.add_argument(
+            '--segmentations_folder', type=str, default=None, help="Folder with \
+             panoptic COCO format segmentations. Default: X if panoptic_json_file is \
+             X.json"
+        )
+        parser.add_argument('--confidence_thr', type=float, default=0.5,
+                            help="Predicted segments with smaller confidences than the threshold are filtered out")
+        parser.add_argument('--overlap_thr', type=float, default=0.5,
+                            help="Segments that have higher that the threshold ratio of \
+                            their area being overlapped by segments with higher confidence are filtered out")
+        parser.add_argument('--stuff_area_limit', type=float, default=64*64,
+                            help="Stuff segments with area smaller that the limit are filtered out")
+        args = parser.parse_args()
+        combine_predictions(args.semseg_json_file,
+                            args.instseg_json_file,
+                            args.images_json_file,
+                            args.categories_json_file,
+                            args.segmentations_folder,
+                            args.panoptic_json_file,
+                            args.confidence_thr,
+                            args.overlap_thr,
+                            args.stuff_area_limit)

converted_data/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore