Benchmark

Pixel-Level Semantic Segmentation Task

The segmentation benchmark involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {1. .... , 27}, where the first 26 classes corresponds to the level3Ids (see Overview, for details of the level 3 ids) and the class 27 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 1080p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 27) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leader board.

Results
Team/Uploader Name Method Name mIoU for L3 IDs at 1080p mIoU for L2 IDs at 1080p mIoU for L1 IDs at 1080p
Baseline*DRN-D-38 [3]0.6656--
Baseline*ERFNet [2]0.5541--
Mapillary Research (AutoNUE Challenge) Inplace ABN 0.7432 0.7789 0.8972
BDAI (AutoNUE Challenge) PSPNET+++ 0.7412 0.7796 0.8992
Vinda (AutoNUE Challenge) Joint Channel-Spatial Attention... 0.7407 0.78 0.8986
Geelpen (AutoNUE Challenge) Places365 model feature trained 0.7376 0.7788 0.8954
HUST_IALab (AutoNUE Challenge) DenseScaleNetwork 0.7339 0.7745 0.8955
DeepScene (AutoNUE Challenge) Easpp+DenseAspp 0.7111 0.7584 0.8823
Team7 (AutoNUE Challenge) DRN-D-105 modified 0.6794 0.738 0.8696
Anonymous Anonymous 0.4784 0.5704 0.6707
Attention-Net Attention U-net Based Segmentation 0.3422 0.4716 0.6281
Sabari nathan Attention U-net 0.3305 0.4419 0.6423
Sabari nathan Attention U-net Based Segmentation 0.0187 0.1345 0.3423

* Baseline was run by the organizers using the code released by the authors (ERFNet [2] here: https://github.com/Eromera/erfnet_pytorch) and (DRN [3] here: https://github.com/fyu/drn)

Instance-Level Semantic Segmentation Task

In the instance segmentation benchmark, the model is expected to segment each instance of a class separately. Instance segments are only expected of "things" classes which are all level3Ids under living things and vehicles (ie. level3Ids 4-12).

Output Format & Metric

The output format and metric is the same as Cityscape's instance segmentation [1].

The predictions should use "id" specified in : https://github.com/AutoNUE/public-code/blob/master/helpers/anue_labels.py , unlike the semantic segmentation challenge where level3Ids were used.

Results
Team/Uploader Name Method Name AP AP 50%
TUTU (AutoNUE Challenge) PANET 0.3918 0.6753
Anonymous Anonymous 0.2766 0.5001
Poly (AutoNUE Challenge) RESNET101 MASK RCNN 0.2681 0.4991
Dynamove_IITM (AutoNUE Challenge) Mask RCNN 0.1857 0.3873
DV (AutoNUE Challenge) Finetuned MaskRCNN 0.1036 0.1998
Anonymous Anonymous 0.0 0.0

References

  1. The Cityscapes Dataset for Semantic Urban Scene Understanding.
    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele.
    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3213-3223
  2. ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation.
    E. Romera, J. M. Alvarez, L. M. Bergasa & R. Arroyo,
    Transactions on Intelligent Transportation Systems (T-ITS), December 2017. [pdf]
  3. Dilated Residual Networks.
    Fisher Yu, Vladlen Koltun & Thomas Funkhouser
    Computer Vision and Pattern Recognition (CVPR) 2017. [code]

AutoNue '19 - Localization Challenge

The goal in the localization challenge is, given the start-location (GPS) of a trip by a vehicle, using images, and/or any of the multimodal data (stereo, LIDAR, vehicle parameters) or its combinations, localize the vehicle in real-world, and estimate the trip route, and end-of-trip – as positions (GPS) of the vehicle.

Dataset

Directions for use:

  1. Register to this site, with event selected as "AutoNUE Challenge 2019"
  2. Go to Download > Download page in the menu.
  3. Download the IDD Multimodal - Primary, Secondary and Suplement which has data from various sensors.
  4. Make submissions of predictions (as specified here: https://github.com/AutoNUE/autonue2019_localization) on the test data at Dataset > Submit Result.
  5. See the scores of the metric computed on the test set.

IDD Multimodal - Primary, Secondary and Supplement has the bellow mentioned data:

  • Stereo images from front camera (15 fps)
  • GPS points (15 Hz) – latitude & longitude
  • LIDAR
  • OBD

Output Format & Metric

The evaluation scripts and a specification of the submission format can be found here: https://github.com/AutoNUE/autonue2019_localization

A submission for this challenge consists of translation vectors \(\hat v^t_r\) for timestamps \(t = 1\cdots N\) and routes \(r = 0,1,2\) relative to the starting point of the test data. Our evaluation script rescales the translation vectors to best fit the ground truth translation vectors of the corresponding routes using the Umeyama's Algorithm. Let \(\hat u^t_r\) be the vectors after rescaling. Furthermore the translation vectors are converted to GPS coordinate (lat, log, alt) using the standard Inverse Mercator projection to obtain \(\hat w^t_r\). Then the following metric on \(\hat w^t_r\) is used as benchmark: $$\frac{1}{3\times N} \sum^{N,2}_{i=1,r=0} \text{dist}\left(\hat w^t_r,w^t_r \right) $$ where \(w^t_r\) is the ground truth GPS coordinates for the correponding timestamp \(t\) and route \( r\) and \(\text{dist}\), the distance in meters between the two coordinates.

Results
Team/Uploader Name Method Name Error
Mapillary AI Research (MAIR) mair_sfm 2178.9384
Mapillary AI Research (MAIR) mair2 1083.9888

AutoNue '19 - Segmentation Challenge

The segmentation benchmark involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).

Results
Team/Uploader Name Method Name mIoU for L3 IDs at 720p mIoU for L2 IDs at 720p mIoU for L1 IDs at 720p
Mapillary AI Research (MAIR) tba 0.7597 0.7831 0.8969
DeepBlueAI unknown 0.7506 0.7795 0.8961
west brook test754sin 0.7429 0.7735 0.8941
Anonymous Anonymous 0.7408 0.7706 0.8916
Anonymous test 0.7359 0.7668 0.8883
Anonymous test 0.7303 0.7636 0.8865
DeepScene AdapNet++ 0.7246 0.7601 0.8779
SEG hdc_fpa 0.7224 0.7583 0.8807
westbrook test 0.7192 0.7497 0.8656
Anonymous Anonymous 0.716 0.754 0.8785
SEG hdc 0.7148 0.7527 0.8747
Anonymous TBA 0.7146 0.7469 0.8682
Anonymous Anonymous 0.7146 0.7469 0.8682
DeepScene AdapNet++ 0.6996 0.7437 0.8691
Hang Zhou test 0.6983 0.7429 0.8725
Sally-BITS Pilani DeepLab V3+ with DPC 0.6858 0.7259 0.8543
USSB DeeplabV3+ 0.6841 0.739 0.8675
SGGS DeeplabV3+ 0.6813 0.7375 0.8657
Anonymous Anonymous 0.6754 0.7179 0.8465
SGGS Deeplabv3 b13 0.6581 0.7169 0.854
ASKM PSPNet 0.5256 0.6224 0.7619
Sumukh Aithal K tba 0.4967 0.5939 0.7466
Anonymous Anonymous 0.4876 0.5702 0.6734
RAS DeeplabV3+ 0.4321 0.5412 0.6781
ASKM PSPNet 0.4152 0.5083 0.6291
RAS DeeplabV3+ 0.3969 0.5075 0.6221
Sally-BITS Pilani DeepLab V3+ with DPC 0.0343 0.0496 0.1193
Hang Zhou deeplabv3 0.0199 0.0056 0.0046
CBT attention 0.0199 0.0056 0.0046
chenxi drn_d_38_700 0.0081 0.1929 0.3765
PHJA Anonymous 0.0 0.0147 0.0831

AutoNue '19 - Constrained Devices Challenge

The segmentation in constrained devices requires models with restricted runtime requirements. The participants will be required to run the inference code of their models, in docker containers with restricted memory, cpus and runtime.

Results
Team/Uploader Name Method Name mIoU for L1 IDs at 128p
DeepBlueAI tba 0.5379
DYZH Unknown 0.5009
DeepBlueAI tba 0.5914
ASKM Unknown 0.5884
Mapillary AI Research (MAIR) tba 0.6146
DYZH Unknown 0.5535
ASKM Unknown 0.5671
ASKM Unknown 0.5687
ASKM Unknown 0.5391
DeepBlueAI tba 0.5704
Mapillary AI Research (MAIR) tba 0.5727
DeepBlueAI tba 0.6041
Mapillary AI Research (MAIR) tba 0.6144
DeepBlueAI tba 0.6072
Mapillary AI Research (MAIR) tba 0.6184
ASKM Unknown 0.5911