# AutoNUE 2019 Benchmark

The challenge will have 4 benchmarks, details of which can be seen bellow:

## 1. AutoNue '19 - Segmentation

The segmentation challenge involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).

### Dataset

IDD20K is an expanded version of the IDD dataset released last year with fine annotations for 20K images.

### Directions for Participation

1. Register an account at http://idd.insaan.iiit.ac.in/, with event selected as "AutoNUE Challenge 2019".
4. First part is the IDD - Segmentation (IDD 20k Part I) that was released last year.
5. The other part is IDD 20k Part II.
6. Extract both the downloaded compressed files in to the same folder.
7. Please run the data preparation code for generating ground truth segmentation mask as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers$C
8. Once you have built a model, and have the predictions of the model in any of the split (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds$PRED --num-workers $C Your predictions is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level3Ids (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size. 9. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/ ### Output Format The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes corresponds to the level3Ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class. ### Metric We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge. Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leader board. Evaluation scripts are available here: https://github.com/AutoNUE/public-code ##### Results Team/Uploader Name Method Name mIoU, L3mIoU for L3 IDs at 720p mIoU, L2mIoU for L2 IDs at 720p mIoU, L1mIoU for L1 IDs at 720p Mapillary AI Research (MAIR) tba 0.7597 0.7831 0.8969 DeepBlueAI unknown 0.7506 0.7795 0.8961 west brook test754sin 0.7429 0.7735 0.8941 Anonymous Anonymous 0.7408 0.7706 0.8916 Anonymous test 0.7359 0.7668 0.8883 Anonymous test 0.7303 0.7636 0.8865 DeepScene AdapNet++ 0.7246 0.7601 0.8779 SEG hdc_fpa 0.7224 0.7583 0.8807 westbrook test 0.7192 0.7497 0.8656 Anonymous Anonymous 0.716 0.754 0.8785 SEG hdc 0.7148 0.7527 0.8747 Anonymous TBA 0.7146 0.7469 0.8682 Anonymous Anonymous 0.7146 0.7469 0.8682 DeepScene AdapNet++ 0.6996 0.7437 0.8691 Hang Zhou test 0.6983 0.7429 0.8725 Sally-BITS Pilani DeepLab V3+ with DPC 0.6858 0.7259 0.8543 USSB DeeplabV3+ 0.6841 0.739 0.8675 SGGS DeeplabV3+ 0.6813 0.7375 0.8657 Anonymous Anonymous 0.6754 0.7179 0.8465 SGGS Deeplabv3 b13 0.6581 0.7169 0.854 ASKM PSPNet 0.5256 0.6224 0.7619 Sumukh Aithal K tba 0.4967 0.5939 0.7466 Anonymous Anonymous 0.4876 0.5702 0.6734 RAS DeeplabV3+ 0.4321 0.5412 0.6781 ASKM PSPNet 0.4152 0.5083 0.6291 RAS DeeplabV3+ 0.3969 0.5075 0.6221 Sally-BITS Pilani DeepLab V3+ with DPC 0.0343 0.0496 0.1193 Hang Zhou deeplabv3 0.0199 0.0056 0.0046 CBT attention 0.0199 0.0056 0.0046 chenxi drn_d_38_700 0.0081 0.1929 0.3765 PHJA Anonymous 0.0 0.0147 0.0831 ## 2. AutoNue '19 - Panoptic Segmentation In the panoptic segmentation benchmark, the model is expected to segment each instance of a class separately. Instance segments are only expected of "things" classes which are all level3Ids under living things and vehicles (ie. level3Ids 4-12). We are also planning to have a panoptic segmentation challenge. More details regarding this task and metrics can be found in the links bellow: • Panoptic Segmentation: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár. "Panoptic Segmentation." CVPR (2019). [paper] ### Directions for Participation 1. Register an account at http://idd.insaan.iiit.ac.in/, with event selected as "AutoNUE Challenge 2019". 2. Go to Dataset > Download page in the menu. 3. Dataset consists of 2 parts which are available for download. 4. First part is the IDD - Segmentation (IDD 20k Part I) that was released last year. 5. The other part is IDD 20k Part II. 6. Extract both the downloaded compressed files in to the same folder. 7. Please run the data preparation code for generating ground truth panoptic segmentation mask using the code here: https://github.com/AutoNUE/public-code. After downloading the code, add the helpers folder to PYTHONPATH. Then use the following command for panoptic mask generation: python preperation/createLabels.py --datadir$ANUE --id-type level3Id --panoptic True --num-workers $C Above generates the panoptic ground truths in the COCO panoptic format (See https://research.mapillary.com/eccv18/#panoptic). The ids used for the labels follow the level3Ids. The panoptic masks are generated at 1280x720 resolution which is used in the leaderboard for benchmarking. The files generated (a json file and a folder for each split) can be found in the gtFine folder of the dataset, after the above script has finished running. 8. Once you have built a model, and have the predictions of the model in any of the split (train, val), you can evaluate the metric using the code here: https://github.com/AutoNUE/panopticapi. Use the following command for panoptic evaluation: pip3 install git+https://github.com/AutoNUE/panopticapi.git python3 -m panopticapi.evaluation --gt_json_file gt_panoptic.json \ --pred_json_file pred_panoptic.json \ --gt_folder gt_panoptic \ --pred_folder pred_panoptic  The format used is similar to the COCO panoptic format (See https://research.mapillary.com/eccv18/#panoptic). The ids used for the labels follow the level3Ids. 9. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/ ### Results Team/Uploader Name Method Name PQ SQ RQ Mapillary AI Research (MAIR) SeamSeg 53.3 79.6 65.9 DeepBlueAI Unknown 50.1 77.2 63.4 Anonymous Anonymous 45.4 76.5 57.5 SJTU_317CV Unknown 45.4 76.5 57.5 Anonymous Anonymous 45.4 76.5 57.5 Anonymous Anonymous 45.4 76.5 57.5 ## 3. AutoNue '19 - Segmentation in Constrained Devices The segmentation in constrained devices requires models with restricted runtime requirements. The participants will be required to run the inference code of their models, in docker containers with restricted memory, cpus and runtime. Following are some publicly available containers: • Pytorch CPU 1.1: gcr.io/deeplearning-platform-release/pytorch-cpu.1-1@sha256:5ed1b2c57fc9139fd571a1216a81275dc3fd08c89d4d61d9a576b1fff8fee081 • Tensorflow CPU 1.14: gcr.io/deeplearning-platform-release/tf-cpu.1-14@sha256:cc76de15705cb0e8cbc7d0d1ce4584a27fc5e334e55a811770506059207fa05e • Tensorflow CPU 2.0: gcr.io/deeplearning-platform-release/tf2-cpu.2-0@sha256:b1b0076dbd08c45f0d463f873ad1b52c82e7f58bd251009fb72f7dead8dde4ba The participants needs to: • Setup docker in their linux system (we test on Ubuntu) using the instructions in the docker website. • Cgroups should be enabled in docker to impose constraints on memory, cpus, runtime, by running the following:  edit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash cgroup_enable=memory swapaccount=1" sudo update-grub reboot  • Run docker pull command, with one of the container urls above.  docker pull [image_url]  • Run the docker container with cpus 4, memory 512MB and runtime using the command:  docker run --rm -it --name autonue-chal --cpus=4 --memory=512m --memory-swap=512m /bin/bash  • Copy the IDD Lite dataset and your inference code to the running docker container using: scp -r [user]@[ip]:[file_path] . scp -r [user]@[ip]:[idd_lite] .  • Run the inference code on the test images, to generate test predictions. The test predictions generated need to be uploaded to the leaderboard in format similar to the semantic segmentation challenge. Winning participants will be required to demonstrate the code running in the environment as above to the organizers. ### Dataset An expanded version of the IDD Lite dataset with a similar label set will be used. The expanded version will be used for the challenge. Participants can start building their models on the IDD Lite dataset that has been released at the IDD Homepage: http://idd.insaan.iiit.ac.in/. ### Directions for Participation 1. Register an account at http://idd.insaan.iiit.ac.in/, with event selected as "AutoNUE Challenge 2019". 2. Go to Dataset > Download page in the menu. 3. Download the IDD-Lite (Under Datasets available) 4. Segmentations masks have all been already generated for IDD Lite in the download. 5. Once you have built a model, and have the predictions of the model in any of the split (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation: python evaluate/idd_lite_evaluate_mIoU.py --gts$GT --preds $PRED --num-workers$C
Your predictions is a png image, which has the size of 256x128. Each pixel of this image contains the label as level1Ids (see labels code) of the corresponding image (resized to 256x128). The evaluation code above resizes both your prediction and ground truth png files to 256x128, in case they are not of that size.
6. Finally you can upload the predictions for the test split, to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/

### Output Format & Metric

The output formats and metrics are similar to the semantic segmentation challenge. The only difference is that we will be using the IDD Lite dataset with is a subsampled version of the IDD dataset with lower image resolutions and labels (level 1). You can find the information about label ids in the processing scripts:

https://github.com/AutoNUE/public-code/blob/master/helpers/anue_labels.py#L40

##### Results
Team/Uploader Name Method Name mIoUmIoU for L1 IDs at 128p
DeepBlueAI tba 0.5379
DYZH Unknown 0.5009
DeepBlueAI tba 0.5914
Mapillary AI Research (MAIR) tba 0.6146
SJTU_317CV Unknown 0.5535
DeepBlueAI tba 0.5704
Mapillary AI Research (MAIR) tba 0.5727
DeepBlueAI tba 0.6041
Mapillary AI Research (MAIR) tba 0.6144
DeepBlueAI tba 0.6072
Mapillary AI Research (MAIR) tba 0.6184

## 4. AutoNue '19 - Localization

The goal in the localization challenge is, given the start-location (GPS) of a trip by a vehicle, using images, and/or any of the multimodal data (stereo, LIDAR, vehicle parameters) or its combinations, localize the vehicle in real-world, and estimate the trip route, and end-of-trip – as positions (GPS) of the vehicle.

### Dataset

Directions for use:

1. Register to http://idd.insaan.iiit.ac.in/, with event selected as "AutoNUE Challenge 2019"
3. Download the IDD Multimodal - Primary, Secondary and Supplement which has data from various sensors.
4. Make submissions of predictions (as specified here: https://github.com/AutoNUE/autonue2019_localization) on the test data at Dataset > Submit Result.
5. See the scores of the metric computed on the test set.

IDD Multimodal - Primary, Secondary and Supplement has the bellow mentioned data:

• Stereo images from front camera (15 fps)
• GPS points (15 Hz) – latitude & longitude
• LIDAR
• OBD

### Output Format & Metric

The evaluation scripts and a specification of the submission format can be found here: https://github.com/AutoNUE/autonue2019_localization

A submission for this challenge consists of translation vectors $$\hat v^t_r$$ for timestamps $$t = 1\cdots N$$ and routes $$r = 0,1,2$$ relative to the starting point of the test data. Our evaluation script rescales the translation vectors to best fit the ground truth translation vectors of the corresponding routes using the Umeyama's Algorithm. Let $$\hat u^t_r$$ be the vectors after rescaling. Furthermore the translation vectors are converted to GPS coordinate (lat, log, alt) using the standard Inverse Mercator projection to obtain $$\hat w^t_r$$. Then the following metric on $$\hat w^t_r$$ is used as benchmark: $$\frac{1}{3\times N} \sum^{N,2}_{i=1,r=0} \text{dist}\left(\hat w^t_r,w^t_r \right)$$ where $$w^t_r$$ is the ground truth GPS coordinates for the corresponding timestamp $$t$$ and route $$r$$ and $$\text{dist}$$, the distance in meters between the two coordinates.