The challenge will have 4 benchmarks, details of which can be seen bellow:
The segmentation challenge involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).
IDD20K is an expanded version of the IDD dataset released last year with fine annotations for 20K images.
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes corresponds to the level3Ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leader board. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Team/Uploader Name | Method Name | mIoU for L3 IDs at 720p | mIoU for L2 IDs at 720p | mIoU for L1 IDs at 720p |
---|---|---|---|---|
Mapillary AI Research (MAIR) | tba | 0.7597 | 0.7831 | 0.8969 |
DeepBlueAI | unknown | 0.7506 | 0.7795 | 0.8961 |
west brook | test754sin | 0.7429 | 0.7735 | 0.8941 |
Anonymous | Anonymous | 0.7408 | 0.7706 | 0.8916 |
Anonymous | test | 0.7359 | 0.7668 | 0.8883 |
Anonymous | test | 0.7303 | 0.7636 | 0.8865 |
DeepScene | AdapNet++ | 0.7246 | 0.7601 | 0.8779 |
SEG | hdc_fpa | 0.7224 | 0.7583 | 0.8807 |
westbrook | test | 0.7192 | 0.7497 | 0.8656 |
Anonymous | Anonymous | 0.716 | 0.754 | 0.8785 |
SEG | hdc | 0.7148 | 0.7527 | 0.8747 |
Anonymous | TBA | 0.7146 | 0.7469 | 0.8682 |
Anonymous | Anonymous | 0.7146 | 0.7469 | 0.8682 |
DeepScene | AdapNet++ | 0.6996 | 0.7437 | 0.8691 |
Hang Zhou | test | 0.6983 | 0.7429 | 0.8725 |
Sally-BITS Pilani | DeepLab V3+ with DPC | 0.6858 | 0.7259 | 0.8543 |
USSB | DeeplabV3+ | 0.6841 | 0.739 | 0.8675 |
SGGS | DeeplabV3+ | 0.6813 | 0.7375 | 0.8657 |
Anonymous | Anonymous | 0.6754 | 0.7179 | 0.8465 |
SGGS | Deeplabv3 b13 | 0.6581 | 0.7169 | 0.854 |
ASKM | PSPNet | 0.5256 | 0.6224 | 0.7619 |
Sumukh Aithal K | tba | 0.4967 | 0.5939 | 0.7466 |
Anonymous | Anonymous | 0.4876 | 0.5702 | 0.6734 |
RAS | DeeplabV3+ | 0.4321 | 0.5412 | 0.6781 |
ASKM | PSPNet | 0.4152 | 0.5083 | 0.6291 |
RAS | DeeplabV3+ | 0.3969 | 0.5075 | 0.6221 |
Sally-BITS Pilani | DeepLab V3+ with DPC | 0.0343 | 0.0496 | 0.1193 |
Hang Zhou | deeplabv3 | 0.0199 | 0.0056 | 0.0046 |
CBT | attention | 0.0199 | 0.0056 | 0.0046 |
chenxi | drn_d_38_700 | 0.0081 | 0.1929 | 0.3765 |
PHJA | Anonymous | 0.0 | 0.0147 | 0.0831 |
In the panoptic segmentation benchmark, the model is expected to segment each instance of a class separately. Instance segments are only expected of "things" classes which are all level3Ids under living things and vehicles (ie. level3Ids 4-12). We are also planning to have a panoptic segmentation challenge. More details regarding this task and metrics can be found in the links bellow:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --panoptic True --num-workers $C
pip3 install git+https://github.com/AutoNUE/panopticapi.git python3 -m panopticapi.evaluation --gt_json_file gt_panoptic.json \ --pred_json_file pred_panoptic.json \ --gt_folder gt_panoptic \ --pred_folder pred_panopticThe format used is similar to the COCO panoptic format (See https://research.mapillary.com/eccv18/#panoptic). The ids used for the labels follow the level3Ids.
Team/Uploader Name | Method Name | PQ | SQ | RQ |
---|---|---|---|---|
Mapillary AI Research (MAIR) | SeamSeg | 53.3 | 79.6 | 65.9 |
DeepBlueAI | Unknown | 50.1 | 77.2 | 63.4 |
Anonymous | Anonymous | 45.4 | 76.5 | 57.5 |
SJTU_317CV | Unknown | 45.4 | 76.5 | 57.5 |
Anonymous | Anonymous | 45.4 | 76.5 | 57.5 |
Anonymous | Anonymous | 45.4 | 76.5 | 57.5 |
The segmentation in constrained devices requires models with restricted runtime requirements. The participants will be required to run the inference code of their models, in docker containers with restricted memory, cpus and runtime.
Following are some publicly available containers:
The participants needs to:
edit /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash cgroup_enable=memory swapaccount=1"
sudo update-grub
reboot
docker pull [image_url]
docker run --rm -it --name autonue-chal --cpus=4 --memory=512m --memory-swap=512m /bin/bash
scp -r [user]@[ip]:[file_path] .
scp -r [user]@[ip]:[idd_lite] .
The test predictions generated need to be uploaded to the leaderboard in format similar to the semantic segmentation challenge. Winning participants will be required to demonstrate the code running in the environment as above to the organizers.
An expanded version of the IDD Lite dataset with a similar label set will be used. The expanded version will be used for the challenge. Participants can start building their models on the IDD Lite dataset that has been released at the IDD Homepage: http://idd.insaan.iiit.ac.in/.
python evaluate/idd_lite_evaluate_mIoU.py --gts $GT --preds $PRED --num-workers
$C
The output formats and metrics are similar to the semantic segmentation challenge. The only difference is that we will be using the IDD Lite dataset with is a subsampled version of the IDD dataset with lower image resolutions and labels (level 1). You can find the information about label ids in the processing scripts:
https://github.com/AutoNUE/public-code/blob/master/helpers/anue_labels.py#L40
Team/Uploader Name | Method Name | mIoU for L1 IDs at 128p |
---|---|---|
DeepBlueAI | tba | 0.5379 |
DYZH | Unknown | 0.5009 |
DeepBlueAI | tba | 0.5914 |
ASKM | Unknown | 0.5884 |
Mapillary AI Research (MAIR) | tba | 0.6146 |
SJTU_317CV | Unknown | 0.5535 |
ASKM | Unknown | 0.5671 |
ASKM | Unknown | 0.5687 |
ASKM | Unknown | 0.5391 |
DeepBlueAI | tba | 0.5704 |
Mapillary AI Research (MAIR) | tba | 0.5727 |
DeepBlueAI | tba | 0.6041 |
Mapillary AI Research (MAIR) | tba | 0.6144 |
DeepBlueAI | tba | 0.6072 |
Mapillary AI Research (MAIR) | tba | 0.6184 |
ASKM | Unknown | 0.5911 |
The goal in the localization challenge is, given the start-location (GPS) of a trip by a vehicle, using images, and/or any of the multimodal data (stereo, LIDAR, vehicle parameters) or its combinations, localize the vehicle in real-world, and estimate the trip route, and end-of-trip – as positions (GPS) of the vehicle.
Directions for use:
IDD Multimodal - Primary, Secondary and Supplement has the bellow mentioned data:
The evaluation scripts and a specification of the submission format can be found here: https://github.com/AutoNUE/autonue2019_localization
A submission for this challenge consists of translation vectors \(\hat v^t_r\) for timestamps \(t = 1\cdots N\) and routes \(r = 0,1,2\) relative to the starting point of the test data. Our evaluation script rescales the translation vectors to best fit the ground truth translation vectors of the corresponding routes using the Umeyama's Algorithm. Let \(\hat u^t_r\) be the vectors after rescaling. Furthermore the translation vectors are converted to GPS coordinate (lat, log, alt) using the standard Inverse Mercator projection to obtain \(\hat w^t_r\). Then the following metric on \(\hat w^t_r\) is used as benchmark: $$\frac{1}{3\times N} \sum^{N,2}_{i=1,r=0} \text{dist}\left(\hat w^t_r,w^t_r \right) $$ where \(w^t_r\) is the ground truth GPS coordinates for the corresponding timestamp \(t\) and route \( r\) and \(\text{dist}\), the distance in meters between the two coordinates.
Team/Uploader Name | Method Name | Error |
---|---|---|
Mapillary AI Research (MAIR) | mair_sfm | 2178.9384 |
Mapillary AI Research (MAIR) | mair2 | 1083.9888 |