AutoNUE 2021 Instructions & Benchmark
1. Supervised Domain Adaptation
This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.
Directions for Participation
- Register an account at http://idd.insaan.iiit.ac.in/,
with the event selected as "AutoNUE Challenge 2021".
- Go to Dataset > Download page in the menu.
- Dataset consists of 2 parts which are available for download.
- The first part is the IDD Segmentation (IDD 20k Part I).
- The other part is IDD Segmentation (IDD 20k Part II).
- Extract both the downloaded compressed files into the same folder.
- Please run the data preparation code for generating ground truth segmentation masks as documented
here: https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C
- Download the target datasets from original websites (all except GTA require registrations), given
below for easy reference:
- https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
- https://bdd-data.berkeley.edu/
(you
might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
- https://download.visinf.tu-darmstadt.de/data/from_games/
(merge all images and labels into a pair folders)
- https://www.cityscapes-dataset.com/examples/#fine-annotations
(make sure that you only download dataset with fine annotations)
- Please run the data preparation code for generating sampled source datasets as documented here:
https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
./domain_adaptation/source/prep_all.sh
This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find
the images and annotations for the source dataset to be used for this challenge.
- Use IDD training and validation datasets for the target.
- Once you have built a model, and have the predictions of the model in any of the splits (train,
val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation.
Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains
the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The
evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case
they are not of that size.
- Finally you can upload the predictions for the test split (4k; 2k each from the two parts of
IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
- Sample format is given in the submission link.
- The suffix of each file should be "_leftImg8bit".
Output Format
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
Metric
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Results
Team/Uploader Name |
Method Name |
mIoU, L3mIoU for L3 IDs at 720p |
mIoU, L2mIoU for L2 IDs at 720p |
mIoU, L1mIoU for L1 IDs at 720p |
BASELINE |
DRND 38 |
0.5615 |
0.6489 |
0.8026 |
2. Semi-supervised Domain Adaptation
This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.
Directions for Participation
- Register an account at http://idd.insaan.iiit.ac.in/,
with the event selected as "AutoNUE Challenge 2021".
- Go to Dataset > Download page in the menu.
- Dataset consists of 2 parts which are available for download.
- The first part is the IDD Segmentation (IDD 20k Part I).
- The other part is IDD Segmentation (IDD 20k Part II).
- Extract both the downloaded compressed files into the same folder.
- Please run the data preparation code for generating ground truth segmentation masks as documented
here: https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --semisup_da True
Note that only selected train masks, which can be used for this challenge, will be generated for the training stage. All validation masks will be generated for the evaluation stage (refer to step 10 below).
- Download the target datasets from original websites (all except GTA require registrations), given
below for easy reference:
- https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
- https://bdd-data.berkeley.edu/
(you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
- https://download.visinf.tu-darmstadt.de/data/from_games/
(merge all images and labels into a pair folders)
- https://www.cityscapes-dataset.com/examples/#fine-annotations
(make sure that you only download dataset with fine annotations)
- Please run the data preparation code for generating sampled source datasets as documented here:
https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
./domain_adaptation/source/prep_all.sh
This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find
the images and annotations for the source dataset to be used for this challenge.
- Once you have built a model, and have the predictions of the model in any of the splits (train,
val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation.
Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains
the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The
evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case
they are not of that size.
- Finally you can upload the predictions for the test split (4k; 2k each from the two parts of
IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
- Sample format is given in the submission link.
- The suffix of each file should be "_leftImg8bit".
Output Format
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
Metric
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Results
Team/Uploader Name |
Method Name |
mIoU, L3mIoU for L3 IDs at 720p |
mIoU, L2mIoU for L2 IDs at 720p |
mIoU, L1mIoU for L1 IDs at 720p |
BASELINE |
USSS |
0.3044 |
0.4031 |
0.5173 |
3. Weakly-supervised Domain Adaptation
This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.
Directions for Participation
- Register an account at http://idd.insaan.iiit.ac.in/,
with the event selected as "AutoNUE Challenge 2021".
- Go to Dataset > Download page in the menu.
- Dataset consists of 2 parts which are available for download.
- The first part is the IDD Segmentation (IDD 20k Part I).
- The other part is IDD Segmentation (IDD 20k Part II).
- Extract both the downloaded compressed files into the same folder.
- Please run the data preparation code for generating ground truth segmentation masks as documented
here: https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --weaksup_da True
Note that only validation masks will be generated for this challenge towards the evaluation stage (refer to step 11 below). Bounding box annotations to be used for training in this challenge are present here: https://github.com/AutoNUE/public-code/tree/master/domain_adaptation/target/weakly-supervised
- Download the target datasets from original websites (all except GTA require registrations), given
below for easy reference:
- https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
- https://bdd-data.berkeley.edu/
(you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
- https://download.visinf.tu-darmstadt.de/data/from_games/
(merge all images and labels into a pair folders)
- https://www.cityscapes-dataset.com/examples/#fine-annotations
(make sure that you only download dataset with fine annotations)
- Please run the data preparation code for generating sampled source datasets as documented here:
https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
./domain_adaptation/source/prep_all.sh
This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find
the images and annotations for the source dataset to be used for this challenge.
- Once you have built a model, and have the predictions of the model in any of the splits (train,
val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation.
Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains
the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The
evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case
they are not of that size.
- Finally you can upload the predictions for the test split (4k; 2k each from the two parts of
IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
- Sample format is given in the submission link.
- The suffix of each file should be "_leftImg8bit".
Output Format
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
Metric
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Results
Team/Uploader Name |
Method Name |
mIoU, L3mIoU for L3 IDs at 720p |
mIoU, L2mIoU for L2 IDs at 720p |
mIoU, L1mIoU for L1 IDs at 720p |
No submissions yet. |
4. Unsupervised Domain Adaptation
This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.
Directions for Participation
- Register an account at http://idd.insaan.iiit.ac.in/,
with the event selected as "AutoNUE Challenge 2021".
- Go to Dataset > Download page in the menu.
- Dataset consists of 2 parts which are available for download.
- The first part is the IDD Segmentation (IDD 20k Part I).
- The other part is IDD Segmentation (IDD 20k Part II).
- Extract both the downloaded compressed files into the same folder.
- Please run the data preparation code for generating ground truth segmentation masks as documented
here: https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --unsup_da True
Note that only validation masks will be generated for this challenge towards the evaluation stage (refer to step 11 below). IDD Training labels cannot be used for this challenge. Images from training data can be used.
- Download the target datasets from original websites (all except GTA require registrations), given
below for easy reference:
- https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
- https://bdd-data.berkeley.edu/
(you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
- https://download.visinf.tu-darmstadt.de/data/from_games/
(merge all images and labels into a pair folders)
- https://www.cityscapes-dataset.com/examples/#fine-annotations
(make sure that you only download dataset with fine annotations)
- Please run the data preparation code for generating sampled source datasets as documented here:
https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
./domain_adaptation/source/prep_all.sh
This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find
the images and annotations for the source dataset to be used for this challenge.
- Once you have built a model, and have the predictions of the model in any of the splits (train,
val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation.
Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains
the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The
evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case
they are not of that size.
- Finally you can upload the predictions for the test split (4k; 2k each from the two parts of
IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
- Sample format is given in the submission link.
- The suffix of each file should be "_leftImg8bit".
Output Format
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
Metric
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Results
Team/Uploader Name |
Method Name |
mIoU, L3mIoU for L3 IDs at 720p |
mIoU, L2mIoU for L2 IDs at 720p |
mIoU, L1mIoU for L1 IDs at 720p |
BASELINE |
DRND 22 |
0.2551 |
0.3522 |
0.492 |
5. Semantic Segmentation
The segmentation challenge involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).
Directions for Participation
- Register an account at http://idd.insaan.iiit.ac.in/,
with the event selected as "AutoNUE Challenge 2021".
- Go to Dataset > Download page in the menu.
- Dataset consists of 2 parts which are available for download.
- The first part is the IDD Segmentation (IDD 20k Part I).
- The other part is IDD Segmentation (IDD 20k Part II).
- Extract both the downloaded compressed files into the same folder.
- Please run the data preparation code for generating ground truth segmentation masks as documented
here: https://github.com/AutoNUE/public-code.
Use the following command for segmentation mask generation:
python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C
- Once you have built a model, and have the predictions of the model in any of the splits (train,
val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation.
Use the following command for segmentation evaluation:
python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains
the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The
evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case
they are not of that size.
- Finally you can upload the predictions for the test split (4k; 2k each from the two parts of
IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
- Sample format is given in the submission link.
- The suffix of each file should be "_leftImg8bit".
Output Format
The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.
Metric
We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.
Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code
Results
Team/Uploader Name |
Method Name |
mIoU, L3mIoU for L3 IDs at 720p |
mIoU, L2mIoU for L2 IDs at 720p |
mIoU, L1mIoU for L1 IDs at 720p |
Huy Hung Nguyen |
HRNet |
0.7621 |
0.786 |
0.898 |
Julius Zhang |
deeplab |
0.752 |
0.7783 |
0.8932 |