NCVPRIPG 2019 IDD Benchmark

Semantic segmentation on IDD Lite Dataset

This challenge is open to students in Indian universities and colleges. The challenge will use IDD-Lite, a dataset that is small and compact to fit on any personal computer and so will not huge-compute infrastructure. IDD-Lite is less than 50MB is size, contains 7 classes (compared to 30 in IDD).

Directions for Participation

  1. Register here for receiving updates about the competition.
  2. Register an account at the IDD website: http://idd.insaan.iiit.ac.in/
  3. After logging in to IDD website, download the IDD Lite dataset (http://idd.insaan.iiit.ac.in/dataset/download/)
  4. Segmentations masks have all been already generated for IDD Lite in the download.
  5. Once you have built a model, and have the predictions of the model in any of the split (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/idd_lite_evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your predictions is a png image, which has the size of 256x128. Each pixel of this image contains the label as level1Ids (see labels code) of the corresponding image (resized to 256x128). The evaluation code above resizes both your prediction and ground truth png files to 256x128, in case they are not of that size.
  6. Finally you can upload the predictions for the test split, to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
Results
Team/Uploader Name Method Name mIoUmIoU for L1 IDs at 128p
kt002 trial 0.0321
SSN_CSE Modified FCN 0.3438
KLE Tech_Team CustomNet 0.4563
KLE Tech_Team CustomNet 0.0768
Deepsegmentation Deep Encoder Decoder Network for semantic segmentation 0.5017
Deepsegmentation Deep Encoder Decoder Network for semantic segmentation 0.2271
bekar_phd_students Unemployment ki shakti 0.4809
Deepsegmentation Deep Encoder Decoder Network for semantic segmetnation 0.5017
TCE RS Lab Sematic Segmentation U-net 0.5323
kz001 CNet 0.5232
TCE RS Lab Sematic Segmentation U-net 0.5432
SGGS-IITJ testing 0.5296
KLE Tech_Srikar CustomNet 0.0321
SGGS-IITJ Deep CNN 0.5586
SSN_CSE FCN 2 0.3771
KLE Tech_Team CustomNet 0.0781
kz002 CNet 0.5196
kz003 CNet 0.4802
test test 0.4748
t1 t2 0.4831
bekar_phd_students Unemployment ki shakti 0.5293
SSS Gated Shape CNN 0.4872
BharatAI BharatAI-t1 0.5262
SSN_CSE FCN 3 0.3137
Sanchayan Santra nnet 0.3235
Residual U-Net Residual U-Net2 0.5935
test test 0.3235
bekar_phd_students Unemployment ki shakti 0.4968
SSN_CSE UNet 0.1361
SSN_CSE FCN 0.0926
Deepsegmentation Deep Encoder Decoder Network for semantic segmentation 0.4738
bekar_phd_student bekar 0.4616
tyrion first_try 0.4043
SGGS-IITJ testing 0.5176
KLE Tech_Team CustomNet 0.538
SGGS-IITJ Server 1 0.5588
TCE RS Lab Sematic Segmentation U-net 0.532
First_Resnet Residual U-Net 0.5838
kt001 trial 0.0006
Swetha B Enet 0.0
Swetha B Enet 0.0
Swetha B Enet 0.0
Swetha B Enet 0.0
Swetha B Enet 0.0
Swetha B Enet 0.3986
Swetha B Enet 0.3986
mohit_asudani resnet 0.3885
Testing Testing 0.0959
testing testing 0.0959
tensorcat dlv3p 0.0031
mohit_asudani resnet 0.3885
USSB Try1 0.5614
SSN_CSE FCN 4 0.3407
kz004 CNet 0.4349
Parimal TBd 0.0611
Parimal TBd 0.0611
PK tK 0.0611
PK tK 0.0611
PK1 tK1 0.0611
vision deep 0.0611
SSS Tuned GSCNN 0.5231
BharatAI BharatAI-t2 0.4933
PV=NRT segmentation_model 0.0382
coe deep trying 0.5296
NAS_IITRPR Baseline 0.0003
Ryan Dsouza Spatial pooling pyramid 0.0162
NAS_IITRPR Baseline 0.4435
Ryan Dsouza Gods Touch 0.0212
Ryan Dsouza Gods Touches 0.0445
rajat modi tbd 0.4348
godzilla godzilla 0.4348
godzilla2 godzilla2 0.4354
SSS Resnet-101 0.5676
Ryan Dsouza low score 0.0665
PPR Deep Method 0.0729
PPR Deep Method 0.5611
PV Seg_model 0.0403
USSB Try2 0.5505
SSN_CSE UNet 0.0778
SSN_CSE FCN 1 0.4727
SSN_CSE UNet Modified 0.4705
godzilla3 mo 0.4329
KLE Tech_Team_02 VAENet 0.5232
KLE Tech_Team_02 VAENet 0.5196
godzilla4 godzilla4 0.4433
Himanshu Mittal SemanticSegmentation_Enet_v3 0.0
godzilla5 godzilla5 0.4374
godzilla5 godzilla5 0.4374
godzilla6 godzilla6 0.4373
SGGS-IITJ Deep CNN 0.5588
USSB Try3 0.3388
PARIMAL PARIMAL 0.528
PPR Deep method1 0.5681
Tabasco Trial 0.5438
mohit_asudani resnet 0.4243
Alphamales Baseline 0.2405
tabasco Trial2 0.5511
USSB DCNN 0.5502
SGGS-IITJ Try_Server2 0.5201
mohit_asudani resnet 0.4516
Deepsegmentation Deep Encoder Decoder Network for semantic segmetnation 0.306
Ryan Dsouza hallelujah 0.0403
USSB dxb13 0.507
SSS Tuned GSCNN 0.5926
SSS Tuned GSCNN 0.5785
mohit_asudani resnet 0.4515
USSB DCNN 0.5519
SGGS-IITJ RESNET 0.5391
jai mata ki jai mata ki 0.4777
TCE RS Lab Sematic Segmentation U-net 0.5586
USSB DCNN 0.5193
USSB pur9 0.5926
USSB pur7 0.6009
LYNX DRNet 0.0693
PARIMAL U Net 0.5807
Lynx DRNet2 0.0714
lab lab 0.5246
Lynx DSP 0.0632
kz005 CNet 0.3092
Lynx DSP2 0.0623
Residual U-Net Custom Net 0.6081
coe deep custom_net 0.4845
TCE RS Lab Sematic Segmentation Residual U-net 0.5838
Ryan Dsouza Devils dice 0.3117
parimal_1 secret 0.4183
Cerebro PSPnet 0.0779
PARIMAL U Net 0.5841
Residual U-Net Custom Net 0.6238
SSS Tuned GSCNN 0.5832
SSS Tuned GSCNN 0.6014
onemoretry onemoretry 0.4251
SGGS-IITJ Dilated J-ResNet 0.5661
CV Lab Deep ConvNet 0.1921
Aayush encoder_decoder 0.2509
Cerebro Resnet50 0.0691
USSB DCNN 0.5733
Aayush encoder_decoder 0.2875
Aayush encoder_decoder 0.2271
USSB pirv2 0.611
Cerebro Resnet50 0.068
USSB irv2 0.611
SGGS-IITJ RESNET 0.5384
Aayush encoder_decoder 0.2271
Aayush encoder_decoder 0.3147
Aayush encoder_decoder 0.3147
NAS_IITRPR Baseline 2 0.536
SSS Tuned GSCNN 0.6141
tabasco OCR 0.5338
NAS_IITRPR Meta Baseline 0.5336
PARIMAL U Net + CRF 0.5397
SGGS-IITJ DIl_r_resnet 0.5661
SGGS-IITJ Dil_r_resnet(tuned) 0.564
tensorcat dlv3p 0.5511
Himanshu Mittal MEnet 0.0403
BharatAI BharatAI-t0 0.5035
Himanshu Mittal MEnetv1 0.0403
UNET UNET 0.5369
BharatAI t3 0.5344
CV Lab Deep ConvNet 0.1919
CV Lab Deep ConvNet 0.4875
USSB tuned pur7 0.6009
USSB tuned pirv2 0.613
NAS_IITRPR Model 2 0.5386
Espada ENET 0.4705
USSB e5net 0.6087
SGGS-IITJ custom unet 0.6039
BharatAI t4 0.5748
BharatAI BharatAI-t2 0.5588
Espada UNET 0.5729
lab lab 0.5714
tyrion trying 0.4333
BharatAI Morphological Network 0.5823
BharatAI t4 0.5699
SGGS-IITJ custom-unet 0.6166
BharatAI t4 0.5838
lab lab 0.5519
USSB e7net 0.6175
PARIMAL Tuned U-Net 0.4654
PHJA lnet 0.5276
SSS Tuned GSCNN 0.6167
Espada Ensemble 0.531
Espada Ensemble 0.531
small s1 0.5897
CV Lab Deep ConvNet 0.4949
SGGS-IITJ Tuned Custom U-net 0.5515
PARIMAL Tuned Tiramisu 0.5965
SSS Tuned GSCNN 0.2229
USSB ppirv2 0.6175
SGGS-IITJ Tuned Custom U-Net 0.5515
small s1 0.5788
CV Lab SegNet 0.4884
USSB p eff7net 0.6276
SGGS-IITJ Custom Net_final 0.6117
small s1 0.5897
Espada UNET 0.5577
bekar_phd_students Unemployment ki shakti 0.5718
PARIMAL Tuned Tiramisu 0.5952
BharatAI Exp-Net 0.5897
SSS Tuned GSCNN 0.2849
Aayush encoder_decoder 0.3451
Residual U-Net Custom Net 0.6245