Benchmarks

Monitizer currently supports 9 ID-datasets: MNIST, CIFAR-10, CIFAR-100, German Traffic Sign Recognition Benchmark (GTSRB), SVHN, Describable Textures Dataset (DTD), Fashion-MNIST, Kuzushiji-MNIST, ImageNet.

Note that it is fairly simple to include any other dataset, especially if it already exists in Torchvision. Check out Implementation of a dataset to see how you can include another dataset. There you will see how you can include your own custom dataset.

CIFAR-10

The CIFAR-10 dataset consists of 60,000 32x32 color images in 10 different classes. It is widely used for object recognition tasks.

CIFAR-10 Dataset

CIFAR-10 example image

Monitizer provides four specific OOD classes for CIFAR-10:

  • NewWorld/GTSRB: (handselected) GTSRB images share no similarity with any CIFAR-100 image

  • NewWorld/DTD: DTD images share no similarity with any CIFAR-10 image

  • UnseenObject/Cifar100: the general type of pictures is similar, but the (handselected) CIFAR-100 images contain objects that do not appear in CIFAR-10

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

CIFAR-100

Similar to CIFAR-10 but with 100 classes containing 600 images each, CIFAR-100 offers a more challenging classification problem.

CIFAR-100 Dataset

CIFAR-100 example image

Monitizer provides three specific OOD classes for CIFAR-100:

  • NewWorld/GTSRB: (handselected) GTSRB images share no similarity with any CIFAR-100 image

  • NewWorld/DTD: DTD images share no similarity with any CIFAR-100 image

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

Fashion-MNIST

Fashion-MNIST contains 70,000 grayscale images of 28x28 pixels representing 10 different fashion categories.

Fashion-MNIST Dataset

Fashion-MNIST example image

Monitizer provides one specific OOD classes for FashionMNIST

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

GTSRB (German Traffic Sign Recognition Benchmark)

GTSRB is a multi-class traffic sign recognition dataset with more than 50,000 images of traffic signs in different conditions.

GTSRB Dataset

GTSRB example image

Monitizer provides four specific OOD classes for GTSRB:

  • NewWorld/CIFAR10: (handselected) CIFAR-10 images share no similarity with any GTSRB image

  • NewWorld/DTD: DTD images share no similarity with any GTSRB image

  • NewWorld/SVHN: (handselected) SVHN images that share no similarity with any GTSRB image

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

ImageNet

ImageNet is a large-scale dataset with over 14 million images across more than 20,000 categories, commonly used for deep learning.

ImageNet Dataset

ImageNet example image

KMNIST (Kuzushiji-MNIST)

KMNIST is a dataset of 70,000 grayscale images of Japanese Kuzushiji characters, designed as a drop-in replacement for MNIST.

KMNIST Dataset

KMNIST example image

Monitizer provides one specific OOD classes for FashionMNIST

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

MNIST

MNIST contains 70,000 grayscale images of handwritten digits (0-9) and is a classic benchmark for image classification.

MNIST Dataset

MNIST example image

Monitizer provides six specific OOD classes for MNIST:

  • NewWorld/CIFAR10: CIFAR-10 images share no similarity with any MNIST image

  • NewWorld/DTD: DTD images share no similarity with any MNIST image

  • UnseenEnvironment/SVHN: SVHN also shows numbers but in a different style than MNIST

  • UnseenObject/FashionMNIST: the type of images is the same, but the objects differ

  • UnseenObject/KMNIST: the type of images is the same, but the objects differ

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

SVHN (Street View House Numbers)

SVHN consists of real-world house number images obtained from Google Street View, used for digit recognition tasks.

SVHN Dataset

SVHN example image

Monitizer provides one specific OOD classes for FashionMNIST

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples

DTD (Describing Textures in the Wild)

DTD contains 5,640 images categorized into 47 texture classes and is used for texture recognition and classification.

DTD Dataset

DTD example image

Monitizer provides one specific OOD classes for FashionMNIST

  • WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples