Benchmarks ========== Monitizer currently supports 9 ID-datasets: `MNIST `__, `CIFAR-10 `_, `CIFAR-100 `_, `German Traffic Sign Recognition Benchmark (GTSRB) `_, `SVHN `__, `Describable Textures Dataset (DTD) `_, `Fashion-MNIST `_, `Kuzushiji-MNIST `_, `ImageNet `__. Note that it is fairly simple to include any other dataset, especially if it already exists in `Torchvision `_. Check out :ref:`dataset-implementation` to see how you can include another dataset. There you will see how you can include your own custom dataset. .. _cifar10: CIFAR-10 -------- The CIFAR-10 dataset consists of 60,000 32x32 color images in 10 different classes. It is widely used for object recognition tasks. `CIFAR-10 Dataset `_ .. image:: ../images/CIFAR10.png :alt: CIFAR-10 example image :width: 400px Monitizer provides four specific OOD classes for CIFAR-10: * NewWorld/GTSRB: (handselected) GTSRB images share no similarity with any CIFAR-100 image * NewWorld/DTD: DTD images share no similarity with any CIFAR-10 image * UnseenObject/Cifar100: the general type of pictures is similar, but the (handselected) CIFAR-100 images contain objects that do not appear in CIFAR-10 * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _cifar100: CIFAR-100 --------- Similar to CIFAR-10 but with 100 classes containing 600 images each, CIFAR-100 offers a more challenging classification problem. `CIFAR-100 Dataset `_ .. image:: ../images/CIFAR100.png :alt: CIFAR-100 example image :width: 400px Monitizer provides three specific OOD classes for CIFAR-100: * NewWorld/GTSRB: (handselected) GTSRB images share no similarity with any CIFAR-100 image * NewWorld/DTD: DTD images share no similarity with any CIFAR-100 image * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _fashion_mnist: Fashion-MNIST ------------- Fashion-MNIST contains 70,000 grayscale images of 28x28 pixels representing 10 different fashion categories. `Fashion-MNIST Dataset `_ .. image:: ../images/FashionMNIST.png :alt: Fashion-MNIST example image :width: 400px Monitizer provides one specific OOD classes for FashionMNIST * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _gtsrb: GTSRB (German Traffic Sign Recognition Benchmark) -------------------------------------------------- GTSRB is a multi-class traffic sign recognition dataset with more than 50,000 images of traffic signs in different conditions. `GTSRB Dataset `_ .. image:: ../images/GTSRB.png :alt: GTSRB example image :width: 400px Monitizer provides four specific OOD classes for GTSRB: * NewWorld/CIFAR10: (handselected) CIFAR-10 images share no similarity with any GTSRB image * NewWorld/DTD: DTD images share no similarity with any GTSRB image * NewWorld/SVHN: (handselected) SVHN images that share no similarity with any GTSRB image * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _imagenet: ImageNet -------- ImageNet is a large-scale dataset with over 14 million images across more than 20,000 categories, commonly used for deep learning. `ImageNet Dataset `_ .. image:: ../images/Imagenet.png :alt: ImageNet example image :width: 400px .. _kmnist: KMNIST (Kuzushiji-MNIST) ------------------------ KMNIST is a dataset of 70,000 grayscale images of Japanese Kuzushiji characters, designed as a drop-in replacement for MNIST. `KMNIST Dataset `_ .. image:: ../images/KMNIST.png :alt: KMNIST example image :width: 400px Monitizer provides one specific OOD classes for FashionMNIST * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _mnist: MNIST ----- MNIST contains 70,000 grayscale images of handwritten digits (0-9) and is a classic benchmark for image classification. `MNIST Dataset `_ .. image:: ../images/MNIST.png :alt: MNIST example image :width: 400px Monitizer provides six specific OOD classes for MNIST: * NewWorld/CIFAR10: CIFAR-10 images share no similarity with any MNIST image * NewWorld/DTD: DTD images share no similarity with any MNIST image * UnseenEnvironment/SVHN: SVHN also shows numbers but in a different style than MNIST * UnseenObject/FashionMNIST: the type of images is the same, but the objects differ * UnseenObject/KMNIST: the type of images is the same, but the objects differ * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _svhn: SVHN (Street View House Numbers) -------------------------------- SVHN consists of real-world house number images obtained from Google Street View, used for digit recognition tasks. `SVHN Dataset `_ .. image:: ../images/SVHN.png :alt: SVHN example image :width: 400px Monitizer provides one specific OOD classes for FashionMNIST * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples .. _dtd: DTD (Describing Textures in the Wild) ------------------------------------- DTD contains 5,640 images categorized into 47 texture classes and is used for texture recognition and classification. `DTD Dataset `_ .. image:: ../images/DTD.png :alt: DTD example image :width: 400px Monitizer provides one specific OOD classes for FashionMNIST * WrongPrediction/FGSM: automatically generated OOD images by using a provided NN to generate adversarial examples