User Manual

Monitizer is an open-source tool developed by LiVe-Lab to facilitate the design, optimization, and evaluation of runtime monitors for neural networks (NNs). Its primary goal is to enhance the reliability of NNs, especially in safety-critical applications, by detecting out-of-distribution (OOD) inputs that the network may not have encountered during training.

Getting Started

Monitizer is available as open source in GitLab https://gitlab.com/live-lab/software/monitizer.

Installation

Monitizer is a python package available via PyPI. Make sure you have Python 3 installed (we support 3.8 and higher).

Download

Download the repository via git using:

git clone https://gitlab.com/live-lab/software/monitizer.git
cd monitizer

Install Requirements

Note that you have to install PyTorch. The default installation is based on GPU-computation. If you do not have a GPU on your machine, please run the following lines before continuing with the installation:

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu

Install all the requirements with:

pip install -r requirements.txt

This will install the following packages:

• ONNX

• numpy

• pandas

• pytorch

• matplotlib

• onnx2pytorch

• plotly

• kaleido

• tqdm

• tabulate

• torch

• typing_extensions

• scikit-learn

Setup

You need to setup Monitizer to already download the relevant datasets. It downloads existing datasets from Torchvision. Select the datasets you are interested in from the supported datasets (see Benchmarks) and replace <dataset> with the name of a dataset that you intend to use, e.g. ./monitizer --setup MNIST CIFAR10. Specify <PATH/TO/DATA> with a folder, where you want the raw data to be stored (default is data within the Monitizer directory). If you already have some datasets downloaded, please consider setting the download path accordingly. This will automatically download all required datasets, including the respective OOD-datasets, that monitizer needs.

monitizer --setup <dataset1> ... <datasetN> --data-folder <PATH/TO/DATA>

For convenience, you can download all implemented datasets by running

monitizer --setup all --data-folder <PATH/TO/DATA>

Note that Imagenet must be downloaded after logging in, so Monitizer cannot take this over for you. However, it will tell you where to download the data and where to put it. Visit https://www.image-net.org/challenges/LSVRC/2012/2012-downloads.php for downloading ILSVRC2012_devkit_t12.tar.gz, ILSVRC2012_img_val.tar, and ILSVRC2012_devkit_t12.tar.gz, and put it in your data folder.

If you are curious, how the setup actually works, check it out here monitizer.monitizer.setup_monitizer().

Your first command

Run

monitizer --evaluate --monitor-template energy --dataset MNIST --neural-network example-networks/MNIST3x100 --optimize --optimization-objective optimization-objective.ini

It optimizes an Energy monitor on a network called “MNIST3x100”, trained on the MNIST dataset. The optimization is random and the resulting monitor is then to be evaluated.

Overview

We provide more information about what Monitizer contains and how it can be used in the following chapters.

• Benchmarks
  • CIFAR-10
  • CIFAR-100
  • Fashion-MNIST
  • GTSRB (German Traffic Sign Recognition Benchmark)
  • ImageNet
  • KMNIST (Kuzushiji-MNIST)
  • MNIST
  • SVHN (Street View House Numbers)
  • DTD (Describing Textures in the Wild)
• Implemented Monitors
  • ASH
  • Box
  • DICE
  • Energy
  • Entropy
  • Gaussian
  • GradNorm
  • KLMatching
  • K-Nearest-Neighbors
  • Mahalanobis Distance
  • Maximum Softmax Probability
  • Maximum Logit
  • ODIN
  • RMD
  • ReAct
  • SHE
  • Temperature Scaling
  • VIM
• Use Cases
  • Use Cases
  • Use Case 0 - The Tester
  • Use Case 1 - The End User
  • Use Case 2 - The Developer of Monitors
  • Use Case 3 - The Scholar