User Manual
===========

Monitizer is an open-source tool developed by `LiVe-Lab <https://live-lab.fi.muni.cz/>`_  to facilitate the design, optimization, and evaluation of runtime monitors for neural networks (NNs). Its primary goal is to enhance the reliability of NNs, especially in safety-critical applications, by detecting out-of-distribution (OOD) inputs that the network may not have encountered during training.

Getting Started
---------------

Monitizer is available as open source in GitLab https://gitlab.com/live-lab/software/monitizer.

Installation
^^^^^^^^^^^^
Monitizer is a python package available via PyPI.
Make sure you have Python 3 installed (we support 3.8 and higher). 

Download
""""""""
Download the repository via git using:

.. code-block:: bash

    git clone https://gitlab.com/live-lab/software/monitizer.git
    cd monitizer
    
Install Requirements
""""""""""""""""""""
Note that you have to install `PyTorch <https://pytorch.org/>`_. 
The default installation is based on GPU-computation. 
If you do **not have a GPU** on your machine, please run the following lines before continuing with the installation:

.. code-block:: bash

    pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
    
Install all the requirements with:

.. code-block:: bash

    pip install -r requirements.txt
   
This will install the following packages:

* ONNX
* numpy
* pandas
* pytorch
* matplotlib
* onnx2pytorch
* plotly
* kaleido
* tqdm
* tabulate
* torch
* typing_extensions
* scikit-learn

.. _setup:

Setup
"""""
You need to setup Monitizer to already download the relevant datasets. 
It downloads existing datasets from `Torchvision <https://docs.pytorch.org/vision/main/datasets.html>`_. 
Select the datasets you are interested in from the supported datasets (see :doc:`benchmarks`) and replace <dataset> with the name of a dataset that you intend to use, e.g. ./monitizer ``--setup MNIST CIFAR10``.
Specify ``<PATH/TO/DATA>`` with a folder, where you want the raw data to be stored (default is ``data`` within the Monitizer directory).
If you already have some datasets downloaded, please consider setting the download path accordingly.
This will automatically download all required datasets, including the respective OOD-datasets, that monitizer needs.

.. code-block:: bash

    monitizer --setup <dataset1> ... <datasetN> --data-folder <PATH/TO/DATA>
    
For convenience, you can download all implemented datasets by running

.. code-block:: bash

    monitizer --setup all --data-folder <PATH/TO/DATA>
    
Note that Imagenet must be downloaded after logging in, so Monitizer cannot take this over for you.
However, it will tell you where to download the data and where to put it.
Visit https://www.image-net.org/challenges/LSVRC/2012/2012-downloads.php for downloading ILSVRC2012_devkit_t12.tar.gz,
ILSVRC2012_img_val.tar, and ILSVRC2012_devkit_t12.tar.gz, and put it in your data folder.


If you are curious, how the setup actually works, check it out here :meth:`monitizer.monitizer.setup_monitizer`.

.. _first-command:

Your first command
""""""""""""""""""

Run

.. code-block:: bash

    monitizer --evaluate --monitor-template energy --dataset MNIST --neural-network example-networks/MNIST3x100 --optimize --optimization-objective optimization-objective.ini
    
It optimizes an `Energy <https://link.springer.com/chapter/10.1007/978-3-030-88494-9_14>`_ monitor on a network called "MNIST3x100", trained on the MNIST dataset. The optimization is random and the resulting monitor is then to be evaluated.


Overview
--------

We provide more information about what Monitizer contains and how it can be used in the following chapters.
    
.. toctree::
   :maxdepth: 2
 
   benchmarks
   implemented-monitors
   usecases