Developer Manual¶

Describe how Monitizer is structured. Main files, main folders, main classes.

Overview¶

In this list, you can finde the automatically generated documentation of the code. To the best of our knowledge, all functions and classes have descriptions that you can find here.

Monitizer

Implementation of a monitor¶

If you want to implement your own monitor, you must implement the interface monitizer.monitors.Monitor. This entails especially the implementation of the functions monitizer.monitors.Monitor.BaseMonitor.__init__(), monitizer.monitors.Monitor.BaseMonitor.evaluate(), monitizer.monitors.Monitor.BaseMonitor.get_scores(), monitizer.monitors.Monitor.BaseMonitor.get_parameters(), monitizer.monitors.Monitor.BaseMonitor.set_parameters().

Let’s look at each function individually.

init¶

Obviously, you need to initialize your monitor. You can do it like so:

def __init__(self, model: NeuralNetwork, data: Dataset, parameters_for_optimization: [str] = None):
    super().__init__(model,data)

    # define the parameters of your model, e.g. you have a threshold and a scaling parameter
    self.threshold = 0
    self.scaling = 1

    # define the bounds for your parameters
    self._bounds = {
        'threshold': FloatBounds(0,10),
        'scaling': FloatBounds(1,100)
    }

set_parameters¶

This function is called by the optimization to set new parameters in the monitor. Therefore, you must implement the functionality to update your parameters to a new value.

def set_parameters(self, parameters: dict):
    if 'threshold' in parameters:
        self.threshold = parameters['threshold']
    if 'scaling' in parameters:
        self.scaling = parameters['scaling']

This function is either called during optimization or when you specify parameters for your monitor as input to Monitizer via -p (see Input Parameters).

get_parameters¶

This function is also used by the optimization to gradually change the values of the monitor.

def get_parameters(self) -> dict:
    return {
        'threshold' : self.threshold,
        'scaling' : self.scaling
    }

evaluate or get_scores¶

These functions are the core functions of your monitor. Given some input to the NN, what does your monitor predict? If your monitor uses a threshold, the function evaluate is very simple. You call the function get_scores and compare the result to the threshold. Note that some monitors (like the monitizer.monitors.GaussianMonitor and the monitizer.monitors.BoxMonitor) do not have a threshold and thus have no implementation of the function get_scores.

Note: The latter is relevant for the computation of the AUROC!

def evaluate(self, input: DataLoader) -> np.ndarray:
    scores = self.get_scores(input)
    return scores > self.threshold

Note that the input is a torch.utils.data.DataLoader, which we can iterate over and feed to the NN. As output, however, we expect a numpy.ndarray. You can transform a torch.Tensor to an numpy-array by using .numpy().

For the scores, take for example the implementation of the monitizer.monitors.Energy:

def get_scores(self, input: DataLoader) -> np.ndarray:
    output = []
    # iterate over the inputs
    for idx, (images, labels) in enumerate(input):
        with torch.no_grad():
            # pass the inputs through the NN
            logits = self._model(images)
            # compute the energy score
            energy = self.temperature * torch.logsumexp(logits / self.temperature, axis=1)
            output.append(energy)
    if self._model.device()=='cuda':
        torch.cuda.empty_cache()
    return torch.cat(output).cpu().detach().numpy()

You can see how we iterate over the dataloader to get the inputs and how they are passed through the NN stored in self._model.

Using your monitor¶

To input your monitor to Monitizer, use the input parameter -mu, i.e.

monitizer -mu PATH/TO/YOUR/MONITOR.py -mu-name YourMonitorClassName ...

Make sure to either call your monitor Monitor or to input the correct class name with -mu-name.

Implementation of a dataset¶

There are 2 steps to take for adding your own dataset. Let’s assume your dataset is called FancySet:

Add a custom class for your dataset FancyDataset(Dataset) implementing the interface monitizer.benchmark.dataset.Dataset.
Update the function monitizer.benchmark.load() to point to your class.

Let’s look at these steps one after the other.

1. Custom Class¶

You need to implement the interface monitizer.benchmark.dataset.Dataset for your dataset. This includes several functions that we will visit one by one.

`import ...`¶

You can use our function monitizer.benchmark.datasets.datasets.get_dataloader() to make your life a bit easier:

from monitizer.benchmark.dataset import Dataset
from monitizer.benchmark.datasets.datasets import get_dataloader

It creates a torch.utils.data.DataLoader as it is expected as input to the monitor (see evaluate or get_scores).

`get_dataset`¶

You need to create a function that loads your dataset, similar to the one below.

def get_fancy_set(usecase: Literal['val', 'train', 'test'], transform: Callable, validation_split: float = 0.2, data_folder: str = './') -> torch.utils.data.Dataset:
    # load your data from the data_folder
    # ...
    my_fancy_data = torch.utils.data.TensorDataset(fancy_input, fancy_label)

    # split the dataset on validation, train and test set
    if usecase=='val':
        dataset = Subset(my_fancy_data, validation_indices)
    elif usecase == 'train':
        dataset = Subset(my_fancy_data, train_indices)
    else: usecase == 'test':
        dataset = Subset(my_fancy_data, test_indices)

    return dataset

Optionally, you can store your function in the file monitizer.benchmark.datasets.datasets, where all of Monitizer’s loading functions are stored.

`init`¶

This function sets all relevant information about your dataset.

class FancySet(Dataset):
    def __init__(self, name: str,
        get_dataset_function: typing.Callable, # a function that returns a torch.utils.data.Dataset containing your data
        data_folder='./data', # the location of the datset on your hard drive
        specification_root='./', # the location of additional specifications (if existing)
        preprocessing_transformers=[transforms.ToTensor()], # the preprocessing transformations
        image_size=(32, 32), # the size of the images
        num_workers=os.cpu_count() - 1 # number of workers for the dataloader
        )
        super().__init__(name, get_mnist, data_folder, root, num_workers=num_workers)

        # the validation split is by default 0.2, you can set it manually
        self.validation_split = 0.1

        # define the datasets
        self.test_set = get_dataset_function('test', self.preprocessing_transformers)
        self.train_set = get_dataset_function('train', self.preprocessing_transformers)
        self.validation_set = get_dataset_function('val', self.preprocessing_transformers)

This function will be called by the function monitizer.benchmark.load(). The name of the dataset, the location of the data and optional additional specifications, the preprocessing transformations, the image size and the data loading function are automatically stored by the interface by calling the super initialization super().__init__.

get_dataset_function is the function you implemented above.
data_folder is the location of the data on your hard-drive. This is also where Monitizer stores all downloaded data (see Setup).
specification_root is an additional location on your hard-drive, where you might have stored additional information. For example, Monitizer uses this to store information about the applied distortions for the generated OOD inputs (see Description and Background).
preprocessing_transformers is a list of transformations that are applied to the inputs before they are fed to the network. For example, this contains parsing to torch.Tensor, rescaling, cropping, …
image_size defines the size of the inputs, e.g. 28x28 for MNIST.
num_workers defines the number of workers for the data loading (only really relevant on a GPU). Ignore, if you don’t know about it.

`get_OOD_data_specific`¶

In this function, you can define specific OOD datasets for your dataset. This can be hand-selected inputs from other datasets or you can use our implemented datasets, if you find them useful. Look at the example from the CIFAR-10 dataset below

import monitizer.benchmark.datasets.CIFAR10.cifar10_dataloaders as cifar10_data

def get_OOD_data_specific(self, name, usecase) -> DataLoader:
    if name == "NewWorld/GTSRB":
        return cifar10_data.get_cifar10_gtsrb_dataloader(usecase, data_folder=self.data_folder,
                                                         root=self.specification_root, num_workers=self.num_workers)
    elif name == ...
        ...

You can directly use the functions monitizer.benchmark.datasets.CIFAR10.cifar10_dataloaders from any of the datasets if they fit your preference (especially image size). Otherwise, adapt them to your liking or define your own OOD-classes. Just note that the function must return a torch.utils.data.DataLoader.

`get_all_subset_names`¶

This function servers as an overview that returns all possible OOD classes for your custom dataset. It can look like this

def get_all_subset_names(self) -> list:
    generated_ood = self.get_generated_ood_names()
    specific_ood = ["MY_OOD_CLASS1", "MY_OOD_CLASS2", ...]
    return generated_ood + specific_ood

Note that the names must be the same as in the function get_OOD_data_specific (see get_OOD_data_specific)!

`get_ID_train`¶

This function loads the ID training dataset and returns a data loader for further processing. It can look like this:

def get_ID_train(self):
    return get_dataloader(self.train_set, num_workers=self.num_workers)

`get_ID_train_labels`¶

This function loads only the labels of your training dataset. It can look like this:

def get_ID_train_labels(self) -> torch.Tensor:
    return self.train_set.dataset.targets[self.train_set.indices]

`get_ID_val`¶

This function loads the ID validation dataset and returns a data loader for further processing. It can look like this

def get_ID_val_labels(self) -> torch.Tensor:
    return self.validation_set.dataset.targets[self.validation_set.indices]

`get_ID_val_labels`¶

This function loads only the labels of your validation dataset. It can look like this:

def get_ID_val(self):
    return get_dataloader(self.validation_set, num_workers=self.num_workers)

`get_ID_test`¶

This function loads the ID test dataset and returns a data loader for further processing. It can look like this

def get_ID_test(self):
    return get_dataloader(self.test_set, num_workers=self.num_workers)

2. Update `load`¶

Open the file monitizer.benchmark.load and modify the function monitizer.benchmark.load.load():

def load(dataset_name: str, data_folder: str, num_workers: int = os.cpu_count() - 1) -> Dataset:
    ...
    ...
    elif dataset_name.lower() == "kmnist":
        from monitizer.benchmark.datasets.KMNIST.kmnist_dataset import KMNISTDataset
        return KMNISTDataset(dataset_name, data_folder=data_folder, num_workers=num_workers)
    # ENTER YOUR CODE HERE
    elif dataset_name.lower() == "fancyset":
        from monitizer.benchmark.dataset.YOURFANCYSET.yourfancyset import FancySet
        return FancySet(dataset_name, data_folder=data_folder, num_workers=num_workers)
    # END OF YOUR CODE
    else:
        raise NotImplementedError(f"{dataset_name} is not known!")

Use your dataset¶

You can now call Monitizer on your dataset by calling

monitizer --evaluate --monitor-template all --dataset FancySet --neural-network fancynet --optimize --optimization-objective optimization-objective.ini

under the assumption that you have an NN called “fancynet” in your folder. Look at Use Case 1 - The End User for more information on that.

Optimization configuration¶

The default file optimization-objective.ini contains this:

[Optimization]
[Optimization.Objective]
type = single-objective
function = OptimalForOODClassSubjectToFNR
file =
# single-objective currently contains: OptimalForOODClass, OptimalForOODClassSubjectToFNR
# multi-objective currently contains: OptimalForOODClasses, OptimalForOODClassesSubjectToFNR

[Optimization.Objective.Specification]
ood_classes = Noise / Gaussian
tnr_minimum = 70

[Optimization.Optimizer]
type = random

[Optimization.Optimizer.Random]
episodes = 1

[Optimization.Optimizer.Grid-search]
grid_count = 4

[Optimization.Optimizer.Gradient-descent]
episodes = 1
learning_rate = 1

[Multi - Objective]
num_splits = 4

This configures the optimization and can easily be adapted. There are two main parts: the Optimization.Objective and the Optimization.Optimizer.

Objective¶

This defines what your optimzation goal is, i.e. the objective of the optimization. We provide two different types of objectives: single and multi-objectives. Single objectives optimize for a singular objective and try to maximize it. Multi objectives contain several objectives. Therefore, there is no single maximum, but a set of pareto optimal points (see Wikipedia).

You have to choose between type = single-objective and type = multi-objective, which then further limits your choices. The optimization objective is defined under function. We currently offer two for each the single- and the multi-objective case, described below.

You can also define your own optimization objective which you can input here as a file file = Path/To/Your/Objective/YourObjective.py. Refer to Implement an optimization objective for instructions how to implement your own objective.

Optimization objectives¶

OptimalForOODClass (single objective) optimizes the monitor to perform as good as possible on a specified OOD class. For example, you can optimize the monitor to perform as good as possible on noisy data. Check out monitizer.optimizers.single_objectives.OptimalForOODClass() for the implementation.

OptimalForOODClasses (multi objective) is similar to the above, only that you can specify a list of OOD classes that Monitizer optimizes for. Check out monitizer.optimizers.multi_objectives.OptimalForOODClasses() for the implementation.

OptimalForOODClassSubjectToFNR (single objective) optimizes the monitor to perform as good as possible on a specified OOD class under the constraint the false-negative rate on the ID data should be better than a specified threshold. Check out monitizer.optimizers.single_objectives.OptimalForOODClassSubjectToFNR() for the implementation.

OptimalForOODClassesSubjectToFNR (multi objective) is similar to the above, only that you can specify a list of OOD classes that Monitizer optimizes for. Check out monitizer.optimizers.multi_objectives.OptimalForOODClassesSubjectToFNR() for the implementation.

Depending on the optimization function, you need to add specifications, e.g. the OOD class you are optimizing for. This is done in the section [Optimization.Objective.Specification]. For the pre-implemented functions, there are two specifications.

the OOD class(es): for example ood_classes = Noise/Gaussian or ood_classes = Noise/Gaussian,Noise/SaltAndPepper. The OOD class names must be defined in the dataset (check The OOD classes for the automatically generated classes, and Benchmarks for all implemented datasets).
(optional) the minimum true-negative rate on the ID dataset, e.g. tnr_minimum = 50.

For the multi objective, we need an additional parameter. Since it is intractable to compute the full Pareto frontier, we sample points and compute a value for them. We need to specify this number of samples by

[Multi - Objective]
num_splits = 4

Optimizer¶

This defines, how the optimization is done. It only contains one information type =. You can choose between thre three offered optimization methods: random,``grid-search``, and gradient-descent (see Optimization methods for their description).

For each of the optimization methods, you need to define their parameters.

Random Add the number of epsiodes, i.e. the number of random draws:

[Optimization.Optimizer.Random]
episodes = 1

Grid-search Add the number of splits per parameter. Be careful to avoid a combinatorical explosion!

[Optimization.Optimizer.Grid - search]
grid_count = 4

Gradient-descent Add the number of episodes and the learning rate.

[Optimization.Optimizer.Gradient - descent]
episodes = 1
learning_rate = 1

Implement an optimization objective¶

Implement the interface monitizer.optimizers.objective.Objective or monitizer.optimizers.multi_objectives.MultiObjective for the single and mutli objective case respectively.

Single-Objective¶

The implementation of a single objective requires the implementation of two base functions: __init__ and __call__.

`init`¶

Setup the internal parameters in monitizer.optimizers.objective.Objective.__init__(). For example, define the ood classes:

def __init__(self, config: OptimizationConfig):
    super().__init__(config)
    self.ood_class = config.additional_objective_info['ood_classes'].split(',')[0]
    self.tnr_minimum = config.additional_objective_info.getint('tnr_minimum')

Note that you can access the configuration directly via config._config, which is of type configparser.ConfigParser.

`call`¶

Define the main computation of your objective in monitizer.optimizers.objective.Objective.__call__().

def __call__(self, monitor: BaseMonitor) -> float:
    assert monitor.is_initialized()
    ## compute something
    ...
    return result

Note that you can access the data and the NN directly through the monitor. For example, you can evaluate the monitor on the ID validation set by calling monitor.evaluate(monitor.data.get_ID_val()) or on an ood set by calling monitor.evaluate(monitor.data.get_OOD_val(OOD_CLASS_NAME)).

Multi-Objective¶

The implementation of a multi objective requires a bit more work than for the single objective. The interface is defined in monitizer.optimizers.multi_objective.MultiObjective'. ``__init__` and __call__ are the same. However, now, we have several objectives that are weighed by a set of weights. The value of the multi objective is then a weighted sum of the containing objectives times the weights. Additionally, there is get_num_objectives, evaluate, get_columns.

`get_num_objectives`¶

Returns the number of objectives that are contained in the multi objective, see monitizer.optimizers.multi_objective.MultiObjective.get_num_objectives(). For example:

def get_num_objectives(self) -> int:
    return len(self.ood_classes)

`evaluate`¶

Evaluates the monitor on the contained single objectives and returns the weights and the results of the objectives, see monitizer.optimizers.multi_objective.MultiObjective.evaluate().

def evaluate(self, monitor: BaseMonitor) -> (list, list):
    ...
    return weights, result

`get_columns`¶

This is a helper-function to store the results. It provides column names for a table, such that the results from evaluate can be stored in such a table (see an example output in Multi-Objective optimization). It returns something like this [“weight-1”,”weight-2”,”result-1”,”result-2”]. For example

def get_columns(self) -> list:
    columns = [f"weight-{ood_class}" for ood_class in self.ood_classes]
    columns += [f"result-{ood_class}" for ood_class in self.ood_classes]
    return columns

Bounds¶

For the parameters in the monitor templates, we need bounds for the optimization. Instead of defining them manually inside the monitor, we decided to have an object structure for that.

Any type of bound is inherently a monitizer.monitor.Bounds.Bounds and has two functions: get_random_draw and get_grid_params. The first randomly draws within the bounds. The second returns a list of parameters when we split the range uniformly into a grid.

We provide four types of bounds:

for floats (monitizer.monitors.Bounds.FloatBounds), e.g. a real number between 0 and 1
for integers (monitizer.monitors.Bounds.IntegerBounds), e.g. an integer between 0 and 10
for lists (monitizer.monitors.Bounds.UniqueList), e.g. a value from a specified list.
for lists of bounds (monitizer.monitors.Bounds.ListOfBounds), e.g. for a parameter that is a vector

They must be initialized with lower and uper value or a list respectively.

float_bounds = FloatBounds(lower=0, upper=1)
integer_bounds = IntegerBounds(lower=0, upper=10)
list_bounds = UniqueList([7,2.2,8.1])
list_of_bounds = ListOfBounds([float_bounds,integer_bounds,list_bounds])

Monitor Configuration (BETA)¶

To ease the input of monitors and to store the input, we provide a method to input a configuration file instead of the input parameters. To this end, use -mc CONFIG instead of -m TEMPLATE.

The file should look like this

[Monitor]
location = monitizer
name = "gaussian"

[Monitor.Parameter.Optimization]
# define the paramter that should be optimized
parameters = ["thresholds"]
parameter_bounds =

[Monitor.Parameter.Default]
parameter_values = {"layer_indices": ["6"]}

The first section defines the location of the file that contains the monitor. If you use a monitor that already exists in Monitizer, use location = monitizer and change the name to the monitor template you want to use (see Implemented Monitors for an overview). Otherwise, add your path to the python file and change the name of the monitor to the name of your class (see Using your monitor).

In the second section, you can define all parameters that shall be optimized. In the third section, you set the default values for all the parameters that shall not be optimized. In the example above, the available parameters are threshold and layer_indices. threshold shall be optimized, but layer_indices is fixed to ["6"].

Developer Manual¶

Overview¶

Implementation of a monitor¶

init¶

set_parameters¶

get_parameters¶

evaluate or get_scores¶

Using your monitor¶

Implementation of a dataset¶

1. Custom Class¶

`import ...`¶

`get_dataset`¶

`init`¶

`get_OOD_data_specific`¶

`get_all_subset_names`¶

`get_ID_train`¶

`get_ID_train_labels`¶

`get_ID_val`¶

`get_ID_val_labels`¶

`get_ID_test`¶

2. Update `load`¶

Use your dataset¶

Optimization configuration¶

Objective¶

Optimization objectives¶

Optimizer¶

Implement an optimization objective¶

Single-Objective¶

`init`¶

`call`¶

Multi-Objective¶

`get_num_objectives`¶

`evaluate`¶

`get_columns`¶

Bounds¶

Monitor Configuration (BETA)¶

Monitizer

Navigation

Related Topics

Developer Manual¶

Overview¶

Implementation of a monitor¶

init¶

set_parameters¶

get_parameters¶

evaluate or get_scores¶

Using your monitor¶

Implementation of a dataset¶

1. Custom Class¶

import ...¶

get_dataset¶

__init__¶

get_OOD_data_specific¶

get_all_subset_names¶

get_ID_train¶

get_ID_train_labels¶

get_ID_val¶

get_ID_val_labels¶

get_ID_test¶

2. Update load¶

Use your dataset¶

Optimization configuration¶

Objective¶

Optimization objectives¶

Optimizer¶

Implement an optimization objective¶

Single-Objective¶

__init__¶

__call__¶

Multi-Objective¶

get_num_objectives¶

evaluate¶

get_columns¶

Bounds¶

Monitor Configuration (BETA)¶

`import ...`¶

`get_dataset`¶

`init`¶

`get_OOD_data_specific`¶

`get_all_subset_names`¶

`get_ID_train`¶

`get_ID_train_labels`¶

`get_ID_val`¶

`get_ID_val_labels`¶

`get_ID_test`¶

2. Update `load`¶

`init`¶

`call`¶

`get_num_objectives`¶

`evaluate`¶

`get_columns`¶