Developer Manual ================ Describe how Monitizer is structured. Main files, main folders, main classes. Overview -------- In this list, you can finde the automatically generated documentation of the code. To the best of our knowledge, all functions and classes have descriptions that you can find here. .. toctree:: :maxdepth: 2 monitizer-tool .. _monitor-implementation: Implementation of a monitor --------------------------- If you want to implement your own monitor, you must implement the interface :mod:`monitizer.monitors.Monitor`. This entails especially the implementation of the functions :meth:`monitizer.monitors.Monitor.BaseMonitor.__init__`, :meth:`monitizer.monitors.Monitor.BaseMonitor.evaluate`, :meth:`monitizer.monitors.Monitor.BaseMonitor.get_scores`, :meth:`monitizer.monitors.Monitor.BaseMonitor.get_parameters`, :meth:`monitizer.monitors.Monitor.BaseMonitor.set_parameters`. Let's look at each function individually. **init** ^^^^^^^^ Obviously, you need to initialize your monitor. You can do it like so: .. code-block:: python def __init__(self, model: NeuralNetwork, data: Dataset, parameters_for_optimization: [str] = None): super().__init__(model,data) # define the parameters of your model, e.g. you have a threshold and a scaling parameter self.threshold = 0 self.scaling = 1 # define the bounds for your parameters self._bounds = { 'threshold': FloatBounds(0,10), 'scaling': FloatBounds(1,100) } **set_parameters** ^^^^^^^^^^^^^^^^^^ This function is called by the optimization to set new parameters in the monitor. Therefore, you must implement the functionality to update your parameters to a new value. .. code-block:: python def set_parameters(self, parameters: dict): if 'threshold' in parameters: self.threshold = parameters['threshold'] if 'scaling' in parameters: self.scaling = parameters['scaling'] This function is either called during optimization or when you specify parameters for your monitor as input to Monitizer via `-p` (see :doc:`input-parameters`). **get_parameters** ^^^^^^^^^^^^^^^^^^ This function is also used by the optimization to gradually change the values of the monitor. .. code-block:: python def get_parameters(self) -> dict: return { 'threshold' : self.threshold, 'scaling' : self.scaling } .. _evaluate-monitor: **evaluate** or **get_scores** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ These functions are the core functions of your monitor. Given some input to the NN, what does your monitor predict? If your monitor uses a threshold, the function ``evaluate`` is very simple. You call the function ``get_scores`` and compare the result to the threshold. Note that some monitors (like the :mod:`monitizer.monitors.GaussianMonitor` and the :mod:`monitizer.monitors.BoxMonitor`) do not have a threshold and thus have no implementation of the function ``get_scores``. **Note**: The latter is relevant for the computation of the AUROC! .. code-block:: python def evaluate(self, input: DataLoader) -> np.ndarray: scores = self.get_scores(input) return scores > self.threshold Note that the input is a :mod:`torch.utils.data.DataLoader`, which we can iterate over and feed to the NN. As output, however, we expect a :mod:`numpy.ndarray`. You can transform a :mod:`torch.Tensor` to an numpy-array by using `.numpy()`. For the scores, take for example the implementation of the :mod:`monitizer.monitors.Energy`: .. code-block:: python def get_scores(self, input: DataLoader) -> np.ndarray: output = [] # iterate over the inputs for idx, (images, labels) in enumerate(input): with torch.no_grad(): # pass the inputs through the NN logits = self._model(images) # compute the energy score energy = self.temperature * torch.logsumexp(logits / self.temperature, axis=1) output.append(energy) if self._model.device()=='cuda': torch.cuda.empty_cache() return torch.cat(output).cpu().detach().numpy() You can see how we iterate over the dataloader to get the inputs and how they are passed through the NN stored in ``self._model``. .. _monitor-by-user: Using your monitor ^^^^^^^^^^^^^^^^^^^ To input your monitor to Monitizer, use the input parameter ``-mu``, i.e. .. code-block:: bash monitizer -mu PATH/TO/YOUR/MONITOR.py -mu-name YourMonitorClassName ... Make sure to either call your monitor ``Monitor`` or to input the correct class name with ``-mu-name``. .. _dataset-implementation: Implementation of a dataset --------------------------- There are 2 steps to take for adding your own dataset. Let's assume your dataset is called `FancySet`: 1. Add a custom class for your dataset `FancyDataset(Dataset)` implementing the interface :mod:`monitizer.benchmark.dataset.Dataset`. 2. Update the function :meth:`monitizer.benchmark.load` to point to your class. Let's look at these steps one after the other. 1. Custom Class ^^^^^^^^^^^^^^^ You need to implement the interface :mod:`monitizer.benchmark.dataset.Dataset` for your dataset. This includes several functions that we will visit one by one. ``import ...`` """""""""""""" You can use our function :meth:`monitizer.benchmark.datasets.datasets.get_dataloader` to make your life a bit easier: .. code-block:: python from monitizer.benchmark.dataset import Dataset from monitizer.benchmark.datasets.datasets import get_dataloader It creates a :mod:`torch.utils.data.DataLoader` as it is expected as input to the monitor (see :ref:`evaluate-monitor`). ``get_dataset`` ^^^^^^^^^^^^^^^ You need to create a function that loads your dataset, similar to the one below. .. code-block:: python def get_fancy_set(usecase: Literal['val', 'train', 'test'], transform: Callable, validation_split: float = 0.2, data_folder: str = './') -> torch.utils.data.Dataset: # load your data from the data_folder # ... my_fancy_data = torch.utils.data.TensorDataset(fancy_input, fancy_label) # split the dataset on validation, train and test set if usecase=='val': dataset = Subset(my_fancy_data, validation_indices) elif usecase == 'train': dataset = Subset(my_fancy_data, train_indices) else: usecase == 'test': dataset = Subset(my_fancy_data, test_indices) return dataset Optionally, you can store your function in the file :mod:`monitizer.benchmark.datasets.datasets`, where all of Monitizer's loading functions are stored. ``__init__`` """""""""""" This function sets all relevant information about your dataset. .. code-block:: python class FancySet(Dataset): def __init__(self, name: str, get_dataset_function: typing.Callable, # a function that returns a torch.utils.data.Dataset containing your data data_folder='./data', # the location of the datset on your hard drive specification_root='./', # the location of additional specifications (if existing) preprocessing_transformers=[transforms.ToTensor()], # the preprocessing transformations image_size=(32, 32), # the size of the images num_workers=os.cpu_count() - 1 # number of workers for the dataloader ) super().__init__(name, get_mnist, data_folder, root, num_workers=num_workers) # the validation split is by default 0.2, you can set it manually self.validation_split = 0.1 # define the datasets self.test_set = get_dataset_function('test', self.preprocessing_transformers) self.train_set = get_dataset_function('train', self.preprocessing_transformers) self.validation_set = get_dataset_function('val', self.preprocessing_transformers) This function will be called by the function :meth:`monitizer.benchmark.load`. The name of the dataset, the location of the data and optional additional specifications, the preprocessing transformations, the image size and the data loading function are automatically stored by the interface by calling the super initialization `super().__init__`. * ``get_dataset_function`` is the function you implemented above. * ``data_folder`` is the location of the data on your hard-drive. This is also where Monitizer stores all downloaded data (see :ref:`setup`). * ``specification_root`` is an additional location on your hard-drive, where you might have stored additional information. For example, Monitizer uses this to store information about the applied distortions for the generated OOD inputs (see :ref:`ood-explanation`). * ``preprocessing_transformers`` is a list of transformations that are applied to the inputs before they are fed to the network. For example, this contains parsing to torch.Tensor, rescaling, cropping, ... * ``image_size`` defines the size of the inputs, e.g. 28x28 for MNIST. * ``num_workers`` defines the number of workers for the data loading (only really relevant on a GPU). Ignore, if you don't know about it. .. _ood-specific: ``get_OOD_data_specific`` """"""""""""""""""""""""" In this function, you can define specific OOD datasets for your dataset. This can be hand-selected inputs from other datasets or you can use our implemented datasets, if you find them useful. Look at the example from the CIFAR-10 dataset below .. code-block:: python import monitizer.benchmark.datasets.CIFAR10.cifar10_dataloaders as cifar10_data def get_OOD_data_specific(self, name, usecase) -> DataLoader: if name == "NewWorld/GTSRB": return cifar10_data.get_cifar10_gtsrb_dataloader(usecase, data_folder=self.data_folder, root=self.specification_root, num_workers=self.num_workers) elif name == ... ... You can directly use the functions :mod:`monitizer.benchmark.datasets.CIFAR10.cifar10_dataloaders` from any of the datasets if they fit your preference (especially image size). Otherwise, adapt them to your liking or define your own OOD-classes. Just note that the function must return a :mod:`torch.utils.data.DataLoader`. ``get_all_subset_names`` """""""""""""""""""""""" This function servers as an overview that returns all possible OOD classes for your custom dataset. It can look like this .. code-block:: python def get_all_subset_names(self) -> list: generated_ood = self.get_generated_ood_names() specific_ood = ["MY_OOD_CLASS1", "MY_OOD_CLASS2", ...] return generated_ood + specific_ood Note that the names must be the same as in the function ``get_OOD_data_specific`` (see :ref:`ood-specific`)! ``get_ID_train`` """""""""""""""" This function loads the ID training dataset and returns a data loader for further processing. It can look like this: .. code-block:: python def get_ID_train(self): return get_dataloader(self.train_set, num_workers=self.num_workers) ``get_ID_train_labels`` """"""""""""""""""""""" This function loads only the labels of your training dataset. It can look like this: .. code-block:: python def get_ID_train_labels(self) -> torch.Tensor: return self.train_set.dataset.targets[self.train_set.indices] ``get_ID_val`` """""""""""""" This function loads the ID validation dataset and returns a data loader for further processing. It can look like this .. code-block:: python def get_ID_val_labels(self) -> torch.Tensor: return self.validation_set.dataset.targets[self.validation_set.indices] ``get_ID_val_labels`` """"""""""""""""""""" This function loads only the labels of your validation dataset. It can look like this: .. code-block:: python def get_ID_val(self): return get_dataloader(self.validation_set, num_workers=self.num_workers) ``get_ID_test`` """"""""""""""" This function loads the ID test dataset and returns a data loader for further processing. It can look like this .. code-block:: python def get_ID_test(self): return get_dataloader(self.test_set, num_workers=self.num_workers) 2. Update ``load`` ^^^^^^^^^^^^^^^^^^ Open the file :mod:`monitizer.benchmark.load` and modify the function :meth:`monitizer.benchmark.load.load`: .. code-block:: python def load(dataset_name: str, data_folder: str, num_workers: int = os.cpu_count() - 1) -> Dataset: ... ... elif dataset_name.lower() == "kmnist": from monitizer.benchmark.datasets.KMNIST.kmnist_dataset import KMNISTDataset return KMNISTDataset(dataset_name, data_folder=data_folder, num_workers=num_workers) # ENTER YOUR CODE HERE elif dataset_name.lower() == "fancyset": from monitizer.benchmark.dataset.YOURFANCYSET.yourfancyset import FancySet return FancySet(dataset_name, data_folder=data_folder, num_workers=num_workers) # END OF YOUR CODE else: raise NotImplementedError(f"{dataset_name} is not known!") Use your dataset ^^^^^^^^^^^^^^^^ You can now call Monitizer on your dataset by calling .. code-block:: bash monitizer --evaluate --monitor-template all --dataset FancySet --neural-network fancynet --optimize --optimization-objective optimization-objective.ini under the assumption that you have an NN called "fancynet" in your folder. Look at :ref:`use-case1` for more information on that. .. _optimization-config: Optimization configuration -------------------------- The default file `optimization-objective.ini` contains this: .. code-block:: [Optimization] [Optimization.Objective] type = single-objective function = OptimalForOODClassSubjectToFNR file = # single-objective currently contains: OptimalForOODClass, OptimalForOODClassSubjectToFNR # multi-objective currently contains: OptimalForOODClasses, OptimalForOODClassesSubjectToFNR [Optimization.Objective.Specification] ood_classes = Noise / Gaussian tnr_minimum = 70 [Optimization.Optimizer] type = random [Optimization.Optimizer.Random] episodes = 1 [Optimization.Optimizer.Grid-search] grid_count = 4 [Optimization.Optimizer.Gradient-descent] episodes = 1 learning_rate = 1 [Multi - Objective] num_splits = 4 This configures the optimization and can easily be adapted. There are two main parts: the ``Optimization.Objective`` and the ``Optimization.Optimizer``. Objective ^^^^^^^^^ This defines what your optimzation **goal** is, i.e. the **objective** of the optimization. We provide two different types of objectives: single and multi-objectives. *Single objectives* optimize for a singular objective and try to maximize it. *Multi objectives* contain several objectives. Therefore, there is no single maximum, but a set of pareto optimal points (see `Wikipedia `_). You have to choose between ``type = single-objective`` and ``type = multi-objective``, which then further limits your choices. The optimization objective is defined under ``function``. We currently offer two for each the single- and the multi-objective case, described below. You can also define your own optimization objective which you can input here as a file ``file = Path/To/Your/Objective/YourObjective.py``. Refer to :ref:`opti-objective-implementation` for instructions how to implement your own objective. Optimization objectives """"""""""""""""""""""" **OptimalForOODClass** (single objective) optimizes the monitor to perform as good as possible on a specified OOD class. For example, you can optimize the monitor to perform as good as possible on noisy data. Check out :meth:`monitizer.optimizers.single_objectives.OptimalForOODClass` for the implementation. **OptimalForOODClasses** (multi objective) is similar to the above, only that you can specify a list of OOD classes that Monitizer optimizes for. Check out :meth:`monitizer.optimizers.multi_objectives.OptimalForOODClasses` for the implementation. **OptimalForOODClassSubjectToFNR** (single objective) optimizes the monitor to perform as good as possible on a specified OOD class under the constraint the false-negative rate on the ID data should be better than a specified threshold. Check out :meth:`monitizer.optimizers.single_objectives.OptimalForOODClassSubjectToFNR` for the implementation. **OptimalForOODClassesSubjectToFNR** (multi objective) is similar to the above, only that you can specify a list of OOD classes that Monitizer optimizes for. Check out :meth:`monitizer.optimizers.multi_objectives.OptimalForOODClassesSubjectToFNR` for the implementation. Depending on the optimization function, you need to add specifications, e.g. the OOD class you are optimizing for. This is done in the section ``[Optimization.Objective.Specification]``. For the pre-implemented functions, there are two specifications. * the OOD class(es): for example ``ood_classes = Noise/Gaussian`` or ``ood_classes = Noise/Gaussian,Noise/SaltAndPepper``. The OOD class names must be defined in the dataset (check :ref:`ood-classes` for the automatically generated classes, and :doc:`benchmarks` for all implemented datasets). * (optional) the minimum true-negative rate on the ID dataset, e.g. ``tnr_minimum = 50``. For the multi objective, we need an additional parameter. Since it is intractable to compute the full Pareto frontier, we sample points and compute a value for them. We need to specify this number of samples by .. code-block:: [Multi - Objective] num_splits = 4 Optimizer ^^^^^^^^^ This defines, how the optimization is done. It only contains one information ``type =``. You can choose between thre three offered optimization methods: ``random``,``grid-search``, and ``gradient-descent`` (see :ref:`optimization-methods` for their description). For each of the optimization methods, you need to define their parameters. **Random** Add the number of epsiodes, i.e. the number of random draws: .. code-block:: [Optimization.Optimizer.Random] episodes = 1 **Grid-search** Add the number of splits per parameter. Be careful to avoid a combinatorical explosion! .. code-block:: [Optimization.Optimizer.Grid - search] grid_count = 4 **Gradient-descent** Add the number of episodes and the learning rate. .. code-block:: [Optimization.Optimizer.Gradient - descent] episodes = 1 learning_rate = 1 .. _opti-objective-implementation: Implement an optimization objective ----------------------------------- Implement the interface :mod:`monitizer.optimizers.objective.Objective` or :mod:`monitizer.optimizers.multi_objectives.MultiObjective` for the single and mutli objective case respectively. Single-Objective ^^^^^^^^^^^^^^^^ The implementation of a single objective requires the implementation of two base functions: ``__init__`` and ``__call__``. ``__init__`` """""""""""" Setup the internal parameters in :meth:`monitizer.optimizers.objective.Objective.__init__`. For example, define the ood classes: .. code-block:: python def __init__(self, config: OptimizationConfig): super().__init__(config) self.ood_class = config.additional_objective_info['ood_classes'].split(',')[0] self.tnr_minimum = config.additional_objective_info.getint('tnr_minimum') **Note** that you can access the configuration directly via ``config._config``, which is of type :mod:`configparser.ConfigParser`. ``__call__`` """""""""""" Define the main computation of your objective in :meth:`monitizer.optimizers.objective.Objective.__call__`. .. code-block:: python def __call__(self, monitor: BaseMonitor) -> float: assert monitor.is_initialized() ## compute something ... return result **Note** that you can access the data and the NN directly through the monitor. For example, you can evaluate the monitor on the ID validation set by calling ``monitor.evaluate(monitor.data.get_ID_val())`` or on an ood set by calling ``monitor.evaluate(monitor.data.get_OOD_val(OOD_CLASS_NAME))``. .. _multi-objective: Multi-Objective ^^^^^^^^^^^^^^^ The implementation of a multi objective requires a bit more work than for the single objective. The interface is defined in :mod:`monitizer.optimizers.multi_objective.MultiObjective'. ``__init__`` and ``__call__`` are the same. However, now, we have several objectives that are weighed by a set of weights. The value of the multi objective is then a weighted sum of the containing objectives times the weights. Additionally, there is ``get_num_objectives``, ``evaluate``, ``get_columns``. ``get_num_objectives`` """""""""""""""""""""" Returns the number of objectives that are contained in the multi objective, see :meth:`monitizer.optimizers.multi_objective.MultiObjective.get_num_objectives`. For example: .. code-block:: python def get_num_objectives(self) -> int: return len(self.ood_classes) ``evaluate`` """""""""""" Evaluates the monitor on the contained single objectives and returns the weights and the results of the objectives, see :meth:`monitizer.optimizers.multi_objective.MultiObjective.evaluate`. .. code-block:: python def evaluate(self, monitor: BaseMonitor) -> (list, list): ... return weights, result ``get_columns`` """"""""""""""" This is a helper-function to store the results. It provides column names for a table, such that the results from ``evaluate`` can be stored in such a table (see an example output in :ref:`multi-obj-output`). It returns something like this ["weight-1","weight-2","result-1","result-2"]. For example .. code-block:: python def get_columns(self) -> list: columns = [f"weight-{ood_class}" for ood_class in self.ood_classes] columns += [f"result-{ood_class}" for ood_class in self.ood_classes] return columns .. _bounds: Bounds ------ For the parameters in the monitor templates, we need bounds for the optimization. Instead of defining them manually inside the monitor, we decided to have an object structure for that. Any type of bound is inherently a :mod:`monitizer.monitor.Bounds.Bounds` and has two functions: ``get_random_draw`` and ``get_grid_params``. The first randomly draws within the bounds. The second returns a list of parameters when we split the range uniformly into a grid. We provide four types of bounds: * for **floats** (:mod:`monitizer.monitors.Bounds.FloatBounds`), e.g. a real number between 0 and 1 * for **integers** (:mod:`monitizer.monitors.Bounds.IntegerBounds`), e.g. an integer between 0 and 10 * for **lists** (:mod:`monitizer.monitors.Bounds.UniqueList`), e.g. a value from a specified list. * for **lists of bounds** (:mod:`monitizer.monitors.Bounds.ListOfBounds`), e.g. for a parameter that is a vector They must be initialized with lower and uper value or a list respectively. .. code-block:: python float_bounds = FloatBounds(lower=0, upper=1) integer_bounds = IntegerBounds(lower=0, upper=10) list_bounds = UniqueList([7,2.2,8.1]) list_of_bounds = ListOfBounds([float_bounds,integer_bounds,list_bounds]) .. _monitor-config: Monitor Configuration (BETA) ----------------------------- To ease the input of monitors and to store the input, we provide a method to input a configuration file instead of the input parameters. To this end, use ``-mc CONFIG`` instead of ``-m TEMPLATE``. The file should look like this .. code-block:: [Monitor] location = monitizer name = "gaussian" [Monitor.Parameter.Optimization] # define the paramter that should be optimized parameters = ["thresholds"] parameter_bounds = [Monitor.Parameter.Default] parameter_values = {"layer_indices": ["6"]} The first section defines the location of the file that contains the monitor. If you use a monitor that already exists in Monitizer, use ``location = monitizer`` and change the name to the monitor template you want to use (see :doc:`implemented-monitors` for an overview). Otherwise, add your path to the python file and change the name of the monitor to the name of your class (see :ref:`monitor-by-user`). In the second section, you can define all parameters that shall be optimized. In the third section, you set the default values for all the parameters that shall not be optimized. In the example above, the available parameters are ``threshold`` and ``layer_indices``. ``threshold`` shall be optimized, but ``layer_indices`` is fixed to ``["6"]``.