FAQ

Let me be honest: These are not exactly frequently asked questions. They are questions I imagine will be asked.

Questions

What is Monitizer and what does it do?

Monitizer is a tool that automates the configuration, optimization, and evaluation of runtime monitors for neural networks, especially for detecting out-of-distribution (OOD) data. Refer to the Introduction for the high-level introduction and to Monitizer Framework for a detailed description of how Monitizer works.

What types of monitors does Monitizer support?

Monitizer includes a library of 19 monitor types from the literature (see Implemented Monitors for a detailed description), and it supports the addition of custom monitors using Python (see Can I implement my own monitor?).

What inputs are required to use Monitizer?

You can run Monitizer without additional input from the examples that are provided. Beyond that, you can vary one or all of the inputs: the NN, the dataset, the monitor template, the optimization configuration. Refer to Use Cases for a detailed explanation.

What are monitor templates?

Monitor templates are unconfigured monitors that define the monitoring method and structure. They need to be optimized for the given model and data before evaluation.

How does Monitizer optimize monitor templates?

Monitizer uses techniques like random search, grid search, or gradient descent to automatically tune monitor parameters.

How does Monitizer evaluate monitor performance?

It evaluates monitors based on detection performance metrics such as precision, recall, F1-score, false positive/negative rates, and AUROC on various OOD classes.

Can I compare multiple monitors at once?

Yes, Monitizer can optimize and evaluate several monitors in one run and provide a comparative report ranking them by performance.

Is Monitizer suitable for real-time monitoring?

Monitizer is primarily intended for offline evaluation and optimization, but the output monitors can be integrated into real-time systems.

Where can I find examples or tutorials?

Look at Use Cases for some example use cases of Monitizer.

Can I implement my own monitor?

Of course! This was one of our main intentions. Refer to Use Case 2 - The Developer of Monitors for the high-level idea and Implementation of a monitor for the developer instructions.

Can I integrate my own dataset?

Certainly! Look at Use Case 1 - The End User for a high-level description and at Implementation of a dataset for detailed implementation instructions.

What does –PARAMETER do?

Check out Input Parameters. We explain every possible input to Monitizer.

How do I choose the best optimization method (grid search, random search, gradient descent)?

The choice depends on the number of parameters and search space complexity. Grid search is exhaustive but slow; random search is faster; gradient descent is best if the loss landscape is smooth and differentiable.

Can I save and reload optimized monitors?

Yes, Monitizer supports saving the optimized monitor configurations, which can later be reloaded for evaluation or deployment.

Is there a way to visualize monitor performance (e.g., ROC curves, confusion matrices)?

Yes, Monitizer can generate plots of key performance metrics during the evaluation phase if visualization modules are enabled.

What programming language is required to use Monitizer?

Monitizer is written in Python, and users should be familiar with Python to configure, extend, or customize the tool.

Can I use Monitizer without a GPU?

Yes, but training and evaluating models—especially with deep networks may be significantly slower on CPU.

How do I integrate Monitizer’s output with my production model?

Optimized monitors can be exported and embedded in your inference pipeline using a simple API or Python script.

What happens if a monitor fails during optimization or evaluation?

Monitizer logs errors and can skip to the next monitor. It provides error reports for debugging.

Can I evaluate monitors on multiple OOD classes independently?

Yes, Monitizer can evaluate each monitor against separate OOD classes and provide class-wise performance metrics.

I don’t want to optimize, can I evaluate my monitor nevertheless?

Sure! You can evaluate the AUROC instead, which works if your monitor provides one single threshold. Use -ec auroc as an input parameter. Note that some monitors do not compute a singular threshold and can thus not comput an AUROC score.

The optimization takes ages. What can I do?

You can either manually fix one parameter of your monitor template (remove it from self._parameters) or you can use the monitor configuration (BETA - see Monitor Configuration (BETA)).

How does the multi-objective optimization work?

For general information, refer to Optimization objectives. A more detailed instruction on how the multi-objectives look like can be found in Multi-Objective. To see, what the output would look like, check out multi-output.