FAQ
===

Let me be honest: These are not exactly **frequently** asked questions. They are questions I imagine will be asked.

Questions
---------

**What is Monitizer and what does it do?** 
""""""""""""""""""""""""""""""""""""""""""
Monitizer is a tool that automates the configuration, optimization, and evaluation of runtime monitors for neural networks, especially for detecting out-of-distribution (OOD) data.
Refer to the :ref:`intro` for the high-level introduction and to :doc:`framework` for a detailed description of how Monitizer works.

**What types of monitors does Monitizer support?** 
""""""""""""""""""""""""""""""""""""""""""""""""""
Monitizer includes a library of 19 monitor types from the literature (see :doc:`implemented-monitors` for a detailed description), and it supports the addition of custom monitors using Python (see :ref:`faq-monitor`).

**What inputs are required to use Monitizer?**
""""""""""""""""""""""""""""""""""""""""""""""
You can run Monitizer without additional input from the examples that are provided. Beyond that, you can vary one or all of the inputs: the NN, the dataset, the monitor template, the optimization configuration. Refer to :doc:`usecases` for a detailed explanation.

**What are monitor templates?**
"""""""""""""""""""""""""""""""
Monitor templates are unconfigured monitors that define the monitoring method and structure. They need to be optimized for the given model and data before evaluation.


**How does Monitizer optimize monitor templates?**
""""""""""""""""""""""""""""""""""""""""""""""""""
Monitizer uses techniques like random search, grid search, or gradient descent to automatically tune monitor parameters.

**How does Monitizer evaluate monitor performance?**
""""""""""""""""""""""""""""""""""""""""""""""""""""
It evaluates monitors based on detection performance metrics such as precision, recall, F1-score, false positive/negative rates, and AUROC on various OOD classes.

**Can I compare multiple monitors at once?**
""""""""""""""""""""""""""""""""""""""""""""
Yes, Monitizer can optimize and evaluate several monitors in one run and provide a comparative report ranking them by performance.

**Is Monitizer suitable for real-time monitoring?**
"""""""""""""""""""""""""""""""""""""""""""""""""""
Monitizer is primarily intended for offline evaluation and optimization, but the output monitors can be integrated into real-time systems.

**Where can I find examples or tutorials?**
"""""""""""""""""""""""""""""""""""""""""""
Look at :doc:`usecases` for some example use cases of Monitizer.

.. _faq-monitor:

**Can I implement my own monitor?** 
"""""""""""""""""""""""""""""""""""
Of course! This was one of our main intentions. Refer to :ref:`use-case2` for the high-level idea and :ref:`monitor-implementation` for the developer instructions.

**Can I integrate my own dataset?**
"""""""""""""""""""""""""""""""""""
Certainly! Look at :ref:`use-case1` for a high-level description and at :ref:`dataset-implementation` for detailed implementation instructions.

**What does --PARAMETER do?**
"""""""""""""""""""""""""""""
Check out :doc:`input-parameters`. We explain every possible input to Monitizer.
 
**How do I choose the best optimization method (grid search, random search, gradient descent)?**
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The choice depends on the number of parameters and search space complexity. Grid search is exhaustive but slow; random search is faster; gradient descent is best if the loss landscape is smooth and differentiable.
 
**Can I save and reload optimized monitors?**
"""""""""""""""""""""""""""""""""""""""""""""
Yes, Monitizer supports saving the optimized monitor configurations, which can later be reloaded for evaluation or deployment.

**Is there a way to visualize monitor performance (e.g., ROC curves, confusion matrices)?**
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Yes, Monitizer can generate plots of key performance metrics during the evaluation phase if visualization modules are enabled.

**What programming language is required to use Monitizer?**
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Monitizer is written in Python, and users should be familiar with Python to configure, extend, or customize the tool.

**Can I use Monitizer without a GPU?**
""""""""""""""""""""""""""""""""""""""
Yes, but training and evaluating models—especially with deep networks may be significantly slower on CPU.

**How do I integrate Monitizer's output with my production model?**
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Optimized monitors can be exported and embedded in your inference pipeline using a simple API or Python script.

**What happens if a monitor fails during optimization or evaluation?**
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Monitizer logs errors and can skip to the next monitor. It provides error reports for debugging.

**Can I evaluate monitors on multiple OOD classes independently?**
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Yes, Monitizer can evaluate each monitor against separate OOD classes and provide class-wise performance metrics.

**I don't want to optimize, can I evaluate my monitor nevertheless?**
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Sure! You can evaluate the AUROC instead, which works if your monitor provides one single threshold. Use ``-ec auroc`` as an input parameter. Note that some monitors do not compute a singular threshold and can thus not comput an AUROC score.

**The optimization takes ages. What can I do?**
"""""""""""""""""""""""""""""""""""""""""""""""
You can either manually fix one parameter of your monitor template (remove it from ``self._parameters``) or you can use the monitor configuration (BETA - see :ref:`monitor-config`).

**How does the multi-objective optimization work?**
""""""""""""""""""""""""""""""""""""""""""""""""""""
For general information, refer to :ref:`optimization-objectives`. A more detailed instruction on how the multi-objectives look like can be found in :ref:`multi-objective`. To see, what the output would look like, check out :ref:`multi-output`.