.. Monitizer documentation master file, created by
   sphinx-quickstart on Fri May 23 08:14:17 2025.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Monitizer documentation
========================

.. _intro:

Introduction
************

Monitizer is a research tool for configuring, evaluating, and optimizing *neural network monitors* - systems that detect out-of-distribution inputs, assess model uncertainty, and raise alerts when a predictive model is at risk of failure. First introduced in the paper *"Monitizer: A Configurable and Optimizable Framework for Monitoring Machine Learning Models"* [:ref:`monitizer-paper`], Monitizer addresses a critical gap in the machine learning lifecycle: systematically analyzing and improving the *monitors* that keep models safe and reliable in deployment.

Monitoring is an increasingly important component of trustworthy AI, yet the tools to support rigorous monitor development are limited. Monitizer changes this by offering a modular and extensible framework that supports multiple types of monitors, datasets, and evaluation protocols. Whether you're working with out-of-distribution detection, confidence-based monitoring, or custom-defined heuristics, Monitizer allows you to plug in your monitor and systematically assess its performance.

This manual is intended to guide you through Monitizer’s capabilities. You’ll find an overview of its key features, detailed setup instructions, and descriptions of supported use cases. The manual also includes a developer section that explains how to implement custom monitors and datasets, as well as advanced functionality like optimization and evaluation workflows.

Monitizer is especially useful for researchers interested in comparing monitoring approaches across different tasks and datasets. With built-in support for hyperparameter optimization, ROC analysis, performance metrics, and benchmark datasets, it facilitates robust, reproducible experiments.

We invite you to explore Monitizer, whether you're an ML practitioner looking to audit model behavior or a researcher developing the next generation of reliable AI monitoring techniques.

The full source code is available here: https://gitlab.com/live-lab/software/monitizer


What is where?
**************

* Are you new to Monitizer? - Start with the :doc:`userman` and check out the :doc:`usecases`.
* Do you want to integrate your own implementation? - Look at the :doc:`developerman`.
* Do you want to know how Monitizer works? - Start with the :doc:`framework`.


.. _monitizer-paper:

References
**********
Azeem, M., Grobelna, M., Kanav, S., Křetínský, J., Mohr, S., Rieder, S. (2024). Monitizer: Automating Design and Evaluation of Neural Network Monitors. In: Gurfinkel, A., Ganesh, V. (eds) Computer Aided Verification. CAV 2024. Lecture Notes in Computer Science, vol 14682. Springer, Cham. https://doi.org/10.1007/978-3-031-65630-9_14

Table of Contents
*****************

.. toctree::
   :maxdepth: 2
 
   userman
   framework
   developerman
   faq
   


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`