Monitizer documentation¶
Introduction¶
Monitizer is a research tool for configuring, evaluating, and optimizing neural network monitors - systems that detect out-of-distribution inputs, assess model uncertainty, and raise alerts when a predictive model is at risk of failure. First introduced in the paper “Monitizer: A Configurable and Optimizable Framework for Monitoring Machine Learning Models” [References], Monitizer addresses a critical gap in the machine learning lifecycle: systematically analyzing and improving the monitors that keep models safe and reliable in deployment.
Monitoring is an increasingly important component of trustworthy AI, yet the tools to support rigorous monitor development are limited. Monitizer changes this by offering a modular and extensible framework that supports multiple types of monitors, datasets, and evaluation protocols. Whether you’re working with out-of-distribution detection, confidence-based monitoring, or custom-defined heuristics, Monitizer allows you to plug in your monitor and systematically assess its performance.
This manual is intended to guide you through Monitizer’s capabilities. You’ll find an overview of its key features, detailed setup instructions, and descriptions of supported use cases. The manual also includes a developer section that explains how to implement custom monitors and datasets, as well as advanced functionality like optimization and evaluation workflows.
Monitizer is especially useful for researchers interested in comparing monitoring approaches across different tasks and datasets. With built-in support for hyperparameter optimization, ROC analysis, performance metrics, and benchmark datasets, it facilitates robust, reproducible experiments.
We invite you to explore Monitizer, whether you’re an ML practitioner looking to audit model behavior or a researcher developing the next generation of reliable AI monitoring techniques.
The full source code is available here: https://gitlab.com/live-lab/software/monitizer
What is where?¶
Are you new to Monitizer? - Start with the User Manual and check out the Use Cases.
Do you want to integrate your own implementation? - Look at the Developer Manual.
Do you want to know how Monitizer works? - Start with the Monitizer Framework.
References¶
Azeem, M., Grobelna, M., Kanav, S., Křetínský, J., Mohr, S., Rieder, S. (2024). Monitizer: Automating Design and Evaluation of Neural Network Monitors. In: Gurfinkel, A., Ganesh, V. (eds) Computer Aided Verification. CAV 2024. Lecture Notes in Computer Science, vol 14682. Springer, Cham. https://doi.org/10.1007/978-3-031-65630-9_14