Optimization ============ The optimization in Monitizer automatically tunes the parameters of monitor templates to improve their performance on specific tasks. It searches through the space of possible parameter configurations using methods like grid search, random search, or gradient descent. This process enables users to find monitor setups that yield better accuracy, calibration, or other user-defined objectives without manual trial and error. In any optimization, there are two key components: the optimization **objective** (e.g. a loss function) and the optimization **function** (e.g. an optimizer). In Monitizer, we provide different types of objectives and optimization functions. .. _optimization-objectives: Optimization objectives ----------------------- We provide two different types of objectives: single and multi-objectives. *Single objectives* optimize for a singular objective and try to maximize it. *Multi objectives* contain several objectives. Therefore, there is no single maximum, but a set of pareto optimal points (see `Wikipedia `_). We provide four implementations of objectives: each two for single and multi objective cases. One objective tries to maximize the performance of a monitor template on a specific OOD class. The second objective additionally restricts to the case where the performance on the ID data stays within a threshold. For the multi-objective case, one can specify a set of OOD classes for which it should optimize. The single objectives are stored in :mod:`monitizer.optimizers.single_objectives` and the multi objectives in :mod:`monitizer.optimizers.multi_objectives`. .. _optimization-methods: Optimization methods -------------------- Grid Search """"""""""" Stored at :meth:`monitizer.optimizers.optimization_functions.optimize_monitor_grid_search` - **How it works:** Exhaustively evaluates all combinations of predefined parameter values. - **Use case:** Suitable for small parameter spaces where brute-force is feasible. - **Pros:** Simple, deterministic, guarantees best result in the grid. - **Cons:** Computationally expensive for large parameter spaces. Random Search """"""""""""" Stored at :meth:`monitizer.optimizers.optimization_functions.optimize_monitor_random` - **How it works:** Samples parameter combinations randomly from a defined distribution. - **Use case:** Useful for large or high-dimensional search spaces. - **Pros:** Often more efficient than grid search in practice. - **Cons:** Results can vary; may miss optimal configurations without enough samples. Gradient-Based Optimization """"""""""""""""""""""""""""""""" Stored at :meth:`monitizer.optimizers.optimization_functions.optimize_monitor_gradient_descent` - **How it works:** Uses gradients of a differentiable loss function to iteratively update parameters toward an optimum (e.g., SGD, Adam). - **Use case:** Works well when monitors have differentiable components. - **Pros:** Efficient for continuous and high-dimensional spaces. - **Cons:** Requires differentiability and may get stuck in local minima. Custom Optimizers """""""""""""""""""""" - **How it works:** Users can implement their own optimization routine by extending Monitizer’s optimizer interface. - **Use case:** Advanced users with specific needs or domain knowledge. - **Pros:** Full flexibility. - **Cons:** Requires development effort.