Optimization

The optimization in Monitizer automatically tunes the parameters of monitor templates to improve their performance on specific tasks. It searches through the space of possible parameter configurations using methods like grid search, random search, or gradient descent. This process enables users to find monitor setups that yield better accuracy, calibration, or other user-defined objectives without manual trial and error.

In any optimization, there are two key components: the optimization objective (e.g. a loss function) and the optimization function (e.g. an optimizer).

In Monitizer, we provide different types of objectives and optimization functions.

Optimization objectives

We provide two different types of objectives: single and multi-objectives. Single objectives optimize for a singular objective and try to maximize it. Multi objectives contain several objectives. Therefore, there is no single maximum, but a set of pareto optimal points (see Wikipedia).

We provide four implementations of objectives: each two for single and multi objective cases. One objective tries to maximize the performance of a monitor template on a specific OOD class. The second objective additionally restricts to the case where the performance on the ID data stays within a threshold. For the multi-objective case, one can specify a set of OOD classes for which it should optimize.

The single objectives are stored in monitizer.optimizers.single_objectives and the multi objectives in monitizer.optimizers.multi_objectives.

Optimization methods

Grid Search

Stored at monitizer.optimizers.optimization_functions.optimize_monitor_grid_search()

• How it works: Exhaustively evaluates all combinations of predefined parameter values.

• Use case: Suitable for small parameter spaces where brute-force is feasible.

• Pros: Simple, deterministic, guarantees best result in the grid.

• Cons: Computationally expensive for large parameter spaces.

Random Search

Stored at monitizer.optimizers.optimization_functions.optimize_monitor_random()

• How it works: Samples parameter combinations randomly from a defined distribution.

• Use case: Useful for large or high-dimensional search spaces.

• Pros: Often more efficient than grid search in practice.

• Cons: Results can vary; may miss optimal configurations without enough samples.

Gradient-Based Optimization

Stored at monitizer.optimizers.optimization_functions.optimize_monitor_gradient_descent()

• How it works: Uses gradients of a differentiable loss function to iteratively update parameters toward an optimum (e.g., SGD, Adam).

• Use case: Works well when monitors have differentiable components.

• Pros: Efficient for continuous and high-dimensional spaces.

• Cons: Requires differentiability and may get stuck in local minima.

Custom Optimizers

• How it works: Users can implement their own optimization routine by extending Monitizer’s optimizer interface.

• Use case: Advanced users with specific needs or domain knowledge.

• Pros: Full flexibility.

• Cons: Requires development effort.