QoIMetric¶

class axtreme.metrics.QoIMetric(name: str, qoi_estimator: QoIEstimator, minimum_data_points: int = 5, *, lower_is_better: bool | None = None, properties: dict[str, Any] | None = None, attach_transforms: bool = False)¶

Bases: Metric

Helper for recording the QoI estimate over the course of an Experiment.

This helper records the score of a QoIEstimator. Internally it trains a GP on all non-tracking metrics for trials up to the current trial (e.g if the Experiment has 10 trials total, the QoI estimate for trial 5 will use all trials 0-5 inclusive in the GP). The QoIEstimator is then run using the GP.

This metric should be used as a tracking metric (e.g _tracking_metrics attribute Experiment). _tracking_metrics signify that these metrics should not be modelled with a GP (Note: this still need to be explicitly set when creating GPs).

Pseudo code Example: >>> qoi_estimator = qoi.GPBruteForce(…) # or any other QoIEstimator >>> qoi_metric = QoIMetric(name=”QoIMetric”, qoi_estimator=qoi_estimator, attach_transforms=True) >>> experiment = Experiment(…) >>> experiment.add_tracking_metric(qoi_metric)

__init__(name: str, qoi_estimator: QoIEstimator, minimum_data_points: int = 5, *, lower_is_better: bool | None = None, properties: dict[str, Any] | None = None, attach_transforms: bool = False) → None¶

Initialize the QoIMetric.

Parameters:

name – The name of the metric.
qoi_estimator – The QoI estimator to use to calculate the metric.
minimum_data_points – The minimum number of data points the experiment must have before the GP is trained and the QoI is actually run.
lower_is_better – Flag for metrics which should be minimized. Typically we are not interested in minimising/maximising QoIs so this values should be None.
properties – Dictionary of this metric’s properties.
attach_transforms – If True, attaches the input and outcome transforms required of the GP to operate in the problem space. This assume qoi_estimator have attributes input_transform and outcome_transform where these should be attached. TODO(sw 11-4-25): remove when issue #19 is addressed.

Methods

`__init__`(name, qoi_estimator[, ...])	Initialize the QoIMetric.
`bulk_fetch_experiment_data`(experiment, metrics)	Fetch multiple metrics data for multiple trials on an experiment, using instance attributes of the metrics.
`bulk_fetch_trial_data`(trial, metrics, **kwargs)	Fetch multiple metrics data for one trial, using instance attributes of the metrics.
`clone`()	Create a copy of this Metric.
`deserialize_init_args`(args[, ...])	Given a dictionary, deserialize the properties needed to initialize the object.
`fetch_data_prefer_lookup`(experiment, metrics)	Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running.
`fetch_experiment_data_multi`(experiment, metrics)	Fetch multiple metrics data for an experiment.
`fetch_trial_data`(trial, **kwargs)	Fetch the data for a trial.
`fetch_trial_data_multi`(trial, metrics, **kwargs)	Fetch multiple metrics data for one trial.
`is_available_while_running`()	Whether metrics of this class are available while the trial is running.
`maybe_raise_deprecation_warning_on_class_methods`()
`period_of_new_data_after_trial_completion`()	Period of time metrics of this class are still expecting new data to arrive after trial completion.
`serialize_init_args`(obj)	Serialize the properties needed to initialize the object.

Attributes

`db_id`
`fetch_multi_group_by_metric`	Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.
`name`	Get name of metric.
`summary_dict`	Returns a dictionary containing the metric's name and properties.

fetch_trial_data(trial: BaseTrial, **kwargs: Any) → Result[Data, MetricFetchE]¶

Fetch the data for a trial.

See class docstring for overview.

Returns:

The data returned is effect by the total amount of data available by this trial (e.g this trial and earlier trials).

available data < minimum_data-points: mean and sem are NaN.
available data >= minimum_data-points: mean is the mean QoIEstimate, sem is the standard error of the measure deviation. The standard error corresponds to the standard deviation of the distribution as each sample is a prediction of the measure (the QoI).