axtreme.experiment

Helper functions for ax Experiments.

Functions

add_json_data_to_experiment(experiment, ...)	Adds the data from a dictionary to an ax Experiment.
add_metric_data_to_experiment(experiment, ...)	Add metric data to an experiment.
add_simulation_data_to_experiment(...)	Add raw simulator data to an experiment.
add_sobol_points_to_experiment(experiment[, ...])	Adds some points (chosen by sobol) to an experiment.
extract_data_from_experiment_as_json(experiment)	Extracts the data from an ax Experiment.
make_experiment(simulator, search_space, dist)	Returns an experiment according to the given simulator, search_space, and dist_class.

axtreme.experiment.add_json_data_to_experiment(experiment: Experiment, json_data: dict[int, dict[str, dict[str, dict[str, float | dict[str, float]]]]]) → None

Adds the data from a dictionary to an ax Experiment.

Parameters:

• experiment – The ax Experiment to add the data to.

• json_data – The data to add to the ax Experiment. The structure should be the same as the output of extract_data_from_experiment.

Example

{
    "trial_index": {
        "arm_name": {
            "parameters": {
                "parameter_name": "parameter_value"
            },
            "metrics": {
                "metric_name": {
                    "mean": 0.0,
                    "sem": 0.0
                }
            }
        }
    }
}

axtreme.experiment.add_metric_data_to_experiment(experiment: Experiment, parameterizations: Iterable[Dict[str, None | str | bool | float | int]], metric_data: Iterable[Mapping[str, float | tuple[float, float | None] | dict[str, float | None]]]) → tuple[Data, int]

Add metric data to an experiment.

This function is used to add data from previously run simulations to an experiment. To use this function one needs to have an estimate of the mean and standard error for each metric. This is useful if the simulator is slow or expensive to run and the data is already available.

Parameters:

• experiment – The ax Experiment to add the data to.

• parameterizations –

  The parameterizations used in the simulation.

  • Iterable where each element is a dict parameter names and values ({[param_name]: value}).

• metric_data –

  The metric data from the simulation.

  • Iterable where each element is a dict of metric names and values.

  • The values can be a float, a tuple of (mean, sem), or a dict of {“mean”: mean, “sem”: sem}.

  • If a float is provided, the sem is assumed to be 0.

Returns:

• The data that was attached to the experiment.

• The trial index of the trial that was created.

axtreme.experiment.add_simulation_data_to_experiment(experiment: Experiment, parameters: list[str], simulation_inputs: Iterable[ndarray[Any, dtype[float64]]], simulation_outputs: Iterable[ndarray[Any, dtype[float64]]]) → tuple[Data, int]

Add raw simulator data to an experiment.

This function is used to add data from previously run simulations to an experiment. This is very useful if the simulator is slow or expensive to run and the data is already available. Using this function allows you to add the data to the experiment and then use it to train the model which can be used to collect new data by running new simulations.

Parameters:

• experiment – The ax Experiment to add the data to.

• parameters –

  The names of the parameters used as input to the simulator.

  • This has to match the names of the parameters in the search space of the experiment.

  • This should match the order of the input dimensions in the simulation_inputs.

• simulation_inputs –

  The inputs to the simulator.

  • Iterable where each element is the parameters used in the simulation.

  • Each element should be a numpy array with shape (n_input_dims,)

• simulation_outputs –

  The outputs from the simulator.

  • Iterable where each element is multiple output from the simulation ran with the same input.

  • Each element should be a numpy array with shape (n_simulations_per_point,)

  • Only supports a single output dimension for now.

Returns:

• The data that was attached to the experiment.

• The trial index of the trial that was created.

axtreme.experiment.add_sobol_points_to_experiment(experiment: Experiment, n_iter: int = 5, seed: int | None = None) → None

Adds some points (chosen by sobol) to an experiment.

Typically used to initialise the experiment so a GP has data for training.

axtreme.experiment.extract_data_from_experiment_as_json(experiment: Experiment) → dict[int, dict[str, dict[str, dict[str, float | dict[str, float]]]]]

Extracts the data from an ax Experiment.

Parameters:

experiment – The Ax experiment to extract the data from.

Returns:

{
    "trial_index": {
        "arm_name": {
            "parameters": {
                "parameter_name": "parameter_value"
            },
            "metrics": {
                "metric_name": {
                    "mean": 0.0,
                    "sem": 0.0
                }
            }
        }
    }
}

Return type:

A dictionary with the following structure

axtreme.experiment.make_experiment(simulator: Simulator, search_space: SearchSpace, dist: rv_continuous, n_simulations_per_point: int = 100) → Experiment

Returns an experiment according to the given simulator, search_space, and dist_class.

Parameters:

• simulator – The simulator to use for the experiment.

• search_space – The ax search space to use for the experiment.

• dist – The distribution that the result of a simulation is assumed to follow.

• n_simulations_per_point – The number of simulations to run for each point in the experiment.

Returns:

An ax Experiment.