benchmarl

benchmarl.run

hydra_experiment(cfg: DictConfig) → None

Runs an experiment loading its config from hydra.

This function is decorated as @hydra.main and is called by running

python benchmarl/run.py algorithm=mappo task=vmas/balance

Parameters:

cfg (DictConfig) – the hydra config dictionary

benchmarl.hydra_config

load_experiment_from_hydra(cfg: DictConfig, task_name: str, callbacks=()) → Experiment

Creates an Experiment from hydra config.

Parameters:

• cfg (DictConfig) – the config dictionary from hydra main

• task_name (str) – the name of the task to load

Returns:

Experiment

load_task_config_from_hydra(cfg: DictConfig, task_name: str) → TaskClass

Returns a Task from hydra config.

Parameters:

• cfg (DictConfig) – the task config dictionary from hydra

• task_name (str) – the name of the task to load

Returns:

Task

load_experiment_config_from_hydra(cfg: DictConfig) → ExperimentConfig

Returns a ExperimentConfig from hydra config.

Parameters:

cfg (DictConfig) – the experiment config dictionary from hydra

Returns:

ExperimentConfig

load_algorithm_config_from_hydra(cfg: DictConfig) → AlgorithmConfig

Returns a AlgorithmConfig from hydra config.

Parameters:

cfg (DictConfig) – the algorithm config dictionary from hydra

Returns:

AlgorithmConfig

load_model_config_from_hydra(cfg: DictConfig) → ModelConfig

Returns a ModelConfig from hydra config.

Parameters:

cfg (DictConfig) – the model config dictionary from hydra

Returns:

ModelConfig

reload_experiment_from_file(restore_file: str) → Experiment

Reloads the experiment from a given restore file.

Requires a .hydra folder containing config.yaml, hydra.yaml, and overrides.yaml at max three directory levels higher than the checkpoint file. This should be automatically created by hydra.

Parameters:

restore_file (str) – The checkpoint file of the experiment reload.

benchmarl.eval_results

get_raw_dict_from_multirun_folder(multirun_folder: str) → Dict

Get the marl-eval input dictionary from the folder of a hydra multirun.

Examples

from benchmarl.eval_results import get_raw_dict_from_multirun_folder, Plotting
raw_dict = get_raw_dict_from_multirun_folder(
    multirun_folder="some_prefix/multirun/2023-09-22/17-21-34"
)
processed_data = Plotting.process_data(raw_dict)

Parameters:

multirun_folder (str) – the absolute path to the multirun folder

Returns:

the dict obtained by merging all the json files in the multirun

load_and_merge_json_dicts(json_input_files: List[str], json_output_file: str | None = None) → Dict

Loads and merges json dictionaries to form the marl-eval input dictionary .

Parameters:

• json_input_files (list of str) – a list containing the absolute paths to the json files

• json_output_file (str, optional) – if specified, the merged dictionary will be also written to the file in this absolute path

Returns:

the dict obtained by merging all the json files

class Plotting

Class containing static utilities for plotting in marl-eval.

Examples

>>> from benchmarl.eval_results import get_raw_dict_from_multirun_folder, Plotting
>>> raw_dict = get_raw_dict_from_multirun_folder(
... multirun_folder="some_prefix/multirun/2023-09-22/17-21-34"
... )
>>> processed_data = Plotting.process_data(raw_dict)
... (
...     environment_comparison_matrix,
...     sample_efficiency_matrix,
... ) = Plotting.create_matrices(processed_data, env_name="vmas")
>>> Plotting.performance_profile_figure(
...     environment_comparison_matrix=environment_comparison_matrix
... )
>>> Plotting.aggregate_scores(
...     environment_comparison_matrix=environment_comparison_matrix
... )
>>> Plotting.environemnt_sample_efficiency_curves(
...     sample_effeciency_matrix=sample_efficiency_matrix
... )
>>> Plotting.task_sample_efficiency_curves(
...     processed_data=processed_data, env="vmas", task="navigation"
... )
>>> plt.show()

METRICS_TO_NORMALIZE = ['return']

METRIC_TO_PLOT = 'return'

static process_data(raw_data: Dict, metrics_to_normalize: List[str] | None = ['return']) → Dict

Call data_process_pipeline to normalize the chosen metrics and to clean the data

Parameters:

• raw_data (dict) – the input data

• metrics_to_normalize (optional, List[str]) – the list of metrics to normalize

Returns:

the processed dict

static create_matrices(processed_data, env_name: str, metrics_to_normalize: List[str] | None = ['return'])

static performance_profile_figure(environment_comparison_matrix, metric_name: str | None = 'return', metrics_to_normalize: List[str] | None = ['return'], **kwargs)

static aggregate_scores(environment_comparison_matrix, metric_name: str | None = 'return', metrics_to_normalize: List[str] | None = ['return'], save_tabular_as_latex: bool = True, **kwargs)

static probability_of_improvement(environment_comparison_matrix, algorithms_to_compare: List[List[str]], metric_name: str | None = 'return', metrics_to_normalize: List[str] | None = ['return'], **kwargs)

static environemnt_sample_efficiency_curves(sample_effeciency_matrix, metric_name: str | None = 'return', metrics_to_normalize: List[str] | None = ['return'], **kwargs)

static task_sample_efficiency_curves(processed_data, task, env, metric_name: str | None = 'return', metrics_to_normalize: List[str] | None = ['return'], **kwargs)