benchmarl.benchmark

Benchmark

class Benchmark(algorithm_configs: Sequence[AlgorithmConfig], model_config: ModelConfig, tasks: Sequence[Task], seeds: Set[int], experiment_config: ExperimentConfig, critic_model_config: ModelConfig | None = None)[source]

A benchmark.

Benchmarks are collections of experiments to compare.

Parameters:

  • algorithm_configs (list of AlgorithmConfig) – the algorithms to benchmark

  • model_config (ModelConfig) – the config of the policy model

  • tasks (list of Task) – the tasks to benchmark

  • seeds (set of int) – the seeds for the benchmark

  • experiment_config (ExperimentConfig) – the experiment config

  • critic_model_config (ModelConfig, optional) – the config of the critic model. Defaults to model_config

property n_experiments

The number of experiments in the benchmark.

get_experiments() → Iterator[Experiment][source]

Yields one experiment at a time

run_sequential()[source]

Run all the experiments in the benchmark in a sequence.