Running

Experiments are launched with a default configuration that can be overridden in many ways. To learn how to customize and override configurations please refer to the Configuring section.

To see a list of all available algorithms, tasks, and models, pleas refer to the Components section.

Command line

To launch an experiment from the command line you can do

python benchmarl/run.py algorithm=mappo task=vmas/balance

Example

Thanks to hydra, you can run benchmarks as multi-runs like:

python benchmarl/run.py -m algorithm=mappo,qmix,masac task=vmas/balance,vmas/sampling seed=0,1

Example

The default implementation for hydra multi-runs is sequential, but parallel and slurm launchers are also available.

Script

You can also load and launch your Experiment from within a script

from benchmarl.algorithms import MappoConfig
from benchmarl.environments import VmasTask
from benchmarl.experiment import Experiment, ExperimentConfig
from benchmarl.models.mlp import MlpConfig

experiment = Experiment(
   task=VmasTask.BALANCE.get_from_yaml(),
   algorithm_config=MappoConfig.get_from_yaml(),
   model_config=MlpConfig.get_from_yaml(),
   critic_model_config=MlpConfig.get_from_yaml(),
   seed=0,
   config=ExperimentConfig.get_from_yaml(),
)
experiment.run()

Example

You can also run multiple experiments in a Benchmark.

from benchmarl.algorithms import MappoConfig, MasacConfig, QmixConfig
from benchmarl.benchmark import Benchmark
from benchmarl.environments import VmasTask
from benchmarl.experiment import ExperimentConfig
from benchmarl.models.mlp import MlpConfig

benchmark = Benchmark(
    algorithm_configs=[
        MappoConfig.get_from_yaml(),
        QmixConfig.get_from_yaml(),
        MasacConfig.get_from_yaml(),
    ],
    tasks=[
        VmasTask.BALANCE.get_from_yaml(),
        VmasTask.SAMPLING.get_from_yaml(),
    ],
    seeds={0, 1},
    experiment_config=ExperimentConfig.get_from_yaml(),
    model_config=MlpConfig.get_from_yaml(),
    critic_model_config=MlpConfig.get_from_yaml(),
)
benchmark.run_sequential()

Example