benchmarl.algorithms.Iddpg

class Iddpg(share_param_critic: bool, loss_function: str, delay_value: bool, use_tanh_mapping: bool, **kwargs)[source]

Bases: Algorithm

Same as Maddpg (from https://arxiv.org/abs/1706.02275) but with decentralized critics.

Parameters:

  • share_param_critic (bool) – Whether to share the parameters of the critics withing agent groups

  • loss_function (str) – loss function for the value discrepancy. Can be one of “l1”, “l2” or “smooth_l1”.

  • delay_value (bool) – whether to separate the target value networks from the value networks used for data collection.

  • use_tanh_mapping (bool) – if True, use squash actions (output by the policy) into the action range, otherwise clip them.

_get_loss(group: str, policy_for_loss: TensorDictModule, continuous: bool) → Tuple[LossModule, bool][source]

Implement this function to return the LossModule for a specific group.

Parameters:

  • group (str) – agent group of the loss

  • policy_for_loss (TensorDictModule) – the policy to use in the loss

  • continuous (bool) – whether to return a loss for continuous or discrete actions

Returns: LossModule and a bool representing if the loss should have target parameters

_get_parameters(group: str, loss: LossModule) → Dict[str, Iterable][source]

Get the dictionary mapping loss names to the relative parameters to optimize for a given group loss.

Returns: a dictionary mapping loss names to a parameters’ list

_get_policy_for_loss(group: str, model_config: ModelConfig, continuous: bool) → TensorDictModule[source]

Get the non-explorative policy for a specific group.

Parameters:

  • group (str) – agent group of the policy

  • model_config (ModelConfig) – model config class

  • continuous (bool) – whether the policy should be continuous or discrete

Returns: TensorDictModule representing the policy

_get_policy_for_collection(policy_for_loss: TensorDictModule, group: str, continuous: bool) → TensorDictModule[source]

Implement this function to add an explorative layer to the policy used in the loss.

Parameters:

  • policy_for_loss (TensorDictModule) – the group policy used in the loss

  • group (str) – agent group

  • continuous (bool) – whether the policy is continuous or discrete

Returns: TensorDictModule representing the explorative policy

process_batch(group: str, batch: TensorDictBase) → TensorDictBase[source]

This function can be used to reshape data coming from collection before it is passed to the policy.

Parameters:

  • group (str) – agent group

  • batch (TensorDictBase) – the batch of data coming from the collector

Returns: the processed batch

get_value_module(group: str) → TensorDictModule[source]
_abc_impl = <_abc._abc_data object>