ray: [RLlib] Bugs in PyTorch version of DDPG

@sven1977 would it be possible to keep the issue open until #9683 is merged ?

@sven1977 @raphaelavalos Other bug: Pytorch state_dict method is used to get a dict of model parameters, which is wrong. state_dict returns something way more complex than that, including attributes. named_parameters should be used instead. Moreover, it is sometimes returning a dict instead of an OrderedDict as it should be. The issue occurs here for DDPG. It should be instead:

    def policy_variables(self, as_dict=False):
        """Return the list of variables for the policy net."""
        if as_dict:
            return OrderedDict(self.policy_model.named_parameters())
        return list(self.policy_model.parameters())

    def q_variables(self, as_dict=False):
        """Return the list of variables for Q / twin Q nets."""
        if as_dict:
            return OrderedDict((
                *self.q_model.named_parameters(),
                *(self.twin_q_model.named_parameters() if self.twin_q_model else [])
            ))
        return [*self.q_model.parameters(),
                *(self.twin_q_model.parameters() if self.twin_q_model else [])]

@duburcqa @raphaelavalos Btw, we are very close to setting up daily automatic GPU + “heavy” regression tests (Atari, MuJoCo) to catch these things much earlier than we do right now. Hopefully, this will eliminate issues like the GPU one here altogether.

On the GPU: Yes, we are no longer checking, whether we have a GPU, but whether the config actually says: num_gpus > 0 and only then place the model on the GPU.

The target model is placed on the gpu even if ray was not configure to use the gpu. This should be fixed in the current master.

I would add other issue: clamp is used to clamp the action at several places here and here, while min/max combinaison should be used instead. Indeed, clamp does not support element-wise operation and this limitation is currently circumvent by clamping with respect to the first low/high bounds of the action space, which does not make any sense.