ray: [Bug] ValueError: ('Observation ({}) outside given space ({})!'

Search before asking

  • I searched the issues and found no similar issues.

Ray Component

Ray Tune, Ray Train, RLlib

Issue Severity

High: It blocks me to complete my task.

What happened + What you expected to happen

rrlib’s preprocessor.py line 62 gets stuck in the value error ValueError: ('Observation ({}) outside given space ({})!. But the observation is not outside the given space. Observation

ValueError: ('Observation ({}) outside given space ({})!', array([0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.68      , 0.71333333, 0.74666667, 0.78      , 0.81333333,
       0.84666667, 0.88      , 0.91333333, 0.94666667, 0.98      ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.68      , 0.71333333, 0.74666667, 0.78      , 0.81333333,
       0.84666667, 0.88      , 0.91333333, 0.94666667, 0.98      ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.68      , 0.71333333, 0.74666667, 0.78      , 0.81333333,
       0.84666667, 0.88      , 0.91333333, 0.94666667, 0.98      ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.68      , 0.71333333, 0.74666667, 0.78      , 0.81333333,
       0.84666667, 0.88      , 0.91333333, 0.94666667, 0.98      ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.27272727, 0.27272727, 0.27272727, 0.27272727, 0.27272727,
       0.27272727, 0.27272727, 0.27272727, 0.27272727, 0.27272727,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.18181818, 0.18181818, 0.18181818, 0.18181818, 0.18181818,
       0.18181818, 0.18181818, 0.18181818, 0.18181818, 0.18181818,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.54545455, 0.54545455, 0.54545455, 0.54545455, 0.54545455,
       0.54545455, 0.54545455, 0.54545455, 0.54545455, 0.54545455,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364,
       0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       2.        , 1.        , 2.        , 1.        , 1.        ,
       1.        , 1.        , 1.        , 2.        , 1.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       1.        , 1.        , 1.        , 1.        , 1.        ,
       1.        , 1.        , 2.        , 1.        , 2.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       1.        , 2.        , 2.        , 2.        , 1.        ,
       1.        , 1.        , 1.        , 1.        , 1.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       2.        , 1.        , 2.        , 1.        , 1.        ,
       1.        , 1.        , 1.        , 1.        , 1.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.        , 0.        ,
       0.        , 0.        , 0.        ]), Box([0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], [3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.
 3. 3. 3. 3. 3. 3. 3. 3. 3. 3. 3.], (323,), float32))

Space Box(low=0., high=3, shape = 323, dtype=np.float32)

Versions / Dependencies

Ubuntu 18.04 Ray:1.3.0 python:3.8.5

Reproduction script

def check_shape(self, observation: Any) -> None:                                       
       """Checks the shape of the given observation."""
       if self._i % OBS_VALIDATION_INTERVAL == 0:
           if type(observation) is list and isinstance(
                   self._obs_space, gym.spaces.Box):
               observation = np.array(observation)
           try:
               if not self._obs_space.contains(observation):
                   raise ValueError(
                       "Observation ({}) outside given space ({})!",
                       observation, self._obs_space)
           except AttributeError:
               raise ValueError(
                   "Observation for a Box/MultiBinary/MultiDiscrete space "
                   "should be an np.array, not a Python list.", observation)
       self._i += 1

Anything else

I find the similar problem in #12940, but my observation only has a float32 dtype, which means it doesn’t a a float32 vs float64 problem.

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

Hey @brodermind , could you post a small, fully self-sufficient repro script? We seem to be unable to reproduce this.

Either way, we should probably invest some time into providing a much more specific error message in our env-checker when checking these large, complex dict/tuple spaces. As a user, I would like to know, which exact sub-key was failing and why (shape, dtype, or data range wrong?).

@sven1977 Hi, thank you for your reply, I have fixed my problem by degrading the ray version.

just tried with ray==1.12 and can not reproduce – I get the same NotImplementedError as you got above. Adding a very simple step-method the trainer seems to instantiate just fine – so I guess as far as I’m concerned this has been fixed in 1.12.

In my older report linked above as a duplicate (#22429) I did include a self-contained reproduction script – I can’t test against latest master right now though.

Hey @smita-09 , one of your values in your given observation (value=1.2) is definitely outside the range of your observation space, which has ranges from 0.0 to 1.0, only. This would be one reason, why you see this error. Try increasing your observation space’s range or fix the observation outputs from your environment to fall into the [0.0, 1.0] range.

I tried to reproduce this with Python 3.8 on master and with this script. Can someone confirm that this should produce the issue if it where present on master?