robomimic: Cannot reproduce tool_hang results on image observations

Hi,

I use python robomimic/scripts/generate_paper_configs.py --output_dir /tmp/experiment_results to generate the config for reproducing the results in the paper, and then use python robomimic/scripts/train.py --config robomimic/exps/paper/core/tool_hang/ph/image/bc_rnn.json to run the training process of the tool_hang task on image observations.

However, The best success rate I can get is 24% (at 580th epoch) in this 600 epochs training process. It is far lower than the 67.3 ± 4.1 reported in the robomimic study paper (https://arxiv.org/pdf/2108.03298) in Table 3.

I believe I have strictly followed the instructions in the robomimic document to reproduce the results in the paper. Why the success rate different is so large? Any people met the same issue?

Thanks a lot!

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

I apologize - I just realized that the offline_study branch of robosuite is likely still using the mujoco-py backend. You will likely need to install mujoco-py. For the error you mentioned, you should make a directory (.mujoco) in your home directory, and add the unzipped mujoco library there

When you downgrade robosuite, you shouldn’t need to re-install it. I would make sure mujoco-py is uninstalled, and I’d stick with mujoco==2.3.2 if possible.

yes

For robosuite just checkout the offline_study branch. For robomimic you could checkout v0.2 (https://github.com/ARISE-Initiative/robomimic/releases/tag/v0.2.0)

Thanks for sharing!

You shouldn’t be able to download the image Tool Hang dataset for v1.4.1 of robosuite (see this PR). When I run your first command I get “Skipping tool_hang-ph-image, no url for dataset exists. Create this dataset locally by running the appropriate command from robomimic/scripts/extract_obs_from_raw_datasets.sh.”

I would recommend two different things to try:

  1. Try downloading the raw hdf5 and then postprocessing to get the image hdf5 (and you can follow up with the commands used here if you want confirmation).

  2. Try downgrading robosuite to the offline_study branch, and then use the Tool Hang image dataset from robomimic v0.2: https://robomimic.github.io/docs/v0.2/datasets/robomimic_v0.1.html

It’d help to provide some more details. For example, how did you download / postprocess the Tool Hang dataset? What version of robosuite are you on (branch / commit / version)?