robomimic: Cannot reproduce tool_hang results on image observations
Hi,
I use
python robomimic/scripts/generate_paper_configs.py --output_dir /tmp/experiment_results
to generate the config for reproducing the results in the paper, and then use
python robomimic/scripts/train.py --config robomimic/exps/paper/core/tool_hang/ph/image/bc_rnn.json
to run the training process of the tool_hang task on image observations.
However, The best success rate I can get is 24% (at 580th epoch) in this 600 epochs training process. It is far lower than the 67.3 ± 4.1 reported in the robomimic study paper (https://arxiv.org/pdf/2108.03298) in Table 3.
I believe I have strictly followed the instructions in the robomimic document to reproduce the results in the paper. Why the success rate different is so large? Any people met the same issue?
Thanks a lot!
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (6 by maintainers)
I apologize - I just realized that the
offline_studybranch of robosuite is likely still using the mujoco-py backend. You will likely need to install mujoco-py. For the error you mentioned, you should make a directory (.mujoco) in your home directory, and add the unzipped mujoco library thereWhen you downgrade robosuite, you shouldn’t need to re-install it. I would make sure mujoco-py is uninstalled, and I’d stick with mujoco==2.3.2 if possible.
yes
For robosuite just checkout the
offline_studybranch. For robomimic you could checkout v0.2 (https://github.com/ARISE-Initiative/robomimic/releases/tag/v0.2.0)Thanks for sharing!
You shouldn’t be able to download the image Tool Hang dataset for v1.4.1 of robosuite (see this PR). When I run your first command I get “Skipping tool_hang-ph-image, no url for dataset exists. Create this dataset locally by running the appropriate command from robomimic/scripts/extract_obs_from_raw_datasets.sh.”
I would recommend two different things to try:
Try downloading the raw hdf5 and then postprocessing to get the image hdf5 (and you can follow up with the commands used here if you want confirmation).
Try downgrading robosuite to the offline_study branch, and then use the Tool Hang image dataset from robomimic v0.2: https://robomimic.github.io/docs/v0.2/datasets/robomimic_v0.1.html
It’d help to provide some more details. For example, how did you download / postprocess the Tool Hang dataset? What version of robosuite are you on (branch / commit / version)?