catkin: ImportError: No module named catkin.environment_cache

We’re getting an intermittent issue affecting our CI builds. It pops up at random on different packages, but always takes this form:

00:12:49.053   File "/tmp/buildd/workspace/debian/tmp/build/<pkg>/catkin_generated/generate_cached_setup.py", line 20, in <module>
00:12:49.053     from catkin.environment_cache import generate_environment_script
00:12:49.053 ImportError: No module named catkin.environment_cache

This is running inside a cowbuilder, on vmware, with no additional containerization or isolation. The overall build is a large parallel affair being managed by catkin_tools. I’ve reported this there too (https://github.com/catkin/catkin_tools/issues/378), but @jbohren suggests (and I’m inclined to agree) that this may only be fixable within catkin itself.

Looking at the templated logic here, I’m wondering if we’re looking at a race condition with the copy-on-write filesystem?

It’s not clear to me where an additional sync call is required, or even how to reliably reproduce this problem, but I’d be delighted to accept counsel on either of those points. In the meantime, would we consider a workaround which, say, pauses for 100ms and then retries the whole import/try-block in the event of the second import failing?