For future readers, the solution for this problem is to pin docutils<0.18 in your requirements.txt file, and have a .readthedocs.yaml file with these contents:

version: 2

python:
   install:
   - requirements: docs/requirements.txt

More information in our docs. See also our page on reproducible builds to better understand how to use requirements files on Read the Docs.

Please update the examples in the RTD documentation (https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html).

We now know that e.g.

✅ Good: The specified versions will be used for all your builds, in all platforms, and won’t be updated unexpectedly.

File: docs/requirements.txt

sphinx==3.4.3 sphinx_rtd_theme==0.5.1 readthedocs-sphinx-search==0.1.0rc3

is actually not good (as it pins a legacy Sphinx version but not the related Docutils).

With the requirements relaxed to

sphinx<4.0
sphinx_rtd_theme==0.5.1

users will get the backported fix (Docutils pin added in 3.5). A a pin on not-fixed legacy Sphinx versions must be amended with a Docutils pin like, e.g.

sphinx==3.4.3
docutils < 0.17
sphinx_rtd_theme==0.5.1
readthedocs-sphinx-search==0.1.0rc3

Quick addendum:

• If your project pins the version of sphinx using extras_require keyword in your setup.py, and
• In RTD you’re installing your project via the python keyword of your .readthedocs.yaml config file

Then you can also fix the problem by making sure you set extra_requirements in your RTD config file.

I’ve seen this trip up projects where folks were already pinning Sphinx, only to find out that RTD wasn’t picking up those extra requirements.

Great summary originally posted by @joshuacwnewton in https://github.com/sphinx-doc/sphinx/issues/9783#issuecomment-952950115:

We’ve run into this issue too for several projects. Here’s a summary of the problem:

• The issue is due to the recent release of docutils-0.18 (which is incompatible with Sphinx at the moment, see https://github.com/sphinx-doc/sphinx/issues/9777).
• You can avoid auto-installing docutils-0.18 if your Sphinx installation is up-to-date: https://github.com/sphinx-doc/sphinx/blob/2be9d6b092965a2f9354da66b645bf5ea76ce288/setup.py#L26
• However, if you’re building on RTD, and your project was created prior to October 2020, and you haven’t explicitly specified a version of Sphinx to use, RTD will default to sphinx<2.
• So, sphinx<2 + docutils-0.18 will be installed, causing the error.

The solution we settled on was to make sure we explicitly specified a more recent version of Sphinx in our RTD requirements. Please refer to RTD’s “Reproducible Builds” page (and specifically the pinning dependencies section) for more information on RTD’s recommendations here.

Update: Sphinx 1.8.6 was released this week, and it pins docutils to <0.18 (thanks @tk0miya ❤️)

However, users of old pip versions (for example on Python 2) might not benefit from this, and will need to pin docutils themselves anyway. Otherwise, they will still observe the error, plus this warning on the logs:

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
sphinx 1.8.6 requires docutils<0.18,>=0.11, but you'll have docutils 0.18 which is incompatible.

Well, there are some problems with trying to guess the version of docutils that users need. We actually used to install a pinned version of docutils as a pre-install step, but every dependency that we pin leads to confusion for users. Because of this, and all the confusing UX resulting from this, we’ve been moving away from pinning dependencies for users, and instead expect users to tell us what to install. At least this way you can easily replicate errors locally by just installing your dependencies. With Read the Docs magic helping in the middle, this isn’t possible and it gets even more confusing.

The problem is that if we pin docutils==0.16 as a pre install step, that now satisfies the dependency from Sphinx 4 for docutils<0.18 and docutils is never upgraded. Very few projects would ever know what docutils does, so this particular dependency is not likely to be pinned in a projects dependencies. Additionally, we aren’t doing any inspection of which version of Sphinx is installed, so would need to pre-install a docutils version for all projects.

The best solution here is probably for Sphinx to backport release 1.x and 2.x releases, neither of which pin docutils at the moment. Sphinx 3.x does pin docutils>=0.12,<0.17 at least, as does Sphinx 4.x. Alternatively, there needs to be a better error message emitted from docutils so users are left absolutely lost all of a sudden.

Hi @anitagraser , I see in the logs that the setup.py install step is effectively bringing docutils 0.17, and still the error appears. I still don’t know what could have happened, feel free to open a new issue to analyze this case in depth.

The version of docutils is not pinned and a new version was released.

The build that passed installed docutils-0.17.1-py2.py3-none-any.whl and the one that failed, installed docutils-0.18-py2.py3-none-any.whl