Re: docs: requirements.txt has stopped working again

[Jani Nikula <jani.nikula@linux.intel.com>]

On Tue, 23 Jan 2024, Akira Yokosawa <akiyks@gmail.com> wrote:
> Hi all,
>
> Annoyingly, installing Sphinx 2.4.4 using requirements.txt of v6.8-rc1
> ends up in a run-time error which looks similar to the one Vegard
> reported in commit f4cac0f74658 ("Documentation: constrain alabaster
> package to older versions").
>
> The new error is from sphinxcontrib.applehelp which reads:
>
>     Sphinx version error:
>     The sphinxcontrib.applehelp extension used by this project needs
>     at least Sphinx v5.0; it therefore cannot be built with this version.
>
> Do we want to continue whack-a-mole update for Sphinx 2.4.4?

I looked into the root cause here. Sphinx since early versions
(predating even 2.4.4) depends on packages such as

- sphinxcontrib-applehelp
- sphinxcontrib-devhelp
- sphinxcontrib-htmlhelp
- sphinxcontrib-qthelp
- sphinxcontrib-serializinghtml

but does so without any version constraints. Recently, all of those
*packages* started first depending on Sphinx >= 5. Apparently this
prevented installation of incompatible combos, but resulted in circular
dependency issues in pip.

To fix the circular dependencies, all of those *packages* made the
dependency on Sphinx >= 5 optional, but made the *extensions* in them
app.require_sphinx('5.0') runtime. This happened in the past two weeks.

What this means is that installing any Sphinx < 5 succeeds, it'll pull
in the latest versions of the above mentioned packages because they meet
the package dependencies, but all of them will fail runtime for any
Sphinx version before 5.0. Even if we don't actually use or need any of
those extensions.

I ran git blame and git tag --contains on them to figure out the
packages compatible with Sphinx < 5. These should do it:

diff --git a/Documentation/sphinx/requirements.txt b/Documentation/sphinx/requirements.txt
index 5d47ed443949..0aa4fdb84632 100644
--- a/Documentation/sphinx/requirements.txt
+++ b/Documentation/sphinx/requirements.txt
@@ -2,5 +2,10 @@
 jinja2<3.1
 # alabaster>=0.7.14 is not compatible with Sphinx<=3.3
 alabaster<0.7.14
+sphinxcontrib-applehelp==1.0.4
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-htmlhelp==2.0.1
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.5
 Sphinx==2.4.4
 pyyaml

Conclusions:

- Distro packages probably work, because they maintain deps separately.

- Installing latest from pip probably works. There may be occasional
  hiccups, but they'll likely get noticed and fixed quickly.

- Installing known good requirements from pip freeze works.

- Installing some random version from pip is likely to fail.

- Installing a mix of distro and pip packages is likely to fail.

So maybe we'll need a minimum Sphinx version for distro packages, and
three separate requirements. One with just the top level deps (sphinx
and pyyaml, no versions specified), one which is basically a pip freeze
of that with know good config, updated regularly, and then a pip freeze
of the conservative (2.4.4?) versions, rarely updated.


BR,
Jani.


-- 
Jani Nikula, Intel