Re: Sphinx parallel build errors

[Markus Heiser <markus.heiser@darmarit.de>]

Am 28.10.19 um 20:07 schrieb Kees Cook:
> On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote:
>> Em Sun, 27 Oct 2019 11:30:22 -0700
>> Randy Dunlap <rdunlap@infradead.org> escreveu:
>>
>>> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
>>>> On Sun, 27 Oct 2019 11:11:57 -0700
>>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>    
>>>>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>>>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>>>>
>>>>>> I can see anywhere from 1 to 4 sphinx-build processes running.
>>>>>
>>>>> I continue to have oom-kills when building docs.
>>
>> My 2 cents here.
>>
>> I would keep the default to use more CPUs, as nowadays I suspect that
>> most devs have a way more than 8GB installed. The worse machine I use
>> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a
>> personal laptop I bought 5 years ago).
>>
>> Yet, I agree that doc build should be saner on low-cap machines.
>>

In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
(nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
about 5GB (*more CPUs need more mem*).

@Randy you said the build machine hase 8GB .. and how much CPUs?

>> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
>> detect OOM possible conditions and reduce the number of used CPUs
>> it the max available memory is too low.
> 
> How can we determine that dynamically, though? e.g. how do we determine
> how much is needed per thread in a way that will move with the future
> needs of Sphinx, etc?

d'accord

>> As there's now a script at Kernel, maybe the script could be modified
>> to detect if the max available memory is too low, passing -j1 if,
>> let's say, the machine has less than (let's say) 16 GB RAM.
> 
> Randy, are you able to just use -j2 or similar with your builds to avoid
> this?
> 

Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string,
nothing special .. maybe we have a mem leak in one of our extensions?

@Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything
strange?

Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. 
needs investigation.

@Randy: my first questions are:

- cpu / mem?
- are OOM-kills when building target htmldocs, pdfdocs or ..?
- which py & sphinx version are used? Reference installation see 'virtualenv'
   in [1]

-- Markus --


[1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install