Sphinx parallel build errors

Sphinx parallel build errors

[Randy Dunlap]

Sphinx parallel build error:
OSError: [Errno 12] Cannot allocate memory

I have had this error 4 times in the last 3 days.

The first time it oom-killed Thunderbird.  No big deal.
The second time it oom-killed gimp, which had 24 photos open.
Still no big deal.  Just a nuisance.

My little laptop has only 8 GB of RAM.

Is there a way that I can limit the amount of parallelism?

thanks.
-- 
~Randy

Re: Sphinx parallel build errors

[Jonathan Corbet]

On Wed, 9 Oct 2019 14:37:51 -0700
Randy Dunlap <rdunlap@infradead.org> wrote:

> Sphinx parallel build error:
> OSError: [Errno 12] Cannot allocate memory
> 
> I have had this error 4 times in the last 3 days.
> 
> The first time it oom-killed Thunderbird.  No big deal.
> The second time it oom-killed gimp, which had 24 photos open.
> Still no big deal.  Just a nuisance.
> 
> My little laptop has only 8 GB of RAM.
> 
> Is there a way that I can limit the amount of parallelism?

Is this a change with current docs-next, or have you seen this for a
while?  The parallelism logic changed a bit in docs-next, but the default
behavior should be the same as it was before.

"make -j 1 htmldocs" (in current docs-next) should give you a nice, slow,
serial build.

jon

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/9/19 2:40 PM, Jonathan Corbet wrote:
> On Wed, 9 Oct 2019 14:37:51 -0700
> Randy Dunlap <rdunlap@infradead.org> wrote:
> 
>> Sphinx parallel build error:
>> OSError: [Errno 12] Cannot allocate memory
>>
>> I have had this error 4 times in the last 3 days.
>>
>> The first time it oom-killed Thunderbird.  No big deal.
>> The second time it oom-killed gimp, which had 24 photos open.
>> Still no big deal.  Just a nuisance.
>>
>> My little laptop has only 8 GB of RAM.
>>
>> Is there a way that I can limit the amount of parallelism?
> 
> Is this a change with current docs-next, or have you seen this for a
> while?  The parallelism logic changed a bit in docs-next, but the default
> behavior should be the same as it was before.

I'm not sure.  Today's 3 instances were all in linux-next but I don't know
about the oom failure a few days ago.

> "make -j 1 htmldocs" (in current docs-next) should give you a nice, slow,
> serial build.

I don't want to limit is _that much_.  :(

-- 
~Randy

Re: Sphinx parallel build errors

[Kees Cook]

On Wed, Oct 09, 2019 at 02:37:51PM -0700, Randy Dunlap wrote:
> 
> Sphinx parallel build error:
> OSError: [Errno 12] Cannot allocate memory
> 
> I have had this error 4 times in the last 3 days.
> 
> The first time it oom-killed Thunderbird.  No big deal.
> The second time it oom-killed gimp, which had 24 photos open.
> Still no big deal.  Just a nuisance.

Eek, sorry! OOM-killer once again chooses ... not the culprit. ;)

> My little laptop has only 8 GB of RAM.
> 
> Is there a way that I can limit the amount of parallelism?

As mentioned, "make -jN htmldocs" should be respected now (before it was
as many CPUs as it had). I'm wondering if it's actually the addition of
the MAINTAINERS file parsing -- that's a really big parse and maybe that
pushed things over the edge?

-- 
Kees Cook

Re: Sphinx parallel build errors

[Bhaskar Chowdhury]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On 19:30 Wed 09 Oct 2019, Kees Cook wrote:
>On Wed, Oct 09, 2019 at 02:37:51PM -0700, Randy Dunlap wrote:
>>
>> Sphinx parallel build error:
>> OSError: [Errno 12] Cannot allocate memory
>>
>> I have had this error 4 times in the last 3 days.
>>
>> The first time it oom-killed Thunderbird.  No big deal.
>> The second time it oom-killed gimp, which had 24 photos open.
>> Still no big deal.  Just a nuisance.
>
>Eek, sorry! OOM-killer once again chooses ... not the culprit. ;)
>
>> My little laptop has only 8 GB of RAM.
>>
>> Is there a way that I can limit the amount of parallelism?
>
>As mentioned, "make -jN htmldocs" should be respected now (before it was
>as many CPUs as it had). I'm wondering if it's actually the addition of
>the MAINTAINERS file parsing -- that's a really big parse and maybe that
>pushed things over the edge?
>
>-- 
>Kees Cook

Seriously, looking/parsing MAINTAINERS file becomes serious pain in the
arse. It big and very easy to get lost in the way.
 
 Okay, complaining for the sake of complaining doesn't make things go
 far ...but until we find something more efficient ,alas! We have deal
 with it...that kinda sucks.

-Bhaskar

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Sphinx parallel build errors

[Jonathan Corbet]

On Wed, 9 Oct 2019 19:30:56 -0700
Kees Cook <keescook@chromium.org> wrote:

> I'm wondering if it's actually the addition of
> the MAINTAINERS file parsing -- that's a really big parse and maybe that
> pushed things over the edge?

That does seem like a likely culprit, doesn't it?  Watching a build here,
though, I don't see that any one of the sphinx-build processes balloons to
a horrific size, which is what I would expect if one file were causing the
problem.

In general, the sphinx build doesn't really begin to approach the memory
requirements of, say, firefox or my mail client on my system.

Randy, I'd be curious to know if you see the same behavior with current
mainline.  Also, can you see how many sphinx-build processes you have
running?

Thanks,

jon

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/10/19 11:48 AM, Jonathan Corbet wrote:
> On Wed, 9 Oct 2019 19:30:56 -0700
> Kees Cook <keescook@chromium.org> wrote:
> 
>> I'm wondering if it's actually the addition of
>> the MAINTAINERS file parsing -- that's a really big parse and maybe that
>> pushed things over the edge?
> 
> That does seem like a likely culprit, doesn't it?  Watching a build here,
> though, I don't see that any one of the sphinx-build processes balloons to
> a horrific size, which is what I would expect if one file were causing the
> problem.
> 
> In general, the sphinx build doesn't really begin to approach the memory
> requirements of, say, firefox or my mail client on my system.
> 
> Randy, I'd be curious to know if you see the same behavior with current
> mainline.  Also, can you see how many sphinx-build processes you have
> running?

Yes, I see oom-kills when building docs for 5.4-rc2.
But only after I ran 'gimp *.jpg' for 22 photos.

I can see anywhere from 1 to 4 sphinx-build processes running.

> Thanks,
> 
> jon

-- 
~Randy

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/10/19 1:15 PM, Randy Dunlap wrote:
> On 10/10/19 11:48 AM, Jonathan Corbet wrote:
>> On Wed, 9 Oct 2019 19:30:56 -0700
>> Kees Cook <keescook@chromium.org> wrote:
>>
>>> I'm wondering if it's actually the addition of
>>> the MAINTAINERS file parsing -- that's a really big parse and maybe that
>>> pushed things over the edge?
>>
>> That does seem like a likely culprit, doesn't it?  Watching a build here,
>> though, I don't see that any one of the sphinx-build processes balloons to
>> a horrific size, which is what I would expect if one file were causing the
>> problem.
>>
>> In general, the sphinx build doesn't really begin to approach the memory
>> requirements of, say, firefox or my mail client on my system.
>>
>> Randy, I'd be curious to know if you see the same behavior with current
>> mainline.  Also, can you see how many sphinx-build processes you have
>> running?
> 
> Yes, I see oom-kills when building docs for 5.4-rc2.
> But only after I ran 'gimp *.jpg' for 22 photos.
> 
> I can see anywhere from 1 to 4 sphinx-build processes running.

I continue to have oom-kills when building docs.
At least 2 times I also had system freezes where I had to power
cycle to reboot.

I suppose I can only hope that other people start running into this
issue.  :(

-- 
~Randy

Re: Sphinx parallel build errors

[Jonathan Corbet]

On Sun, 27 Oct 2019 11:11:57 -0700
Randy Dunlap <rdunlap@infradead.org> wrote:

> > Yes, I see oom-kills when building docs for 5.4-rc2.
> > But only after I ran 'gimp *.jpg' for 22 photos.
> > 
> > I can see anywhere from 1 to 4 sphinx-build processes running.  
> 
> I continue to have oom-kills when building docs.
> At least 2 times I also had system freezes where I had to power
> cycle to reboot.
> 
> I suppose I can only hope that other people start running into this
> issue.  :(

Are you still running gimp at the same time?

Also, are you doing a straight "make htmldocs" or something else?

Thanks,

jon

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/27/19 11:17 AM, Jonathan Corbet wrote:
> On Sun, 27 Oct 2019 11:11:57 -0700
> Randy Dunlap <rdunlap@infradead.org> wrote:
> 
>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>
>>> I can see anywhere from 1 to 4 sphinx-build processes running.  
>>
>> I continue to have oom-kills when building docs.
>> At least 2 times I also had system freezes where I had to power
>> cycle to reboot.
>>
>> I suppose I can only hope that other people start running into this
>> issue.  :(
> 
> Are you still running gimp at the same time?

no.

> Also, are you doing a straight "make htmldocs" or something else?

straight "make htmldocs".

-- 
~Randy

Re: Sphinx parallel build errors

[Mauro Carvalho Chehab]

Em Sun, 27 Oct 2019 11:30:22 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
> > On Sun, 27 Oct 2019 11:11:57 -0700
> > Randy Dunlap <rdunlap@infradead.org> wrote:
> >   
> >>> Yes, I see oom-kills when building docs for 5.4-rc2.
> >>> But only after I ran 'gimp *.jpg' for 22 photos.
> >>>
> >>> I can see anywhere from 1 to 4 sphinx-build processes running.    
> >>
> >> I continue to have oom-kills when building docs.

My 2 cents here.

I would keep the default to use more CPUs, as nowadays I suspect that
most devs have a way more than 8GB installed. The worse machine I use
here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a 
personal laptop I bought 5 years ago).

Yet, I agree that doc build should be saner on low-cap machines.

Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
detect OOM possible conditions and reduce the number of used CPUs
it the max available memory is too low.

As there's now a script at Kernel, maybe the script could be modified 
to detect if the max available memory is too low, passing -j1 if,
let's say, the machine has less than (let's say) 16 GB RAM.

> >> At least 2 times I also had system freezes where I had to power
> >> cycle to reboot.

That sounds weird. I can't imagine why you would experience system
freeze with doc build. Are you sure this is not your CPU overheating?

> >>
> >> I suppose I can only hope that other people start running into this
> >> issue.  :(  
> > 
> > Are you still running gimp at the same time?  
> 
> no.
> 
> > Also, are you doing a straight "make htmldocs" or something else?  
> 
> straight "make htmldocs".


Thanks,
Mauro

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/27/19 12:05 PM, Mauro Carvalho Chehab wrote:
> Em Sun, 27 Oct 2019 11:30:22 -0700
> Randy Dunlap <rdunlap@infradead.org> escreveu:
> 
>> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
>>> On Sun, 27 Oct 2019 11:11:57 -0700
>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>   
>>>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>>>
>>>>> I can see anywhere from 1 to 4 sphinx-build processes running.    
>>>>
>>>> I continue to have oom-kills when building docs.
> 
> My 2 cents here.
> 
> I would keep the default to use more CPUs, as nowadays I suspect that
> most devs have a way more than 8GB installed. The worse machine I use
> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a 
> personal laptop I bought 5 years ago).
> 
> Yet, I agree that doc build should be saner on low-cap machines.
> 
> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
> detect OOM possible conditions and reduce the number of used CPUs
> it the max available memory is too low.
> 
> As there's now a script at Kernel, maybe the script could be modified 
> to detect if the max available memory is too low, passing -j1 if,
> let's say, the machine has less than (let's say) 16 GB RAM.
> 
>>>> At least 2 times I also had system freezes where I had to power
>>>> cycle to reboot.
> 
> That sounds weird. I can't imagine why you would experience system
> freeze with doc build. Are you sure this is not your CPU overheating?
> 

No, I'm not sure.  I'm just sure that it only happens when I am
building kernel docs.

>>>> I suppose I can only hope that other people start running into this
>>>> issue.  :(  
>>>
>>> Are you still running gimp at the same time?  
>>
>> no.
>>
>>> Also, are you doing a straight "make htmldocs" or something else?  
>>
>> straight "make htmldocs".
> 
> 
> Thanks,
> Mauro


-- 
~Randy
Reported-by: Randy Dunlap <rdunlap@infradead.org>

Re: Sphinx parallel build errors

[Kees Cook]

On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote:
> Em Sun, 27 Oct 2019 11:30:22 -0700
> Randy Dunlap <rdunlap@infradead.org> escreveu:
> 
> > On 10/27/19 11:17 AM, Jonathan Corbet wrote:
> > > On Sun, 27 Oct 2019 11:11:57 -0700
> > > Randy Dunlap <rdunlap@infradead.org> wrote:
> > >   
> > >>> Yes, I see oom-kills when building docs for 5.4-rc2.
> > >>> But only after I ran 'gimp *.jpg' for 22 photos.
> > >>>
> > >>> I can see anywhere from 1 to 4 sphinx-build processes running.    
> > >>
> > >> I continue to have oom-kills when building docs.
> 
> My 2 cents here.
> 
> I would keep the default to use more CPUs, as nowadays I suspect that
> most devs have a way more than 8GB installed. The worse machine I use
> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a 
> personal laptop I bought 5 years ago).
> 
> Yet, I agree that doc build should be saner on low-cap machines.
> 
> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
> detect OOM possible conditions and reduce the number of used CPUs
> it the max available memory is too low.

How can we determine that dynamically, though? e.g. how do we determine
how much is needed per thread in a way that will move with the future
needs of Sphinx, etc?

> As there's now a script at Kernel, maybe the script could be modified 
> to detect if the max available memory is too low, passing -j1 if,
> let's say, the machine has less than (let's say) 16 GB RAM.

Randy, are you able to just use -j2 or similar with your builds to avoid
this?

-- 
Kees Cook

Re: Sphinx parallel build errors

[Markus Heiser]

Am 28.10.19 um 20:07 schrieb Kees Cook:
> On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote:
>> Em Sun, 27 Oct 2019 11:30:22 -0700
>> Randy Dunlap <rdunlap@infradead.org> escreveu:
>>
>>> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
>>>> On Sun, 27 Oct 2019 11:11:57 -0700
>>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>    
>>>>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>>>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>>>>
>>>>>> I can see anywhere from 1 to 4 sphinx-build processes running.
>>>>>
>>>>> I continue to have oom-kills when building docs.
>>
>> My 2 cents here.
>>
>> I would keep the default to use more CPUs, as nowadays I suspect that
>> most devs have a way more than 8GB installed. The worse machine I use
>> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a
>> personal laptop I bought 5 years ago).
>>
>> Yet, I agree that doc build should be saner on low-cap machines.
>>

In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
(nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
about 5GB (*more CPUs need more mem*).

@Randy you said the build machine hase 8GB .. and how much CPUs?

>> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
>> detect OOM possible conditions and reduce the number of used CPUs
>> it the max available memory is too low.
> 
> How can we determine that dynamically, though? e.g. how do we determine
> how much is needed per thread in a way that will move with the future
> needs of Sphinx, etc?

d'accord

>> As there's now a script at Kernel, maybe the script could be modified
>> to detect if the max available memory is too low, passing -j1 if,
>> let's say, the machine has less than (let's say) 16 GB RAM.
> 
> Randy, are you able to just use -j2 or similar with your builds to avoid
> this?
> 

Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string,
nothing special .. maybe we have a mem leak in one of our extensions?

@Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything
strange?

Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. 
needs investigation.

@Randy: my first questions are:

- cpu / mem?
- are OOM-kills when building target htmldocs, pdfdocs or ..?
- which py & sphinx version are used? Reference installation see 'virtualenv'
   in [1]

-- Markus --


[1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install

Re: Sphinx parallel build errors

[Kees Cook]

On Tue, Oct 29, 2019 at 09:16:06AM +0100, Markus Heiser wrote:
> @Kees: why did you mentioning MAINTAINERS parsing .. did you observe anything
> strange?

No, nothing strange exactly, but it was another change that happened
around the same time and is a pretty large parse. So I was wondering if
adding it might have tipped the scale.

-- 
Kees Cook

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/29/19 1:16 AM, Markus Heiser wrote:
> Am 28.10.19 um 20:07 schrieb Kees Cook:
>> On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote:
>>> Em Sun, 27 Oct 2019 11:30:22 -0700
>>> Randy Dunlap <rdunlap@infradead.org> escreveu:
>>>
>>>> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
>>>>> On Sun, 27 Oct 2019 11:11:57 -0700
>>>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>>   
>>>>>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>>>>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>>>>>
>>>>>>> I can see anywhere from 1 to 4 sphinx-build processes running.
>>>>>>
>>>>>> I continue to have oom-kills when building docs.
>>>
>>> My 2 cents here.
>>>
>>> I would keep the default to use more CPUs, as nowadays I suspect that
>>> most devs have a way more than 8GB installed. The worse machine I use
>>> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a
>>> personal laptop I bought 5 years ago).
>>>
>>> Yet, I agree that doc build should be saner on low-cap machines.
>>>
> 
> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
> about 5GB (*more CPUs need more mem*).
> 
> @Randy you said the build machine hase 8GB .. and how much CPUs?

4.

>>> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
>>> detect OOM possible conditions and reduce the number of used CPUs
>>> it the max available memory is too low.
>>
>> How can we determine that dynamically, though? e.g. how do we determine
>> how much is needed per thread in a way that will move with the future
>> needs of Sphinx, etc?
> 
> d'accord
> 
>>> As there's now a script at Kernel, maybe the script could be modified
>>> to detect if the max available memory is too low, passing -j1 if,
>>> let's say, the machine has less than (let's say) 16 GB RAM.
>>
>> Randy, are you able to just use -j2 or similar with your builds to avoid
>> this?

No, sometimes even -j1 fails with oom-killer.

> 
> Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string,
> nothing special .. maybe we have a mem leak in one of our extensions?
> 
> @Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything
> strange?
> 
> Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. needs investigation.
> 
> @Randy: my first questions are:
> 
> - cpu / mem?
> - are OOM-kills when building target htmldocs, pdfdocs or ..?

htmldocs

> - which py & sphinx version are used? Reference installation see 'virtualenv'
>   in [1]

Python 2.7.14
sphinx-build 1.7.6

> -- Markus --
> 
> 
> [1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install

thanks.
-- 
~Randy

Re: Sphinx parallel build errors

[Markus Heiser]

Am 29.10.19 um 20:41 schrieb Randy Dunlap:
>> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
>> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
>> about 5GB (*more CPUs need more mem*).
>>
>> @Randy you said the build machine hase 8GB .. and how much CPUs?
> 
> 4.
> 

>>> Randy, are you able to just use -j2 or similar with your builds to avoid
>>> this?
> 
> No, sometimes even -j1 fails with oom-killer. >
>> @Randy: my first questions are:
>>
>> - cpu / mem?
>> - are OOM-kills when building target htmldocs, pdfdocs or ..?
> 
> htmldocs
> 
>> - which py & sphinx version are used? Reference installation see 'virtualenv'
>>    in [1]
> 
> Python 2.7.14
> sphinx-build 1.7.6

Seems nothing special in your environment .. its strange to me.

Installing new sphinx-doc will not help, you are fighting with some
other problems.  Hoped that I can help, but realize that I don't have
a clue what it is / sorry.

   -- Markus --

Re: Sphinx parallel build errors

[Randy Dunlap]

On 10/29/19 12:57 PM, Markus Heiser wrote:
> Am 29.10.19 um 20:41 schrieb Randy Dunlap:
>>> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
>>> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
>>> about 5GB (*more CPUs need more mem*).
>>>
>>> @Randy you said the build machine hase 8GB .. and how much CPUs?
>>
>> 4.
>>
> 
>>>> Randy, are you able to just use -j2 or similar with your builds to avoid
>>>> this?
>>
>> No, sometimes even -j1 fails with oom-killer. >
>>> @Randy: my first questions are:
>>>
>>> - cpu / mem?
>>> - are OOM-kills when building target htmldocs, pdfdocs or ..?
>>
>> htmldocs
>>
>>> - which py & sphinx version are used? Reference installation see 'virtualenv'
>>>    in [1]
>>
>> Python 2.7.14
>> sphinx-build 1.7.6
> 
> Seems nothing special in your environment .. its strange to me.
> 
> Installing new sphinx-doc will not help, you are fighting with some
> other problems.  Hoped that I can help, but realize that I don't have
> a clue what it is / sorry.

No problem.  Thanks for looking.

-- 
~Randy