* Sphinx parallel build errors @ 2019-10-09 21:37 Randy Dunlap 2019-10-09 21:40 ` Jonathan Corbet 2019-10-10 2:30 ` Kees Cook 0 siblings, 2 replies; 18+ messages in thread From: Randy Dunlap @ 2019-10-09 21:37 UTC (permalink / raw) To: linux-doc, Jonathan Corbet; +Cc: Kees Cook, Mauro Carvalho Chehab Sphinx parallel build error: OSError: [Errno 12] Cannot allocate memory I have had this error 4 times in the last 3 days. The first time it oom-killed Thunderbird. No big deal. The second time it oom-killed gimp, which had 24 photos open. Still no big deal. Just a nuisance. My little laptop has only 8 GB of RAM. Is there a way that I can limit the amount of parallelism? thanks. -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-09 21:37 Sphinx parallel build errors Randy Dunlap @ 2019-10-09 21:40 ` Jonathan Corbet 2019-10-09 22:18 ` Randy Dunlap 2019-10-10 2:30 ` Kees Cook 1 sibling, 1 reply; 18+ messages in thread From: Jonathan Corbet @ 2019-10-09 21:40 UTC (permalink / raw) To: Randy Dunlap; +Cc: linux-doc, Kees Cook, Mauro Carvalho Chehab On Wed, 9 Oct 2019 14:37:51 -0700 Randy Dunlap <rdunlap@infradead.org> wrote: > Sphinx parallel build error: > OSError: [Errno 12] Cannot allocate memory > > I have had this error 4 times in the last 3 days. > > The first time it oom-killed Thunderbird. No big deal. > The second time it oom-killed gimp, which had 24 photos open. > Still no big deal. Just a nuisance. > > My little laptop has only 8 GB of RAM. > > Is there a way that I can limit the amount of parallelism? Is this a change with current docs-next, or have you seen this for a while? The parallelism logic changed a bit in docs-next, but the default behavior should be the same as it was before. "make -j 1 htmldocs" (in current docs-next) should give you a nice, slow, serial build. jon ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-09 21:40 ` Jonathan Corbet @ 2019-10-09 22:18 ` Randy Dunlap 0 siblings, 0 replies; 18+ messages in thread From: Randy Dunlap @ 2019-10-09 22:18 UTC (permalink / raw) To: Jonathan Corbet; +Cc: linux-doc, Kees Cook, Mauro Carvalho Chehab On 10/9/19 2:40 PM, Jonathan Corbet wrote: > On Wed, 9 Oct 2019 14:37:51 -0700 > Randy Dunlap <rdunlap@infradead.org> wrote: > >> Sphinx parallel build error: >> OSError: [Errno 12] Cannot allocate memory >> >> I have had this error 4 times in the last 3 days. >> >> The first time it oom-killed Thunderbird. No big deal. >> The second time it oom-killed gimp, which had 24 photos open. >> Still no big deal. Just a nuisance. >> >> My little laptop has only 8 GB of RAM. >> >> Is there a way that I can limit the amount of parallelism? > > Is this a change with current docs-next, or have you seen this for a > while? The parallelism logic changed a bit in docs-next, but the default > behavior should be the same as it was before. I'm not sure. Today's 3 instances were all in linux-next but I don't know about the oom failure a few days ago. > "make -j 1 htmldocs" (in current docs-next) should give you a nice, slow, > serial build. I don't want to limit is _that much_. :( -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-09 21:37 Sphinx parallel build errors Randy Dunlap 2019-10-09 21:40 ` Jonathan Corbet @ 2019-10-10 2:30 ` Kees Cook 2019-10-10 3:00 ` Bhaskar Chowdhury 2019-10-10 18:48 ` Jonathan Corbet 1 sibling, 2 replies; 18+ messages in thread From: Kees Cook @ 2019-10-10 2:30 UTC (permalink / raw) To: Randy Dunlap; +Cc: linux-doc, Jonathan Corbet, Mauro Carvalho Chehab On Wed, Oct 09, 2019 at 02:37:51PM -0700, Randy Dunlap wrote: > > Sphinx parallel build error: > OSError: [Errno 12] Cannot allocate memory > > I have had this error 4 times in the last 3 days. > > The first time it oom-killed Thunderbird. No big deal. > The second time it oom-killed gimp, which had 24 photos open. > Still no big deal. Just a nuisance. Eek, sorry! OOM-killer once again chooses ... not the culprit. ;) > My little laptop has only 8 GB of RAM. > > Is there a way that I can limit the amount of parallelism? As mentioned, "make -jN htmldocs" should be respected now (before it was as many CPUs as it had). I'm wondering if it's actually the addition of the MAINTAINERS file parsing -- that's a really big parse and maybe that pushed things over the edge? -- Kees Cook ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-10 2:30 ` Kees Cook @ 2019-10-10 3:00 ` Bhaskar Chowdhury 2019-10-10 18:48 ` Jonathan Corbet 1 sibling, 0 replies; 18+ messages in thread From: Bhaskar Chowdhury @ 2019-10-10 3:00 UTC (permalink / raw) To: Kees Cook; +Cc: Randy Dunlap, linux-doc, Jonathan Corbet, Mauro Carvalho Chehab [-- Attachment #1: Type: text/plain, Size: 1199 bytes --] On 19:30 Wed 09 Oct 2019, Kees Cook wrote: >On Wed, Oct 09, 2019 at 02:37:51PM -0700, Randy Dunlap wrote: >> >> Sphinx parallel build error: >> OSError: [Errno 12] Cannot allocate memory >> >> I have had this error 4 times in the last 3 days. >> >> The first time it oom-killed Thunderbird. No big deal. >> The second time it oom-killed gimp, which had 24 photos open. >> Still no big deal. Just a nuisance. > >Eek, sorry! OOM-killer once again chooses ... not the culprit. ;) > >> My little laptop has only 8 GB of RAM. >> >> Is there a way that I can limit the amount of parallelism? > >As mentioned, "make -jN htmldocs" should be respected now (before it was >as many CPUs as it had). I'm wondering if it's actually the addition of >the MAINTAINERS file parsing -- that's a really big parse and maybe that >pushed things over the edge? > >-- >Kees Cook Seriously, looking/parsing MAINTAINERS file becomes serious pain in the arse. It big and very easy to get lost in the way. Okay, complaining for the sake of complaining doesn't make things go far ...but until we find something more efficient ,alas! We have deal with it...that kinda sucks. -Bhaskar [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-10 2:30 ` Kees Cook 2019-10-10 3:00 ` Bhaskar Chowdhury @ 2019-10-10 18:48 ` Jonathan Corbet 2019-10-10 20:15 ` Randy Dunlap 1 sibling, 1 reply; 18+ messages in thread From: Jonathan Corbet @ 2019-10-10 18:48 UTC (permalink / raw) To: Kees Cook; +Cc: Randy Dunlap, linux-doc, Mauro Carvalho Chehab On Wed, 9 Oct 2019 19:30:56 -0700 Kees Cook <keescook@chromium.org> wrote: > I'm wondering if it's actually the addition of > the MAINTAINERS file parsing -- that's a really big parse and maybe that > pushed things over the edge? That does seem like a likely culprit, doesn't it? Watching a build here, though, I don't see that any one of the sphinx-build processes balloons to a horrific size, which is what I would expect if one file were causing the problem. In general, the sphinx build doesn't really begin to approach the memory requirements of, say, firefox or my mail client on my system. Randy, I'd be curious to know if you see the same behavior with current mainline. Also, can you see how many sphinx-build processes you have running? Thanks, jon ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-10 18:48 ` Jonathan Corbet @ 2019-10-10 20:15 ` Randy Dunlap 2019-10-27 18:11 ` Randy Dunlap 0 siblings, 1 reply; 18+ messages in thread From: Randy Dunlap @ 2019-10-10 20:15 UTC (permalink / raw) To: Jonathan Corbet, Kees Cook; +Cc: linux-doc, Mauro Carvalho Chehab On 10/10/19 11:48 AM, Jonathan Corbet wrote: > On Wed, 9 Oct 2019 19:30:56 -0700 > Kees Cook <keescook@chromium.org> wrote: > >> I'm wondering if it's actually the addition of >> the MAINTAINERS file parsing -- that's a really big parse and maybe that >> pushed things over the edge? > > That does seem like a likely culprit, doesn't it? Watching a build here, > though, I don't see that any one of the sphinx-build processes balloons to > a horrific size, which is what I would expect if one file were causing the > problem. > > In general, the sphinx build doesn't really begin to approach the memory > requirements of, say, firefox or my mail client on my system. > > Randy, I'd be curious to know if you see the same behavior with current > mainline. Also, can you see how many sphinx-build processes you have > running? Yes, I see oom-kills when building docs for 5.4-rc2. But only after I ran 'gimp *.jpg' for 22 photos. I can see anywhere from 1 to 4 sphinx-build processes running. > Thanks, > > jon -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-10 20:15 ` Randy Dunlap @ 2019-10-27 18:11 ` Randy Dunlap 2019-10-27 18:17 ` Jonathan Corbet 0 siblings, 1 reply; 18+ messages in thread From: Randy Dunlap @ 2019-10-27 18:11 UTC (permalink / raw) To: Jonathan Corbet, Kees Cook; +Cc: linux-doc, Mauro Carvalho Chehab On 10/10/19 1:15 PM, Randy Dunlap wrote: > On 10/10/19 11:48 AM, Jonathan Corbet wrote: >> On Wed, 9 Oct 2019 19:30:56 -0700 >> Kees Cook <keescook@chromium.org> wrote: >> >>> I'm wondering if it's actually the addition of >>> the MAINTAINERS file parsing -- that's a really big parse and maybe that >>> pushed things over the edge? >> >> That does seem like a likely culprit, doesn't it? Watching a build here, >> though, I don't see that any one of the sphinx-build processes balloons to >> a horrific size, which is what I would expect if one file were causing the >> problem. >> >> In general, the sphinx build doesn't really begin to approach the memory >> requirements of, say, firefox or my mail client on my system. >> >> Randy, I'd be curious to know if you see the same behavior with current >> mainline. Also, can you see how many sphinx-build processes you have >> running? > > Yes, I see oom-kills when building docs for 5.4-rc2. > But only after I ran 'gimp *.jpg' for 22 photos. > > I can see anywhere from 1 to 4 sphinx-build processes running. I continue to have oom-kills when building docs. At least 2 times I also had system freezes where I had to power cycle to reboot. I suppose I can only hope that other people start running into this issue. :( -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-27 18:11 ` Randy Dunlap @ 2019-10-27 18:17 ` Jonathan Corbet 2019-10-27 18:30 ` Randy Dunlap 0 siblings, 1 reply; 18+ messages in thread From: Jonathan Corbet @ 2019-10-27 18:17 UTC (permalink / raw) To: Randy Dunlap; +Cc: Kees Cook, linux-doc, Mauro Carvalho Chehab On Sun, 27 Oct 2019 11:11:57 -0700 Randy Dunlap <rdunlap@infradead.org> wrote: > > Yes, I see oom-kills when building docs for 5.4-rc2. > > But only after I ran 'gimp *.jpg' for 22 photos. > > > > I can see anywhere from 1 to 4 sphinx-build processes running. > > I continue to have oom-kills when building docs. > At least 2 times I also had system freezes where I had to power > cycle to reboot. > > I suppose I can only hope that other people start running into this > issue. :( Are you still running gimp at the same time? Also, are you doing a straight "make htmldocs" or something else? Thanks, jon ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-27 18:17 ` Jonathan Corbet @ 2019-10-27 18:30 ` Randy Dunlap 2019-10-27 19:05 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 18+ messages in thread From: Randy Dunlap @ 2019-10-27 18:30 UTC (permalink / raw) To: Jonathan Corbet; +Cc: Kees Cook, linux-doc, Mauro Carvalho Chehab On 10/27/19 11:17 AM, Jonathan Corbet wrote: > On Sun, 27 Oct 2019 11:11:57 -0700 > Randy Dunlap <rdunlap@infradead.org> wrote: > >>> Yes, I see oom-kills when building docs for 5.4-rc2. >>> But only after I ran 'gimp *.jpg' for 22 photos. >>> >>> I can see anywhere from 1 to 4 sphinx-build processes running. >> >> I continue to have oom-kills when building docs. >> At least 2 times I also had system freezes where I had to power >> cycle to reboot. >> >> I suppose I can only hope that other people start running into this >> issue. :( > > Are you still running gimp at the same time? no. > Also, are you doing a straight "make htmldocs" or something else? straight "make htmldocs". -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-27 18:30 ` Randy Dunlap @ 2019-10-27 19:05 ` Mauro Carvalho Chehab 2019-10-27 19:36 ` Randy Dunlap 2019-10-28 19:07 ` Kees Cook 0 siblings, 2 replies; 18+ messages in thread From: Mauro Carvalho Chehab @ 2019-10-27 19:05 UTC (permalink / raw) To: Randy Dunlap; +Cc: Jonathan Corbet, Kees Cook, linux-doc Em Sun, 27 Oct 2019 11:30:22 -0700 Randy Dunlap <rdunlap@infradead.org> escreveu: > On 10/27/19 11:17 AM, Jonathan Corbet wrote: > > On Sun, 27 Oct 2019 11:11:57 -0700 > > Randy Dunlap <rdunlap@infradead.org> wrote: > > > >>> Yes, I see oom-kills when building docs for 5.4-rc2. > >>> But only after I ran 'gimp *.jpg' for 22 photos. > >>> > >>> I can see anywhere from 1 to 4 sphinx-build processes running. > >> > >> I continue to have oom-kills when building docs. My 2 cents here. I would keep the default to use more CPUs, as nowadays I suspect that most devs have a way more than 8GB installed. The worse machine I use here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a personal laptop I bought 5 years ago). Yet, I agree that doc build should be saner on low-cap machines. Ideally, the fix should be, IMHO, at sphinx-build: -jauto should detect OOM possible conditions and reduce the number of used CPUs it the max available memory is too low. As there's now a script at Kernel, maybe the script could be modified to detect if the max available memory is too low, passing -j1 if, let's say, the machine has less than (let's say) 16 GB RAM. > >> At least 2 times I also had system freezes where I had to power > >> cycle to reboot. That sounds weird. I can't imagine why you would experience system freeze with doc build. Are you sure this is not your CPU overheating? > >> > >> I suppose I can only hope that other people start running into this > >> issue. :( > > > > Are you still running gimp at the same time? > > no. > > > Also, are you doing a straight "make htmldocs" or something else? > > straight "make htmldocs". Thanks, Mauro ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-27 19:05 ` Mauro Carvalho Chehab @ 2019-10-27 19:36 ` Randy Dunlap 2019-10-28 19:07 ` Kees Cook 1 sibling, 0 replies; 18+ messages in thread From: Randy Dunlap @ 2019-10-27 19:36 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, Kees Cook, linux-doc On 10/27/19 12:05 PM, Mauro Carvalho Chehab wrote: > Em Sun, 27 Oct 2019 11:30:22 -0700 > Randy Dunlap <rdunlap@infradead.org> escreveu: > >> On 10/27/19 11:17 AM, Jonathan Corbet wrote: >>> On Sun, 27 Oct 2019 11:11:57 -0700 >>> Randy Dunlap <rdunlap@infradead.org> wrote: >>> >>>>> Yes, I see oom-kills when building docs for 5.4-rc2. >>>>> But only after I ran 'gimp *.jpg' for 22 photos. >>>>> >>>>> I can see anywhere from 1 to 4 sphinx-build processes running. >>>> >>>> I continue to have oom-kills when building docs. > > My 2 cents here. > > I would keep the default to use more CPUs, as nowadays I suspect that > most devs have a way more than 8GB installed. The worse machine I use > here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a > personal laptop I bought 5 years ago). > > Yet, I agree that doc build should be saner on low-cap machines. > > Ideally, the fix should be, IMHO, at sphinx-build: -jauto should > detect OOM possible conditions and reduce the number of used CPUs > it the max available memory is too low. > > As there's now a script at Kernel, maybe the script could be modified > to detect if the max available memory is too low, passing -j1 if, > let's say, the machine has less than (let's say) 16 GB RAM. > >>>> At least 2 times I also had system freezes where I had to power >>>> cycle to reboot. > > That sounds weird. I can't imagine why you would experience system > freeze with doc build. Are you sure this is not your CPU overheating? > No, I'm not sure. I'm just sure that it only happens when I am building kernel docs. >>>> I suppose I can only hope that other people start running into this >>>> issue. :( >>> >>> Are you still running gimp at the same time? >> >> no. >> >>> Also, are you doing a straight "make htmldocs" or something else? >> >> straight "make htmldocs". > > > Thanks, > Mauro -- ~Randy Reported-by: Randy Dunlap <rdunlap@infradead.org> ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-27 19:05 ` Mauro Carvalho Chehab 2019-10-27 19:36 ` Randy Dunlap @ 2019-10-28 19:07 ` Kees Cook 2019-10-29 8:16 ` Markus Heiser 1 sibling, 1 reply; 18+ messages in thread From: Kees Cook @ 2019-10-28 19:07 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Randy Dunlap, Jonathan Corbet, linux-doc On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote: > Em Sun, 27 Oct 2019 11:30:22 -0700 > Randy Dunlap <rdunlap@infradead.org> escreveu: > > > On 10/27/19 11:17 AM, Jonathan Corbet wrote: > > > On Sun, 27 Oct 2019 11:11:57 -0700 > > > Randy Dunlap <rdunlap@infradead.org> wrote: > > > > > >>> Yes, I see oom-kills when building docs for 5.4-rc2. > > >>> But only after I ran 'gimp *.jpg' for 22 photos. > > >>> > > >>> I can see anywhere from 1 to 4 sphinx-build processes running. > > >> > > >> I continue to have oom-kills when building docs. > > My 2 cents here. > > I would keep the default to use more CPUs, as nowadays I suspect that > most devs have a way more than 8GB installed. The worse machine I use > here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a > personal laptop I bought 5 years ago). > > Yet, I agree that doc build should be saner on low-cap machines. > > Ideally, the fix should be, IMHO, at sphinx-build: -jauto should > detect OOM possible conditions and reduce the number of used CPUs > it the max available memory is too low. How can we determine that dynamically, though? e.g. how do we determine how much is needed per thread in a way that will move with the future needs of Sphinx, etc? > As there's now a script at Kernel, maybe the script could be modified > to detect if the max available memory is too low, passing -j1 if, > let's say, the machine has less than (let's say) 16 GB RAM. Randy, are you able to just use -j2 or similar with your builds to avoid this? -- Kees Cook ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-28 19:07 ` Kees Cook @ 2019-10-29 8:16 ` Markus Heiser 2019-10-29 16:09 ` Kees Cook 2019-10-29 19:41 ` Randy Dunlap 0 siblings, 2 replies; 18+ messages in thread From: Markus Heiser @ 2019-10-29 8:16 UTC (permalink / raw) To: Kees Cook, Mauro Carvalho Chehab, Randy Dunlap; +Cc: Jonathan Corbet, linux-doc Am 28.10.19 um 20:07 schrieb Kees Cook: > On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote: >> Em Sun, 27 Oct 2019 11:30:22 -0700 >> Randy Dunlap <rdunlap@infradead.org> escreveu: >> >>> On 10/27/19 11:17 AM, Jonathan Corbet wrote: >>>> On Sun, 27 Oct 2019 11:11:57 -0700 >>>> Randy Dunlap <rdunlap@infradead.org> wrote: >>>> >>>>>> Yes, I see oom-kills when building docs for 5.4-rc2. >>>>>> But only after I ran 'gimp *.jpg' for 22 photos. >>>>>> >>>>>> I can see anywhere from 1 to 4 sphinx-build processes running. >>>>> >>>>> I continue to have oom-kills when building docs. >> >> My 2 cents here. >> >> I would keep the default to use more CPUs, as nowadays I suspect that >> most devs have a way more than 8GB installed. The worse machine I use >> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a >> personal laptop I bought 5 years ago). >> >> Yet, I agree that doc build should be saner on low-cap machines. >> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems. (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is about 5GB (*more CPUs need more mem*). @Randy you said the build machine hase 8GB .. and how much CPUs? >> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should >> detect OOM possible conditions and reduce the number of used CPUs >> it the max available memory is too low. > > How can we determine that dynamically, though? e.g. how do we determine > how much is needed per thread in a way that will move with the future > needs of Sphinx, etc? d'accord >> As there's now a script at Kernel, maybe the script could be modified >> to detect if the max available memory is too low, passing -j1 if, >> let's say, the machine has less than (let's say) 16 GB RAM. > > Randy, are you able to just use -j2 or similar with your builds to avoid > this? > Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string, nothing special .. maybe we have a mem leak in one of our extensions? @Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything strange? Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. needs investigation. @Randy: my first questions are: - cpu / mem? - are OOM-kills when building target htmldocs, pdfdocs or ..? - which py & sphinx version are used? Reference installation see 'virtualenv' in [1] -- Markus -- [1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-29 8:16 ` Markus Heiser @ 2019-10-29 16:09 ` Kees Cook 2019-10-29 19:41 ` Randy Dunlap 1 sibling, 0 replies; 18+ messages in thread From: Kees Cook @ 2019-10-29 16:09 UTC (permalink / raw) To: Markus Heiser Cc: Mauro Carvalho Chehab, Randy Dunlap, Jonathan Corbet, linux-doc On Tue, Oct 29, 2019 at 09:16:06AM +0100, Markus Heiser wrote: > @Kees: why did you mentioning MAINTAINERS parsing .. did you observe anything > strange? No, nothing strange exactly, but it was another change that happened around the same time and is a pretty large parse. So I was wondering if adding it might have tipped the scale. -- Kees Cook ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-29 8:16 ` Markus Heiser 2019-10-29 16:09 ` Kees Cook @ 2019-10-29 19:41 ` Randy Dunlap 2019-10-29 19:57 ` Markus Heiser 1 sibling, 1 reply; 18+ messages in thread From: Randy Dunlap @ 2019-10-29 19:41 UTC (permalink / raw) To: Markus Heiser, Kees Cook, Mauro Carvalho Chehab Cc: Jonathan Corbet, linux-doc On 10/29/19 1:16 AM, Markus Heiser wrote: > Am 28.10.19 um 20:07 schrieb Kees Cook: >> On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote: >>> Em Sun, 27 Oct 2019 11:30:22 -0700 >>> Randy Dunlap <rdunlap@infradead.org> escreveu: >>> >>>> On 10/27/19 11:17 AM, Jonathan Corbet wrote: >>>>> On Sun, 27 Oct 2019 11:11:57 -0700 >>>>> Randy Dunlap <rdunlap@infradead.org> wrote: >>>>> >>>>>>> Yes, I see oom-kills when building docs for 5.4-rc2. >>>>>>> But only after I ran 'gimp *.jpg' for 22 photos. >>>>>>> >>>>>>> I can see anywhere from 1 to 4 sphinx-build processes running. >>>>>> >>>>>> I continue to have oom-kills when building docs. >>> >>> My 2 cents here. >>> >>> I would keep the default to use more CPUs, as nowadays I suspect that >>> most devs have a way more than 8GB installed. The worse machine I use >>> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a >>> personal laptop I bought 5 years ago). >>> >>> Yet, I agree that doc build should be saner on low-cap machines. >>> > > In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems. > (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is > about 5GB (*more CPUs need more mem*). > > @Randy you said the build machine hase 8GB .. and how much CPUs? 4. >>> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should >>> detect OOM possible conditions and reduce the number of used CPUs >>> it the max available memory is too low. >> >> How can we determine that dynamically, though? e.g. how do we determine >> how much is needed per thread in a way that will move with the future >> needs of Sphinx, etc? > > d'accord > >>> As there's now a script at Kernel, maybe the script could be modified >>> to detect if the max available memory is too low, passing -j1 if, >>> let's say, the machine has less than (let's say) 16 GB RAM. >> >> Randy, are you able to just use -j2 or similar with your builds to avoid >> this? No, sometimes even -j1 fails with oom-killer. > > Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string, > nothing special .. maybe we have a mem leak in one of our extensions? > > @Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything > strange? > > Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. needs investigation. > > @Randy: my first questions are: > > - cpu / mem? > - are OOM-kills when building target htmldocs, pdfdocs or ..? htmldocs > - which py & sphinx version are used? Reference installation see 'virtualenv' > in [1] Python 2.7.14 sphinx-build 1.7.6 > -- Markus -- > > > [1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install thanks. -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-29 19:41 ` Randy Dunlap @ 2019-10-29 19:57 ` Markus Heiser 2019-10-29 20:10 ` Randy Dunlap 0 siblings, 1 reply; 18+ messages in thread From: Markus Heiser @ 2019-10-29 19:57 UTC (permalink / raw) To: Randy Dunlap, Kees Cook, Mauro Carvalho Chehab; +Cc: Jonathan Corbet, linux-doc Am 29.10.19 um 20:41 schrieb Randy Dunlap: >> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems. >> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is >> about 5GB (*more CPUs need more mem*). >> >> @Randy you said the build machine hase 8GB .. and how much CPUs? > > 4. > >>> Randy, are you able to just use -j2 or similar with your builds to avoid >>> this? > > No, sometimes even -j1 fails with oom-killer. > >> @Randy: my first questions are: >> >> - cpu / mem? >> - are OOM-kills when building target htmldocs, pdfdocs or ..? > > htmldocs > >> - which py & sphinx version are used? Reference installation see 'virtualenv' >> in [1] > > Python 2.7.14 > sphinx-build 1.7.6 Seems nothing special in your environment .. its strange to me. Installing new sphinx-doc will not help, you are fighting with some other problems. Hoped that I can help, but realize that I don't have a clue what it is / sorry. -- Markus -- ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Sphinx parallel build errors 2019-10-29 19:57 ` Markus Heiser @ 2019-10-29 20:10 ` Randy Dunlap 0 siblings, 0 replies; 18+ messages in thread From: Randy Dunlap @ 2019-10-29 20:10 UTC (permalink / raw) To: Markus Heiser, Kees Cook, Mauro Carvalho Chehab Cc: Jonathan Corbet, linux-doc On 10/29/19 12:57 PM, Markus Heiser wrote: > Am 29.10.19 um 20:41 schrieb Randy Dunlap: >>> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems. >>> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is >>> about 5GB (*more CPUs need more mem*). >>> >>> @Randy you said the build machine hase 8GB .. and how much CPUs? >> >> 4. >> > >>>> Randy, are you able to just use -j2 or similar with your builds to avoid >>>> this? >> >> No, sometimes even -j1 fails with oom-killer. > >>> @Randy: my first questions are: >>> >>> - cpu / mem? >>> - are OOM-kills when building target htmldocs, pdfdocs or ..? >> >> htmldocs >> >>> - which py & sphinx version are used? Reference installation see 'virtualenv' >>> in [1] >> >> Python 2.7.14 >> sphinx-build 1.7.6 > > Seems nothing special in your environment .. its strange to me. > > Installing new sphinx-doc will not help, you are fighting with some > other problems. Hoped that I can help, but realize that I don't have > a clue what it is / sorry. No problem. Thanks for looking. -- ~Randy ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2019-10-29 20:10 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-10-09 21:37 Sphinx parallel build errors Randy Dunlap 2019-10-09 21:40 ` Jonathan Corbet 2019-10-09 22:18 ` Randy Dunlap 2019-10-10 2:30 ` Kees Cook 2019-10-10 3:00 ` Bhaskar Chowdhury 2019-10-10 18:48 ` Jonathan Corbet 2019-10-10 20:15 ` Randy Dunlap 2019-10-27 18:11 ` Randy Dunlap 2019-10-27 18:17 ` Jonathan Corbet 2019-10-27 18:30 ` Randy Dunlap 2019-10-27 19:05 ` Mauro Carvalho Chehab 2019-10-27 19:36 ` Randy Dunlap 2019-10-28 19:07 ` Kees Cook 2019-10-29 8:16 ` Markus Heiser 2019-10-29 16:09 ` Kees Cook 2019-10-29 19:41 ` Randy Dunlap 2019-10-29 19:57 ` Markus Heiser 2019-10-29 20:10 ` Randy Dunlap
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.