[PATCH 0/1] Fix webkit-gtk 'Memory Exhaust' issue

[PATCH 0/1] Fix webkit-gtk 'Memory Exhaust' issue

[Kai Kang]

The following changes since commit 31e6eee860b5f9f4ac9ef0889bcff5648de6e3f9:

  poky-tiny.conf: blacklist core-image-weston option (2013-07-18 21:26:58 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib kangkai/webkit-gtk
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/webkit-gtk

Kai Kang (1):
  webkit-gtk: fix 'Memory exhausted' error

 meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
 1 file changed, 2 insertions(+)

-- 
1.8.1.2

[PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Kai Kang]

Build webkit-gtk occupies lot of memory and fails on some host:
| ...tmp/sysroots/i686-linux/usr/libexec/x86_64-wrs-linux/gcc/x86_64-wrs-linux/4.8.1/ld:
failed to set dynamic section sizes: Memory exhausted
| collect2: error: ld returned 1 exit status
| make[1]: *** [libwebkitgtk-1.0.la] Error 1

Add ld option to fix it.

Refer to:
http://comments.gmane.org/gmane.os.opendarwin.webkit.devel/20078

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
index 5691d3f..90625a3 100644
--- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
+++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
@@ -60,6 +60,8 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
 
 EXTRA_AUTORECONF = " -I Source/autotools "
 
+# Fix compile error: "ld: failed to set dynamic section sizes: Memory exhausted"
+LDFLAGS += " -Wl,--no-keep-memory"
 
 #| ./Source/JavaScriptCore/heap/HandleTypes.h: In static member function 'static T* JSC::HandleTypes<T>::getFromSlot(JSC::HandleSlot) [with T = JSC::Structure, JSC::HandleTypes<T>::ExternalType = JSC::Structure*, JSC::HandleSlot = JSC::JSValue*]':
 #| ./Source/JavaScriptCore/heap/Handle.h:141:79:   instantiated from 'JSC::Handle<T>::ExternalType JSC::Handle<T>::get() const [with T = JSC::Structure, JSC::Handle<T>::ExternalType = JSC::Structure*]'
-- 
1.8.1.2

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[André Draszik]

Hi,

Instead of penalising everybody, would it be possible to make this
change dependent on the build machine (ram + swap)? Or to only add
-Wl,--no-keep-memory if the build failed without it?

Not sure if it would make any difference, but do you see this only with
full debug enabled (-ggdb)? If yes, can the debug flags be changed, or
that linker flag only be added, if full debug is enabled?


Cheers,
Andre'

On Mon, 2013-07-22 at 15:51 +0800, Kai Kang wrote:
> Build webkit-gtk occupies lot of memory and fails on some host:
> | ...tmp/sysroots/i686-linux/usr/libexec/x86_64-wrs-linux/gcc/x86_64-wrs-linux/4.8.1/ld:
> failed to set dynamic section sizes: Memory exhausted
> | collect2: error: ld returned 1 exit status
> | make[1]: *** [libwebkitgtk-1.0.la] Error 1
> 
> Add ld option to fix it.
> 
> Refer to:
> http://comments.gmane.org/gmane.os.opendarwin.webkit.devel/20078
> 
> Signed-off-by: Kai Kang <kai.kang@windriver.com>
> ---
>  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> index 5691d3f..90625a3 100644
> --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> @@ -60,6 +60,8 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
>  
>  EXTRA_AUTORECONF = " -I Source/autotools "
>  
> +# Fix compile error: "ld: failed to set dynamic section sizes: Memory exhausted"
> +LDFLAGS += " -Wl,--no-keep-memory"
>  
>  #| ./Source/JavaScriptCore/heap/HandleTypes.h: In static member function 'static T* JSC::HandleTypes<T>::getFromSlot(JSC::HandleSlot) [with T = JSC::Structure, JSC::HandleTypes<T>::ExternalType = JSC::Structure*, JSC::HandleSlot = JSC::JSValue*]':
>  #| ./Source/JavaScriptCore/heap/Handle.h:141:79:   instantiated from 'JSC::Handle<T>::ExternalType JSC::Handle<T>::get() const [with T = JSC::Structure, JSC::Handle<T>::ExternalType = JSC::Structure*]'

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Phil Blundell]

Agreed, I think this sort of thing belongs in site.conf or some such
place, not in the recipe.

p.

On Mon, 2013-07-22 at 09:22 +0100, André Draszik wrote:
> Hi,
> 
> Instead of penalising everybody, would it be possible to make this
> change dependent on the build machine (ram + swap)? Or to only add
> -Wl,--no-keep-memory if the build failed without it?
> 
> Not sure if it would make any difference, but do you see this only with
> full debug enabled (-ggdb)? If yes, can the debug flags be changed, or
> that linker flag only be added, if full debug is enabled?
> 
> 
> Cheers,
> Andre'
> 
> On Mon, 2013-07-22 at 15:51 +0800, Kai Kang wrote:
> > Build webkit-gtk occupies lot of memory and fails on some host:
> > | ...tmp/sysroots/i686-linux/usr/libexec/x86_64-wrs-linux/gcc/x86_64-wrs-linux/4.8.1/ld:
> > failed to set dynamic section sizes: Memory exhausted
> > | collect2: error: ld returned 1 exit status
> > | make[1]: *** [libwebkitgtk-1.0.la] Error 1
> > 
> > Add ld option to fix it.
> > 
> > Refer to:
> > http://comments.gmane.org/gmane.os.opendarwin.webkit.devel/20078
> > 
> > Signed-off-by: Kai Kang <kai.kang@windriver.com>
> > ---
> >  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > index 5691d3f..90625a3 100644
> > --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > @@ -60,6 +60,8 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
> >  
> >  EXTRA_AUTORECONF = " -I Source/autotools "
> >  
> > +# Fix compile error: "ld: failed to set dynamic section sizes: Memory exhausted"
> > +LDFLAGS += " -Wl,--no-keep-memory"
> >  
> >  #| ./Source/JavaScriptCore/heap/HandleTypes.h: In static member function 'static T* JSC::HandleTypes<T>::getFromSlot(JSC::HandleSlot) [with T = JSC::Structure, JSC::HandleTypes<T>::ExternalType = JSC::Structure*, JSC::HandleSlot = JSC::JSValue*]':
> >  #| ./Source/JavaScriptCore/heap/Handle.h:141:79:   instantiated from 'JSC::Handle<T>::ExternalType JSC::Handle<T>::get() const [with T = JSC::Structure, JSC::Handle<T>::ExternalType = JSC::Structure*]'
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Kang Kai]

On 2013年07月22日 16:22, André Draszik wrote:
> Hi,
>
> Instead of penalising everybody, would it be possible to make this
> change dependent on the build machine (ram + swap)? Or to only add
> -Wl,--no-keep-memory if the build failed without it?

Yes, it doesn't fail on every build machine. But it seems that build 
fails randomly on different build machines.

>
> Not sure if it would make any difference, but do you see this only with
> full debug enabled (-ggdb)? If yes, can the debug flags be changed, or
> that linker flag only be added, if full debug is enabled?

webkit-gtk set configure option "--enable-debug=no" to turn off debug. 
And no such "-ggdb" flags.

Thanks,
Kai

>
>
> Cheers,
> Andre'
>
> On Mon, 2013-07-22 at 15:51 +0800, Kai Kang wrote:
>> Build webkit-gtk occupies lot of memory and fails on some host:
>> | ...tmp/sysroots/i686-linux/usr/libexec/x86_64-wrs-linux/gcc/x86_64-wrs-linux/4.8.1/ld:
>> failed to set dynamic section sizes: Memory exhausted
>> | collect2: error: ld returned 1 exit status
>> | make[1]: *** [libwebkitgtk-1.0.la] Error 1
>>
>> Add ld option to fix it.
>>
>> Refer to:
>> http://comments.gmane.org/gmane.os.opendarwin.webkit.devel/20078
>>
>> Signed-off-by: Kai Kang <kai.kang@windriver.com>
>> ---
>>   meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> index 5691d3f..90625a3 100644
>> --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> @@ -60,6 +60,8 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
>>   
>>   EXTRA_AUTORECONF = " -I Source/autotools "
>>   
>> +# Fix compile error: "ld: failed to set dynamic section sizes: Memory exhausted"
>> +LDFLAGS += " -Wl,--no-keep-memory"
>>   
>>   #| ./Source/JavaScriptCore/heap/HandleTypes.h: In static member function 'static T* JSC::HandleTypes<T>::getFromSlot(JSC::HandleSlot) [with T = JSC::Structure, JSC::HandleTypes<T>::ExternalType = JSC::Structure*, JSC::HandleSlot = JSC::JSValue*]':
>>   #| ./Source/JavaScriptCore/heap/Handle.h:141:79:   instantiated from 'JSC::Handle<T>::ExternalType JSC::Handle<T>::get() const [with T = JSC::Structure, JSC::Handle<T>::ExternalType = JSC::Structure*]'
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>


-- 
Regards,
Neil | Kai Kang

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Burton, Ross]

On 24 July 2013 03:19, Kang Kai <Kai.Kang@windriver.com> wrote:
>> Instead of penalising everybody, would it be possible to make this
>> change dependent on the build machine (ram + swap)? Or to only add
>> -Wl,--no-keep-memory if the build failed without it?
>
> Yes, it doesn't fail on every build machine. But it seems that build fails
> randomly on different build machines.

I'm in agreement with the others - this is how individual build
machine admins make a webkit-gtk build work on their machine if it
doesn't have enough memory available.  We should probably document how
to do this in local.conf in the recipe for reference, but not make it
a default.

Ross

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Paul Barker]

On 24 July 2013 08:49, Burton, Ross <ross.burton@intel.com> wrote:
> On 24 July 2013 03:19, Kang Kai <Kai.Kang@windriver.com> wrote:
>>> Instead of penalising everybody, would it be possible to make this
>>> change dependent on the build machine (ram + swap)? Or to only add
>>> -Wl,--no-keep-memory if the build failed without it?
>>
>> Yes, it doesn't fail on every build machine. But it seems that build fails
>> randomly on different build machines.
>
> I'm in agreement with the others - this is how individual build
> machine admins make a webkit-gtk build work on their machine if it
> doesn't have enough memory available.  We should probably document how
> to do this in local.conf in the recipe for reference, but not make it
> a default.
>

Perhaps create a bbclass for this that can be globally inherited in
local.conf? I ran into a similar thing with gnuradio and had to drop
PARALLEL_MAKE to get the build to complete with my amount of RAM+swap.
Something like "lean-build.bbclass" that handles known issues for
systems with limited memory.

Would variables like LDFLAGS_webkit-gtk and PARALLEL_MAKE_gnuradio be
applied to the appropriate recipes if they were set in local.conf or a
globally inherited bbclass, or am I misunderstanding how bitbake
parses variables?

--
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Phil Blundell]

On Wed, 2013-07-24 at 09:31 +0100, Paul Barker wrote:
> Would variables like LDFLAGS_webkit-gtk and PARALLEL_MAKE_gnuradio be
> applied to the appropriate recipes if they were set in local.conf or a
> globally inherited bbclass, or am I misunderstanding how bitbake
> parses variables?

LDFLAGS_pn-webkit-gtk and PARALLEL_MAKE_pn-gnuradio, yes.  Any of that
stuff can go in site.conf and/or local.conf, and this is where this sort
of build-environment-specific customisation belongs.

p.

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Randy MacLeod]

On 13-07-24 05:47 AM, Phil Blundell wrote:
> On Wed, 2013-07-24 at 09:31 +0100, Paul Barker wrote:
>> Would variables like LDFLAGS_webkit-gtk and PARALLEL_MAKE_gnuradio be
>> applied to the appropriate recipes if they were set in local.conf or a
>> globally inherited bbclass, or am I misunderstanding how bitbake
>> parses variables?
>
> LDFLAGS_pn-webkit-gtk and PARALLEL_MAKE_pn-gnuradio, yes.  Any of that
> stuff can go in site.conf and/or local.conf, and this is where this sort
> of build-environment-specific customisation belongs.

I disagree.

The default behaviour should be that the system builds as long
as the hosts has some minimum RAM.  If this means that we pay
a < ~5% build time penalty, we should not flinch. I'm not sure
how much extra time I'm willing to accept but for a given package
even a factor of two could be acceptable as long as the overall
build did not take twice as long.

Kai, when you have a chance to test this, can you tell us
what sort of build time differences we are talking about for
the full package as well as just for the link phase if that's easy
to extract? You could use /usr/bin/time to get the cpu and memory
usage.

Are we really expecting all system builders to track all
these limitations and put them in place on low-end machines?
I realize that we need, and most of us have, a beefy box for builds
but a few people are still trying to build using 32 bit, 4 GB systems
and many people likely expect that 8 GB of RAM is plenty, in fact
the system where this problem occurred had 8 GB and was running
with:
   BB_NUMBER_THREADS ?= "2"
   PARALLEL_MAKE ?= "-j 4"
so there shouldn't have been much memory pressure.

Do we have minimum build system specs?

Regardless of what default behaviour we pick, switching a build to be
tuned for a high/low end machine should be easy to do at
set-up time as Paul suggested. Embedding such limitations in
individual recipes and activating them by setting a single
variable in local.conf is appealing.

// Randy


> p.
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Re: [PATCH 1/1] webkit-gtk: fix 'Memory exhausted' error

[Khem Raj]

On Jul 22, 2013, at 1:22 AM, André Draszik <andre.draszik@linaro.org> wrote:

> Hi,
> 
> Instead of penalising everybody, would it be possible to make this
> change dependent on the build machine (ram + swap)? Or to only add
> -Wl,--no-keep-memory if the build failed without it?

penalty is not as much when you do parallel builds. However it will be good
to measure how much extra built time is needed when this options is active.
if its regressing the build time considerably then we need to make a choice.


> 
> Not sure if it would make any difference, but do you see this only with
> full debug enabled (-ggdb)? If yes, can the debug flags be changed, or
> that linker flag only be added, if full debug is enabled?
> 
> 
> Cheers,
> Andre'
> 
> On Mon, 2013-07-22 at 15:51 +0800, Kai Kang wrote:
>> Build webkit-gtk occupies lot of memory and fails on some host:
>> | ...tmp/sysroots/i686-linux/usr/libexec/x86_64-wrs-linux/gcc/x86_64-wrs-linux/4.8.1/ld:
>> failed to set dynamic section sizes: Memory exhausted
>> | collect2: error: ld returned 1 exit status
>> | make[1]: *** [libwebkitgtk-1.0.la] Error 1
>> 
>> Add ld option to fix it.
>> 
>> Refer to:
>> http://comments.gmane.org/gmane.os.opendarwin.webkit.devel/20078
>> 
>> Signed-off-by: Kai Kang <kai.kang@windriver.com>
>> ---
>> meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb | 2 ++
>> 1 file changed, 2 insertions(+)
>> 
>> diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> index 5691d3f..90625a3 100644
>> --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
>> @@ -60,6 +60,8 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
>> 
>> EXTRA_AUTORECONF = " -I Source/autotools "
>> 
>> +# Fix compile error: "ld: failed to set dynamic section sizes: Memory exhausted"
>> +LDFLAGS += " -Wl,--no-keep-memory"
>> 
>> #| ./Source/JavaScriptCore/heap/HandleTypes.h: In static member function 'static T* JSC::HandleTypes<T>::getFromSlot(JSC::HandleSlot) [with T = JSC::Structure, JSC::HandleTypes<T>::ExternalType = JSC::Structure*, JSC::HandleSlot = JSC::JSValue*]':
>> #| ./Source/JavaScriptCore/heap/Handle.h:141:79:   instantiated from 'JSC::Handle<T>::ExternalType JSC::Handle<T>::get() const [with T = JSC::Structure, JSC::Handle<T>::ExternalType = JSC::Structure*]'
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core