[PATCH] core-image-sato: qemumips use 512 mem

[PATCH] core-image-sato: qemumips use 512 mem

[akuster]

From: Armin Kuster <akuster808@gmail.comt>

Fixes:

WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password
WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
root
<<< run_serial(): command timed out after 60 seconds without output >>>

In another run, this error was seen:

Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 14.3M are free, but a safety buffer of 16.0M is enforced.

With updates to systemd and Qemu we should revert: 499a31cf06 core-image-sato: don't use 512MB in qemumips

Signed-off-by: Armin Kuster  <akuster808@gmail.com>
---
 meta/recipes-sato/images/core-image-sato.bb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-sato/images/core-image-sato.bb b/meta/recipes-sato/images/core-image-sato.bb
index 673106eb6d..878e0eaa8d 100644
--- a/meta/recipes-sato/images/core-image-sato.bb
+++ b/meta/recipes-sato/images/core-image-sato.bb
@@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-intltool nativesdk-glib-2.0"
 TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-intltool nativesdk-glib-2.0"
 
 QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
-QB_MEM_qemumips = "-m 256"
-- 
2.17.1

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Khem Raj]

On 9/12/20 8:59 PM, akuster wrote:
> From: Armin Kuster <akuster808@gmail.comt>
> 
> Fixes:
> 
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into serial console as root using blank password
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
> root
> <<< run_serial(): command timed out after 60 seconds without output >>>
> 
> In another run, this error was seen:
> 
> Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 14.3M are free, but a safety buffer of 16.0M is enforced.
> 
> With updates to systemd and Qemu we should revert: 499a31cf06 core-image-sato: don't use 512MB in qemumips
> 
> Signed-off-by: Armin Kuster  <akuster808@gmail.com>
> ---
>   meta/recipes-sato/images/core-image-sato.bb | 1 -
>   1 file changed, 1 deletion(-)
> 
> diff --git a/meta/recipes-sato/images/core-image-sato.bb b/meta/recipes-sato/images/core-image-sato.bb
> index 673106eb6d..878e0eaa8d 100644
> --- a/meta/recipes-sato/images/core-image-sato.bb
> +++ b/meta/recipes-sato/images/core-image-sato.bb
> @@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-intltool nativesdk-glib-2.0"
>   TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-intltool nativesdk-glib-2.0"
>   
>   QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m 512", "-m 256", d)}'
> -QB_MEM_qemumips = "-m 256"

Incidently, I was testing similar patch, which essentially is a revert 
of [1], I think we have to ensure that these issues seen by Ross a year 
ago are hopefully gone with time

[1] 
https://git.openembedded.org/openembedded-core/commit/?id=499a31cf06c934b40b5866d5eb77c9ad20c9a15f

> 
> 
> 
> 

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Richard Purdie]

On Sat, 2020-09-12 at 20:59 -0700, akuster wrote:
> From: Armin Kuster <akuster808@gmail.comt>
> 
> Fixes:
> 
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into
> serial console as root using blank password
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
> root
> <<< run_serial(): command timed out after 60 seconds without output
> >>>

I must admit I was sceptical but I reran my timing tests and this does
appear to significantly reduce the amount of time the serial login
takes for sato-sdk so you are probably right in claiming that.

Its a pretty bad 'advert' for systemd though :(.

> In another run, this error was seen:
> 
> Failed to reload daemon: Refusing to reload, not enough space
> available on /run/systemd. Currently, 14.3M are free, but a safety
> buffer of 16.0M is enforced.
> 
> With updates to systemd and Qemu we should revert: 499a31cf06 core-
> image-sato: don't use 512MB in qemumips

The original hard qemu hangs for qemumips were a real pain to debug so
I remain worried we're going to swap one set of problems for another :(

Cheers,

Richard

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Richard Purdie]

On Sat, 2020-09-12 at 20:59 -0700, akuster wrote:
> From: Armin Kuster <akuster808@gmail.comt>
> 
> Fixes:
> 
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into
> serial console as root using blank password
> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
> root
> <<< run_serial(): command timed out after 60 seconds without output
> >>>
> 
> In another run, this error was seen:
> 
> Failed to reload daemon: Refusing to reload, not enough space
> available on /run/systemd. Currently, 14.3M are free, but a safety
> buffer of 16.0M is enforced.
> 
> With updates to systemd and Qemu we should revert: 499a31cf06 core-
> image-sato: don't use 512MB in qemumips
> 
> Signed-off-by: Armin Kuster  <akuster808@gmail.com>
> ---
>  meta/recipes-sato/images/core-image-sato.bb | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/meta/recipes-sato/images/core-image-sato.bb
> b/meta/recipes-sato/images/core-image-sato.bb
> index 673106eb6d..878e0eaa8d 100644
> --- a/meta/recipes-sato/images/core-image-sato.bb
> +++ b/meta/recipes-sato/images/core-image-sato.bb
> @@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-intltool
> nativesdk-glib-2.0"
>  TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-
> intltool nativesdk-glib-2.0"
>  
>  QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m
> 512", "-m 256", d)}'
> -QB_MEM_qemumips = "-m 256"


We have now seen:

https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/2452

with this applied, not sure if its related. We have seen this kind of
network dropout elsewhere but we don't seem to have good debug when it
happens.

Cheers,

Richard

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Khem Raj]

On 9/14/20 12:13 AM, Richard Purdie wrote:
> On Sat, 2020-09-12 at 20:59 -0700, akuster wrote:
>> From: Armin Kuster <akuster808@gmail.comt>
>>
>> Fixes:
>>
>> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login into
>> serial console as root using blank password
>> WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
>> root
>> <<< run_serial(): command timed out after 60 seconds without output
>>>>>
>>
>> In another run, this error was seen:
>>
>> Failed to reload daemon: Refusing to reload, not enough space
>> available on /run/systemd. Currently, 14.3M are free, but a safety
>> buffer of 16.0M is enforced.
>>
>> With updates to systemd and Qemu we should revert: 499a31cf06 core-
>> image-sato: don't use 512MB in qemumips
>>
>> Signed-off-by: Armin Kuster  <akuster808@gmail.com>
>> ---
>>   meta/recipes-sato/images/core-image-sato.bb | 1 -
>>   1 file changed, 1 deletion(-)
>>
>> diff --git a/meta/recipes-sato/images/core-image-sato.bb
>> b/meta/recipes-sato/images/core-image-sato.bb
>> index 673106eb6d..878e0eaa8d 100644
>> --- a/meta/recipes-sato/images/core-image-sato.bb
>> +++ b/meta/recipes-sato/images/core-image-sato.bb
>> @@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-intltool
>> nativesdk-glib-2.0"
>>   TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-
>> intltool nativesdk-glib-2.0"
>>   
>>   QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m
>> 512", "-m 256", d)}'
>> -QB_MEM_qemumips = "-m 256"
> 
> 
> We have now seen:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/2452
> 
> with this applied, not sure if its related. We have seen this kind of
> network dropout elsewhere but we don't seem to have good debug when it
> happens.
> 

I wonder if ssh daemon is dead in this case. I also see its invoking 
compilers on target( kernelmodule case), are they usign -jX during make 
?, 512M might not be enough to run parallel compiles

> Cheers,
> 
> Richard
> 
> 
> 
> 

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Richard Purdie]

On Mon, 2020-09-14 at 10:27 -0700, Khem Raj wrote:
> 
> On 9/14/20 12:13 AM, Richard Purdie wrote:
> > On Sat, 2020-09-12 at 20:59 -0700, akuster wrote:
> > > From: Armin Kuster <akuster808@gmail.comt>
> > > 
> > > Fixes:
> > > 
> > > WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Couldn't login
> > > into
> > > serial console as root using blank password
> > > WARNING: core-image-sato-sdk-1.0-r0 do_testimage: The output:
> > > root
> > > <<< run_serial(): command timed out after 60 seconds without
> > > output
> > > 
> > > In another run, this error was seen:
> > > 
> > > Failed to reload daemon: Refusing to reload, not enough space
> > > available on /run/systemd. Currently, 14.3M are free, but a
> > > safety
> > > buffer of 16.0M is enforced.
> > > 
> > > With updates to systemd and Qemu we should revert: 499a31cf06
> > > core-
> > > image-sato: don't use 512MB in qemumips
> > > 
> > > Signed-off-by: Armin Kuster  <akuster808@gmail.com>
> > > ---
> > >   meta/recipes-sato/images/core-image-sato.bb | 1 -
> > >   1 file changed, 1 deletion(-)
> > > 
> > > diff --git a/meta/recipes-sato/images/core-image-sato.bb
> > > b/meta/recipes-sato/images/core-image-sato.bb
> > > index 673106eb6d..878e0eaa8d 100644
> > > --- a/meta/recipes-sato/images/core-image-sato.bb
> > > +++ b/meta/recipes-sato/images/core-image-sato.bb
> > > @@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-
> > > intltool
> > > nativesdk-glib-2.0"
> > >   TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-
> > > intltool nativesdk-glib-2.0"
> > >   
> > >   QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m
> > > 512", "-m 256", d)}'
> > > -QB_MEM_qemumips = "-m 256"
> > 
> > We have now seen:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/2452
> > 
> > with this applied, not sure if its related. We have seen this kind
> > of
> > network dropout elsewhere but we don't seem to have good debug when
> > it
> > happens.
> > 
> 
> I wonder if ssh daemon is dead in this case. I also see its invoking 
> compilers on target( kernelmodule case), are they usign -jX during
> make 
> ?, 512M might not be enough to run parallel compiles

Shouldn't be using -j for these tests. I think we need better debugging
using the serial console when these failures occur...

Cheers,

Richard

Re: [OE-core] [PATCH] core-image-sato: qemumips use 512 mem

[Richard Purdie]

On Mon, 2020-09-14 at 22:02 +0100, Richard Purdie via lists.openembedded.org wrote:
> On Mon, 2020-09-14 at 10:27 -0700, Khem Raj wrote:
> > On 9/14/20 12:13 AM, Richard Purdie wrote:
> > > On Sat, 2020-09-12 at 20:59 -0700, akuster wrote:
> > > > diff --git a/meta/recipes-sato/images/core-image-sato.bb
> > > > b/meta/recipes-sato/images/core-image-sato.bb
> > > > index 673106eb6d..878e0eaa8d 100644
> > > > --- a/meta/recipes-sato/images/core-image-sato.bb
> > > > +++ b/meta/recipes-sato/images/core-image-sato.bb
> > > > @@ -12,4 +12,3 @@ TOOLCHAIN_HOST_TASK_append = " nativesdk-
> > > > intltool
> > > > nativesdk-glib-2.0"
> > > >   TOOLCHAIN_HOST_TASK_remove_task-populate-sdk-ext = " nativesdk-
> > > > intltool nativesdk-glib-2.0"
> > > >   
> > > >   QB_MEM = '${@bb.utils.contains("DISTRO_FEATURES", "opengl", "-m
> > > > 512", "-m 256", d)}'
> > > > -QB_MEM_qemumips = "-m 256"
> > > 
> > > We have now seen:
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/2452
> > > 
> > > with this applied, not sure if its related. We have seen this kind
> > > of
> > > network dropout elsewhere but we don't seem to have good debug when
> > > it
> > > happens.
> > > 
> > 
> > I wonder if ssh daemon is dead in this case. I also see its invoking 
> > compilers on target( kernelmodule case), are they usign -jX during
> > make 
> > ?, 512M might not be enough to run parallel compiles
> 
> Shouldn't be using -j for these tests. I think we need better debugging
> using the serial console when these failures occur...

We now also have:

https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/2460

which is hanging in a dnf command.

From the autobuilder, I couldn't ssh into the image, it hangs but
tantalisingly, it would show me the time of the last login. I was
therefore able to run dmesg and confirm there is nothing in the logs
relating to this hang. I can't get a shell (bash or ash), top won't
run. Not sure what else to try but I am pretty sure this is our 512MB
memory bug, its still there. I'll leave it running for now in case
anyone has better ideas about debugging.

Cheers,

Richard