Yocto Project Status WW47’17

Yocto Project Status WW47’17

[Jolley, Stephen K]

[-- Attachment #1: Type: text/plain, Size: 2566 bytes --]

Current Dev Position: YP 2.5 Planning and M1 development

Next Deadline: YP 2.5 M1 cut off of 12/4/17


SWAT team rotation: Juro -> Paul on Nov.17, 2017.

SWAT team rotation: Paul -> Todor on Nov. 24, 2017.

https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team


Key Status/Updates:

·        There is no real change to the status from last week. We continue to suffer intermittent build failures and are continuing to attempt to debug these.

·        Currently open issues are:

o   qemuppc continues to demonstrate random hangs in boot  in userspace

o   Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps)

o   nfs inode count for the sstate/dldir share appears to break periodically causing the disk monitor to halt the builds (bug 12267)

o   a perf build race (bug 12302)

o   An ext filesystem creation/sizing issue (bug 12304)

·        Until we resolve these issues patches will continue to be slow to merge, if at all. This also blocks several core developers from doing any feature work at this point in time (e.g. layer setup tool is on hold, again).

·        We can only continue to stress that unless others step up and help to try and root cause these issues, things will stall with the project.


Planned upcoming dot releases:

YP 2.2.3 is planned and should be out in the next few weeks.

YP 2.3.3 is planned, but date TBD during YP 2.5 planning.

YP 2.4.1 is planned, but date TBD during YP 2.5 planning.


Key YP 2.5 Dates are:

YP 2.5 M1 cut off of 12/4/17

YP 2.5 M1 release of 12/15/17

YP 2.5 M2 cut off of 1/15/18

YP 2.5 M2 release of 1/26/18

YP 2.5 M3 cut off of 2/19/18

YP 2.5 M3 release of 3/2/18

YP 2.5 M4 cut off of 4/2/18

YP 2.5 M4 release of 4/27/18


Tracking Metrics:

            WDD 2655 (last week 2640)

(https://wiki.yoctoproject.org/charts/combo.html)


Key Status Links for YP:

https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status

https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule

https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features

[If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!]

Thanks,

Stephen K. Jolley
Yocto Project Program Manager
INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124
•   Work Telephone:        (503) 712-0534
•    Cell:                              (208) 244-4460
• Email:                            stephen.k.jolley@intel.com


[-- Attachment #2: Type: text/html, Size: 21304 bytes --]

Re: Yocto Project Status WW47’17

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 4007 bytes --]

On 2017-11-20 10:36 AM, Jolley, Stephen K wrote:
>
> Current Dev Position: YP 2.5 Planning and M1 development
>
> Next Deadline: YP 2.5 M1 cut off of 12/4/17
>
> SWAT team rotation: Juro -> Paul on Nov.17, 2017.
>
> SWAT team rotation: Paul -> Todor on Nov. 24, 2017.
>
> https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team
>
> Key Status/Updates:
>
> ·There is no real change to the status from last week. We continue to 
> suffer intermittent build failures and are continuing to attempt to 
> debug these.
>
> ·Currently open issues are:
>

Some US-based people may be on holiday later this week so I'm offering
help from the frozen Northland and more importantly from the team in 
Beijing. ;-)

> oqemuppc continues to demonstrate random hangs in boot  in userspace
>

Is we can create a defect for this and point / copy the wiki notes into 
it, that
would help.
    https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs

I think I had asked Chi to see if he could reproduce this a week or two ago.
When the lack of entropy problem was identified and fix, many people thought
this hang went away as well. Chi can you read the wiki report and see if you
can add anything to them?

> oIssues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed 
> (Suse) and Fedora 26 (attempting to see if 4.13.12 helps)
>

Robert, can you test Fedora 26. It would help to have a defect open with 
steps to reproduce or
something about the typical workflow/ build time/ day of the week/ phase 
of the moon.

> onfs inode count for the sstate/dldir share appears to break 
> periodically causing the disk monitor to halt the builds (bug 12267)
>

Likely specific to the AB server so no plans to do anything for this bug.

> oa perf build race (bug 12302)
>
  I'll take a look to:
  - see if I can duplicate the bug on a fast build host
  - check upstream to see if the bug is known/fixed
  - see if I can spot a race in the build rules.

> oAn ext filesystem creation/sizing issue (bug 12304)
>

Saul, Are you around this week? Do you have any additional information 
before
leaving for Thanksgiving?

Jackie,
Can you look at the code around the image creation and try to reproduce 
this one?

../Randy

> ·Until we resolve these issues patches will continue to be slow to 
> merge, if at all. This also blocks several core developers from doing 
> any feature work at this point in time (e.g. layer setup tool is on 
> hold, again).
>
> ·We can only continue to stress that unless others step up and help to 
> try and root cause these issues, things will stall with the project.
>
> Planned upcoming dot releases:
>
> YP 2.2.3 is planned and should be out in the next few weeks.
>
> YP 2.3.3 is planned, but date TBD during YP 2.5 planning.
>
> YP 2.4.1 is planned, but date TBD during YP 2.5 planning.
>
> Key YP 2.5 Dates are:
>
> YP 2.5 M1 cut off of 12/4/17
>
> YP 2.5 M1 release of 12/15/17
>
> YP 2.5 M2 cut off of 1/15/18
>
> YP 2.5 M2 release of 1/26/18
>
> YP 2.5 M3 cut off of 2/19/18
>
> YP 2.5 M3 release of 3/2/18
>
> YP 2.5 M4 cut off of 4/2/18
>
> YP 2.5 M4 release of 4/27/18
>
> Tracking Metrics:
>
> WDD 2655 (last week 2640)
>
> (https://wiki.yoctoproject.org/charts/combo.html)
>
> Key Status Links for YP:
>
> https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status
>
> https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule
>
> https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features
>
> [If anyone has suggestions for other information you’d like to see on 
> this weekly status update, let us know!]
>
> Thanks,
>
> */Stephen K. Jolley/**//*
>
> *Yocto Project Program Manager*
>
> *INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124 *
>
> (*Work Telephone*: (503) 712-0534
>
> (*Cell*: (208) 244-4460
>
> * *Email*:_stephen.k.jolley@intel.com_
>
>
>

-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company


[-- Attachment #2: Type: text/html, Size: 28345 bytes --]

Re: Yocto Project Status WW47’17

[Huang, Jie (Jackie)]

> 
> 
> From: MacLeod, Randy 
> Sent: Tuesday, November 21, 2017 09:22
> To: openembedded-core@lists.openembedded.org; Armin Kuster; Huang, Jie (Jackie); Xu, Chi; Yang, Liezhi; WOLD, SAUL
> Subject: Re: [OE-core] Yocto Project Status WW47’17
> 
> On 2017-11-20 10:36 AM, Jolley, Stephen K wrote:
> Current Dev Position: YP 2.5 Planning and M1 development
> Next Deadline: YP 2.5 M1 cut off of 12/4/17
>  
> SWAT team rotation: Juro -> Paul on Nov.17, 2017.
> SWAT team rotation: Paul -> Todor on Nov. 24, 2017.
> https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team
>  
> Key Status/Updates:
> ?        There is no real change to the status from last week. We continue to suffer intermittent build failures and are continuing to attempt to debug these.
> ?        Currently open issues are:
> 
> Some US-based people may be on holiday later this week so I'm offering 
> help from the frozen Northland and more importantly from the team in Beijing. ;-)
> 
> 
> o   qemuppc continues to demonstrate random hangs in boot  in userspace
> 
> Is we can create a defect for this and point / copy the wiki notes into it, that
> would help.
>    https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs
> 
> I think I had asked Chi to see if he could reproduce this a week or two ago.
> When the lack of entropy problem was identified and fix, many people thought
> this hang went away as well. Chi can you read the wiki report and see if you
> can add anything to them?
> 
> 
> o   Issues with 4.13.10 host kernels booting kvm x86 guests on Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12 helps)
> 
> Robert, can you test Fedora 26. It would help to have a defect open with steps to reproduce or
> something about the typical workflow/ build time/ day of the week/ phase of the moon. 
> 
> 
> o   nfs inode count for the sstate/dldir share appears to break periodically causing the disk monitor to halt the builds (bug 12267)
> 
> Likely specific to the AB server so no plans to do anything for this bug.
> 
> 
> o   a perf build race (bug 12302)
>  I'll take a look to:
>  - see if I can duplicate the bug on a fast build host
>  - check upstream to see if the bug is known/fixed
>  - see if I can spot a race in the build rules.
> 
> 
> o   An ext filesystem creation/sizing issue (bug 12304)
> 
> Saul, Are you around this week? Do you have any additional information before
> leaving for Thanksgiving?
> 
> Jackie, 
> Can you look at the code around the image creation and try to reproduce this one?

There is no steps to reproduce in this bug and it seems to be a rare issue, but yes, I can 
take a look at the code and info from yocto's autobuilder and try to reproduce it.

Thanks,
Jackie

> 
> ../Randy
> 
> 
> ?        Until we resolve these issues patches will continue to be slow to merge, if at all. This also blocks several core developers from doing any feature work at this point in time (e.g. layer setup tool is on hold, again).
> ?        We can only continue to stress that unless others step up and help to try and root cause these issues, things will stall with the project.
>  
> Planned upcoming dot releases:
> YP 2.2.3 is planned and should be out in the next few weeks.
> YP 2.3.3 is planned, but date TBD during YP 2.5 planning.
> YP 2.4.1 is planned, but date TBD during YP 2.5 planning.
>  
> Key YP 2.5 Dates are:
> YP 2.5 M1 cut off of 12/4/17
> YP 2.5 M1 release of 12/15/17
> YP 2.5 M2 cut off of 1/15/18
> YP 2.5 M2 release of 1/26/18
> YP 2.5 M3 cut off of 2/19/18
> YP 2.5 M3 release of 3/2/18
> YP 2.5 M4 cut off of 4/2/18
> YP 2.5 M4 release of 4/27/18
>  
> Tracking Metrics:
>             WDD 2655 (last week 2640)
>        (https://wiki.yoctoproject.org/charts/combo.html)
>  
> Key Status Links for YP:
> https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.5_Status
> https://wiki.yoctoproject.org/wiki/Yocto_2.5_Schedule
> https://wiki.yoctoproject.org/wiki/Yocto_2.5_Features
> [If anyone has suggestions for other information you’d like to see on this weekly status update, let us know!]
>  
> Thanks,
>  
> Stephen K. Jolley
> Yocto Project Program Manager
> INTEL, MS JF1-255, 2111 N.E. 25th Avenue, Hillsboro, OR 97124 
>     Work Telephone:        (503) 712-0534
> 4    Cell:                              (208) 244-4460
> 00Email:                            stephen.k.jolley@intel.com
>  
> 
> 
> 
> -- 
> # Randy MacLeod.  WR Linux
> # Wind River an Intel Company
>

Re: Yocto Project Status WW47’17

[Richard Purdie]

On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
> On 2017-11-20 10:36 AM, Jolley, Stephen K wrote:
> > Current Dev Position: YP 2.5 Planning and M1 development
> > Next Deadline: YP 2.5 M1 cut off of 12/4/17
> >  
> > SWAT team rotation: Juro -> Paul on Nov.17, 2017.
> > SWAT team rotation: Paul -> Todor on Nov. 24, 2017.
> > https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team
> >  
> > Key Status/Updates:
> > ·        There is no real change to the status from last week. We
> > continue to suffer intermittent build failures and are continuing
> > to attempt to debug these.
> > ·        Currently open issues are:
>  
> Some US-based people may be on holiday later this week so I'm
> offering 
> help from the frozen Northland and more importantly from the team in
> Beijing. ;-)
> 
> > o   qemuppc continues to demonstrate random hangs in boot  in
> > userspace
>  
> Is we can create a defect for this and point / copy the wiki notes
> into it, that
> would help.
>    https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs
> 
> I think I had asked Chi to see if he could reproduce this a week or
> two ago.
> When the lack of entropy problem was identified and fix, many people
> thought
> this hang went away as well. Chi can you read the wiki report and see
> if you
> can add anything to them?

Good news is that the qemuppc issue has been identified as a bug in
qemu ppc locking which breaks timer interrupt handling. I've posted on
the qemu mailing list asking for help in verifying what I think is
happening.

I have a patch ready to merge which should address this one, just
cleaning up my environment and doing some further stress testing.

[There is a defect somewhere for this btw, I created the wiki as it was
a better place to dump and update information as we learnt what it
is/is not without having to follow a train of thought updating in the
bugzilla].

> > o   Issues with 4.13.10 host kernels booting kvm x86 guests on
> > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > helps)
>  
> Robert, can you test Fedora 26. It would help to have a defect open
> with steps to reproduce or
> something about the typical workflow/ build time/ day of the week/
> phase of the moon. 

FWIW we have noticed that the choice of kernel timers seems to vary in
the x86_64 boots but not with a pattern that matches the hangs.

> > o   nfs inode count for the sstate/dldir share appears to break
> > periodically causing the disk monitor to halt the builds (bug
> > 12267)
>  
> Likely specific to the AB server so no plans to do anything for this
> bug.

Agreed, this one is our infrastructure somehow :(. We have a workaround
in -next for this at least.

> > o   a perf build race (bug 12302)
>   I'll take a look to:
>  - see if I can duplicate the bug on a fast build host
>  - check upstream to see if the bug is known/fixed
>  - see if I can spot a race in the build rules.

Sounds good, thanks!

> > o   An ext filesystem creation/sizing issue (bug 12304)
>  
> Saul, Are you around this week? Do you have any additional
> information before
> leaving for Thanksgiving?
> 
> Jackie, 
> Can you look at the code around the image creation and try to
> reproduce this one?

Saul hasn't been able to reproduce. I've asked at the minimum we add
better logging so if/when this happens again, we can debug it properly
next time. I did also wonder about fuzzing the image size code, writing
some code which puts in all possible input values and checks the sanity
of the resulting image size. Its the kind of problem a computer can
probably brute force. Anyone interested in trying that?

Cheers,

Richard

Re: Yocto Project Status WW47’17

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

On 21 November 2017 at 11:09, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> > > o   Issues with 4.13.10 host kernels booting kvm x86 guests on
> > > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > > helps)
> >
> > Robert, can you test Fedora 26. It would help to have a defect open
> > with steps to reproduce or
> > something about the typical workflow/ build time/ day of the week/
> > phase of the moon.
>
> FWIW we have noticed that the choice of kernel timers seems to vary in
> the x86_64 boots but not with a pattern that matches the hangs.



Also of note is that last week the cluster was hitting this once in every
nightly run, but after a complete reboot and kernel upgrade we haven't seen
it so far.  So either it's gone with the latest kernels, or it's a slow
accumulation of non-movable memory that eventually breaks kvm (my money is
on the latter).


> > > o   a perf build race (bug 12302)
> >   I'll take a look to:
> >  - see if I can duplicate the bug on a fast build host
> >  - check upstream to see if the bug is known/fixed
> >  - see if I can spot a race in the build rules.
>
> Sounds good, thanks!


It looks and sounds like a classic makefile dependency bug, should be
reproducible on demand with careful explicit target steps.  The perf
makefiles are not exactly easy to hack though...

Ross

[-- Attachment #2: Type: text/html, Size: 2131 bytes --]

Re: Yocto Project Status WW47’17

[Wold, Saul]

On Tue, 2017-11-21 at 11:09 +0000, Richard Purdie wrote:
> On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
> > On 2017-11-20 10:36 AM, Jolley, Stephen K wrote:
> > > Current Dev Position: YP 2.5 Planning and M1 development
> > > Next Deadline: YP 2.5 M1 cut off of 12/4/17
> > >  
> > > SWAT team rotation: Juro -> Paul on Nov.17, 2017.
> > > SWAT team rotation: Paul -> Todor on Nov. 24, 2017.
> > > https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team
> > >  
> > > Key Status/Updates:
> > > ·        There is no real change to the status from last week. We
> > > continue to suffer intermittent build failures and are continuing
> > > to attempt to debug these.
> > > ·        Currently open issues are:
> > 
> >  
> > Some US-based people may be on holiday later this week so I'm
> > offering 
> > help from the frozen Northland and more importantly from the team
> > in
> > Beijing. ;-)
> > 
> > > o   qemuppc continues to demonstrate random hangs in boot  in
> > > userspace
> > 
> >  
> > Is we can create a defect for this and point / copy the wiki notes
> > into it, that
> > would help.
> >    https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs
> > 
> > I think I had asked Chi to see if he could reproduce this a week or
> > two ago.
> > When the lack of entropy problem was identified and fix, many
> > people
> > thought
> > this hang went away as well. Chi can you read the wiki report and
> > see
> > if you
> > can add anything to them?
> 
> Good news is that the qemuppc issue has been identified as a bug in
> qemu ppc locking which breaks timer interrupt handling. I've posted
> on
> the qemu mailing list asking for help in verifying what I think is
> happening.
> 
> I have a patch ready to merge which should address this one, just
> cleaning up my environment and doing some further stress testing.
> 
This is great news hopefully you will hear back from the qemu ML
verifying your patch.

> [There is a defect somewhere for this btw, I created the wiki as it
> was
> a better place to dump and update information as we learnt what it
> is/is not without having to follow a train of thought updating in the
> bugzilla].
> 
> > > o   Issues with 4.13.10 host kernels booting kvm x86 guests on
> > > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > > helps)
> > 
> >  
> > Robert, can you test Fedora 26. It would help to have a defect open
> > with steps to reproduce or
> > something about the typical workflow/ build time/ day of the week/
> > phase of the moon. 
> 
> FWIW we have noticed that the choice of kernel timers seems to vary
> in
> the x86_64 boots but not with a pattern that matches the hangs.
> 
> > > o   nfs inode count for the sstate/dldir share appears to break
> > > periodically causing the disk monitor to halt the builds (bug
> > > 12267)
> > 
> >  
> > Likely specific to the AB server so no plans to do anything for
> > this
> > bug.
> 
> Agreed, this one is our infrastructure somehow :(. We have a
> workaround
> in -next for this at least.
> 
> > > o   a perf build race (bug 12302)
> > 
> >   I'll take a look to:
> >  - see if I can duplicate the bug on a fast build host
> >  - check upstream to see if the bug is known/fixed
> >  - see if I can spot a race in the build rules.
> 
> Sounds good, thanks!
> 
> > > o   An ext filesystem creation/sizing issue (bug 12304)
> > 
> >  
> > Saul, Are you around this week? Do you have any additional
> > information before
> > leaving for Thanksgiving?
> > 
> > Jackie, 
> > Can you look at the code around the image creation and try to
> > reproduce this one?
> 
> Saul hasn't been able to reproduce. I've asked at the minimum we add
> better logging so if/when this happens again, we can debug it
> properly

Patch sent which provides some additional debugging information to the
log files, ideally, this will be saved with the bug next time this
issue occurs.

Sau!

> next time. I did also wonder about fuzzing the image size code,
> writing
> some code which puts in all possible input values and checks the
> sanity
> of the resulting image size. Its the kind of problem a computer can
> probably brute force. Anyone interested in trying that?
> 
> Cheers,
> 
> Richard
> 
> 

Re: Yocto Project Status WW47’17

[Robert Yang]

On 11/21/2017 07:09 PM, Richard Purdie wrote:
> On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
>> On 2017-11-20 10:36 AM, Jolley, Stephen K wrote:
>>> Current Dev Position: YP 2.5 Planning and M1 development
>>> Next Deadline: YP 2.5 M1 cut off of 12/4/17
>>>   
>>> SWAT team rotation: Juro -> Paul on Nov.17, 2017.
>>> SWAT team rotation: Paul -> Todor on Nov. 24, 2017.
>>> https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team
>>>   
>>> Key Status/Updates:
>>> ·        There is no real change to the status from last week. We
>>> continue to suffer intermittent build failures and are continuing
>>> to attempt to debug these.
>>> ·        Currently open issues are:
>>   
>> Some US-based people may be on holiday later this week so I'm
>> offering
>> help from the frozen Northland and more importantly from the team in
>> Beijing. ;-)
>>
>>> o   qemuppc continues to demonstrate random hangs in boot  in
>>> userspace
>>   
>> Is we can create a defect for this and point / copy the wiki notes
>> into it, that
>> would help.
>>     https://wiki.yoctoproject.org/wiki/Qemuppc_Boot_Hangs
>>
>> I think I had asked Chi to see if he could reproduce this a week or
>> two ago.
>> When the lack of entropy problem was identified and fix, many people
>> thought
>> this hang went away as well. Chi can you read the wiki report and see
>> if you
>> can add anything to them?
> 
> Good news is that the qemuppc issue has been identified as a bug in
> qemu ppc locking which breaks timer interrupt handling. I've posted on
> the qemu mailing list asking for help in verifying what I think is
> happening.
> 
> I have a patch ready to merge which should address this one, just
> cleaning up my environment and doing some further stress testing.
> 
> [There is a defect somewhere for this btw, I created the wiki as it was
> a better place to dump and update information as we learnt what it
> is/is not without having to follow a train of thought updating in the
> bugzilla].
> 
>>> o   Issues with 4.13.10 host kernels booting kvm x86 guests on
>>> Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
>>> helps)
>>   
>> Robert, can you test Fedora 26. It would help to have a defect open
>> with steps to reproduce or
>> something about the typical workflow/ build time/ day of the week/
>> phase of the moon.
> 
> FWIW we have noticed that the choice of kernel timers seems to vary in
> the x86_64 boots but not with a pattern that matches the hangs.
> 
>>> o   nfs inode count for the sstate/dldir share appears to break
>>> periodically causing the disk monitor to halt the builds (bug
>>> 12267)
>>   
>> Likely specific to the AB server so no plans to do anything for this
>> bug.
> 
> Agreed, this one is our infrastructure somehow :(. We have a workaround
> in -next for this at least.
> 
>>> o   a perf build race (bug 12302)
>>    I'll take a look to:
>>   - see if I can duplicate the bug on a fast build host
>>   - check upstream to see if the bug is known/fixed
>>   - see if I can spot a race in the build rules.
> 
> Sounds good, thanks!
> 
>>> o   An ext filesystem creation/sizing issue (bug 12304)
>>   
>> Saul, Are you around this week? Do you have any additional
>> information before
>> leaving for Thanksgiving?
>>
>> Jackie,
>> Can you look at the code around the image creation and try to
>> reproduce this one?
> 
> Saul hasn't been able to reproduce. I've asked at the minimum we add
> better logging so if/when this happens again, we can debug it properly
> next time. I did also wonder about fuzzing the image size code, writing
> some code which puts in all possible input values and checks the sanity
> of the resulting image size. Its the kind of problem a computer can
> probably brute force. Anyone interested in trying that?

I have interests on it, but I don't quite understand what did you mean.
Currently, the image size is from "du -sh <image_rootfs>" * factor, about
the bug 12304, it got 8192 which is the default value of core-image-minimal,
so maybe something is wrong with "du -sh", the normal value is larger than 8192.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: Yocto Project Status WW47’17

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
> > o   Issues with 4.13.10 host kernels booting kvm x86 guests on
> > Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
> > helps)
>  
> Robert, can you test Fedora 26. It would help to have a defect open
> with steps to reproduce or
> something about the typical workflow/ build time/ day of the week/
> phase of the moon. 

We have some further data:

a) The issue occurs on 4.13.12

rpurdie@tumbleweed:/tmp> cat /proc/version 
Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 UTC 2017 (9151c66)

b) The hang usually occurs at the TIMER line in the kernel logs but can
occur after booting into userspace around the udevd line, or
occasionally later in the boot process.

c) The similarity between this and the ppc bug I worked on make me
strongly suspect qemu's timers are stopping firing and the guest is
sitting in the idle loop.

d) I do now have a way to brute force the hangs at will. The attached
"runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
had 10 hung on the autobuilder. I can provide more info on using that
script if its not obvious. It does assume my recent master changes to
the qemuconf files so we don't need to run bitbake -e to run runqemu.

This could well be the same kind of locking issue we saw on ppc. I'll
continue to look into that.

Hopefully this extra information will put us on a good track to
resolving it now. It is continuing to break builds and stop patch
merging.

Cheers,

Richard

[-- Attachment #2: runqemu-parallel.py --]
[-- Type: text/x-python, Size: 2190 bytes --]

#!/usr/bin/env python3

import shutil
import time
import subprocess
import os

procs = list(range(1, 50))

logs = "/tmp/runqemu-log"
errlogs = "/tmp/zrunqemu-log"
logfds = {}
errlogfds = {}
processes = {}
image = "/media/build1/poky/build/tmp-musl/deploy/images/qemuppc/core-image-sato-qemuppc.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemuppc/vmlinux-qemuppc.bin"
machine = "qemuppc"

image = "/media/build1/poky/build/tmp-musl/deploy/images/qemux86-64/core-image-sato-qemux86-64.qemuboot.conf"
kernel = "tmp-musl/deploy/images/qemux86-64/bzImage-qemux86-64.bin"
machine = "qemux86-64"

env = os.environ.copy()
env['MACHINE'] = machine

def start(p):
    logfds[p] = open(logs + ".%d" % p, "w")
    errlogfds[p] = open(errlogs + ".%d" % p, "w")
    #debugopts = "-d unimp,guest_errors,int,out_asm,op,op_opt,op_ind,mmu,pcall,cpu_reset,nochain"
    #exec,cpu,in_asm,
    debugopts = "-d unimp,guest_errors,int"
    debugopts += " -D /tmp/qemu.%d -monitor pty" % p
    processes[p] = subprocess.Popen(["runqemu",machine,"nographic", "snapshot", "kvm", kernel, image, "qemuparams=-pidfile /tmp/zzqemu.%d.pid %s" % (p, debugopts)], stdout=logfds[p], stderr=errlogfds[p], env=env)
    print("Started %d" % p)

for p in procs:
    start(p)

lastsize = {}

while procs:
    sizes = {}
    time.sleep(0.5)

    for p in procs:
        logfile = logs + ".%d" % p
        pidfn = "/tmp/zzqemu.%d.pid" % p
        with open(logfile, "r") as lf:
            for l in lf:
                if "login:" in l:
                    with open(pidfn) as pidfile:
                        pid = int(pidfile.readline().strip())
                    print("Killing %d with pid %d" % (p, pid))
                    os.kill(pid, 9)
                    procs.remove(p)
                    if p in lastsize:
                        del lastsize[p]
                    time.sleep(0.1)
                    logfds[p].close()
                    errlogfds[p].close()
                    start(p)
                    procs.append(p)
        sizes[p] = os.stat(logfile).st_size
        if p in lastsize and sizes[p]:
           if not sizes[p] > lastsize[p]:
               print("%d stalled?" % p)
    lastsize = sizes

Re: Yocto Project Status WW47’17

[Robert Yang]

On 11/23/2017 06:20 PM, Richard Purdie wrote:
> On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
>>> o   Issues with 4.13.10 host kernels booting kvm x86 guests on
>>> Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
>>> helps)
>>   
>> Robert, can you test Fedora 26. It would help to have a defect open
>> with steps to reproduce or
>> something about the typical workflow/ build time/ day of the week/
>> phase of the moon.
> 
> We have some further data:
> 
> a) The issue occurs on 4.13.12
> 
> rpurdie@tumbleweed:/tmp> cat /proc/version
> Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 UTC 2017 (9151c66)
> 
> b) The hang usually occurs at the TIMER line in the kernel logs but can
> occur after booting into userspace around the udevd line, or
> occasionally later in the boot process.
> 
> c) The similarity between this and the ppc bug I worked on make me
> strongly suspect qemu's timers are stopping firing and the guest is
> sitting in the idle loop.
> 
> d) I do now have a way to brute force the hangs at will. The attached
> "runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
> had 10 hung on the autobuilder. I can provide more info on using that
> script if its not obvious. It does assume my recent master changes to
> the qemuconf files so we don't need to run bitbake -e to run runqemu.
> 
> This could well be the same kind of locking issue we saw on ppc. I'll
> continue to look into that.
> 
> Hopefully this extra information will put us on a good track to
> resolving it now. It is continuing to break builds and stop patch
> merging.

I modified the script to run on my host (I used qemux86, not qemux86-64,
the later one is still in building) (Up to date Fedora 26, the kernel
is 4.13.13):

$ cat /proc/version
Linux version 4.13.13-200.fc26.x86_64 
(mockbuild@bkernel01.phx2.fedoraproject.org) (gcc version 7.2.1 20170915 (Red 
Hat 7.2.1-2) (GCC)) #1 SMP Wed Nov 15 15:46:36 UTC 2017

But didn't see any hangs after 10 minutes, all the qemus reached login. So maybe
it had been fixed on 4.13.13 ?

// Robert

> 
> Cheers,
> 
> Richard
> 

Re: Yocto Project Status WW47’17

[Robert Yang]

On 11/28/2017 03:49 PM, Robert Yang wrote:
> 
> On 11/23/2017 06:20 PM, Richard Purdie wrote:
>> On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
>>>> o   Issues with 4.13.10 host kernels booting kvm x86 guests on
>>>> Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
>>>> helps)
>>> Robert, can you test Fedora 26. It would help to have a defect open
>>> with steps to reproduce or
>>> something about the typical workflow/ build time/ day of the week/
>>> phase of the moon.
>>
>> We have some further data:
>>
>> a) The issue occurs on 4.13.12
>>
>> rpurdie@tumbleweed:/tmp> cat /proc/version
>> Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 
>> [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 11:21:09 
>> UTC 2017 (9151c66)
>>
>> b) The hang usually occurs at the TIMER line in the kernel logs but can
>> occur after booting into userspace around the udevd line, or
>> occasionally later in the boot process.
>>
>> c) The similarity between this and the ppc bug I worked on make me
>> strongly suspect qemu's timers are stopping firing and the guest is
>> sitting in the idle loop.
>>
>> d) I do now have a way to brute force the hangs at will. The attached
>> "runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
>> had 10 hung on the autobuilder. I can provide more info on using that
>> script if its not obvious. It does assume my recent master changes to
>> the qemuconf files so we don't need to run bitbake -e to run runqemu.
>>
>> This could well be the same kind of locking issue we saw on ppc. I'll
>> continue to look into that.
>>
>> Hopefully this extra information will put us on a good track to
>> resolving it now. It is continuing to break builds and stop patch
>> merging.
> 
> I modified the script to run on my host (I used qemux86, not qemux86-64,
> the later one is still in building) (Up to date Fedora 26, the kernel
> is 4.13.13):
> 
> $ cat /proc/version
> Linux version 4.13.13-200.fc26.x86_64 
> (mockbuild@bkernel01.phx2.fedoraproject.org) (gcc version 7.2.1 20170915 (Red 
> Hat 7.2.1-2) (GCC)) #1 SMP Wed Nov 15 15:46:36 UTC 2017
> 
> But didn't see any hangs after 10 minutes, all the qemus reached login. So maybe
> it had been fixed on 4.13.13 ?

I tested qemux86-64, the boot became very slow (just like hang), no one booted
after about 2 hours (still in booting). And it only happened when
use qemux86-64 + kvm:

# Works
$ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot

# Doesn't work, like hang:
$ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot kvm

// Robert

> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>

Good news Re: Yocto Project Status WW47’17

[Robert Yang]

Hi RP,

On 11/28/2017 06:47 PM, Robert Yang wrote:
> 
> 
> On 11/28/2017 03:49 PM, Robert Yang wrote:
>>
>> On 11/23/2017 06:20 PM, Richard Purdie wrote:
>>> On Mon, 2017-11-20 at 20:22 -0500, Randy MacLeod wrote:
>>>>> o   Issues with 4.13.10 host kernels booting kvm x86 guests on
>>>>> Tumbleweed (Suse) and Fedora 26 (attempting to see if 4.13.12
>>>>> helps)
>>>> Robert, can you test Fedora 26. It would help to have a defect open
>>>> with steps to reproduce or
>>>> something about the typical workflow/ build time/ day of the week/
>>>> phase of the moon.
>>>
>>> We have some further data:
>>>
>>> a) The issue occurs on 4.13.12
>>>
>>> rpurdie@tumbleweed:/tmp> cat /proc/version
>>> Linux version 4.13.12-1-vanilla (geeko@buildhost) (gcc version 7.2.1 20171020 
>>> [gcc-7-branch revision 253932] (SUSE Linux)) #1 SMP PREEMPT Wed Nov 8 
>>> 11:21:09 UTC 2017 (9151c66)
>>>
>>> b) The hang usually occurs at the TIMER line in the kernel logs but can
>>> occur after booting into userspace around the udevd line, or
>>> occasionally later in the boot process.
>>>
>>> c) The similarity between this and the ppc bug I worked on make me
>>> strongly suspect qemu's timers are stopping firing and the guest is
>>> sitting in the idle loop.
>>>
>>> d) I do now have a way to brute force the hangs at will. The attached
>>> "runqemu-parallel.py" script runs 50 qemus in parallel. In around 45s I
>>> had 10 hung on the autobuilder. I can provide more info on using that
>>> script if its not obvious. It does assume my recent master changes to
>>> the qemuconf files so we don't need to run bitbake -e to run runqemu.
>>>
>>> This could well be the same kind of locking issue we saw on ppc. I'll
>>> continue to look into that.
>>>
>>> Hopefully this extra information will put us on a good track to
>>> resolving it now. It is continuing to break builds and stop patch
>>> merging.
>>
>> I modified the script to run on my host (I used qemux86, not qemux86-64,
>> the later one is still in building) (Up to date Fedora 26, the kernel
>> is 4.13.13):
>>
>> $ cat /proc/version
>> Linux version 4.13.13-200.fc26.x86_64 
>> (mockbuild@bkernel01.phx2.fedoraproject.org) (gcc version 7.2.1 20170915 (Red 
>> Hat 7.2.1-2) (GCC)) #1 SMP Wed Nov 15 15:46:36 UTC 2017
>>
>> But didn't see any hangs after 10 minutes, all the qemus reached login. So maybe
>> it had been fixed on 4.13.13 ?
> 
> I tested qemux86-64, the boot became very slow (just like hang), no one booted
> after about 2 hours (still in booting). And it only happened when
> use qemux86-64 + kvm:
> 
> # Works
> $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot
> 
> # Doesn't work, like hang:
> $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot kvm

I upgraded qemu-native to qemu-2.11.0-rc2, then no hangs any more. I dropped
the conflicted patches during upgrade. I will send a formal upgrade tomorrow
it this is acceptable.

Here is the draft patch for testing:

   git://git.openembedded.org/openembedded-core-contrib rbt/qemu_test
   http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/qemu_test

Robert Yang (1):
   qemu: 2.10.1 -> 2.11.0-rc2 (test)

// Robert

> 
> // Robert
> 
>>
>> // Robert
>>
>>>
>>> Cheers,
>>>
>>> Richard
>>>

Re: Good news Re: Yocto Project Status WW47’17

[Richard Purdie]

On Tue, 2017-11-28 at 20:48 +0800, Robert Yang wrote:
> Hi RP,
> On 11/28/2017 06:47 PM, Robert Yang wrote:
> > # Works
> > $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot
> > 
> > # Doesn't work, like hang:
> > $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot kvm
> I upgraded qemu-native to qemu-2.11.0-rc2, then no hangs any more. I
> dropped
> the conflicted patches during upgrade. I will send a formal upgrade
> tomorrow
> it this is acceptable.
> 
> Here is the draft patch for testing:
> 
>    git://git.openembedded.org/openembedded-core-contrib rbt/qemu_test
>    http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/
> qemu_test

Thanks for the data point. I've been putting my data into:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301

As you can see there, I think there are actually two issues:

a) A boot hang around the ..TIMER line where the guest isn't getting
timer interupts

b) Something which causes the APIC to see "version 0".

I applied the 2.11-rc2 upgrade and it seemed to stop the hangs I was
seeing with "-no-kvm-irqchip" but doesn't stop the ones I see without
that.

I am seeing hangs in guests where the version 0 message appears as well
as where we see 0x14. The 0x14 versions do boot further into userspace
usually.

It also seems host hardware specific as I cannot get my local build
machine to hang as yet. That could mean its timing or could be cpu
revision, I've removed many but not all OS differences :/.

Cheers,

Richard

Re: Good news Re: Yocto Project Status WW47’17

[Robert Yang]

On 11/28/2017 09:15 PM, Richard Purdie wrote:
> On Tue, 2017-11-28 at 20:48 +0800, Robert Yang wrote:
>> Hi RP,
>> On 11/28/2017 06:47 PM, Robert Yang wrote:
>>> # Works
>>> $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot
>>>
>>> # Doesn't work, like hang:
>>> $ runqemu tmp/deploy/images/qemux86-64/ nographic snapshot kvm
>> I upgraded qemu-native to qemu-2.11.0-rc2, then no hangs any more. I
>> dropped
>> the conflicted patches during upgrade. I will send a formal upgrade
>> tomorrow
>> it this is acceptable.
>>
>> Here is the draft patch for testing:
>>
>>     git://git.openembedded.org/openembedded-core-contrib rbt/qemu_test
>>     http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/
>> qemu_test
> 
> Thanks for the data point. I've been putting my data into:
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
> 
> As you can see there, I think there are actually two issues:
> 
> a) A boot hang around the ..TIMER line where the guest isn't getting
> timer interupts
> 
> b) Something which causes the APIC to see "version 0".
> 
> I applied the 2.11-rc2 upgrade and it seemed to stop the hangs I was
> seeing with "-no-kvm-irqchip" but doesn't stop the ones I see without
> that.

Maybe you can try to reboot the host ? It didn't work when I first
upgrade to 2.11.0-rc2, and it worked after I reboot the machine.
My host is fedora 26 server which is running in qemu-kvm, what I did was:

- Upgrade to 2.11.0-rc2
- $ bitbake qemu-helper-native
- Reboot fedora 26
- run runqemu-parallel with qemux86-64
No hangs
- Back to qemu 2.10.1
- $ bitbake qemu-helper-native
- run runqemu-parallel
Hangs
- Upgrade to 2.11.0-rc2 again, no hangs

// Robert

> 
> I am seeing hangs in guests where the version 0 message appears as well
> as where we see 0x14. The 0x14 versions do boot further into userspace
> usually.
> 
> It also seems host hardware specific as I cannot get my local build
> machine to hang as yet. That could mean its timing or could be cpu
> revision, I've removed many but not all OS differences :/.
> 
> Cheers,
> 
> Richard
>