Yocto Technical Team Minutes, Engineering Sync, for July 28 2020

Yocto Technical Team Minutes, Engineering Sync, for July 28 2020

[Trevor Woerner]

Yocto Technical Team Minutes, Engineering Sync, for July 28 2020
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolly, Armin Kuster, Josef Holzmayr, Richard
Purdie, Trevor Gamblin, Joshua Watt, Mark Morton, Ross Burton, Bruce
Ashfield, Steve Sakoman, (phone-in ??), Jon Mason, Randy MacLeod, Scott
Murray, Denys Dmytriyenko

== notes ==
- fixed a race on AB wrt perl
- 3.1.2 into QA
- 3.2-m2 to QA once they’re done with 3.1.2

== general ==
RP: load of perl issues over the weekend, believe to have found issue
RP: lots of qemumips issues, not sure why, perhaps a timeouts issue? (no ping
    within 30 seconds)
RP: some infrastructure issues (one worker has become corrupted)

RP: curious to know what people think about 3.1.2: there is a bitbake issue
    with toaster, should we release then fix or wait for fix before release?
SS: maybe we should get feedback from people using toaster
RP: the fix is simple, maybe release 3.1.2 (with release note) then fix

RP: merged re-arrangement of package manager code, this puts more of our code
    into python libraries instead of bbclass files, which is the direction i
    want to take going forward
RP: i need to start a conversation on oe-architecture for future directions
RP: e.g. binary package feeds, we want to capture these use-cases into the
    wiki

[NB: done! see https://wiki.yoctoproject.org/wiki/Future_Directions]

RP: we also posted the current agreement on “inclusive language” as agreed
    on by various groups (oe board, oe-tsc, yp-tsc)

TW: what’s the thinking behind moving code to libraries vs bbclass?
RP: pro: parsing speed con: variable dependency tracking is lost
RP: lots of python code gets written to logs etc, which slows it down
JPEW: does bitbake automatically detect variable dependencies in library code?
RP: no. maybe we just need to do the scan once?
TW: do we lose flexibility? (e.g. it’s easy to copy+paste+modify a bbclass,
    will that be possible?)
RP: there will still be all the same entry points, so it should be possible
RP: i’m worried about the size of the datastore in memory and it is a
    bottleneck

RP: i wonder why qemumips is so slow?
Randy: i think Cisco are the only ones caring about mips
RP: yes, and Comcast too i think

MM: have been wrapped up in $WORK stuff, but will be working on docs
    conversion soon
RP: MH has been putting infrastructure in place and Nico is looking forward to
    getting this going
MM: do we want to try to reproduce our current colours/formatting or try
    something new
RP: i like what we have, the project does have an existing scheme, but we can
    change if something else makes sense

TW: i do lots of builds, never seen issues, why are there so many failures on
    AB?
RP: do you do oe-selftest or testsdk builds?
TW: no
RP: do you do mips builds?
TW: no
RP: so these are the areas where we see these failures, “regular” builds
    are okay
SS: i don’t see issues on my builds but that’s probably because of working
    these issues out on the AB
RP: infrastructure issues are low, it’s mostly race issues that come out of
    the extreme load of the AB or intermittent architecture issues that are
    rarely tested

Re: [yocto] Yocto Technical Team Minutes, Engineering Sync, for July 28 2020

[Nicolas Dechesne]

On Wed, Jul 29, 2020 at 12:42 AM Trevor Woerner <twoerner@gmail.com> wrote:
>
> Yocto Technical Team Minutes, Engineering Sync, for July 28 2020
> archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit
>
> == disclaimer ==
> Best efforts are made to ensure the below is accurate and valid. However,
> errors sometimes happen. If any errors or omissions are found, please feel
> free to reply to this email with any corrections.
>
> == attendees ==
> Trevor Woerner, Stephen Jolly, Armin Kuster, Josef Holzmayr, Richard
> Purdie, Trevor Gamblin, Joshua Watt, Mark Morton, Ross Burton, Bruce
> Ashfield, Steve Sakoman, (phone-in ??), Jon Mason, Randy MacLeod, Scott
> Murray, Denys Dmytriyenko
>
> == notes ==
> - fixed a race on AB wrt perl
> - 3.1.2 into QA
> - 3.2-m2 to QA once they’re done with 3.1.2
>
> == general ==
> RP: load of perl issues over the weekend, believe to have found issue
> RP: lots of qemumips issues, not sure why, perhaps a timeouts issue? (no ping
>     within 30 seconds)
> RP: some infrastructure issues (one worker has become corrupted)
>
> RP: curious to know what people think about 3.1.2: there is a bitbake issue
>     with toaster, should we release then fix or wait for fix before release?
> SS: maybe we should get feedback from people using toaster
> RP: the fix is simple, maybe release 3.1.2 (with release note) then fix
>
> RP: merged re-arrangement of package manager code, this puts more of our code
>     into python libraries instead of bbclass files, which is the direction i
>     want to take going forward
> RP: i need to start a conversation on oe-architecture for future directions
> RP: e.g. binary package feeds, we want to capture these use-cases into the
>     wiki
>
> [NB: done! see https://wiki.yoctoproject.org/wiki/Future_Directions]
>
> RP: we also posted the current agreement on “inclusive language” as agreed
>     on by various groups (oe board, oe-tsc, yp-tsc)
>
> TW: what’s the thinking behind moving code to libraries vs bbclass?
> RP: pro: parsing speed con: variable dependency tracking is lost
> RP: lots of python code gets written to logs etc, which slows it down
> JPEW: does bitbake automatically detect variable dependencies in library code?
> RP: no. maybe we just need to do the scan once?
> TW: do we lose flexibility? (e.g. it’s easy to copy+paste+modify a bbclass,
>     will that be possible?)
> RP: there will still be all the same entry points, so it should be possible
> RP: i’m worried about the size of the datastore in memory and it is a
>     bottleneck
>
> RP: i wonder why qemumips is so slow?
> Randy: i think Cisco are the only ones caring about mips
> RP: yes, and Comcast too i think
>
> MM: have been wrapped up in $WORK stuff, but will be working on docs
>     conversion soon
> RP: MH has been putting infrastructure in place and Nico is looking forward to
>     getting this going
> MM: do we want to try to reproduce our current colours/formatting or try
>     something new
> RP: i like what we have, the project does have an existing scheme, but we can
>     change if something else makes sense

I have some updates on Sphinx..
I recall that the work in progress can be found here:
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx

The automated conversion from DocBook to Sphinx is managing 'internal'
links quite badly, so it results in a significant amount of work to
fixup links. Also since I wanted to use the Sphinx builtin 'glossary'
for terms and variables in the reference manual, that also required me
to update how links work for terms and variables. That's why it's
taking more time than I would like. I manage to use some
automation/regexp, and do a bit of manual fix up as well.

I also started to experiment with customization in the CSS theme, to
change some color/formatting. It's not done yet, but it has got
better:
https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-manual.html

For now we generate a single large documentation 'website' that
includes all existing docs, instead of publishing each document
separately. I think it's an important change compared to how our
documentation was published so far. I personally like that quite a
bit, and think it's 'better' this way, but I would really want to hear
from you and that we agree this is what we want for the project. We
still need to look at how we want to generate PDF/offline
documentation though. And we need to decide how to manage the bitbake
manual which sits in the bitbake git tree, so right now it's published
separately.

Side note: the VSCode extension for reST is quite good
(https://docs.restructuredtext.net/index.html)

>
> TW: i do lots of builds, never seen issues, why are there so many failures on
>     AB?
> RP: do you do oe-selftest or testsdk builds?
> TW: no
> RP: do you do mips builds?
> TW: no
> RP: so these are the areas where we see these failures, “regular” builds
>     are okay
> SS: i don’t see issues on my builds but that’s probably because of working
>     these issues out on the AB
> RP: infrastructure issues are low, it’s mostly race issues that come out of
>     the extreme load of the AB or intermittent architecture issues that are
>     rarely tested
> 

Re: [docs] [yocto] Yocto Technical Team Minutes, Engineering Sync, for July 28 2020

[Mark Morton]

> -----Original Message-----
> From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf
> Of Nicolas Dechesne
> Sent: Wednesday, July 29, 2020 12:37 AM
> To: Trevor Woerner <twoerner@gmail.com>
> Cc: docs@lists.yoctoproject.org; Morton, Mark
> <Mark.Morton@windriver.com>
> Subject: Re: [docs] [yocto] Yocto Technical Team Minutes, Engineering Sync,
> for July 28 2020
> 
> On Wed, Jul 29, 2020 at 12:42 AM Trevor Woerner <twoerner@gmail.com>
> wrote:
> >
> > Yocto Technical Team Minutes, Engineering Sync, for July 28 2020
> > archive:
> >
> https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7
> QwKC5w
> > WVDg9dDH4/edit
> >
> > == disclaimer ==
> > Best efforts are made to ensure the below is accurate and valid.
> > However, errors sometimes happen. If any errors or omissions are
> > found, please feel free to reply to this email with any corrections.
> >
> > == attendees ==
> > Trevor Woerner, Stephen Jolly, Armin Kuster, Josef Holzmayr, Richard
> > Purdie, Trevor Gamblin, Joshua Watt, Mark Morton, Ross Burton, Bruce
> > Ashfield, Steve Sakoman, (phone-in ??), Jon Mason, Randy MacLeod,
> > Scott Murray, Denys Dmytriyenko
> >
> > == notes ==
> > - fixed a race on AB wrt perl
> > - 3.1.2 into QA
> > - 3.2-m2 to QA once they’re done with 3.1.2
> >
> > == general ==
> > RP: load of perl issues over the weekend, believe to have found issue
> > RP: lots of qemumips issues, not sure why, perhaps a timeouts issue? (no
> ping
> >     within 30 seconds)
> > RP: some infrastructure issues (one worker has become corrupted)
> >
> > RP: curious to know what people think about 3.1.2: there is a bitbake issue
> >     with toaster, should we release then fix or wait for fix before release?
> > SS: maybe we should get feedback from people using toaster
> > RP: the fix is simple, maybe release 3.1.2 (with release note) then
> > fix
> >
> > RP: merged re-arrangement of package manager code, this puts more of
> our code
> >     into python libraries instead of bbclass files, which is the direction i
> >     want to take going forward
> > RP: i need to start a conversation on oe-architecture for future
> > directions
> > RP: e.g. binary package feeds, we want to capture these use-cases into the
> >     wiki
> >
> > [NB: done! see https://wiki.yoctoproject.org/wiki/Future_Directions]
> >
> > RP: we also posted the current agreement on “inclusive language” as agreed
> >     on by various groups (oe board, oe-tsc, yp-tsc)
> >
> > TW: what’s the thinking behind moving code to libraries vs bbclass?
> > RP: pro: parsing speed con: variable dependency tracking is lost
> > RP: lots of python code gets written to logs etc, which slows it down
> > JPEW: does bitbake automatically detect variable dependencies in library
> code?
> > RP: no. maybe we just need to do the scan once?
> > TW: do we lose flexibility? (e.g. it’s easy to copy+paste+modify a bbclass,
> >     will that be possible?)
> > RP: there will still be all the same entry points, so it should be
> > possible
> > RP: i’m worried about the size of the datastore in memory and it is a
> >     bottleneck
> >
> > RP: i wonder why qemumips is so slow?
> > Randy: i think Cisco are the only ones caring about mips
> > RP: yes, and Comcast too i think
> >
> > MM: have been wrapped up in $WORK stuff, but will be working on docs
> >     conversion soon
> > RP: MH has been putting infrastructure in place and Nico is looking forward
> to
> >     getting this going
> > MM: do we want to try to reproduce our current colours/formatting or try
> >     something new
> > RP: i like what we have, the project does have an existing scheme, but we
> can
> >     change if something else makes sense
> 
> I have some updates on Sphinx..
> I recall that the work in progress can be found here:
> https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx
> 
> The automated conversion from DocBook to Sphinx is managing 'internal'
> links quite badly, so it results in a significant amount of work to fixup links. Also
> since I wanted to use the Sphinx builtin 'glossary'
> for terms and variables in the reference manual, that also required me to
> update how links work for terms and variables. That's why it's taking more
> time than I would like. I manage to use some automation/regexp, and do a bit
> of manual fix up as well.
> 
> I also started to experiment with customization in the CSS theme, to change
> some color/formatting. It's not done yet, but it has got
> better:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-
> manual.html

I see you got rid of some of the more obvius "funny colors" - the Notes look much better, for example.
I cloned the new sphinx branch and started looking at the implementation some. To make these changes, are you updating the repoDir/documentation/sphinx-static/theme_overrides.css file, or is there somewhere else you're making the changes?

Thanks,
-Mark M

> 
> For now we generate a single large documentation 'website' that includes all
> existing docs, instead of publishing each document separately. I think it's an
> important change compared to how our documentation was published so far. I
> personally like that quite a bit, and think it's 'better' this way, but I would really
> want to hear from you and that we agree this is what we want for the project.
> We still need to look at how we want to generate PDF/offline documentation
> though. And we need to decide how to manage the bitbake manual which sits
> in the bitbake git tree, so right now it's published separately.
> 
> Side note: the VSCode extension for reST is quite good
> (https://docs.restructuredtext.net/index.html)
> 
> >
> > TW: i do lots of builds, never seen issues, why are there so many failures on
> >     AB?
> > RP: do you do oe-selftest or testsdk builds?
> > TW: no
> > RP: do you do mips builds?
> > TW: no
> > RP: so these are the areas where we see these failures, “regular” builds
> >     are okay
> > SS: i don’t see issues on my builds but that’s probably because of working
> >     these issues out on the AB
> > RP: infrastructure issues are low, it’s mostly race issues that come out of
> >     the extreme load of the AB or intermittent architecture issues that are
> >     rarely tested
> >

Re: [docs] [yocto] Yocto Technical Team Minutes, Engineering Sync, for July 28 2020

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 7091 bytes --]

Le ven. 31 juil. 2020 à 22:06, Morton, Mark <Mark.Morton@windriver.com> a
écrit :

>
>
> > -----Original Message-----
> > From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On
> Behalf
> > Of Nicolas Dechesne
> > Sent: Wednesday, July 29, 2020 12:37 AM
> > To: Trevor Woerner <twoerner@gmail.com>
> > Cc: docs@lists.yoctoproject.org; Morton, Mark
> > <Mark.Morton@windriver.com>
> > Subject: Re: [docs] [yocto] Yocto Technical Team Minutes, Engineering
> Sync,
> > for July 28 2020
> >
> > On Wed, Jul 29, 2020 at 12:42 AM Trevor Woerner <twoerner@gmail.com>
> > wrote:
> > >
> > > Yocto Technical Team Minutes, Engineering Sync, for July 28 2020
> > > archive:
> > >
> > https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7
> > QwKC5w
> > > WVDg9dDH4/edit
> > >
> > > == disclaimer ==
> > > Best efforts are made to ensure the below is accurate and valid.
> > > However, errors sometimes happen. If any errors or omissions are
> > > found, please feel free to reply to this email with any corrections.
> > >
> > > == attendees ==
> > > Trevor Woerner, Stephen Jolly, Armin Kuster, Josef Holzmayr, Richard
> > > Purdie, Trevor Gamblin, Joshua Watt, Mark Morton, Ross Burton, Bruce
> > > Ashfield, Steve Sakoman, (phone-in ??), Jon Mason, Randy MacLeod,
> > > Scott Murray, Denys Dmytriyenko
> > >
> > > == notes ==
> > > - fixed a race on AB wrt perl
> > > - 3.1.2 into QA
> > > - 3.2-m2 to QA once they’re done with 3.1.2
> > >
> > > == general ==
> > > RP: load of perl issues over the weekend, believe to have found issue
> > > RP: lots of qemumips issues, not sure why, perhaps a timeouts issue?
> (no
> > ping
> > >     within 30 seconds)
> > > RP: some infrastructure issues (one worker has become corrupted)
> > >
> > > RP: curious to know what people think about 3.1.2: there is a bitbake
> issue
> > >     with toaster, should we release then fix or wait for fix before
> release?
> > > SS: maybe we should get feedback from people using toaster
> > > RP: the fix is simple, maybe release 3.1.2 (with release note) then
> > > fix
> > >
> > > RP: merged re-arrangement of package manager code, this puts more of
> > our code
> > >     into python libraries instead of bbclass files, which is the
> direction i
> > >     want to take going forward
> > > RP: i need to start a conversation on oe-architecture for future
> > > directions
> > > RP: e.g. binary package feeds, we want to capture these use-cases into
> the
> > >     wiki
> > >
> > > [NB: done! see https://wiki.yoctoproject.org/wiki/Future_Directions]
> > >
> > > RP: we also posted the current agreement on “inclusive language” as
> agreed
> > >     on by various groups (oe board, oe-tsc, yp-tsc)
> > >
> > > TW: what’s the thinking behind moving code to libraries vs bbclass?
> > > RP: pro: parsing speed con: variable dependency tracking is lost
> > > RP: lots of python code gets written to logs etc, which slows it down
> > > JPEW: does bitbake automatically detect variable dependencies in
> library
> > code?
> > > RP: no. maybe we just need to do the scan once?
> > > TW: do we lose flexibility? (e.g. it’s easy to copy+paste+modify a
> bbclass,
> > >     will that be possible?)
> > > RP: there will still be all the same entry points, so it should be
> > > possible
> > > RP: i’m worried about the size of the datastore in memory and it is a
> > >     bottleneck
> > >
> > > RP: i wonder why qemumips is so slow?
> > > Randy: i think Cisco are the only ones caring about mips
> > > RP: yes, and Comcast too i think
> > >
> > > MM: have been wrapped up in $WORK stuff, but will be working on docs
> > >     conversion soon
> > > RP: MH has been putting infrastructure in place and Nico is looking
> forward
> > to
> > >     getting this going
> > > MM: do we want to try to reproduce our current colours/formatting or
> try
> > >     something new
> > > RP: i like what we have, the project does have an existing scheme, but
> we
> > can
> > >     change if something else makes sense
> >
> > I have some updates on Sphinx..
> > I recall that the work in progress can be found here:
> > https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx
> >
> > The automated conversion from DocBook to Sphinx is managing 'internal'
> > links quite badly, so it results in a significant amount of work to
> fixup links. Also
> > since I wanted to use the Sphinx builtin 'glossary'
> > for terms and variables in the reference manual, that also required me to
> > update how links work for terms and variables. That's why it's taking
> more
> > time than I would like. I manage to use some automation/regexp, and do a
> bit
> > of manual fix up as well.
> >
> > I also started to experiment with customization in the CSS theme, to
> change
> > some color/formatting. It's not done yet, but it has got
> > better:
> > https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-
> > manual.html
>
> I see you got rid of some of the more obvius "funny colors" - the Notes
> look much better, for example.
> I cloned the new sphinx branch and started looking at the implementation
> some. To make these changes, are you updating the
> repoDir/documentation/sphinx-static/theme_overrides.css file, or is there
> somewhere else you're making the changes?


Yes. All CSS customization is done in this file. It overrides the css
coming from the base theme template that we use.

Thanks for your help! I definitely need some help!


>
> Thanks,
> -Mark M
>
> >
> > For now we generate a single large documentation 'website' that includes
> all
> > existing docs, instead of publishing each document separately. I think
> it's an
> > important change compared to how our documentation was published so far.
> I
> > personally like that quite a bit, and think it's 'better' this way, but
> I would really
> > want to hear from you and that we agree this is what we want for the
> project.
> > We still need to look at how we want to generate PDF/offline
> documentation
> > though. And we need to decide how to manage the bitbake manual which sits
> > in the bitbake git tree, so right now it's published separately.
> >
> > Side note: the VSCode extension for reST is quite good
> > (https://docs.restructuredtext.net/index.html)
> >
> > >
> > > TW: i do lots of builds, never seen issues, why are there so many
> failures on
> > >     AB?
> > > RP: do you do oe-selftest or testsdk builds?
> > > TW: no
> > > RP: do you do mips builds?
> > > TW: no
> > > RP: so these are the areas where we see these failures, “regular”
> builds
> > >     are okay
> > > SS: i don’t see issues on my builds but that’s probably because of
> working
> > >     these issues out on the AB
> > > RP: infrastructure issues are low, it’s mostly race issues that come
> out of
> > >     the extreme load of the AB or intermittent architecture issues
> that are
> > >     rarely tested
> > >
>

[-- Attachment #2: Type: text/html, Size: 9630 bytes --]