[Qemu-devel] QEMU Summit 2017: minutes

[Qemu-devel] QEMU Summit 2017: minutes

[Peter Maydell]

As usual, during this year's KVM Forum we also held the
QEMU Summit, which is where the more active subsystem
maintainers meet up for a discussion of various maintenance
and other project issues. As usual, none of this is set-in-stone
decisions; further input and discussion on-list is welcome.



qemu.org hosting update:
 * Rackspace announced they were ending our free sponsorship, and it would
   be too expensive to stay with them at their charged rates
 * We were planning to migrate to hosting with a consultancy company
   who offered to host us
 * However subsequently to the Summit Rackspace had a change of heart and
   are extending the free hosting indefinitely, so we will stay with them
   (at least for now)
 * CDN sponsorship would help us split off downloads, which are the
   bulk of our bandwidth use

qemu-next:
 * Problem 1: Contributors cannot get patches merged during freeze
   (bad experience)
 * Problem 2: Block layer subtrees have high chance of conflict towards
   end of freeze
 * Peter Maydell: ...but the block layer seems to be about the only
   area with any significant conflict problems currently
 * Markus Armbruster: Problem 1 is solved if maintainers keep their own
   -next trees
 * Paolo Bonzini: Maintaining -next could slow down or create work for
   -freeze (e.g. who does backports)
 * Action: Maintainers mustn't tell submitters to go away just because
   we're in a release freeze (it's up to them whether they prefer to
   maintain a "-next" tree for their subsystem with patches queued for
   the following release, or track which patches they've accepted
   some other way)
 * We're not going to have an official project-wide "-next" tree, though

Firmware blobs:
 * Gerd Hoffmann: Most users don't use firmware blobs from QEMU repo (they
   get them from distros), blobs are becoming larger and bloating
   download size
 * QEMU now has a /usr/lib/qemu-firmware path where it can load firmware
 * Alex Graf: Even though many distros recompile blobs, some blobs are
   hard to recompile so they are shipped without recompilation
 * Paolo Bonzini: pc-bios/ blobs which have sources in the QEMU tree
   need to stay in the QEMU repo (ie pc-bios/{optionrom,s390-ccw,spapr-rtas})
 * Peter Maydell: Unlikely that adding entire UEFI source tree to QEMU
   repo makes sense, but on the other hand we'd like our from-source
   users to be able to run UEFI on ARM and x86 boards
 * Some firmware blobs need lockstep updates with QEMU (eg sparc openbios,
   likely others), so having them in a separate repo is awkward
 * No clear solution at the moment -- would somebody like to volunteer to
   post a proposal to the list (or just try to capture the requirements)?
Security bug handling:
 * Daniel Berrange: QEMU doesn't document security bugs fixed in last
   release (required by CII Best Practices)
 * Peter Maydell: Currently we effectively delegate providing an actual
   secure QEMU usable in production to distros -- we don't do prompt
   releases with security fixes in
 * Stefan Hajnoczi: Stable tree makes sense if distros can share backporting
   work, if distros are not using the same QEMU point release then the
   stable release tends to be not used much
 * Action: Daniel Berrange will propose process for recording CVEs

QEMU Summit Inclusion:
 * Paolo Bonzini: Currently top contributors plus a set of people with
   responsibilities (e.g. -stable maintainer, qemu.org system administrators)
   are invited.  Should we open the summit to anyone who is interested?
 * Peter Maydell: Adding more people may make discussion unwieldy -- we
   would probably need to make the process more formal with proposed agenda
   items having a defined desired decision or outcome attached. (But that
   might be a good idea anyway!)
 * Peter Maydell: Do the Kernel Summit or Device Tree Summit provide
   anything we can learn from? Perhaps we should take a step back and
   ask why we're doing this anyway...
 * Cornelia Huck: Birds of a Feather sessions could be connected with QEMU
   Summit to discuss maintainer topics
 * No action

Continuous Integration:
 * Christian Borntraeger: qemu-iotests have broken a lot, they should be
   run before patches are merged
 * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
   so if something is regularly regressing then it needs to be added to
   "make check".
 * Peter Maydell: Personal build test scripts are being used on qemu.git,
   looking for help to generalize them so others can do the same build
   testing. (Likely to be awkward since half of it is machines I have
   personal login access to and wouldn't want/be able to give others
   access to.)
 * Action: Stefan Hajnoczi to ask Fam Zheng about status of qemu-iotests
   in patchew

Deprecation Cycle:
 * Daniel Berrange: Current policy gives a 2 release grace period before
   features are removed
 * Markus Armbruster: Sometimes it's painful to keep features that are
   costly to maintain for 2 releases, can we make exceptions to the policy?
 * Daniel Berrange: It's important to stick to the policy so users (e.g.
   management tools) have time to adapt
 * There seems to be some scope for limiting deprecation policy for some
   areas of code

thanks
-- PMM

Re: [Qemu-devel] QEMU Summit 2017: minutes

[John Snow]

On 11/23/2017 11:31 AM, Peter Maydell wrote:
> As usual, during this year's KVM Forum we also held the
> QEMU Summit, which is where the more active subsystem
> maintainers meet up for a discussion of various maintenance
> and other project issues. As usual, none of this is set-in-stone
> decisions; further input and discussion on-list is welcome.
> 
> 
> 
> qemu.org hosting update:
>  * Rackspace announced they were ending our free sponsorship, and it would
>    be too expensive to stay with them at their charged rates
>  * We were planning to migrate to hosting with a consultancy company
>    who offered to host us
>  * However subsequently to the Summit Rackspace had a change of heart and
>    are extending the free hosting indefinitely, so we will stay with them
>    (at least for now)
>  * CDN sponsorship would help us split off downloads, which are the
>    bulk of our bandwidth use
> 
> qemu-next:
>  * Problem 1: Contributors cannot get patches merged during freeze
>    (bad experience)
>  * Problem 2: Block layer subtrees have high chance of conflict towards
>    end of freeze
>  * Peter Maydell: ...but the block layer seems to be about the only
>    area with any significant conflict problems currently
>  * Markus Armbruster: Problem 1 is solved if maintainers keep their own
>    -next trees
>  * Paolo Bonzini: Maintaining -next could slow down or create work for
>    -freeze (e.g. who does backports)
>  * Action: Maintainers mustn't tell submitters to go away just because
>    we're in a release freeze (it's up to them whether they prefer to
>    maintain a "-next" tree for their subsystem with patches queued for
>    the following release, or track which patches they've accepted
>    some other way)
>  * We're not going to have an official project-wide "-next" tree, though

Maybe we ought to have a single block-next tree if we're the main source
of that problem. We already more-or-less do with Kevin's tree, but the
tree with the most patches likely to conflict can change depending on
the week (Kevin? Max? Stefan?)

this decentralization works well for us in general when master is open,
but maybe is a source of frustration for drive-by contributors when
master is closed.

Is there room for a block-layer meta-maintainer to pull and test from
all the different sublayers before passing it onward and upward, or does
that just create more problems than it solves?

> 
> Firmware blobs:
>  * Gerd Hoffmann: Most users don't use firmware blobs from QEMU repo (they
>    get them from distros), blobs are becoming larger and bloating
>    download size
>  * QEMU now has a /usr/lib/qemu-firmware path where it can load firmware
>  * Alex Graf: Even though many distros recompile blobs, some blobs are
>    hard to recompile so they are shipped without recompilation
>  * Paolo Bonzini: pc-bios/ blobs which have sources in the QEMU tree
>    need to stay in the QEMU repo (ie pc-bios/{optionrom,s390-ccw,spapr-rtas})
>  * Peter Maydell: Unlikely that adding entire UEFI source tree to QEMU
>    repo makes sense, but on the other hand we'd like our from-source
>    users to be able to run UEFI on ARM and x86 boards
>  * Some firmware blobs need lockstep updates with QEMU (eg sparc openbios,
>    likely others), so having them in a separate repo is awkward
>  * No clear solution at the moment -- would somebody like to volunteer to
>    post a proposal to the list (or just try to capture the requirements)?
> Security bug handling:
>  * Daniel Berrange: QEMU doesn't document security bugs fixed in last
>    release (required by CII Best Practices)
>  * Peter Maydell: Currently we effectively delegate providing an actual
>    secure QEMU usable in production to distros -- we don't do prompt
>    releases with security fixes in
>  * Stefan Hajnoczi: Stable tree makes sense if distros can share backporting
>    work, if distros are not using the same QEMU point release then the
>    stable release tends to be not used much
>  * Action: Daniel Berrange will propose process for recording CVEs
> 
> QEMU Summit Inclusion:
>  * Paolo Bonzini: Currently top contributors plus a set of people with
>    responsibilities (e.g. -stable maintainer, qemu.org system administrators)
>    are invited.  Should we open the summit to anyone who is interested?
>  * Peter Maydell: Adding more people may make discussion unwieldy -- we
>    would probably need to make the process more formal with proposed agenda
>    items having a defined desired decision or outcome attached. (But that
>    might be a good idea anyway!)
>  * Peter Maydell: Do the Kernel Summit or Device Tree Summit provide
>    anything we can learn from? Perhaps we should take a step back and
>    ask why we're doing this anyway...
>  * Cornelia Huck: Birds of a Feather sessions could be connected with QEMU
>    Summit to discuss maintainer topics
>  * No action
> 

Do you grab representatives from top contributing companies? Might help
lessen the RH-centricity of things which might increase the rate of
patchflow.

> Continuous Integration:
>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
>    run before patches are merged

This, rather unfortunately, is a huge testing burden. I try to make sure
I do it for everything I submit, but for the volume of block patches it
really does rely CI. The more we add (to our pitifully sparse iotesting,
I might add) the longer it takes. Ensuring per-patch testing begins to
take prohibitively long.

Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
implement a block-next amalgam we'd be able to batch our testing on a
weekly basis.

>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
>    so if something is regularly regressing then it needs to be added to
>    "make check".

Is this tenable long term? We can't conceivably state that we will never
test things that aren't in "make check" -- we ought to have different
tiers, at least. The full testing suite should run for RC tags at least,
but it's not feasible (I think?) to run the entire battery of tests on
every commit... but that shouldn't stop us from running them /sometimes/...

>  * Peter Maydell: Personal build test scripts are being used on qemu.git,
>    looking for help to generalize them so others can do the same build
>    testing. (Likely to be awkward since half of it is machines I have
>    personal login access to and wouldn't want/be able to give others
>    access to.)
>  * Action: Stefan Hajnoczi to ask Fam Zheng about status of qemu-iotests
>    in patchew
> 
> Deprecation Cycle:
>  * Daniel Berrange: Current policy gives a 2 release grace period before
>    features are removed
>  * Markus Armbruster: Sometimes it's painful to keep features that are
>    costly to maintain for 2 releases, can we make exceptions to the policy?
>  * Daniel Berrange: It's important to stick to the policy so users (e.g.
>    management tools) have time to adapt
>  * There seems to be some scope for limiting deprecation policy for some
>    areas of code
> 
> thanks
> -- PMM
>

Re: [Qemu-devel] QEMU Summit 2017: minutes

[Thomas Huth]

On 27.11.2017 23:03, John Snow wrote:
> 
> On 11/23/2017 11:31 AM, Peter Maydell wrote:
[...]
>> Continuous Integration:
>>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
>>    run before patches are merged
> 
> This, rather unfortunately, is a huge testing burden. I try to make sure
> I do it for everything I submit, but for the volume of block patches it
> really does rely CI. The more we add (to our pitifully sparse iotesting,
> I might add) the longer it takes. Ensuring per-patch testing begins to
> take prohibitively long.
> 
> Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> implement a block-next amalgam we'd be able to batch our testing on a
> weekly basis.

I think you block-layer folks should do at least run the qemu-iotests
before sending a pull request to Peter. The iotests should really not be
broken in upstream master.

>>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
>>    so if something is regularly regressing then it needs to be added to
>>    "make check".
> 
> Is this tenable long term? We can't conceivably state that we will never
> test things that aren't in "make check" -- we ought to have different
> tiers, at least. The full testing suite should run for RC tags at least,
> but it's not feasible (I think?) to run the entire battery of tests on
> every commit... but that shouldn't stop us from running them /sometimes/...

We've already got "make check SPEED=slow" for running tests that take a
lot of time. So maybe you could do that in the iotests as well, so that
the normal, quick tests can be run during "make check" and the full
iotest suite is only run during "make check SPEED=slow" ?

 Thomas

Re: [Qemu-devel] QEMU Summit 2017: minutes

[Cornelia Huck]

On Tue, 28 Nov 2017 09:33:52 +0100
Thomas Huth <thuth@redhat.com> wrote:

> On 27.11.2017 23:03, John Snow wrote:
> > 
> > On 11/23/2017 11:31 AM, Peter Maydell wrote:  
> [...]
> >> Continuous Integration:
> >>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
> >>    run before patches are merged  
> > 
> > This, rather unfortunately, is a huge testing burden. I try to make sure
> > I do it for everything I submit, but for the volume of block patches it
> > really does rely CI. The more we add (to our pitifully sparse iotesting,
> > I might add) the longer it takes. Ensuring per-patch testing begins to
> > take prohibitively long.
> > 
> > Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> > implement a block-next amalgam we'd be able to batch our testing on a
> > weekly basis.  
> 
> I think you block-layer folks should do at least run the qemu-iotests
> before sending a pull request to Peter. The iotests should really not be
> broken in upstream master.

This is unlikely to cover the iotest failures on s390 (due to usage of
ccw, strange backing devices, etc.), though. We have basically two
options here:
- Continue to rely on the IBM folks finding those problems (which will
  likely be post-merge, but better than nothing.)
- Have patchew (which has a bot on s390) execute the iotests - which is
  time-consuming.

> 
> >>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
> >>    so if something is regularly regressing then it needs to be added to
> >>    "make check".  
> > 
> > Is this tenable long term? We can't conceivably state that we will never
> > test things that aren't in "make check" -- we ought to have different
> > tiers, at least. The full testing suite should run for RC tags at least,
> > but it's not feasible (I think?) to run the entire battery of tests on
> > every commit... but that shouldn't stop us from running them /sometimes/...  
> 
> We've already got "make check SPEED=slow" for running tests that take a
> lot of time. So maybe you could do that in the iotests as well, so that
> the normal, quick tests can be run during "make check" and the full
> iotest suite is only run during "make check SPEED=slow" ?

+1 to that. Having a subset covered by default is better than nothing
at all.

Re: [Qemu-devel] QEMU Summit 2017: minutes

[John Snow]

On 11/28/2017 04:36 AM, Cornelia Huck wrote:
> On Tue, 28 Nov 2017 09:33:52 +0100
> Thomas Huth <thuth@redhat.com> wrote:
> 
>> On 27.11.2017 23:03, John Snow wrote:
>>>
>>> On 11/23/2017 11:31 AM, Peter Maydell wrote:  
>> [...]
>>>> Continuous Integration:
>>>>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
>>>>    run before patches are merged  
>>>
>>> This, rather unfortunately, is a huge testing burden. I try to make sure
>>> I do it for everything I submit, but for the volume of block patches it
>>> really does rely CI. The more we add (to our pitifully sparse iotesting,
>>> I might add) the longer it takes. Ensuring per-patch testing begins to
>>> take prohibitively long.
>>>
>>> Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
>>> implement a block-next amalgam we'd be able to batch our testing on a
>>> weekly basis.  
>>
>> I think you block-layer folks should do at least run the qemu-iotests
>> before sending a pull request to Peter. The iotests should really not be
>> broken in upstream master.
> 
> This is unlikely to cover the iotest failures on s390 (due to usage of
> ccw, strange backing devices, etc.), though. We have basically two
> options here:
> - Continue to rely on the IBM folks finding those problems (which will
>   likely be post-merge, but better than nothing.)
> - Have patchew (which has a bot on s390) execute the iotests - which is
>   time-consuming.
> 

Does patchew test pull requests? Perhaps Peter could wait for an ACK
from patchew before committing. Peter and patchew could check PRs in
tandem and perhaps he can commit fully only when patchew ACKs.

for PRs specifically, perhaps patchew can indeed send an affirmative ACK
to the list indicating success.

>>
>>>>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
>>>>    so if something is regularly regressing then it needs to be added to
>>>>    "make check".  
>>>
>>> Is this tenable long term? We can't conceivably state that we will never
>>> test things that aren't in "make check" -- we ought to have different
>>> tiers, at least. The full testing suite should run for RC tags at least,
>>> but it's not feasible (I think?) to run the entire battery of tests on
>>> every commit... but that shouldn't stop us from running them /sometimes/...  
>>
>> We've already got "make check SPEED=slow" for running tests that take a
>> lot of time. So maybe you could do that in the iotests as well, so that
>> the normal, quick tests can be run during "make check" and the full
>> iotest suite is only run during "make check SPEED=slow" ?
> 
> +1 to that. Having a subset covered by default is better than nothing
> at all.
> 

Sure, I was just taking issue with Peter's phrasing of 'If it isn't
tested by "make check" then it isn't tested' -- this sounds like policy
we should change and indeed formalize fast and slow paths.

Of course, this is the block maintainers' job more than anyone else's,
hence some other musings above and in my initial reply.

Re: [Qemu-devel] QEMU Summit 2017: minutes

[Cornelia Huck]

On Tue, 28 Nov 2017 13:30:23 -0500
John Snow <jsnow@redhat.com> wrote:

> On 11/28/2017 04:36 AM, Cornelia Huck wrote:
> > On Tue, 28 Nov 2017 09:33:52 +0100
> > Thomas Huth <thuth@redhat.com> wrote:
> >   
> >> On 27.11.2017 23:03, John Snow wrote:  
> >>>
> >>> On 11/23/2017 11:31 AM, Peter Maydell wrote:    
> >> [...]  
> >>>> Continuous Integration:
> >>>>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
> >>>>    run before patches are merged    
> >>>
> >>> This, rather unfortunately, is a huge testing burden. I try to make sure
> >>> I do it for everything I submit, but for the volume of block patches it
> >>> really does rely CI. The more we add (to our pitifully sparse iotesting,
> >>> I might add) the longer it takes. Ensuring per-patch testing begins to
> >>> take prohibitively long.
> >>>
> >>> Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> >>> implement a block-next amalgam we'd be able to batch our testing on a
> >>> weekly basis.    
> >>
> >> I think you block-layer folks should do at least run the qemu-iotests
> >> before sending a pull request to Peter. The iotests should really not be
> >> broken in upstream master.  
> > 
> > This is unlikely to cover the iotest failures on s390 (due to usage of
> > ccw, strange backing devices, etc.), though. We have basically two
> > options here:
> > - Continue to rely on the IBM folks finding those problems (which will
> >   likely be post-merge, but better than nothing.)
> > - Have patchew (which has a bot on s390) execute the iotests - which is
> >   time-consuming.
> >   
> 
> Does patchew test pull requests? Perhaps Peter could wait for an ACK
> from patchew before committing. Peter and patchew could check PRs in
> tandem and perhaps he can commit fully only when patchew ACKs.
> 
> for PRs specifically, perhaps patchew can indeed send an affirmative ACK
> to the list indicating success.

I'd assume patchew can figure out whether it deals with a pull request
by checking for 'PULL', and we post all patches in a pull request, so
some special handling might be feasible.

Fam, what do you think?

Re: [Qemu-devel] QEMU Summit 2017: minutes

[Fam Zheng]

On Wed, 11/29 09:31, Cornelia Huck wrote:
> On Tue, 28 Nov 2017 13:30:23 -0500
> John Snow <jsnow@redhat.com> wrote:
> 
> > On 11/28/2017 04:36 AM, Cornelia Huck wrote:
> > > On Tue, 28 Nov 2017 09:33:52 +0100
> > > Thomas Huth <thuth@redhat.com> wrote:
> > >   
> > >> On 27.11.2017 23:03, John Snow wrote:  
> > >>>
> > >>> On 11/23/2017 11:31 AM, Peter Maydell wrote:    
> > >> [...]  
> > >>>> Continuous Integration:
> > >>>>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
> > >>>>    run before patches are merged    
> > >>>
> > >>> This, rather unfortunately, is a huge testing burden. I try to make sure
> > >>> I do it for everything I submit, but for the volume of block patches it
> > >>> really does rely CI. The more we add (to our pitifully sparse iotesting,
> > >>> I might add) the longer it takes. Ensuring per-patch testing begins to
> > >>> take prohibitively long.
> > >>>
> > >>> Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> > >>> implement a block-next amalgam we'd be able to batch our testing on a
> > >>> weekly basis.    
> > >>
> > >> I think you block-layer folks should do at least run the qemu-iotests
> > >> before sending a pull request to Peter. The iotests should really not be
> > >> broken in upstream master.  
> > > 
> > > This is unlikely to cover the iotest failures on s390 (due to usage of
> > > ccw, strange backing devices, etc.), though. We have basically two
> > > options here:
> > > - Continue to rely on the IBM folks finding those problems (which will
> > >   likely be post-merge, but better than nothing.)
> > > - Have patchew (which has a bot on s390) execute the iotests - which is
> > >   time-consuming.
> > >   
> > 
> > Does patchew test pull requests? Perhaps Peter could wait for an ACK
> > from patchew before committing. Peter and patchew could check PRs in
> > tandem and perhaps he can commit fully only when patchew ACKs.
> > 
> > for PRs specifically, perhaps patchew can indeed send an affirmative ACK
> > to the list indicating success.
> 
> I'd assume patchew can figure out whether it deals with a pull request
> by checking for 'PULL', and we post all patches in a pull request, so
> some special handling might be feasible.
> 
> Fam, what do you think?

Yes, but I guess we need to first address Peter's specific unhappiness about
patchew reports, before placing it in the pull request process: a lot of lines,
especially in the beginning of the email is not quite useful making the email
unfriendly to read.

If we want explicit ACKs from patchew on PULL process, two more changes are
needed (they apply to pull requests only):

1) Fetch from the pull request's git branch instead of applying. This has two
   advantages: when failed to apply a series, patchew doesn't test it, which
   cannot happend if we git-fetch instead of git-am; quite often a pull request
   v2, v3, ... only includes changed patch in the series, making git-am more
   likely to fail, which again is not a problem if we git-fetch.

2) Report if all tests cannot complete in time (e.g. a tester is down and some
   tests cannot run). The rationale is the same as above.

Fam