* QEMU 5.1: Can we require each new device/machine to provided a test? @ 2020-04-07 10:59 Philippe Mathieu-Daudé 2020-05-15 10:11 ` Thomas Huth 0 siblings, 1 reply; 12+ messages in thread From: Philippe Mathieu-Daudé @ 2020-04-07 10:59 UTC (permalink / raw) To: qemu-devel@nongnu.org Developers, Markus Armbruster, Peter Maydell, Cleber Rosa Cc: Thomas Huth, Eduardo Habkost, qemu-discuss, Gerd Hoffmann, Paolo Bonzini, David Gibson Hello, Following Markus thread on deprecating unmaintained (untested) code (machines) [1] and the effort done to gather the information shared in the replies [2], and the various acceptance tests added, is it feasible to require for the next release that each new device/machine is provided a test covering it? If no, what is missing? Thanks, Phil. [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg604682.html [2] https://wiki.qemu.org/Testing/Acceptance#Machines ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-04-07 10:59 QEMU 5.1: Can we require each new device/machine to provided a test? Philippe Mathieu-Daudé @ 2020-05-15 10:11 ` Thomas Huth 2020-05-15 10:23 ` Daniel P. Berrangé 2020-05-15 10:51 ` Gerd Hoffmann 0 siblings, 2 replies; 12+ messages in thread From: Thomas Huth @ 2020-05-15 10:11 UTC (permalink / raw) To: Philippe Mathieu-Daudé, qemu-devel@nongnu.org Developers, Markus Armbruster, Peter Maydell, Cleber Rosa Cc: Paolo Bonzini, David Gibson, Gerd Hoffmann, Eduardo Habkost, qemu-discuss On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > Hello, > > Following Markus thread on deprecating unmaintained (untested) code > (machines) [1] and the effort done to gather the information shared in > the replies [2], and the various acceptance tests added, is it > feasible to require for the next release that each new device/machine > is provided a test covering it? > > If no, what is missing? If a qtest is feasible, yes, I think we should require one for new devices. But what about machines - you normally need a test image for this. In that case, there is still the question where testing images could be hosted. Not every developer has a web space where they could put their test images onto. And what about images that contain non-free code? Thomas ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-15 10:11 ` Thomas Huth @ 2020-05-15 10:23 ` Daniel P. Berrangé 2020-05-18 19:56 ` John Snow 2020-05-15 10:51 ` Gerd Hoffmann 1 sibling, 1 reply; 12+ messages in thread From: Daniel P. Berrangé @ 2020-05-15 10:23 UTC (permalink / raw) To: Thomas Huth Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Markus Armbruster, Philippe Mathieu-Daudé, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: > On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > > Hello, > > > > Following Markus thread on deprecating unmaintained (untested) code > > (machines) [1] and the effort done to gather the information shared in > > the replies [2], and the various acceptance tests added, is it > > feasible to require for the next release that each new device/machine > > is provided a test covering it? > > > > If no, what is missing? > > If a qtest is feasible, yes, I think we should require one for new > devices. But what about machines - you normally need a test image for > this. In that case, there is still the question where testing images > could be hosted. Not every developer has a web space where they could > put their test images onto. And what about images that contain non-free > code? Yep, it isn't feasible to make this a hard rule. IMHO this is where a support tier classification comes into play - Tier 1: actively maintained, qtest coverage available. Expected to work reliably at all times since every commit is CI tested - Tier 2: actively maintained, no qtest coverage. Should usually work but regression may creep in due to reliance on the maintainer to manually test on adhoc basis - Tier 3: not actively maintained, unknown state but liable to be broken indefinitely Tier 1 is obviously the most desirable state we would like everthing to be at. Contributors will have to fix problems their patches cause as they will be blocked by CI. Tier 2 is an admission that reality gets in the way. Ideally stuff in this tier will graduate to Tier 1 at some point. Even if it doesn't though, it is still valid to keep it in QEMU long term. Contributors shouldn't gratuitously break stuff in these board, but if they do, then the maintainer is ultimately responsible for fixing it, as the contributors don't have a test rig for it. Tier 3 is abandonware. If a maintainer doesn't appear, users should not expect it to continue to exist long term. Contributors are free to send patches which break this, and are under no obligation to fix problems in these boards. We may deprecate & delete it after a while Over time we'll likely add more criteria to stuff in Tier 1. This could lead to some things dropping from Tier 1 to Tier 2. This is OK, as it doesn't make those things worse than they already were. We're just saying that Tier 2 isn't as thoroughly tested as we would like it to be in an ideal world. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-15 10:23 ` Daniel P. Berrangé @ 2020-05-18 19:56 ` John Snow 2020-05-19 9:04 ` Daniel P. Berrangé 0 siblings, 1 reply; 12+ messages in thread From: John Snow @ 2020-05-18 19:56 UTC (permalink / raw) To: Daniel P. Berrangé, Thomas Huth Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: > On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: >> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: >>> Hello, >>> >>> Following Markus thread on deprecating unmaintained (untested) code >>> (machines) [1] and the effort done to gather the information shared in >>> the replies [2], and the various acceptance tests added, is it >>> feasible to require for the next release that each new device/machine >>> is provided a test covering it? >>> >>> If no, what is missing? >> >> If a qtest is feasible, yes, I think we should require one for new >> devices. But what about machines - you normally need a test image for >> this. In that case, there is still the question where testing images >> could be hosted. Not every developer has a web space where they could >> put their test images onto. And what about images that contain non-free >> code? > > Yep, it isn't feasible to make this a hard rule. > > IMHO this is where a support tier classification comes into play > > - Tier 1: actively maintained, qtest coverage available. Expected > to work reliably at all times since every commit is CI > tested > > - Tier 2: actively maintained, no qtest coverage. Should usually > work but regression may creep in due to reliance on the > maintainer to manually test on adhoc basis > > - Tier 3: not actively maintained, unknown state but liable to > be broken indefinitely > > Tier 1 is obviously the most desirable state we would like everthing to > be at. Contributors will have to fix problems their patches cause as > they will be blocked by CI. > > Tier 2 is an admission that reality gets in the way. Ideally stuff in > this tier will graduate to Tier 1 at some point. Even if it doesn't > though, it is still valid to keep it in QEMU long term. Contributors > shouldn't gratuitously break stuff in these board, but if they do, > then the maintainer is ultimately responsible for fixing it, as the > contributors don't have a test rig for it. > > Tier 3 is abandonware. If a maintainer doesn't appear, users should > not expect it to continue to exist long term. Contributors are free > to send patches which break this, and are under no obligation to > fix problems in these boards. We may deprecate & delete it after a > while > > > Over time we'll likely add more criteria to stuff in Tier 1. This > could lead to some things dropping from Tier 1 to Tier 2. This is > OK, as it doesn't make those things worse than they already were. > We're just saying that Tier 2 isn't as thoroughly tested as we > would like it to be in an ideal world. > > Regards, > Daniel > I really like the idea of device support tiers codified directly in the QEMU codebase, to give upstream users some idea of which devices we expect to work and which we ... don't, really. Not every last device we offer is enterprise production ready, but we don't necessarily do a good job of explaining which devices fall into which categories, and we've got quite a few of them. I wonder if a 2.5th tier would be useful; something like a "hobbyist" tier for pet project SoC boards and the like -- they're not abandoned, but we also don't expect them to work, exactly. Mild semantic difference from Tier 3. --js ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-18 19:56 ` John Snow @ 2020-05-19 9:04 ` Daniel P. Berrangé 2020-05-19 23:06 ` John Snow 0 siblings, 1 reply; 12+ messages in thread From: Daniel P. Berrangé @ 2020-05-19 9:04 UTC (permalink / raw) To: John Snow Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: > > > On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: > > On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: > >> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > >>> Hello, > >>> > >>> Following Markus thread on deprecating unmaintained (untested) code > >>> (machines) [1] and the effort done to gather the information shared in > >>> the replies [2], and the various acceptance tests added, is it > >>> feasible to require for the next release that each new device/machine > >>> is provided a test covering it? > >>> > >>> If no, what is missing? > >> > >> If a qtest is feasible, yes, I think we should require one for new > >> devices. But what about machines - you normally need a test image for > >> this. In that case, there is still the question where testing images > >> could be hosted. Not every developer has a web space where they could > >> put their test images onto. And what about images that contain non-free > >> code? > > > > Yep, it isn't feasible to make this a hard rule. > > > > IMHO this is where a support tier classification comes into play > > > > - Tier 1: actively maintained, qtest coverage available. Expected > > to work reliably at all times since every commit is CI > > tested > > > > - Tier 2: actively maintained, no qtest coverage. Should usually > > work but regression may creep in due to reliance on the > > maintainer to manually test on adhoc basis > > > > - Tier 3: not actively maintained, unknown state but liable to > > be broken indefinitely > > > > Tier 1 is obviously the most desirable state we would like everthing to > > be at. Contributors will have to fix problems their patches cause as > > they will be blocked by CI. > > > > Tier 2 is an admission that reality gets in the way. Ideally stuff in > > this tier will graduate to Tier 1 at some point. Even if it doesn't > > though, it is still valid to keep it in QEMU long term. Contributors > > shouldn't gratuitously break stuff in these board, but if they do, > > then the maintainer is ultimately responsible for fixing it, as the > > contributors don't have a test rig for it. > > > > Tier 3 is abandonware. If a maintainer doesn't appear, users should > > not expect it to continue to exist long term. Contributors are free > > to send patches which break this, and are under no obligation to > > fix problems in these boards. We may deprecate & delete it after a > > while > > > > > > Over time we'll likely add more criteria to stuff in Tier 1. This > > could lead to some things dropping from Tier 1 to Tier 2. This is > > OK, as it doesn't make those things worse than they already were. > > We're just saying that Tier 2 isn't as thoroughly tested as we > > would like it to be in an ideal world. > > I really like the idea of device support tiers codified directly in the > QEMU codebase, to give upstream users some idea of which devices we > expect to work and which we ... don't, really. > > Not every last device we offer is enterprise production ready, but we > don't necessarily do a good job of explaining which devices fall into > which categories, and we've got quite a few of them. > > I wonder if a 2.5th tier would be useful; something like a "hobbyist" > tier for pet project SoC boards and the like -- they're not abandoned, > but we also don't expect them to work, exactly. > > Mild semantic difference from Tier 3. I guess I was thinking such hobbyist stuff would fall into tier 2 if the hobbyist maintainer actually responds to fixing stuff, or tier 3 if they largely aren't active on the mailing list responding to issues/questions. We add have a 4 tier system overall and put hobbyist stuff at tier 3, and abandonware at tier 4. Probably shouldn't go beyond 4 tiers though, as the more criteria we add the harder it is to clearly decide which tier something should go into. The tier 1 vs 2 divison is clearly split based on CI which is a simple classification to decide on. The tier 2 vs 3 division is moderately clearly split based on whether there is a frequently active maintainer. We can probably squeeze in the 4th tier without too much ambiguity in the classisfication if we think it is adding something worthwhile either from our POV as maintainers, or for users consuming it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-19 9:04 ` Daniel P. Berrangé @ 2020-05-19 23:06 ` John Snow 2020-05-20 6:13 ` Thomas Huth 2020-05-20 8:57 ` Daniel P. Berrangé 0 siblings, 2 replies; 12+ messages in thread From: John Snow @ 2020-05-19 23:06 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: > On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: >> >> >> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: >>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: >>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: >>>>> Hello, >>>>> >>>>> Following Markus thread on deprecating unmaintained (untested) code >>>>> (machines) [1] and the effort done to gather the information shared in >>>>> the replies [2], and the various acceptance tests added, is it >>>>> feasible to require for the next release that each new device/machine >>>>> is provided a test covering it? >>>>> >>>>> If no, what is missing? >>>> >>>> If a qtest is feasible, yes, I think we should require one for new >>>> devices. But what about machines - you normally need a test image for >>>> this. In that case, there is still the question where testing images >>>> could be hosted. Not every developer has a web space where they could >>>> put their test images onto. And what about images that contain non-free >>>> code? >>> >>> Yep, it isn't feasible to make this a hard rule. >>> >>> IMHO this is where a support tier classification comes into play >>> >>> - Tier 1: actively maintained, qtest coverage available. Expected >>> to work reliably at all times since every commit is CI >>> tested >>> >>> - Tier 2: actively maintained, no qtest coverage. Should usually >>> work but regression may creep in due to reliance on the >>> maintainer to manually test on adhoc basis >>> >>> - Tier 3: not actively maintained, unknown state but liable to >>> be broken indefinitely >>> >>> Tier 1 is obviously the most desirable state we would like everthing to >>> be at. Contributors will have to fix problems their patches cause as >>> they will be blocked by CI. >>> >>> Tier 2 is an admission that reality gets in the way. Ideally stuff in >>> this tier will graduate to Tier 1 at some point. Even if it doesn't >>> though, it is still valid to keep it in QEMU long term. Contributors >>> shouldn't gratuitously break stuff in these board, but if they do, >>> then the maintainer is ultimately responsible for fixing it, as the >>> contributors don't have a test rig for it. >>> >>> Tier 3 is abandonware. If a maintainer doesn't appear, users should >>> not expect it to continue to exist long term. Contributors are free >>> to send patches which break this, and are under no obligation to >>> fix problems in these boards. We may deprecate & delete it after a >>> while >>> >>> >>> Over time we'll likely add more criteria to stuff in Tier 1. This >>> could lead to some things dropping from Tier 1 to Tier 2. This is >>> OK, as it doesn't make those things worse than they already were. >>> We're just saying that Tier 2 isn't as thoroughly tested as we >>> would like it to be in an ideal world. >> >> I really like the idea of device support tiers codified directly in the >> QEMU codebase, to give upstream users some idea of which devices we >> expect to work and which we ... don't, really. >> >> Not every last device we offer is enterprise production ready, but we >> don't necessarily do a good job of explaining which devices fall into >> which categories, and we've got quite a few of them. >> >> I wonder if a 2.5th tier would be useful; something like a "hobbyist" >> tier for pet project SoC boards and the like -- they're not abandoned, >> but we also don't expect them to work, exactly. >> >> Mild semantic difference from Tier 3. > > I guess I was thinking such hobbyist stuff would fall into tier 2 if the > hobbyist maintainer actually responds to fixing stuff, or tier 3 if they > largely aren't active on the mailing list responding to issues/questions. > > We add have a 4 tier system overall and put hobbyist stuff at tier 3, > and abandonware at tier 4. > > Probably shouldn't go beyond 4 tiers though, as the more criteria we add > the harder it is to clearly decide which tier something should go into. > > The tier 1 vs 2 divison is clearly split based on CI which is a simple > classification to decide on. > > The tier 2 vs 3 division is moderately clearly split based on whether > there is a frequently active maintainer. > > We can probably squeeze in the 4th tier without too much ambiguity in > the classisfication if we think it is adding something worthwhile either > from our POV as maintainers, or for users consuming it. Yes, I didn't mean to start watering it down into a 1,380 tier system that we're never able to properly utilize. I was thinking more along the lines of: - Device works and is well loved - Device works and is well loved (but we have to test manually) - Device doesn't work, but is well loved (Use at your own peril, please file a bug report) - Device doesn't work, and is unloved Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where things can shift from 1A to 1B as their test coverage allows, but it's not meant to indicate general status otherwise. Mostly, I would just like some way for users to avoid accidentally running tier 2 or 3 devices /by accident/, or the ability to compile QEMU versions that only allow tier 1 devices to be used. It's all arbitrary -- but I think we agree more than not! I'd love to have a list of first-class boards and devices that we promise to test and have working. --js ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-19 23:06 ` John Snow @ 2020-05-20 6:13 ` Thomas Huth 2020-05-20 9:02 ` Daniel P. Berrangé 2020-05-20 14:53 ` John Snow 2020-05-20 8:57 ` Daniel P. Berrangé 1 sibling, 2 replies; 12+ messages in thread From: Thomas Huth @ 2020-05-20 6:13 UTC (permalink / raw) To: John Snow, Daniel P. Berrangé Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On 20/05/2020 01.06, John Snow wrote: > > > On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: >> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: >>> >>> >>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: >>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: >>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: >>>>>> Hello, >>>>>> >>>>>> Following Markus thread on deprecating unmaintained (untested) code >>>>>> (machines) [1] and the effort done to gather the information shared in >>>>>> the replies [2], and the various acceptance tests added, is it >>>>>> feasible to require for the next release that each new device/machine >>>>>> is provided a test covering it? >>>>>> >>>>>> If no, what is missing? >>>>> >>>>> If a qtest is feasible, yes, I think we should require one for new >>>>> devices. But what about machines - you normally need a test image for >>>>> this. In that case, there is still the question where testing images >>>>> could be hosted. Not every developer has a web space where they could >>>>> put their test images onto. And what about images that contain non-free >>>>> code? >>>> >>>> Yep, it isn't feasible to make this a hard rule. >>>> >>>> IMHO this is where a support tier classification comes into play >>>> >>>> - Tier 1: actively maintained, qtest coverage available. Expected >>>> to work reliably at all times since every commit is CI >>>> tested >>>> >>>> - Tier 2: actively maintained, no qtest coverage. Should usually >>>> work but regression may creep in due to reliance on the >>>> maintainer to manually test on adhoc basis >>>> >>>> - Tier 3: not actively maintained, unknown state but liable to >>>> be broken indefinitely >>>> >>>> Tier 1 is obviously the most desirable state we would like everthing to >>>> be at. Contributors will have to fix problems their patches cause as >>>> they will be blocked by CI. >>>> >>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in >>>> this tier will graduate to Tier 1 at some point. Even if it doesn't >>>> though, it is still valid to keep it in QEMU long term. Contributors >>>> shouldn't gratuitously break stuff in these board, but if they do, >>>> then the maintainer is ultimately responsible for fixing it, as the >>>> contributors don't have a test rig for it. >>>> >>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should >>>> not expect it to continue to exist long term. Contributors are free >>>> to send patches which break this, and are under no obligation to >>>> fix problems in these boards. We may deprecate & delete it after a >>>> while >>>> >>>> >>>> Over time we'll likely add more criteria to stuff in Tier 1. This >>>> could lead to some things dropping from Tier 1 to Tier 2. This is >>>> OK, as it doesn't make those things worse than they already were. >>>> We're just saying that Tier 2 isn't as thoroughly tested as we >>>> would like it to be in an ideal world. >>> >>> I really like the idea of device support tiers codified directly in the >>> QEMU codebase, to give upstream users some idea of which devices we >>> expect to work and which we ... don't, really. >>> >>> Not every last device we offer is enterprise production ready, but we >>> don't necessarily do a good job of explaining which devices fall into >>> which categories, and we've got quite a few of them. >>> >>> I wonder if a 2.5th tier would be useful; something like a "hobbyist" >>> tier for pet project SoC boards and the like -- they're not abandoned, >>> but we also don't expect them to work, exactly. >>> >>> Mild semantic difference from Tier 3. >> >> I guess I was thinking such hobbyist stuff would fall into tier 2 if the >> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they >> largely aren't active on the mailing list responding to issues/questions. >> >> We add have a 4 tier system overall and put hobbyist stuff at tier 3, >> and abandonware at tier 4. >> >> Probably shouldn't go beyond 4 tiers though, as the more criteria we add >> the harder it is to clearly decide which tier something should go into. >> >> The tier 1 vs 2 divison is clearly split based on CI which is a simple >> classification to decide on. >> >> The tier 2 vs 3 division is moderately clearly split based on whether >> there is a frequently active maintainer. >> >> We can probably squeeze in the 4th tier without too much ambiguity in >> the classisfication if we think it is adding something worthwhile either >> from our POV as maintainers, or for users consuming it. > > Yes, I didn't mean to start watering it down into a 1,380 tier system > that we're never able to properly utilize. > > I was thinking more along the lines of: > > - Device works and is well loved > - Device works and is well loved (but we have to test manually) > - Device doesn't work, but is well loved > (Use at your own peril, please file a bug report) > - Device doesn't work, and is unloved > > Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where > things can shift from 1A to 1B as their test coverage allows, but it's > not meant to indicate general status otherwise. All that sounds somewhat similar to the classification that we already use in our MAINTAINERS file - Supported, Maintained, Odd-Fixes, Orphan, Obsolete ... maybe we can avoid to introduce yet another classification system and merge the two (e.g. by also changing the classification system in MAINTAINERS a little bit?). Thomas ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-20 6:13 ` Thomas Huth @ 2020-05-20 9:02 ` Daniel P. Berrangé 2020-05-20 14:53 ` John Snow 1 sibling, 0 replies; 12+ messages in thread From: Daniel P. Berrangé @ 2020-05-20 9:02 UTC (permalink / raw) To: Thomas Huth Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, John Snow, David Gibson On Wed, May 20, 2020 at 08:13:07AM +0200, Thomas Huth wrote: > On 20/05/2020 01.06, John Snow wrote: > > > > > > On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: > >> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: > >>> > >>> > >>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: > >>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: > >>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > >>>>>> Hello, > >>>>>> > >>>>>> Following Markus thread on deprecating unmaintained (untested) code > >>>>>> (machines) [1] and the effort done to gather the information shared in > >>>>>> the replies [2], and the various acceptance tests added, is it > >>>>>> feasible to require for the next release that each new device/machine > >>>>>> is provided a test covering it? > >>>>>> > >>>>>> If no, what is missing? > >>>>> > >>>>> If a qtest is feasible, yes, I think we should require one for new > >>>>> devices. But what about machines - you normally need a test image for > >>>>> this. In that case, there is still the question where testing images > >>>>> could be hosted. Not every developer has a web space where they could > >>>>> put their test images onto. And what about images that contain non-free > >>>>> code? > >>>> > >>>> Yep, it isn't feasible to make this a hard rule. > >>>> > >>>> IMHO this is where a support tier classification comes into play > >>>> > >>>> - Tier 1: actively maintained, qtest coverage available. Expected > >>>> to work reliably at all times since every commit is CI > >>>> tested > >>>> > >>>> - Tier 2: actively maintained, no qtest coverage. Should usually > >>>> work but regression may creep in due to reliance on the > >>>> maintainer to manually test on adhoc basis > >>>> > >>>> - Tier 3: not actively maintained, unknown state but liable to > >>>> be broken indefinitely > >>>> > >>>> Tier 1 is obviously the most desirable state we would like everthing to > >>>> be at. Contributors will have to fix problems their patches cause as > >>>> they will be blocked by CI. > >>>> > >>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in > >>>> this tier will graduate to Tier 1 at some point. Even if it doesn't > >>>> though, it is still valid to keep it in QEMU long term. Contributors > >>>> shouldn't gratuitously break stuff in these board, but if they do, > >>>> then the maintainer is ultimately responsible for fixing it, as the > >>>> contributors don't have a test rig for it. > >>>> > >>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should > >>>> not expect it to continue to exist long term. Contributors are free > >>>> to send patches which break this, and are under no obligation to > >>>> fix problems in these boards. We may deprecate & delete it after a > >>>> while > >>>> > >>>> > >>>> Over time we'll likely add more criteria to stuff in Tier 1. This > >>>> could lead to some things dropping from Tier 1 to Tier 2. This is > >>>> OK, as it doesn't make those things worse than they already were. > >>>> We're just saying that Tier 2 isn't as thoroughly tested as we > >>>> would like it to be in an ideal world. > >>> > >>> I really like the idea of device support tiers codified directly in the > >>> QEMU codebase, to give upstream users some idea of which devices we > >>> expect to work and which we ... don't, really. > >>> > >>> Not every last device we offer is enterprise production ready, but we > >>> don't necessarily do a good job of explaining which devices fall into > >>> which categories, and we've got quite a few of them. > >>> > >>> I wonder if a 2.5th tier would be useful; something like a "hobbyist" > >>> tier for pet project SoC boards and the like -- they're not abandoned, > >>> but we also don't expect them to work, exactly. > >>> > >>> Mild semantic difference from Tier 3. > >> > >> I guess I was thinking such hobbyist stuff would fall into tier 2 if the > >> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they > >> largely aren't active on the mailing list responding to issues/questions. > >> > >> We add have a 4 tier system overall and put hobbyist stuff at tier 3, > >> and abandonware at tier 4. > >> > >> Probably shouldn't go beyond 4 tiers though, as the more criteria we add > >> the harder it is to clearly decide which tier something should go into. > >> > >> The tier 1 vs 2 divison is clearly split based on CI which is a simple > >> classification to decide on. > >> > >> The tier 2 vs 3 division is moderately clearly split based on whether > >> there is a frequently active maintainer. > >> > >> We can probably squeeze in the 4th tier without too much ambiguity in > >> the classisfication if we think it is adding something worthwhile either > >> from our POV as maintainers, or for users consuming it. > > > > Yes, I didn't mean to start watering it down into a 1,380 tier system > > that we're never able to properly utilize. > > > > I was thinking more along the lines of: > > > > - Device works and is well loved > > - Device works and is well loved (but we have to test manually) > > - Device doesn't work, but is well loved > > (Use at your own peril, please file a bug report) > > - Device doesn't work, and is unloved > > > > Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where > > things can shift from 1A to 1B as their test coverage allows, but it's > > not meant to indicate general status otherwise. > > All that sounds somewhat similar to the classification that we already > use in our MAINTAINERS file - Supported, Maintained, Odd-Fixes, Orphan, > Obsolete ... maybe we can avoid to introduce yet another classification > system and merge the two (e.g. by also changing the classification > system in MAINTAINERS a little bit?). Yes, there is a fair bit of conceptual overlap there. Main difference is probably that we'd want to be a bit more fine grained than the MAINTAINERS file shows, but that seems doable. Also somewhat related here is our security classification where we say we'll only consider bugs security vulnerabilities if they are in components that are targetted towards virtualization uses (ie KVM), and consider anything to do with non-virtualization use cases to be plain bugs, not security bugs. While as developers we can probably say which is which, for end users and bug reporter this is not so clear. So if we're going to classify support status for components, we should try to add a classification for security status at the same time. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-20 6:13 ` Thomas Huth 2020-05-20 9:02 ` Daniel P. Berrangé @ 2020-05-20 14:53 ` John Snow 1 sibling, 0 replies; 12+ messages in thread From: John Snow @ 2020-05-20 14:53 UTC (permalink / raw) To: Thomas Huth, Daniel P. Berrangé Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On 5/20/20 2:13 AM, Thomas Huth wrote: > On 20/05/2020 01.06, John Snow wrote: >> >> >> On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: >>> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: >>>> >>>> >>>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: >>>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: >>>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: >>>>>>> Hello, >>>>>>> >>>>>>> Following Markus thread on deprecating unmaintained (untested) code >>>>>>> (machines) [1] and the effort done to gather the information shared in >>>>>>> the replies [2], and the various acceptance tests added, is it >>>>>>> feasible to require for the next release that each new device/machine >>>>>>> is provided a test covering it? >>>>>>> >>>>>>> If no, what is missing? >>>>>> >>>>>> If a qtest is feasible, yes, I think we should require one for new >>>>>> devices. But what about machines - you normally need a test image for >>>>>> this. In that case, there is still the question where testing images >>>>>> could be hosted. Not every developer has a web space where they could >>>>>> put their test images onto. And what about images that contain non-free >>>>>> code? >>>>> >>>>> Yep, it isn't feasible to make this a hard rule. >>>>> >>>>> IMHO this is where a support tier classification comes into play >>>>> >>>>> - Tier 1: actively maintained, qtest coverage available. Expected >>>>> to work reliably at all times since every commit is CI >>>>> tested >>>>> >>>>> - Tier 2: actively maintained, no qtest coverage. Should usually >>>>> work but regression may creep in due to reliance on the >>>>> maintainer to manually test on adhoc basis >>>>> >>>>> - Tier 3: not actively maintained, unknown state but liable to >>>>> be broken indefinitely >>>>> >>>>> Tier 1 is obviously the most desirable state we would like everthing to >>>>> be at. Contributors will have to fix problems their patches cause as >>>>> they will be blocked by CI. >>>>> >>>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in >>>>> this tier will graduate to Tier 1 at some point. Even if it doesn't >>>>> though, it is still valid to keep it in QEMU long term. Contributors >>>>> shouldn't gratuitously break stuff in these board, but if they do, >>>>> then the maintainer is ultimately responsible for fixing it, as the >>>>> contributors don't have a test rig for it. >>>>> >>>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should >>>>> not expect it to continue to exist long term. Contributors are free >>>>> to send patches which break this, and are under no obligation to >>>>> fix problems in these boards. We may deprecate & delete it after a >>>>> while >>>>> >>>>> >>>>> Over time we'll likely add more criteria to stuff in Tier 1. This >>>>> could lead to some things dropping from Tier 1 to Tier 2. This is >>>>> OK, as it doesn't make those things worse than they already were. >>>>> We're just saying that Tier 2 isn't as thoroughly tested as we >>>>> would like it to be in an ideal world. >>>> >>>> I really like the idea of device support tiers codified directly in the >>>> QEMU codebase, to give upstream users some idea of which devices we >>>> expect to work and which we ... don't, really. >>>> >>>> Not every last device we offer is enterprise production ready, but we >>>> don't necessarily do a good job of explaining which devices fall into >>>> which categories, and we've got quite a few of them. >>>> >>>> I wonder if a 2.5th tier would be useful; something like a "hobbyist" >>>> tier for pet project SoC boards and the like -- they're not abandoned, >>>> but we also don't expect them to work, exactly. >>>> >>>> Mild semantic difference from Tier 3. >>> >>> I guess I was thinking such hobbyist stuff would fall into tier 2 if the >>> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they >>> largely aren't active on the mailing list responding to issues/questions. >>> >>> We add have a 4 tier system overall and put hobbyist stuff at tier 3, >>> and abandonware at tier 4. >>> >>> Probably shouldn't go beyond 4 tiers though, as the more criteria we add >>> the harder it is to clearly decide which tier something should go into. >>> >>> The tier 1 vs 2 divison is clearly split based on CI which is a simple >>> classification to decide on. >>> >>> The tier 2 vs 3 division is moderately clearly split based on whether >>> there is a frequently active maintainer. >>> >>> We can probably squeeze in the 4th tier without too much ambiguity in >>> the classisfication if we think it is adding something worthwhile either >>> from our POV as maintainers, or for users consuming it. >> >> Yes, I didn't mean to start watering it down into a 1,380 tier system >> that we're never able to properly utilize. >> >> I was thinking more along the lines of: >> >> - Device works and is well loved >> - Device works and is well loved (but we have to test manually) >> - Device doesn't work, but is well loved >> (Use at your own peril, please file a bug report) >> - Device doesn't work, and is unloved >> >> Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where >> things can shift from 1A to 1B as their test coverage allows, but it's >> not meant to indicate general status otherwise. > > All that sounds somewhat similar to the classification that we already > use in our MAINTAINERS file - Supported, Maintained, Odd-Fixes, Orphan, > Obsolete ... maybe we can avoid to introduce yet another classification > system and merge the two (e.g. by also changing the classification > system in MAINTAINERS a little bit?). > Yeah, it's similar, but the granularity is per-device instead of per-file, and it would be metadata that is introspectible. (As part of the help files and the CLI, when you look at a list of devices, I'd like to show which devices are in which support tiers.) The scope of each system is meant to be just a little different -- describing source files for git management vs describing logistical in-application API constructs. Still, you're right -- there's a lot of overlap. I'm not sure how to best bridge them. Though I would like to re-engage on the idea that we should be mechanically consuming and testing our MAINTAINERS file to ensure 100% coverage as a CI check. IIRC there was some pushback to this idea because we weren't sure who should own basic/generic files like "LICENSE" or "README" and so on. For files that SHOULD go "maintainerless", for which we would like checkpatch.pl to simply show you "who touched it last", maybe we should create a "Community Managed" section in MAINTAINERS that explicitly lists these files; and get_maintainer amended to understand this section. Maybe if we get the maintainers file getting checked regularly as a first step it will help us bridge the gap into device metadata without worrying about info desync. Just brainstorming. --js (Slightly OT: I am working on an email that contains a quick code-tour overview of what every file in the root of the document is, including which are maintained and unmaintained. Might be a good jumping off point for more MAINTAINER file discussions?) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-19 23:06 ` John Snow 2020-05-20 6:13 ` Thomas Huth @ 2020-05-20 8:57 ` Daniel P. Berrangé 1 sibling, 0 replies; 12+ messages in thread From: Daniel P. Berrangé @ 2020-05-20 8:57 UTC (permalink / raw) To: John Snow Cc: Peter Maydell, Thomas Huth, Eduardo Habkost, qemu-devel@nongnu.org Developers, Philippe Mathieu-Daudé, Markus Armbruster, Gerd Hoffmann, Cleber Rosa, Paolo Bonzini, qemu-discuss, David Gibson On Tue, May 19, 2020 at 07:06:40PM -0400, John Snow wrote: > > > On 5/19/20 5:04 AM, Daniel P. Berrangé wrote: > > On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote: > >> > >> > >> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote: > >>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: > >>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > >>>>> Hello, > >>>>> > >>>>> Following Markus thread on deprecating unmaintained (untested) code > >>>>> (machines) [1] and the effort done to gather the information shared in > >>>>> the replies [2], and the various acceptance tests added, is it > >>>>> feasible to require for the next release that each new device/machine > >>>>> is provided a test covering it? > >>>>> > >>>>> If no, what is missing? > >>>> > >>>> If a qtest is feasible, yes, I think we should require one for new > >>>> devices. But what about machines - you normally need a test image for > >>>> this. In that case, there is still the question where testing images > >>>> could be hosted. Not every developer has a web space where they could > >>>> put their test images onto. And what about images that contain non-free > >>>> code? > >>> > >>> Yep, it isn't feasible to make this a hard rule. > >>> > >>> IMHO this is where a support tier classification comes into play > >>> > >>> - Tier 1: actively maintained, qtest coverage available. Expected > >>> to work reliably at all times since every commit is CI > >>> tested > >>> > >>> - Tier 2: actively maintained, no qtest coverage. Should usually > >>> work but regression may creep in due to reliance on the > >>> maintainer to manually test on adhoc basis > >>> > >>> - Tier 3: not actively maintained, unknown state but liable to > >>> be broken indefinitely > >>> > >>> Tier 1 is obviously the most desirable state we would like everthing to > >>> be at. Contributors will have to fix problems their patches cause as > >>> they will be blocked by CI. > >>> > >>> Tier 2 is an admission that reality gets in the way. Ideally stuff in > >>> this tier will graduate to Tier 1 at some point. Even if it doesn't > >>> though, it is still valid to keep it in QEMU long term. Contributors > >>> shouldn't gratuitously break stuff in these board, but if they do, > >>> then the maintainer is ultimately responsible for fixing it, as the > >>> contributors don't have a test rig for it. > >>> > >>> Tier 3 is abandonware. If a maintainer doesn't appear, users should > >>> not expect it to continue to exist long term. Contributors are free > >>> to send patches which break this, and are under no obligation to > >>> fix problems in these boards. We may deprecate & delete it after a > >>> while > >>> > >>> > >>> Over time we'll likely add more criteria to stuff in Tier 1. This > >>> could lead to some things dropping from Tier 1 to Tier 2. This is > >>> OK, as it doesn't make those things worse than they already were. > >>> We're just saying that Tier 2 isn't as thoroughly tested as we > >>> would like it to be in an ideal world. > >> > >> I really like the idea of device support tiers codified directly in the > >> QEMU codebase, to give upstream users some idea of which devices we > >> expect to work and which we ... don't, really. > >> > >> Not every last device we offer is enterprise production ready, but we > >> don't necessarily do a good job of explaining which devices fall into > >> which categories, and we've got quite a few of them. > >> > >> I wonder if a 2.5th tier would be useful; something like a "hobbyist" > >> tier for pet project SoC boards and the like -- they're not abandoned, > >> but we also don't expect them to work, exactly. > >> > >> Mild semantic difference from Tier 3. > > > > I guess I was thinking such hobbyist stuff would fall into tier 2 if the > > hobbyist maintainer actually responds to fixing stuff, or tier 3 if they > > largely aren't active on the mailing list responding to issues/questions. > > > > We add have a 4 tier system overall and put hobbyist stuff at tier 3, > > and abandonware at tier 4. > > > > Probably shouldn't go beyond 4 tiers though, as the more criteria we add > > the harder it is to clearly decide which tier something should go into. > > > > The tier 1 vs 2 divison is clearly split based on CI which is a simple > > classification to decide on. > > > > The tier 2 vs 3 division is moderately clearly split based on whether > > there is a frequently active maintainer. > > > > We can probably squeeze in the 4th tier without too much ambiguity in > > the classisfication if we think it is adding something worthwhile either > > from our POV as maintainers, or for users consuming it. > > Yes, I didn't mean to start watering it down into a 1,380 tier system > that we're never able to properly utilize. > > I was thinking more along the lines of: > > - Device works and is well loved > - Device works and is well loved (but we have to test manually) > - Device doesn't work, but is well loved > (Use at your own peril, please file a bug report) > - Device doesn't work, and is unloved > > Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where > things can shift from 1A to 1B as their test coverage allows, but it's > not meant to indicate general status otherwise. Yeah 1A/1B would be fairly effective. > Mostly, I would just like some way for users to avoid accidentally > running tier 2 or 3 devices /by accident/, or the ability to compile > QEMU versions that only allow tier 1 devices to be used. > > It's all arbitrary -- but I think we agree more than not! I'd love to > have a list of first-class boards and devices that we promise to test > and have working. Yes, I think we're basically in agreement on the both the goal and way to achieve it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-15 10:11 ` Thomas Huth 2020-05-15 10:23 ` Daniel P. Berrangé @ 2020-05-15 10:51 ` Gerd Hoffmann 2020-05-15 11:24 ` Paolo Bonzini 1 sibling, 1 reply; 12+ messages in thread From: Gerd Hoffmann @ 2020-05-15 10:51 UTC (permalink / raw) To: Thomas Huth Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Markus Armbruster, qemu-discuss, Cleber Rosa, Paolo Bonzini, David Gibson, Philippe Mathieu-Daudé On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote: > On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote: > > Hello, > > > > Following Markus thread on deprecating unmaintained (untested) code > > (machines) [1] and the effort done to gather the information shared in > > the replies [2], and the various acceptance tests added, is it > > feasible to require for the next release that each new device/machine > > is provided a test covering it? > > > > If no, what is missing? > > If a qtest is feasible, yes, I think we should require one for new > devices. qtest is not feasible, at least not for all kinds of devices. You can't talk to usb devices for example, and changing that effectively requires writing uhci/ohci/ehci/xhci drivers so one can submit usb transfers in qtest. The current qtests for usb devices is basically hot-plug and unplug it and check qmp doesn't throw an error. Better that nothing, but there is zero functional testing and changing that is seriously non-trivial. cheers, Gerd ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: QEMU 5.1: Can we require each new device/machine to provided a test? 2020-05-15 10:51 ` Gerd Hoffmann @ 2020-05-15 11:24 ` Paolo Bonzini 0 siblings, 0 replies; 12+ messages in thread From: Paolo Bonzini @ 2020-05-15 11:24 UTC (permalink / raw) To: Gerd Hoffmann, Thomas Huth Cc: Peter Maydell, Eduardo Habkost, qemu-devel@nongnu.org Developers, Markus Armbruster, qemu-discuss, Cleber Rosa, David Gibson, Philippe Mathieu-Daudé On 15/05/20 12:51, Gerd Hoffmann wrote: >> If a qtest is feasible, yes, I think we should require one for new >> devices. > qtest is not feasible, at least not for all kinds of devices. You can't > talk to usb devices for example, and changing that effectively requires > writing uhci/ohci/ehci/xhci drivers so one can submit usb transfers in > qtest. Yes, but that's just because no one has written a test for a USB device. It would be completely feasible to plug USB into qgraph, the code for USB in SeaBIOS is 4.5 kLOC so that's more or less the ballpark of what you'd need. You can then stimulate keyboard or mouse devices via chardev (-serial msmouse), or run SCSI tests for MSD and UAS. Paolo ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-05-20 14:55 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-07 10:59 QEMU 5.1: Can we require each new device/machine to provided a test? Philippe Mathieu-Daudé 2020-05-15 10:11 ` Thomas Huth 2020-05-15 10:23 ` Daniel P. Berrangé 2020-05-18 19:56 ` John Snow 2020-05-19 9:04 ` Daniel P. Berrangé 2020-05-19 23:06 ` John Snow 2020-05-20 6:13 ` Thomas Huth 2020-05-20 9:02 ` Daniel P. Berrangé 2020-05-20 14:53 ` John Snow 2020-05-20 8:57 ` Daniel P. Berrangé 2020-05-15 10:51 ` Gerd Hoffmann 2020-05-15 11:24 ` Paolo Bonzini
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.