Proposal: Feature Maturity Lifecycle

Proposal: Feature Maturity Lifecycle

[Lars Kurth]

Hi all,

following up from http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01775.html I wanted to propose the following convention related to feature classification as a proposal for discussion. I tried to pretty much describe what we do now and hope my understanding is correct. I did propose a new Complete state, to cover for bigger new features which we may not want to mark as supported straight away, for various reasons.

I marked areas which with some of my thoughts or references to other documents in [Note: ...] brackets

Regards
Lars

----

        Feature Maturity Lifecycle
        ==========================

        Purpose and Scope
        -----------------

The purpose of this document is to define the possible states a feature. 
*Features* may be one of the following:
* End user Feature: aka a pice of functionality that is controllable 
  by users of Xen through a command line option exposed via xl or 
  libvirt, a config file or otherwise
* Support for a Hardware platform: e.g. x86 or ARM
* Support for a specific Hardware platforms, e.g. Thunder X, 
  Xilinx ZynqMP, etc.
* A set of APIs, e.g. men-event API, memory sharing APIs, etc.

It is also the intent, that this document is used to set expectations
of contributors to the Xen Project hypervisor, such that they can take 
appropriate steps to ensure that eventually a feature that they develop
is a supported by the community.

Further, it will help users of the Xen Project hypervisors understand
how developers use the feature states. This will help users make informed
decisions about the risk involved in using a feature. 

        Dependencies
        -------------

This document refers to definitions in other files and project conventions,
in particular:

1.      Status of subsystems in the MAINTAINERS file
	The MAINTAINERS file maps individuals against groups of files 
        in the source tree. In there context of this document, we say that
        a feature is *maintained*, iff all components of that feature
        are marked as one of the following
        Supported:  Someone is actually paid to look after the code.
        Maintained: Someone actually looks after the code.

2.      Classification of public APIs and interfaces: 
	APIs and other interfaces are declared stable by agreement
        on the xen-devel@ mailing list. We consider a feature as
        *stable*, iff all public APIs and interfaces a feature depends 
	on have been declared stable.

3. 	Testing
        The Xen Project runs a continuous test and integration system 
        in the project's test lab. We consider a feature *tested*, iff 
        there are test cases that are run in our test lab, that test a 	
        specific feature and consistently pass. For hardware platforms we 
        consider a platform *tested*, iff appropriate hardware for said 
        platform is live and in active use in our test lab and tests
        consistently pass against that platform.

        In some cases, it may not be possible to add hardware to the
        Xen Project test lab (e.g. for cost, space, or other reasons).
        In such cases, it is acceptable for a community member or 
        organisation to run their own tests on behalf of the community.
        In such cases, we consider a feature or platform *tested*
        if said community member tests a feature/platform using their
        own infrastructure and regularly and consistently reports results 
        to the community via xen-devel@. At a minimum, release candidates 
        should be tested.

4.      Documentation
        The Xen Project requires that documentation for user facing 
        features and in-code API documentation (or user guides as 
        appropriate) are provided in tree. We say that a feature as
        *documented*, if relevant documentation has been committed to
        the tree.

        State Definitions
        -----------------

        This section lists state definitions of a *Feature* in terms of
        properties. States are listed in order of increasing number
        of properties. Note that note all features will require to go 
        through each state: for example small and non-risky features
        may go straight from under development to supported. It is up to 
        the development community to judge and discuss, which states
        are necessary based on the size and risk of a feature.

1.	Preview
        - Partially completed, with missing functionality
        - May not be fully functional in all cases
        - May not be tested
        - APIs and interfaces may not be stable
        - The developer is actively looking for user feedback
        - Bugs and issues can be raised on xen-devel@ and will be
          handled on a best-effort basis

[Note:  the term prototype is listed in some cases in 
        http://wiki.xenproject.org/wiki/Xen_Project_Release_Features -
        I propose to purge prototype and replace with Preview]

2.      Experimental
        - Core functionality is fully functional
        - However, not all functionality or platform support may be
          present
        - May not be tested, although there is an expectation that a plan
          to improve testing is in place or worked on
        - APIs and interfaces may not be stable
        - Bugs and issues can be raised on xen-devel@ and will be
          handled on a best-effort basis. However, there is an expectation 
          that issues related to broken core functionality are addressed.

3.      Complete
[Note:  This is a state which has not existed in the past. It is aimed
        at larger new features, which may only be in use or of interest
        to a small number of contributors, or where not enough expertise
        exists in the community to treat the feature as *Supported*. It
        presents an opportunity for developers to prove over one or
        two release cycles that said feature can be *supported* in future.]
        - Intended functionality is fully implemented
        - Feature is *maintained*
        - Feature is *tested*
        - Feature is *stable*
        - Feature is *documented*
        - Bugs and issues can be raised on xen-devel@ and will be
          handled by the developers/maintainers behind the feature. There 
          is an expectation for the developers/maintainers to address 
          bugs and issues over a period of time. 
        - Regressions and blockers in a complete feature do *not* normally
          block new releases. It is at the discretion of the community
          and Release Manager to downgrade the feature. 
        - Security issues would *not* be handled by the security team.
    
4.      Supported
        - Intended functionality is fully implemented
        - Feature is *maintained*
        - Feature is *tested*
        - Feature is *stable*
        - Feature is *documented* 
        - Bugs and issues can be raised on xen-devel@ and will be
          handled by the developers/maintainers/committers within the 
          community.
        - Regressions and blockers in a complete feature do normally
          block new releases. 
        - Security issues would be handled by the security team.

5.      Deprecated
        There are typically two scenarios when a feature would be 
        deprecated. 
5.1.    - If the feature is not relevant any more or has been replaced
          by a new implementation (e.g. xm vs. xl)
5.2     - If we have lost the capability to support a feature.
          For example when we have lost the capability to *maintain*
          the feature, because we do not have maintainers. In such cases
          raising the issue usually will lead to a resolution, if there
          is enough usage by vendors in the eco-system. 
        Features in any state can be deprecated.

        State Changes
        -------------        
[Note:  this assumes that we keep a document in the source tree which
        provides a snapshot of information akin a snapshot of
        http://wiki.xenproject.org/wiki/Xen_Project_Release_Features
        in the source tree. I am volunteering to maintain the wiki
        and initially populate the file based on existing information.
        Andy Cooper suggested he is willing to put together a template
        with some examples. See http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01369.html]

        The intention here is to require that the central file is modified
        with a patch that introduces a feature (state, feature title,
        short description) and that developers which add functionality
        modify accordingly. The release manager and other stake-holders
        such as the community may also propose changes during the release
        cycle (in particular towards the end). 

        Also, if a feature has been for too long on in an incomplete 
        state (e.g. Preview or Experimental) or some of the assumptions
        associated with a state do not hold any more, community members 
        may propose to downgrade a feature. Typically a discussion on
        the list would be expected before a patch to the file is proposed.

Re: Proposal: Feature Maturity Lifecycle

[George Dunlap]

On Fri, Jun 12, 2015 at 4:22 PM, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
> Hi all,
>
> following up from http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01775.html I wanted to propose the following convention related to feature classification as a proposal for discussion. I tried to pretty much describe what we do now and hope my understanding is correct. I did propose a new Complete state, to cover for bigger new features which we may not want to mark as supported straight away, for various reasons.
>
> I marked areas which with some of my thoughts or references to other documents in [Note: ...] brackets
>
> Regards
> Lars
>
> ----
>
>         Feature Maturity Lifecycle
>         ==========================
>
>         Purpose and Scope
>         -----------------
>
> The purpose of this document is to define the possible states a feature.
> *Features* may be one of the following:
> * End user Feature: aka a pice of functionality that is controllable
>   by users of Xen through a command line option exposed via xl or
>   libvirt, a config file or otherwise
> * Support for a Hardware platform: e.g. x86 or ARM
> * Support for a specific Hardware platforms, e.g. Thunder X,
>   Xilinx ZynqMP, etc.
> * A set of APIs, e.g. men-event API, memory sharing APIs, etc.
>
> It is also the intent, that this document is used to set expectations
> of contributors to the Xen Project hypervisor, such that they can take
> appropriate steps to ensure that eventually a feature that they develop
> is a supported by the community.
>
> Further, it will help users of the Xen Project hypervisors understand
> how developers use the feature states. This will help users make informed
> decisions about the risk involved in using a feature.
>
>         Dependencies
>         -------------
>
> This document refers to definitions in other files and project conventions,
> in particular:
>
> 1.      Status of subsystems in the MAINTAINERS file
>         The MAINTAINERS file maps individuals against groups of files
>         in the source tree. In there context of this document, we say that
>         a feature is *maintained*, iff all components of that feature
>         are marked as one of the following
>         Supported:  Someone is actually paid to look after the code.
>         Maintained: Someone actually looks after the code.
>
> 2.      Classification of public APIs and interfaces:
>         APIs and other interfaces are declared stable by agreement
>         on the xen-devel@ mailing list. We consider a feature as
>         *stable*, iff all public APIs and interfaces a feature depends
>         on have been declared stable.
>
> 3.      Testing
>         The Xen Project runs a continuous test and integration system
>         in the project's test lab. We consider a feature *tested*, iff
>         there are test cases that are run in our test lab, that test a
>         specific feature and consistently pass. For hardware platforms we
>         consider a platform *tested*, iff appropriate hardware for said
>         platform is live and in active use in our test lab and tests
>         consistently pass against that platform.
>
>         In some cases, it may not be possible to add hardware to the
>         Xen Project test lab (e.g. for cost, space, or other reasons).
>         In such cases, it is acceptable for a community member or
>         organisation to run their own tests on behalf of the community.
>         In such cases, we consider a feature or platform *tested*
>         if said community member tests a feature/platform using their
>         own infrastructure and regularly and consistently reports results
>         to the community via xen-devel@. At a minimum, release candidates
>         should be tested.
>
> 4.      Documentation
>         The Xen Project requires that documentation for user facing
>         features and in-code API documentation (or user guides as
>         appropriate) are provided in tree. We say that a feature as
>         *documented*, if relevant documentation has been committed to
>         the tree.
>
>         State Definitions
>         -----------------
>
>         This section lists state definitions of a *Feature* in terms of
>         properties. States are listed in order of increasing number
>         of properties. Note that note all features will require to go
>         through each state: for example small and non-risky features
>         may go straight from under development to supported. It is up to
>         the development community to judge and discuss, which states
>         are necessary based on the size and risk of a feature.
>
> 1.      Preview
>         - Partially completed, with missing functionality
>         - May not be fully functional in all cases
>         - May not be tested
>         - APIs and interfaces may not be stable
>         - The developer is actively looking for user feedback
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled on a best-effort basis
>
> [Note:  the term prototype is listed in some cases in
>         http://wiki.xenproject.org/wiki/Xen_Project_Release_Features -
>         I propose to purge prototype and replace with Preview]
>
> 2.      Experimental
>         - Core functionality is fully functional
>         - However, not all functionality or platform support may be
>           present
>         - May not be tested, although there is an expectation that a plan
>           to improve testing is in place or worked on
>         - APIs and interfaces may not be stable
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled on a best-effort basis. However, there is an expectation
>           that issues related to broken core functionality are addressed.
>
> 3.      Complete
> [Note:  This is a state which has not existed in the past. It is aimed
>         at larger new features, which may only be in use or of interest
>         to a small number of contributors, or where not enough expertise
>         exists in the community to treat the feature as *Supported*. It
>         presents an opportunity for developers to prove over one or
>         two release cycles that said feature can be *supported* in future.]
>         - Intended functionality is fully implemented
>         - Feature is *maintained*
>         - Feature is *tested*
>         - Feature is *stable*
>         - Feature is *documented*
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled by the developers/maintainers behind the feature. There
>           is an expectation for the developers/maintainers to address
>           bugs and issues over a period of time.
>         - Regressions and blockers in a complete feature do *not* normally
>           block new releases. It is at the discretion of the community
>           and Release Manager to downgrade the feature.
>         - Security issues would *not* be handled by the security team.
>
> 4.      Supported
>         - Intended functionality is fully implemented
>         - Feature is *maintained*
>         - Feature is *tested*
>         - Feature is *stable*
>         - Feature is *documented*
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled by the developers/maintainers/committers within the
>           community.
>         - Regressions and blockers in a complete feature do normally
>           block new releases.
>         - Security issues would be handled by the security team.

So if we accept these definitions, it would would officially make
adding functionality to osstest a requirement for everybody
contributing new functionality.  I think at the moment, that may be
too high a barrier, unless it becomes simple and straightforward to
write, test, and contribute new tests to osstest.  (I think Stefano
was trying to do this a bit with the raisin project, but that's not
complete yet.)

Another practical issue: There are existing large bits of
functionality that are currently considered "supported" that are not
tested as part of our continuous integration testing.  PCI
pass-through is, I think, one major example.  According to this
classification, a lot of our features would have to move back to
"Experimental" until we managed to get things tested and documented.

Do we want to have another category -- perhaps "Legacy-Stable" or
something like that -- to indicate functionality which has been sort
of "grandfathered" in to the "supported" state?  That would allow us
to indicate that this functionality is actually usable, while keeping
us honest and motivating us to move things into new "stable".

 -George

Re: Proposal: Feature Maturity Lifecycle

[Ian Jackson]

Lars Kurth writes ("Proposal: Feature Maturity Lifecycle"):
> following up from http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01775.html I wanted to propose the following convention related to feature classification as a proposal for discussion. I tried to pretty much describe what we do now and hope my understanding is correct. I did propose a new Complete state, to cover for bigger new features which we may not want to mark as supported straight away, for various reasons.
> 
> I marked areas which with some of my thoughts or references to other documents in [Note: ...] brackets

There is much here that is good.

>         Dependencies
>         -------------
> 
> This document refers to definitions in other files and project conventions,
> in particular:
> 
> 1.      Status of subsystems in the MAINTAINERS file
> 	The MAINTAINERS file maps individuals against groups of files 
>         in the source tree. In there context of this document, we say that
>         a feature is *maintained*, iff all components of that feature
>         are marked as one of the following
>         Supported:  Someone is actually paid to look after the code.
>         Maintained: Someone actually looks after the code.

As part of this, we should rename the `Supported' field in
MAINTAINERS.  I'm not sure that `Supported' makes enough of a
difference from `Maintained' to be worth distinguishing.  So I would
propose to abolish the category `Supported'.

If we do want to continue to distinguish what is currently called
`Supported' it needs a new name.

>         State Changes
>         -------------        
> [Note:  this assumes that we keep a document in the source tree which
>         provides a snapshot of information akin a snapshot of
>         http://wiki.xenproject.org/wiki/Xen_Project_Release_Features
>         in the source tree. I am volunteering to maintain the wiki
>         and initially populate the file based on existing information.
>         Andy Cooper suggested he is willing to put together a template
>         with some examples. See http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01369.html]

I think that we should avoid introducing new manual-transfer tasks.

The document should be in tree (in docs/) in such a way that it is
automatically published (on xenbits) along with the other
documentation.

>         The intention here is to require that the central file is modified
>         with a patch that introduces a feature (state, feature title,
>         short description) and that developers which add functionality
>         modify accordingly. The release manager and other stake-holders
>         such as the community may also propose changes during the release
>         cycle (in particular towards the end). 

Right.

Ian.

Re: Proposal: Feature Maturity Lifecycle

[Lars Kurth]

> On 15 Jun 2015, at 13:21, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> 
> Lars Kurth writes ("Proposal: Feature Maturity Lifecycle"):
>> following up from http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01775.html I wanted to propose the following convention related to feature classification as a proposal for discussion. I tried to pretty much describe what we do now and hope my understanding is correct. I did propose a new Complete state, to cover for bigger new features which we may not want to mark as supported straight away, for various reasons.
>> 
>> I marked areas which with some of my thoughts or references to other documents in [Note: ...] brackets
> 
> There is much here that is good.
> 
>>        Dependencies
>>        -------------
>> 
>> This document refers to definitions in other files and project conventions,
>> in particular:
>> 
>> 1.      Status of subsystems in the MAINTAINERS file
>> 	The MAINTAINERS file maps individuals against groups of files 
>>        in the source tree. In there context of this document, we say that
>>        a feature is *maintained*, iff all components of that feature
>>        are marked as one of the following
>>        Supported:  Someone is actually paid to look after the code.
>>        Maintained: Someone actually looks after the code.
> 
> As part of this, we should rename the `Supported' field in
> MAINTAINERS.  I'm not sure that `Supported' makes enough of a
> difference from `Maintained' to be worth distinguishing.  So I would
> propose to abolish the category `Supported'.
> 
> If we do want to continue to distinguish what is currently called
> `Supported' it needs a new name.

Agreed. How about replacing Supported by "Company-Maintained" or something similar.

> 
>>        State Changes
>>        -------------        
>> [Note:  this assumes that we keep a document in the source tree which
>>        provides a snapshot of information akin a snapshot of
>>        http://wiki.xenproject.org/wiki/Xen_Project_Release_Features
>>        in the source tree. I am volunteering to maintain the wiki
>>        and initially populate the file based on existing information.
>>        Andy Cooper suggested he is willing to put together a template
>>        with some examples. See http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01369.html]
> 
> I think that we should avoid introducing new manual-transfer tasks.
> 
> The document should be in tree (in docs/) in such a way that it is
> automatically published (on xenbits) along with the other
> documentation.

I think we are thinking along the same lines here and the intention is to generate a snapshot of functionality in tree.

However, I do also believe that the time evolution view in http://wiki.xenproject.org/wiki/Xen_Project_Release_Features is valuable and it is probably too hard to do this in an automated fashion, at least initially. So my offer is to maintain that manually, until we have some automated way of doing this.

Regards
Lars

Re: Proposal: Feature Maturity Lifecycle

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] Proposal: Feature Maturity Lifecycle"):
> So if we accept these definitions, it would would officially make
> adding functionality to osstest a requirement for everybody
> contributing new functionality.  I think at the moment, that may be
> too high a barrier, unless it becomes simple and straightforward to
> write, test, and contribute new tests to osstest.  (I think Stefano
> was trying to do this a bit with the raisin project, but that's not
> complete yet.)

Adding tests to osstest is always going to be difficult, at least for
some features or hardware.  It often involves a mixture of hardware
and software, and may involve complicated setup.  Xen by its nature is
slightly awkward to write tests for, because even for a simple test
you need to have a whole machine set up running Xen, and for many
features you need to teach the test suite how to set up the system to
use them.

That's not to say we shouldn't continue to make osstest easier to
extend, and provide other ways to get tests executed (eg, provide some
working in-tree tests in xen.git).  But we need to acknowledge that
there is often some irreducible difficulty in the task of adding test
cases.


A more practical requirement for a feature with "Supported" status is
that _either_:

 * The feature is tested automatically.

 * At least once during each release freeze, the feature's
   maintainers produce a test report (by a deadline specified by
   the release manager).  Features with no test report get
   downgraded from "Supported" to some lower status.


Ian.

Re: Proposal: Feature Maturity Lifecycle

[Ian Jackson]

Lars Kurth writes ("Re: Proposal: Feature Maturity Lifecycle"):
> However, I do also believe that the time evolution view in http://wiki.xenproject.org/wiki/Xen_Project_Release_Features is valuable and it is probably too hard to do this in an automated fashion, at least initially. So my offer is to maintain that manually, until we have some automated way of doing this.

I agree that that is very useful.

We should aim for the feature listings in the in-tree document to be
automatically parsed to extract this kind of information, so that
such a table can be produced automatically, when you get bored of
doing it by hand :-).

Ian.

Re: Proposal: Feature Maturity Lifecycle

[Lars Kurth]

> On 15 Jun 2015, at 12:26, George Dunlap <dunlapg@umich.edu> wrote:
> 
> On Fri, Jun 12, 2015 at 4:22 PM, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
>> Hi all,
>> 
>> <snip>
>> 4.      Supported
>>        - Intended functionality is fully implemented
>>        - Feature is *maintained*
>>        - Feature is *tested*
>>        - Feature is *stable*
>>        - Feature is *documented*
>>        - Bugs and issues can be raised on xen-devel@ and will be
>>          handled by the developers/maintainers/committers within the
>>          community.
>>        - Regressions and blockers in a complete feature do normally
>>          block new releases.
>>        - Security issues would be handled by the security team.
> 
> So if we accept these definitions, it would would officially make
> adding functionality to osstest a requirement for everybody
> contributing new functionality.  I think at the moment, that may be
> too high a barrier, unless it becomes simple and straightforward to
> write, test, and contribute new tests to osstest.  (I think Stefano
> was trying to do this a bit with the raisin project, but that's not
> complete yet.)

That is a good point. Part of the reason for allowing 3rd party testing is to help with that, but this may not be sufficient.
The question is: how much of a gap do have today (or will we have by Xen 4.6)

> Another practical issue: There are existing large bits of
> functionality that are currently considered "supported" that are not
> tested as part of our continuous integration testing.  PCI
> pass-through is, I think, one major example.  According to this
> classification, a lot of our features would have to move back to
> "Experimental" until we managed to get things tested and documented.

PCI passthrough is interesting, because am not convinced we do handle security issues for it. Or are we?
Also, it is maybe something that isn't as easy to test automatically.
What other features that are supported do not get any testing?

> Do we want to have another category -- perhaps "Legacy-Stable" or
> something like that -- to indicate functionality which has been sort
> of "grandfathered" in to the "supported" state?  That would allow us
> to indicate that this functionality is actually usable, while keeping
> us honest and motivating us to move things into new "stable".

Another way would be to allow exceptions to cover for this case (we can use a footnote to that effect in the master document pointing that out). 
I don't mind either way, but it feels to me that we shouldn't introduce a new state for a boundary case. 
It all depends on how much of a problem do we have

Regards
Lars

Re: Proposal: Feature Maturity Lifecycle

[Lars Kurth]

> On 15 Jun 2015, at 13:31, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> 
> George Dunlap writes ("Re: [Xen-devel] Proposal: Feature Maturity Lifecycle"):
>> So if we accept these definitions, it would would officially make
>> adding functionality to osstest a requirement for everybody
>> contributing new functionality.  I think at the moment, that may be
>> too high a barrier, unless it becomes simple and straightforward to
>> write, test, and contribute new tests to osstest.  (I think Stefano
>> was trying to do this a bit with the raisin project, but that's not
>> complete yet.)
> 
> Adding tests to osstest is always going to be difficult, at least for
> some features or hardware.  It often involves a mixture of hardware
> and software, and may involve complicated setup.  Xen by its nature is
> slightly awkward to write tests for, because even for a simple test
> you need to have a whole machine set up running Xen, and for many
> features you need to teach the test suite how to set up the system to
> use them.
> 
> That's not to say we shouldn't continue to make osstest easier to
> extend, and provide other ways to get tests executed (eg, provide some
> working in-tree tests in xen.git).  But we need to acknowledge that
> there is often some irreducible difficulty in the task of adding test
> cases.
> 

Agreed

> 
> A more practical requirement for a feature with "Supported" status is
> that _either_:
> 
> * The feature is tested automatically.
> 
> * At least once during each release freeze, the feature's
>   maintainers produce a test report (by a deadline specified by
>   the release manager).  Features with no test report get
>   downgraded from "Supported" to some lower status.


That would be a reasonable enough compromise. Although it does create extra overhead for the release manager

Lars

Re: Proposal: Feature Maturity Lifecycle

[Ian Jackson]

Lars Kurth writes ("Re: [Xen-devel] Proposal: Feature Maturity Lifecycle"):
> PCI passthrough is interesting, because am not convinced we do
> handle security issues for it. Or are we?

We do handle PCI passthrough security issues, yes.  A large proportion
of XSAs are due to PCI passthrough.

Ian.

Re: Proposal: Feature Maturity Lifecycle

[George Dunlap]

On Mon, Jun 15, 2015 at 1:31 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> George Dunlap writes ("Re: [Xen-devel] Proposal: Feature Maturity Lifecycle"):
>> So if we accept these definitions, it would would officially make
>> adding functionality to osstest a requirement for everybody
>> contributing new functionality.  I think at the moment, that may be
>> too high a barrier, unless it becomes simple and straightforward to
>> write, test, and contribute new tests to osstest.  (I think Stefano
>> was trying to do this a bit with the raisin project, but that's not
>> complete yet.)
>
> Adding tests to osstest is always going to be difficult, at least for
> some features or hardware.  It often involves a mixture of hardware
> and software, and may involve complicated setup.  Xen by its nature is
> slightly awkward to write tests for, because even for a simple test
> you need to have a whole machine set up running Xen, and for many
> features you need to teach the test suite how to set up the system to
> use them.
>
> That's not to say we shouldn't continue to make osstest easier to
> extend, and provide other ways to get tests executed (eg, provide some
> working in-tree tests in xen.git).  But we need to acknowledge that
> there is often some irreducible difficulty in the task of adding test
> cases.

Sure; for example, adding a host usb testcase would involve extending
osstest to know how to find a sensible USB device to pass through.

But many test cases wouldn't necessarily need too much more
infrastructure.  For cpupools, for instance, it would be fairly easy
to write a stand-alone script to randomly perform cpupool operations
with nothing but the ip of the dom0 of the host.

> A more practical requirement for a feature with "Supported" status is
> that _either_:
>
>  * The feature is tested automatically.
>
>  * At least once during each release freeze, the feature's
>    maintainers produce a test report (by a deadline specified by
>    the release manager).  Features with no test report get
>    downgraded from "Supported" to some lower status.

This seems a reasonable compromise.  Rather than forcing a maintainer
to engage with osstest as a condition of considering the feature
flagged Supported, we gently nudge them to automate it to save
themselves the hassle of testing it every release. :-)

 -George

Re: Proposal: Feature Maturity Lifecycle

[Lars Kurth]

> On 15 Jun 2015, at 15:51, George Dunlap <dunlapg@umich.edu> wrote:
> 
>> A more practical requirement for a feature with "Supported" status is
>> that _either_:
>> 
>> * The feature is tested automatically.
>> 
>> * At least once during each release freeze, the feature's
>>   maintainers produce a test report (by a deadline specified by
>>   the release manager).  Features with no test report get
>>   downgraded from "Supported" to some lower status.
> 
> This seems a reasonable compromise.  Rather than forcing a maintainer
> to engage with osstest as a condition of considering the feature
> flagged Supported, we gently nudge them to automate it to save
> themselves the hassle of testing it every release. :-)

Alright. I will make a note of this and amend the next version. 

I was wondering whether any maintainers who will be impacted by this and the Release Manager (current and past) have a view

Also, please let me know of any additional feedback on the main proposal, such that I can make a note of people's views and make modifications.

Best Regards
Lars

Re: Proposal: Feature Maturity Lifecycle

[Wei Liu]

On Mon, Jun 15, 2015 at 05:34:08PM +0100, Lars Kurth wrote:
> 
> > On 15 Jun 2015, at 15:51, George Dunlap <dunlapg@umich.edu> wrote:
> > 
> >> A more practical requirement for a feature with "Supported" status is
> >> that _either_:
> >> 
> >> * The feature is tested automatically.
> >> 
> >> * At least once during each release freeze, the feature's
> >>   maintainers produce a test report (by a deadline specified by
> >>   the release manager).  Features with no test report get
> >>   downgraded from "Supported" to some lower status.
> > 
> > This seems a reasonable compromise.  Rather than forcing a maintainer
> > to engage with osstest as a condition of considering the feature
> > flagged Supported, we gently nudge them to automate it to save
> > themselves the hassle of testing it every release. :-)
> 
> Alright. I will make a note of this and amend the next version. 
> 
> I was wondering whether any maintainers who will be impacted by this and the Release Manager (current and past) have a view
> 

The end result is probably we spam all maintainers for their components,
regardlessly whether that components are tested in OSSTest or not.
Triaging whether a component is tested by OSSTest can only be done by
hand and is tedious and time consuming.

The extra work load may be moderate if we only do it once per release
and have the corresponding tool to generate emails automatically.

Wei.

> Also, please let me know of any additional feedback on the main proposal, such that I can make a note of people's views and make modifications.
> 
> Best Regards
> Lars
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Proposal: Feature Maturity Lifecycle

[George Dunlap]

On Tue, Jun 16, 2015 at 3:06 PM, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jun 15, 2015 at 05:34:08PM +0100, Lars Kurth wrote:
>>
>> > On 15 Jun 2015, at 15:51, George Dunlap <dunlapg@umich.edu> wrote:
>> >
>> >> A more practical requirement for a feature with "Supported" status is
>> >> that _either_:
>> >>
>> >> * The feature is tested automatically.
>> >>
>> >> * At least once during each release freeze, the feature's
>> >>   maintainers produce a test report (by a deadline specified by
>> >>   the release manager).  Features with no test report get
>> >>   downgraded from "Supported" to some lower status.
>> >
>> > This seems a reasonable compromise.  Rather than forcing a maintainer
>> > to engage with osstest as a condition of considering the feature
>> > flagged Supported, we gently nudge them to automate it to save
>> > themselves the hassle of testing it every release. :-)
>>
>> Alright. I will make a note of this and amend the next version.
>>
>> I was wondering whether any maintainers who will be impacted by this and the Release Manager (current and past) have a view
>>
>
> The end result is probably we spam all maintainers for their components,
> regardlessly whether that components are tested in OSSTest or not.
> Triaging whether a component is tested by OSSTest can only be done by
> hand and is tedious and time consuming.
>
> The extra work load may be moderate if we only do it once per release
> and have the corresponding tool to generate emails automatically.

It wouldn't be that hard to mark, in our "supported feature" thing,
which features were in osstest and which features needed manual
testeng.  And then it wouldn't be that hard for a script to give you
1) a list of e-mail addresses of maintainers of features which need
manual testing, and 2) a boilerplate text that contained all the
features and a request for a release testing report.

I would envision it to be something like the monthly "release update"
e-mails -- each person scans through for their own projects and
replies-to-all.

 -George