[RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Lars Kurth]

- Incorporated feedback from
  http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html

Open Issues:
- Did not build and test the doc yet (want to get the content right first)
- Decide on supported status in MAINTAINER file
- Resolve loose ends on where and how to record Feature State

Signed-off-by: Lars Kurth <lars.kurth@xenproject.org>
---
 docs/features/feature-maturity-lifecycle.pandoc | 260 ++++++++++++++++++++++++
 1 file changed, 260 insertions(+)
 create mode 100644 docs/features/feature-maturity-lifecycle.pandoc

diff --git a/docs/features/feature-maturity-lifecycle.pandoc
b/docs/features/feature-maturity-lifecycle.pandoc
new file mode 100644
index 0000000..46f98a7
--- /dev/null
+++ b/docs/features/feature-maturity-lifecycle.pandoc
@@ -0,0 +1,260 @@
+% Feature Maturity Lifecycle
+% Revision 1
+
+\clearpage
+
+# Basics
+--------------- -------------------
+        Status: **Proposal**
+
+     Component: Development Process
+--------------- -------------------
+
+# Purpose and Scope
+
+The purpose of this document is to define the possible states a feature.
+**Features** may be one of the following:
+* End user Feature: aka a pice of functionality that is controllable
+  by users of Xen through a command line option exposed via xl or
+  libvirt, a config file or otherwise
+* Support for a Hardware platform: e.g. x86 or ARM
+* Support for a specific Hardware platforms, e.g. Thunder X,
+  Xilinx ZynqMP, etc.
+* A set of APIs, e.g. mem-event API, memory sharing APIs, etc.
+
+It is also the intent, that this document is used to set expectations
+of contributors to the Xen Project hypervisor, such that they can take
+appropriate steps to ensure that eventually a feature that they develop
+is a supported by the community.
+
+Further, it will help users of the Xen Project hypervisors understand
+how developers use the feature states. This will help users make informed
+decisions about the risk involved in using a feature.
+
+# Dependencies
+
+This document refers to definitions in other files and project conventions,
+in particular:
+
+## Status of subsystems in the MAINTAINERS file
+The [MAINTAINERS]
+(http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=MAINTAINERS;hb=HEAD)
+file maps individuals against groups of files in the source tree. In
+the context of this document, we say that a feature is **maintained**,
+iff all components of that feature are marked as one of the following
+* **Supported**:  Someone is actually paid to look after the code.
+* **Maintained**: Someone actually looks after the code.
+
+## Classification of public APIs and interfaces:
+APIs and other interfaces are declared stable by agreement
+on the xen-devel@ mailing list. We consider a feature as
+**stable**, iff all public APIs and interfaces a feature depends
+on have been declared stable.
+
+## Testing
+The Xen Project runs a continuous test and integration system
+in the project's test lab. We consider a **feature tested**, iff
+one of the following two criteria holds:
+* The feature is tested automatically
+* At least once during each release freeze, the feature's
+  maintainers produce a test report (by a deadline specified by
+  the release manager). Features with no test report get
+  downgraded from **Supported** to some lower status.
+
+For hardware platforms we consider a **platform tested**, iff
+* Appropriate hardware for said platform is live and in active use
+  in our test lab and tests consistently pass against that platform.
+* At least once during each release freeze, the hardware platform's
+  maintainers produce a test report (by a deadline specified by
+  the release manager). Hardware Platforms with no test report get
+  downgraded from **Supported** to some lower status.
+
+## Documentation
+The Xen Project requires that documentation for user facing
+features and in-code API documentation (or user guides as
+appropriate) are provided in tree.
+
+For new larger features, we do also expect that feature documentation
+is submitted to [xen.git / docs / features]
+(http://xenbits.xen.org/gitweb/?p=xen.git;a=tree;f=docs/features;hb=HEAD).
+Whether such documentation is required depends on the discretion of
+the maintainers and committers that review new features.
+
+We say that a feature as **documented**, if relevant documentation has
+been committed to the tree.
+
+# State Definitions
+
+This section lists state definitions of a **Feature** in terms of
+properties. This section talks about Features, for readability, but
+is used interchangeably between **Features** and **Platforms**.
+
+States are listed in order of increasing number
+of properties. Note that note all features will require to go
+through each state: for example small and non-risky features
+may go straight from under development to supported. It is up to
+the development community to judge and discuss, which states
+are necessary based on the size and risk of a feature.
+
+## Preview
+* Partially completed, with missing functionality
+* May **not be fully functional** in all cases
+* May **not be tested**
+* APIs and interfaces may **not be stable**
+* The developer is actively looking for user feedback
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled on a best-effort basis
+
+## Experimental
+* Core functionality is **fully functional**
+* However, not all functionality or platform support may be
+  present
+* May **not be tested**, although there is an expectation that a plan
+  to improve testing is in place or worked on
+* APIs and interfaces may **not be stable**
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled on a best-effort basis. However, there is an expectation
+  that issues related to broken core functionality are addressed.
+
+## Complete
+This is a state which has not existed in the past. It is aimed
+at larger new features, which may only be in use or of interest
+to a small number of contributors, or where not enough expertise
+exists in the community to treat the feature as **Supported**. It
+presents an opportunity for developers to prove over one or
+two release cycles that said feature can be **supported** in future.
+
+* Intended functionality is **fully implemented**
+* Feature is **maintained**
+* Feature is **tested**
+* Feature is **stable**
+* Feature is **documented**
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled by the developers/maintainers behind the feature. There
+  is an expectation for the developers/maintainers to address
+  bugs and issues over a period of time.
+* Regressions and blockers in a complete feature do **not** normally
+  block new releases. It is at the discretion of the community
+  and Release Manager to downgrade the feature.
+  Security issues would **not** be handled by the security team.
+
+## Supported
+* Intended functionality is **fully implemented**
+* Feature is **maintained**
+* Feature is **tested**
+* Feature is **stable**
+* Feature is **documented**
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled by the developers/maintainers/committers within the
+  community.
+* Regressions and blockers in a complete feature do normally
+  block new releases.
+* Security issues would be handled by the security team.
+
+
+## Supported-Legacy-Stable
+According to this classification, a lot of our existing features would
+have to move back to **Experimental** until have tested and documented
+them. In other words, the following criteria may not always hold for
+such features:
+
+* Feature is **tested**
+* Feature is **documented**
+
+However many of these features and platforms are known to work in
+production. For this reason, such features will be marked as
+**Supported-Legacy-Stable** to ease migration to the new **Supported**
+criteria. **Supported-Legacy-Stable** fulfils the following criteria:
+
+* Intended functionality is **fully implemented**
+* Feature is **maintained**
+* Feature is **stable**
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled by the developers/maintainers/committers within the
+  community.
+* Regressions and blockers in a complete feature do normally
+  block new releases.
+* Security issues would be handled by the security team.
+
+## Deprecated
+There are typically two scenarios when a feature would be
+deprecated.
+* If the feature is not relevant any more or has been replaced
+  by a new implementation (e.g. xm vs. xl)
+* If we have lost the capability to support a feature.
+  For example when we have lost the capability to **maintain**
+  the feature, because we do not have maintainers. In such cases
+  raising the issue on xen-devel@ usually will lead to a resolution,
+  if there is enough usage by vendors in the eco-system.
+
+Features in any state can be deprecated.
+
+The following properties apply to deprecated features:
+* Bugs and issues can be raised on xen-devel@ and will be
+  handled at the discretion of the community.
+* Regressions and blockers in a deprecated feature do normally
+  block new releases.
+* Security issues for deprecated features are handled by the security
+  team at their discretion.
+
+# Recording Feature Maturity Status
+
+## State Recording Prior to the introduction of the Feature Maturity Lifecycle
+
+Prior to the introduction of this document, Feature Maturity was listed in
+[Xen Project Release Features]
+(http://wiki.xenproject.org/wiki/Xen_Project_Release_Features).
+
+## State Recording
+
+The intention here is to require that the central file is modified
+with a patch that introduces a feature (state, feature title,
+short description) and that developers which add functionality
+modify accordingly. The release manager and other stake-holders
+such as the community may also propose changes during the release
+cycle (in particular towards the end).
+
+TODO: Sort out the detail. I would propose 3 files, covering *Platforms*,
+*Limits* and *Features*, assuming that their requirements are different.
+One of the open questions is whether duplicating information in files
+such as migration.pandoc and one of the new files would be an issue. If it
+is, then we probably need to write a script, which scrapes information from
+files and generates some info.
+
+# Downgrading Feature Maturity
+If a feature has been for too long on in an incomplete state (e.g.
+**Preview** or **Experimental**) or some of the assumptions associated
+with a state do not hold any more, community members may propose to
+downgrade a feature on xen-devel@. Typically a discussion on
+the list would be expected before a patch to the file is proposed.
+
+# Known Issues
+
+## Supported status in MAINTAINERS
+One comment raised in this [e-mail]
+(http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg02140.html),
+was that we should rename the `Supported' field in [MAINTAINERS]
+(http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=MAINTAINERS;hb=HEAD)
+to avoid confusion. For now, I do not want to block the proposal and
instead just make a note, such that we can address this later.
+
+## Table of Legacy Supported Features
+We need to finalise the format and location for information covering
*Platforms*, *Limits* and *Features* and then update section
+**State Recording** accordingly.
+
+# References
+
+* For an earlier discussion leading to this document, see this
+  [e-mail thread]
+  (http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html).
+
+* Historical Feature support Status can be found in
+  [Xen Project Release Features]
+  (http://wiki.xenproject.org/wiki/Xen_Project_Release_Features)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2015-11-06 1        Xen 4.7  Document written
+---------- -------- -------- -------------------------------------------
-- 
2.1.4

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Wei Liu]

FWIW I think it would be valuable if we can CC some vendors /
contributors and get feedback from them.

Some comments below.

On Fri, Nov 06, 2015 at 12:35:44PM -0500, Lars Kurth wrote:
> - Incorporated feedback from
>   http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html
> 
> Open Issues:
> - Did not build and test the doc yet (want to get the content right first)
> - Decide on supported status in MAINTAINER file
> - Resolve loose ends on where and how to record Feature State
> 
> Signed-off-by: Lars Kurth <lars.kurth@xenproject.org>
> ---
>  docs/features/feature-maturity-lifecycle.pandoc | 260 ++++++++++++++++++++++++
>  1 file changed, 260 insertions(+)
>  create mode 100644 docs/features/feature-maturity-lifecycle.pandoc
> 
> diff --git a/docs/features/feature-maturity-lifecycle.pandoc
> b/docs/features/feature-maturity-lifecycle.pandoc
> new file mode 100644
> index 0000000..46f98a7
> --- /dev/null
> +++ b/docs/features/feature-maturity-lifecycle.pandoc
> @@ -0,0 +1,260 @@
> +% Feature Maturity Lifecycle
> +% Revision 1
> +
> +\clearpage
> +
> +# Basics
> +--------------- -------------------
> +        Status: **Proposal**
> +
> +     Component: Development Process
> +--------------- -------------------
> +
[...]
> +## Complete
> +This is a state which has not existed in the past. It is aimed
> +at larger new features, which may only be in use or of interest
> +to a small number of contributors, or where not enough expertise
> +exists in the community to treat the feature as **Supported**. It
> +presents an opportunity for developers to prove over one or
> +two release cycles that said feature can be **supported** in future.
> +
> +* Intended functionality is **fully implemented**
> +* Feature is **maintained**
> +* Feature is **tested**
> +* Feature is **stable**
> +* Feature is **documented**
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled by the developers/maintainers behind the feature. There
> +  is an expectation for the developers/maintainers to address
> +  bugs and issues over a period of time.
> +* Regressions and blockers in a complete feature do **not** normally

I would just remove "and blockers".

> +  block new releases. It is at the discretion of the community
> +  and Release Manager to downgrade the feature.
> +  Security issues would **not** be handled by the security team.
> +
> +## Supported
> +* Intended functionality is **fully implemented**
> +* Feature is **maintained**
> +* Feature is **tested**
> +* Feature is **stable**
> +* Feature is **documented**
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled by the developers/maintainers/committers within the
> +  community.
> +* Regressions and blockers in a complete feature do normally
> +  block new releases.

... in a supported feature ...

> +* Security issues would be handled by the security team.
> +

When do we downgrade feature with Supported status?  I think it's done
near release like Complete?


> +
> +## Supported-Legacy-Stable
> +According to this classification, a lot of our existing features would
> +have to move back to **Experimental** until have tested and documented
> +them. In other words, the following criteria may not always hold for
> +such features:
> +
> +* Feature is **tested**
> +* Feature is **documented**
> +
> +However many of these features and platforms are known to work in
> +production. For this reason, such features will be marked as
> +**Supported-Legacy-Stable** to ease migration to the new **Supported**
> +criteria. **Supported-Legacy-Stable** fulfils the following criteria:
> +
> +* Intended functionality is **fully implemented**
> +* Feature is **maintained**
> +* Feature is **stable**
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled by the developers/maintainers/committers within the
> +  community.
> +* Regressions and blockers in a complete feature do normally
> +  block new releases.
> +* Security issues would be handled by the security team.
> +
> +## Deprecated
> +There are typically two scenarios when a feature would be
> +deprecated.
> +* If the feature is not relevant any more or has been replaced
> +  by a new implementation (e.g. xm vs. xl)
> +* If we have lost the capability to support a feature.
> +  For example when we have lost the capability to **maintain**
> +  the feature, because we do not have maintainers. In such cases
> +  raising the issue on xen-devel@ usually will lead to a resolution,
> +  if there is enough usage by vendors in the eco-system.
> +
> +Features in any state can be deprecated.
> +
> +The following properties apply to deprecated features:
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled at the discretion of the community.
> +* Regressions and blockers in a deprecated feature do normally
> +  block new releases.

Do or do not?

Since by definition the community doesn't have capacity to support
deprecated feature I don't see much point blocking release with it.

Wei.

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Lars Kurth]

Thank you for the feedback

> On 11 Nov 2015, at 16:33, Wei Liu <wei.liu2@citrix.com> wrote:
> 
> FWIW I think it would be valuable if we can CC some vendors /
> contributors and get feedback from them.

I will bring this up at the next board meeting and highlight the proposal

> Some comments below.
> 
> On Fri, Nov 06, 2015 at 12:35:44PM -0500, Lars Kurth wrote:
>> - Incorporated feedback from
>>  http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html
>> 
>> Open Issues:
>> - Did not build and test the doc yet (want to get the content right first)
>> - Decide on supported status in MAINTAINER file
>> - Resolve loose ends on where and how to record Feature State
>> 
>> Signed-off-by: Lars Kurth <lars.kurth@xenproject.org>
>> ---
>> docs/features/feature-maturity-lifecycle.pandoc | 260 ++++++++++++++++++++++++
>> 1 file changed, 260 insertions(+)
>> create mode 100644 docs/features/feature-maturity-lifecycle.pandoc
>> 
>> diff --git a/docs/features/feature-maturity-lifecycle.pandoc
>> b/docs/features/feature-maturity-lifecycle.pandoc
>> new file mode 100644
>> index 0000000..46f98a7
>> --- /dev/null
>> +++ b/docs/features/feature-maturity-lifecycle.pandoc
>> @@ -0,0 +1,260 @@
>> +% Feature Maturity Lifecycle
>> +% Revision 1
>> +
>> +\clearpage
>> +
>> +# Basics
>> +--------------- -------------------
>> +        Status: **Proposal**
>> +
>> +     Component: Development Process
>> +--------------- -------------------
>> +
> [...]
>> +## Complete
>> +This is a state which has not existed in the past. It is aimed
>> +at larger new features, which may only be in use or of interest
>> +to a small number of contributors, or where not enough expertise
>> +exists in the community to treat the feature as **Supported**. It
>> +presents an opportunity for developers to prove over one or
>> +two release cycles that said feature can be **supported** in future.
>> +
>> +* Intended functionality is **fully implemented**
>> +* Feature is **maintained**
>> +* Feature is **tested**
>> +* Feature is **stable**
>> +* Feature is **documented**
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled by the developers/maintainers behind the feature. There
>> +  is an expectation for the developers/maintainers to address
>> +  bugs and issues over a period of time.
>> +* Regressions and blockers in a complete feature do **not** normally
> 
> I would just remove "and blockers".

ACK

>> +  block new releases. It is at the discretion of the community
>> +  and Release Manager to downgrade the feature.
>> +  Security issues would **not** be handled by the security team.
>> +
>> +## Supported
>> +* Intended functionality is **fully implemented**
>> +* Feature is **maintained**
>> +* Feature is **tested**
>> +* Feature is **stable**
>> +* Feature is **documented**
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled by the developers/maintainers/committers within the
>> +  community.
>> +* Regressions and blockers in a complete feature do normally
>> +  block new releases.
> 
> ... in a supported feature ...

Well spotted

>> +* Security issues would be handled by the security team.
>> +
> 
> When do we downgrade feature with Supported status?  I think it's done
> near release like Complete?

I was thinking that this can happen at any time, but if say a test report where one is expected, doesn't come in, we would have to downgrade during RC stage.

>> +
>> +## Supported-Legacy-Stable
>> +According to this classification, a lot of our existing features would
>> +have to move back to **Experimental** until have tested and documented
>> +them. In other words, the following criteria may not always hold for
>> +such features:
>> +
>> +* Feature is **tested**
>> +* Feature is **documented**
>> +
>> +However many of these features and platforms are known to work in
>> +production. For this reason, such features will be marked as
>> +**Supported-Legacy-Stable** to ease migration to the new **Supported**
>> +criteria. **Supported-Legacy-Stable** fulfils the following criteria:
>> +
>> +* Intended functionality is **fully implemented**
>> +* Feature is **maintained**
>> +* Feature is **stable**
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled by the developers/maintainers/committers within the
>> +  community.
>> +* Regressions and blockers in a complete feature do normally
>> +  block new releases.
>> +* Security issues would be handled by the security team.
>> +
>> +## Deprecated
>> +There are typically two scenarios when a feature would be
>> +deprecated.
>> +* If the feature is not relevant any more or has been replaced
>> +  by a new implementation (e.g. xm vs. xl)
>> +* If we have lost the capability to support a feature.
>> +  For example when we have lost the capability to **maintain**
>> +  the feature, because we do not have maintainers. In such cases
>> +  raising the issue on xen-devel@ usually will lead to a resolution,
>> +  if there is enough usage by vendors in the eco-system.
>> +
>> +Features in any state can be deprecated.
>> +
>> +The following properties apply to deprecated features:
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled at the discretion of the community.
>> +* Regressions and blockers in a deprecated feature do normally
>> +  block new releases.
> 
> Do or do not?

Should read: "do not"

> 
> Since by definition the community doesn't have capacity to support
> deprecated feature I don't see much point blocking release with it.
> 
> Wei.

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Andrew Cooper]

On 06/11/15 17:35, Lars Kurth wrote:
> +# State Definitions
> +
> +This section lists state definitions of a **Feature** in terms of
> +properties. This section talks about Features, for readability, but
> +is used interchangeably between **Features** and **Platforms**.
> +
> +States are listed in order of increasing number
> +of properties. Note that note all features will require to go
> +through each state: for example small and non-risky features
> +may go straight from under development to supported. It is up to
> +the development community to judge and discuss, which states
> +are necessary based on the size and risk of a feature.
> +
> +## Preview
> +* Partially completed, with missing functionality
> +* May **not be fully functional** in all cases
> +* May **not be tested**
> +* APIs and interfaces may **not be stable**

"are not stable".

i.e. maintainers reserve the right to insist that the interfaces are
changed.

For preview and experimental features, it is quite possible that hidden
design considerations become visible after the initial code is committed.

As a result, it is important that nothing is set in stone while the
feature itself is still in development.

> +* The developer is actively looking for user feedback
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled on a best-effort basis

Probably best to explicitly state that there is no security support for
Preview/Experimental features.  By their classification, they are not
ready for production use.

> +
> +## Experimental
> +* Core functionality is **fully functional**
> +* However, not all functionality or platform support may be
> +  present
> +* May **not be tested**, although there is an expectation that a plan
> +  to improve testing is in place or worked on
> +* APIs and interfaces may **not be stable**
> +* Bugs and issues can be raised on xen-devel@ and will be
> +  handled on a best-effort basis. However, there is an expectation
> +  that issues related to broken core functionality are addressed.

I am not completely convinced by splitting Preview and Experiential. 
The difference between the two is quite subjective.

I would hope that the exact state of the feature in the range Nothing ->
Supported is easy to derive from the Limitations, Improvements and Known
Issues sections of its feature document.

~Andrew

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Lars Kurth]

> On 11 Nov 2015, at 16:33, Wei Liu <wei.liu2@citrix.com> wrote:
> 
> FWIW I think it would be valuable if we can CC some vendors /
> contributors and get feedback from them.

I highlighted this at the Advisory Board meeting and the board is supportive. Concrete feedback was
A) "I'm generally supportive of this because I believe it will help reduce ambiguities around the status of features and ultimately help improve quality." This was mirrored by everyone at the last meeting. 
B) The preview vs. experimental classification was deemed to be a bit arbitrary, and maybe we can fold them into one category. The feeling was that the distinction does not provide much value and we should maybe come up with something simpler.
C) A concern raised was that by being explicit about "legacy" features we may create negative PR opportunities, e.g. "the Xen Project has declared 50% of its features to be untested and undocumented". Perhaps instead we just grandfather these features into "stable" rather than having an explicit "legacy-stable" designation.

Another possible solution would be to be clear about what is automatically tested and what requires manual testing (or is currently tested by 3rd party test infrastructure such as XenRT and others) in the "Supported" category. The argument that can be made is that for "legacy-stable" we rely on 3rd party testing. This would help direct test resources during Test Days towards such features. 

The documentation angle is probably less of an issue, because we have user facing docs for pretty much everything in the man pages. So we basically mainly talk about specs and other documents which are either in the wiki or in docs/misc. And of course we can make judgement calls for those docs in pretty much the same way as we do now.

Any views on the pros and cons?
 
Lars

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Lars Kurth]

> On 11 Nov 2015, at 19:38, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> On 06/11/15 17:35, Lars Kurth wrote:
>> +# State Definitions
>> +
>> +This section lists state definitions of a **Feature** in terms of
>> +properties. This section talks about Features, for readability, but
>> +is used interchangeably between **Features** and **Platforms**.
>> +
>> +States are listed in order of increasing number
>> +of properties. Note that note all features will require to go
>> +through each state: for example small and non-risky features
>> +may go straight from under development to supported. It is up to
>> +the development community to judge and discuss, which states
>> +are necessary based on the size and risk of a feature.
>> +
>> +## Preview
>> +* Partially completed, with missing functionality
>> +* May **not be fully functional** in all cases
>> +* May **not be tested**
>> +* APIs and interfaces may **not be stable**
> 
> "are not stable".
> 
> i.e. maintainers reserve the right to insist that the interfaces are
> changed.

OK. That makes sense

> 
> For preview and experimental features, it is quite possible that hidden
> design considerations become visible after the initial code is committed.
> 
> As a result, it is important that nothing is set in stone while the
> feature itself is still in development.

Agreed. 


>> +* The developer is actively looking for user feedback
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled on a best-effort basis
> 
> Probably best to explicitly state that there is no security support for
> Preview/Experimental features.  By their classification, they are not
> ready for production use.

Agreed.

>> +
>> +## Experimental
>> +* Core functionality is **fully functional**
>> +* However, not all functionality or platform support may be
>> +  present
>> +* May **not be tested**, although there is an expectation that a plan
>> +  to improve testing is in place or worked on
>> +* APIs and interfaces may **not be stable**
>> +* Bugs and issues can be raised on xen-devel@ and will be
>> +  handled on a best-effort basis. However, there is an expectation
>> +  that issues related to broken core functionality are addressed.
> 
> I am not completely convinced by splitting Preview and Experiential. 
> The difference between the two is quite subjective.
> 
> I would hope that the exact state of the feature in the range Nothing ->
> Supported is easy to derive from the Limitations, Improvements and Known
> Issues sections of its feature document.

Alright. That makes sense and is also mirrored somewhat by the Advisory Board feedback (see my last mail to Wei).
What is your view on the usefulness of keeping Preview and Experimental separate? It sounds to me that we should fold these into one category and let Limitations, Improvements and Known Issues speak for themselves.

Lars 

Re: [RFC PATCH] Feature doc: Added Feature Maturity Lifecycle

[Wei Liu]

On Tue, Dec 01, 2015 at 12:16:42PM +0000, Lars Kurth wrote:
> 
> > On 11 Nov 2015, at 16:33, Wei Liu <wei.liu2@citrix.com> wrote:
> > 
> > FWIW I think it would be valuable if we can CC some vendors /
> > contributors and get feedback from them.
> 
> I highlighted this at the Advisory Board meeting and the board is
> supportive. Concrete feedback was
>
> A) "I'm generally supportive of this because I believe it will help
> reduce ambiguities around the status of features and ultimately help
> improve quality." This was mirrored by everyone at the last meeting. 
>
> B) The preview vs. experimental classification was deemed to be a bit
> arbitrary, and maybe we can fold them into one category. The feeling
> was that the distinction does not provide much value and we should
> maybe come up with something simpler.
>
> C) A concern raised was that by being explicit about "legacy" features
> we may create negative PR opportunities, e.g. "the Xen Project has
> declared 50% of its features to be untested and undocumented". Perhaps
> instead we just grandfather these features into "stable" rather than
> having an explicit "legacy-stable" designation.
> 

Putting those into "stable" sounds reasonable to me. Plus some
clarification as you mentioned below.

> Another possible solution would be to be clear about what is
> automatically tested and what requires manual testing (or is currently
> tested by 3rd party test infrastructure such as XenRT and others) in
> the "Supported" category. The argument that can be made is that for
> "legacy-stable" we rely on 3rd party testing. This would help direct
> test resources during Test Days towards such features. 
> 
> The documentation angle is probably less of an issue, because we have
> user facing docs for pretty much everything in the man pages. So we
> basically mainly talk about specs and other documents which are either
> in the wiki or in docs/misc. And of course we can make judgement calls
> for those docs in pretty much the same way as we do now.
> 
> Any views on the pros and cons?
>  
> Lars