[PATCH] docs/process/xen-release-management: Lesson to learn

[PATCH] docs/process/xen-release-management: Lesson to learn

[Ian Jackson]

The 4.10 release preparation was significantly more hairy than ideal.
(We seem to have a good overall outcome despite, rather than because
of, our approach.)

This is the second time (at least) that we have come close to failure
by committing to a release date before the exact code to be released
is known and has been made and tested.

Evidently our docs makes it insufficiently clear not to do that.

CC: Lars Kurth <lars.kurth@citrix.com>
CC: Julien Grall <julien.grall@arm.com>
CC: Juergen Gross <jgross@suse.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 docs/process/xen-release-management.pandoc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/docs/process/xen-release-management.pandoc b/docs/process/xen-release-management.pandoc
index 2ff0665..eee5dcf 100644
--- a/docs/process/xen-release-management.pandoc
+++ b/docs/process/xen-release-management.pandoc
@@ -211,6 +211,11 @@ https://wiki.xenproject.org/wiki/Category:Xen_4.9
     Ask them to dry-run their checklist and confirm everything is OK. If not,
     arrange another RC and restart this checklist.
 
+7. Do not commit to a release date until
+
+    * The exact xen.git commit id to be released is known.
+    * That commit id has been satisfactorily tested.
+
 7. Give PR Personnel final go-ahead, and instruct Release Technician to make
 release deliverables (tags and tarballs - will usually be in place the day
 before the release). At this point, PR collateral will be sent to reporters
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Juergen Gross]

On 13/12/17 13:02, Ian Jackson wrote:
> The 4.10 release preparation was significantly more hairy than ideal.
> (We seem to have a good overall outcome despite, rather than because
> of, our approach.)
> 
> This is the second time (at least) that we have come close to failure
> by committing to a release date before the exact code to be released
> is known and has been made and tested.
> 
> Evidently our docs makes it insufficiently clear not to do that.
> 
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Juergen Gross <jgross@suse.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Reviewed-by: Juergen Gross <jgross@suse.com>


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Andrew Cooper]

On 13/12/17 12:02, Ian Jackson wrote:
> The 4.10 release preparation was significantly more hairy than ideal.
> (We seem to have a good overall outcome despite, rather than because
> of, our approach.)
>
> This is the second time (at least) that we have come close to failure
> by committing to a release date before the exact code to be released
> is known and has been made and tested.
>
> Evidently our docs makes it insufficiently clear not to do that.
>
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Juergen Gross <jgross@suse.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  docs/process/xen-release-management.pandoc | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/docs/process/xen-release-management.pandoc b/docs/process/xen-release-management.pandoc
> index 2ff0665..eee5dcf 100644
> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -211,6 +211,11 @@ https://wiki.xenproject.org/wiki/Category:Xen_4.9
>      Ask them to dry-run their checklist and confirm everything is OK. If not,
>      arrange another RC and restart this checklist.
>  
> +7. Do not commit to a release date until
> +
> +    * The exact xen.git commit id to be released is known.
> +    * That commit id has been satisfactorily tested.
> +
>  7. Give PR Personnel final go-ahead, and instruct Release Technician to make

s/7/8/

~Andrew

>  release deliverables (tags and tarballs - will usually be in place the day
>  before the release). At this point, PR collateral will be sent to reporters


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Julien Grall]

Hi Ian,

On 13/12/17 12:02, Ian Jackson wrote:
> The 4.10 release preparation was significantly more hairy than ideal.
> (We seem to have a good overall outcome despite, rather than because
> of, our approach.)
> 
> This is the second time (at least) that we have come close to failure
> by committing to a release date before the exact code to be released
> is known and has been made and tested.

This is a little more complicate than that. As you are part of the 
security team, you are fully aware why we chose this date.

For the other, this release was delayed because of XSA published again 
~2 weeks after the scheduled release. We were also constraint with the 
Christmas period coming up, the latest we could release was the week of 
the 13th.

That's why testing with XSAs was requested on the security-ml before 
hand. However, this was mistakenly done on rc7 rather than rc8.

As it seems, it is becoming more frequent to have XSAs around the 
release. It is getting increasingly more difficult to make a choice on 
the date.

This decision is not helped by the testing that have been quite 
unreliable due to heisenbug.

To be honest, if we had to follow your suggestion below. We would need 
to get the tree completely frozen 2-3 weeks before the actual date.

Cheers,

> 
> Evidently our docs makes it insufficiently clear not to do that.
> 
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Juergen Gross <jgross@suse.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>   docs/process/xen-release-management.pandoc | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/docs/process/xen-release-management.pandoc b/docs/process/xen-release-management.pandoc
> index 2ff0665..eee5dcf 100644
> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -211,6 +211,11 @@ https://wiki.xenproject.org/wiki/Category:Xen_4.9
>       Ask them to dry-run their checklist and confirm everything is OK. If not,
>       arrange another RC and restart this checklist.
>   
> +7. Do not commit to a release date until
> +
> +    * The exact xen.git commit id to be released is known.
> +    * That commit id has been satisfactorily tested.
> +
>   7. Give PR Personnel final go-ahead, and instruct Release Technician to make
>   release deliverables (tags and tarballs - will usually be in place the day
>   before the release). At this point, PR collateral will be sent to reporters
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Ian Jackson]

Julien Grall writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
> That's why testing with XSAs was requested on the security-ml before 
> hand. However, this was mistakenly done on rc7 rather than rc8.

If the proposed release had been constructed in advance, and agreed,
ahead of committing to a release, then there would have been an
opportunity for this mistake to be spotted.

We had no real plan for what to do if the tests showed problems that
we thought meant the proposed release was not suitable.  Indeed, some
of the testing identified a further issue which we have released with,
on the grounds that we are convinced it is not a regression.

IMO, that important bugfixes were still going in after we had
committed to a release date, is indicative of some kind of problem.
The purpose of the freeze is to allow bugs to be shaken out, and
also to allow us to hazard a guess the likely remaining bug density
by looking at the rate at which bugfixes are still going in.

If we commit to a release date when the rate of important bugfixes is
still nontrivial then we are likely to discover a new important bug
after we commit to releasing but before we release, and without
sufficient time for testing (and any necessary rework).  When that
happens we have few good options: we can release with a known
important bug; release with a not-fully-tested and perhaps
itself-buggy bugfix; or abort the release with all the consequences
for reputation and community engagement.


Regarding the testing of the wrong version.  Any task done by humans
involves a risk of errors.  These risks are magnified when: the work
must be done in secret; the kind of work is done fairly rarely; there
are more manual steps; there is new tooling; there is time pressure.

We have a number of opportunities for improvement.  One of them is to
improve our tooling - George's xsatool is part of that.  And another
is that we should have easier tools for setting up custom osstest
flights.  I have been thinking about how to do that.  But, the
introducing of new tooling itself carries a risk of course, as new
tools can have bugs and people can more easily misunderstand them.
Increased automation is not without its downsides, as it can hide what
is going on from the user.

Overall, we will not be able to eliminate human error in this kind of
activity.  We must either tolerate the consequences, or mitigate them.
Strategies for mitigation of human error include review or
double-checking (whether by automation or other humans), and
incorporating contingency time for rework.

IMO if we carry on as we have been doing it is only a matter of time
before we make a release which has some kind of serious and obvious
defect.


> As it seems, it is becoming more frequent to have XSAs around the 
> release. It is getting increasingly more difficult to make a choice on 
> the date.

We may need to formalise the process of releasing immediately after
public release of an XSA.

My proposed requirement to not commit to a release date until we know
what code we are planning to release is not incompatible with
privately preparing the release branch and testing it.

Also, it would be quite possible for the Xen Project Security Team to
adjust the disclosure schedule (which it suggests to vulnerability
discoverers, and which advice discoverers often take).  The Security
Policy says:
 | Our usual starting point for that negotiation, unless there are
 | reasons to diverge
A planned .0 release seems like a "reason ... to diverge" to me.


> This decision is not helped by the testing that have been quite 
> unreliable due to heisenbug.

The release process needs to cope with the actual level of quality in
our codebase, and in the test hardware we have available.

Obviously we have various work in progress to improve both of these
features but this is not easy.  For code quality there are competing
priorities that lead to under-investigation of heisenbugs.  I have
some ideas about how to have osstest push back more on heisenbugs, but
this will inevitably work by increasing the level of pain they cause
to the general flow of Xen development.

For hardware quality there is the competing priority to actually have
some testing of all the architectures we nominally support.

These kind issues seem to be me to be beyond the scope of the release
process.  As I say, the release process has to work with what we have,
not with what we would like to have.


> To be honest, if we had to follow your suggestion below. We would need 
> to get the tree completely frozen 2-3 weeks before the actual date.

That has been done before.


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[George Dunlap]

On Wed, Dec 13, 2017 at 12:02 PM, Ian Jackson <ian.jackson@eu.citrix.com> wrote:
> The 4.10 release preparation was significantly more hairy than ideal.
> (We seem to have a good overall outcome despite, rather than because
> of, our approach.)
>
> This is the second time (at least) that we have come close to failure
> by committing to a release date before the exact code to be released
> is known and has been made and tested.
>
> Evidently our docs makes it insufficiently clear not to do that.
>
> CC: Lars Kurth <lars.kurth@citrix.com>
> CC: Julien Grall <julien.grall@arm.com>
> CC: Juergen Gross <jgross@suse.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  docs/process/xen-release-management.pandoc | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/docs/process/xen-release-management.pandoc b/docs/process/xen-release-management.pandoc
> index 2ff0665..eee5dcf 100644
> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -211,6 +211,11 @@ https://wiki.xenproject.org/wiki/Category:Xen_4.9
>      Ask them to dry-run their checklist and confirm everything is OK. If not,
>      arrange another RC and restart this checklist.
>
> +7. Do not commit to a release date until
> +
> +    * The exact xen.git commit id to be released is known.
> +    * That commit id has been satisfactorily tested.
> +

How would you apply this directive to the particular situation we
found ourselves in this time?

As a reminder:

* Around 3 December, we didn't think we'd be ready to release until 11 December
* The security team had already set an embargo for 12 December
* Our PR people advised us that 13 or 14 December would be the last
suitable day to announce a release in order to have an impact before
Christmas

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Ian Jackson]

George Dunlap writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
> How would you apply this directive to the particular situation we
> found ourselves in this time?
> 
> As a reminder:
> 
> * Around 3 December, we didn't think we'd be ready to release until 11 December
> * The security team had already set an embargo for 12 December
> * Our PR people advised us that 13 or 14 December would be the last
> suitable day to announce a release in order to have an impact before
> Christmas

Well, we could put off the release.

I guess I'm being quite selfish here.  I'm usually the release
technician.  Doing release preparation at the last minute and in
strange ways means lots of opportunity for me to make mistakes.

I would like to put something in the release checklist that stops
people putting me in a difficult position where I am (a) likely to
make mistakes (b) those mistakes will be embarrassing.

Putting this in the release checklist doesn't mean that it always has
to be followed, of course.  Checklists are not rules; they are
guidelines.

But if this guideline is violated, and as a result I mess something up
due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
all in a hurry, then it would be nice if it were obvious that the
cause of the trouble was the decision to take this risk, rather than
my carelessness or lack of attention to detail.

As it happens this last December we lucked out.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Juergen Gross]

On 27/04/18 16:32, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
>> How would you apply this directive to the particular situation we
>> found ourselves in this time?
>>
>> As a reminder:
>>
>> * Around 3 December, we didn't think we'd be ready to release until 11 December
>> * The security team had already set an embargo for 12 December
>> * Our PR people advised us that 13 or 14 December would be the last
>> suitable day to announce a release in order to have an impact before
>> Christmas
> 
> Well, we could put off the release.
> 
> I guess I'm being quite selfish here.  I'm usually the release
> technician.  Doing release preparation at the last minute and in
> strange ways means lots of opportunity for me to make mistakes.
> 
> I would like to put something in the release checklist that stops
> people putting me in a difficult position where I am (a) likely to
> make mistakes (b) those mistakes will be embarrassing.
> 
> Putting this in the release checklist doesn't mean that it always has
> to be followed, of course.  Checklists are not rules; they are
> guidelines.
> 
> But if this guideline is violated, and as a result I mess something up
> due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
> all in a hurry, then it would be nice if it were obvious that the
> cause of the trouble was the decision to take this risk, rather than
> my carelessness or lack of attention to detail.

+1

Relying on luck is nothing we should base our process on.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Julien Grall]

On 27/04/18 15:32, Ian Jackson wrote:
> George Dunlap writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
>> How would you apply this directive to the particular situation we
>> found ourselves in this time?
>>
>> As a reminder:
>>
>> * Around 3 December, we didn't think we'd be ready to release until 11 December
>> * The security team had already set an embargo for 12 December
>> * Our PR people advised us that 13 or 14 December would be the last
>> suitable day to announce a release in order to have an impact before
>> Christmas

If we had missed the 13 or 14 December, we would had to push the release 
beginning of January because in fine we do rely on PR for promoting Xen.

However, we also had Spectre/Meltdown scheduled for beginning of January 
(I was aware of it when we took the decision). It would have been quite 
difficult to release Xen 4.10 without that security fixes. This would 
have meant another push back of the release.

> 
> Well, we could put off the release.
> 
> I guess I'm being quite selfish here.  I'm usually the release
> technician.  Doing release preparation at the last minute and in
> strange ways means lots of opportunity for me to make mistakes.
> 
> I would like to put something in the release checklist that stops
> people putting me in a difficult position where I am (a) likely to
> make mistakes (b) those mistakes will be embarrassing.
> 
> Putting this in the release checklist doesn't mean that it always has
> to be followed, of course.  Checklists are not rules; they are
> guidelines.
> 
> But if this guideline is violated, and as a result I mess something up
> due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
> all in a hurry, then it would be nice if it were obvious that the
> cause of the trouble was the decision to take this risk, rather than
> my carelessness or lack of attention to detail.
> 
> As it happens this last December we lucked out.

While I understand this was not ideal, I still think it was the best 
solution we had at that time. I would be interested to know what you 
would have chosen with the situation we had.
	- Would you have released without the XSAs?
	- When would you have released Xen?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Ian Jackson]

Julien Grall writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
> While I understand this was not ideal, I still think it was the best 
> solution we had at that time. I would be interested to know what you 
> would have chosen with the situation we had.
> 	- Would you have released without the XSAs?
> 	- When would you have released Xen?

I don't have all the information necessary to make that decision.  I
might have tried to avoid getting into this situation by rejecting
patches earlier, or indeed waited until much later, or I might have
decided to do as you did by deliberately breaching this guideline.

I would like to repeat some of my previous posting since it seems to
me that you haven't addressed it.

> > I guess I'm being quite selfish here.  I'm usually the release
> > technician.  Doing release preparation at the last minute and in
> > strange ways means lots of opportunity for me to make mistakes.
> > 
> > I would like to put something in the release checklist that stops
> > people putting me in a difficult position where I am (a) likely to
> > make mistakes (b) those mistakes will be embarrassing.
> > 
> > Putting this in the release checklist doesn't mean that it always has
> > to be followed, of course.  Checklists are not rules; they are
> > guidelines.
> > 
> > But if this guideline is violated, and as a result I mess something up
> > due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
> > all in a hurry, then it would be nice if it were obvious that the
> > cause of the trouble was the decision to take this risk, rather than
> > my carelessness or lack of attention to detail.
> > 
> > As it happens this last December we lucked out.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Lars Kurth]

On 27/04/2018, 16:29, "Julien Grall" <julien.grall@arm.com> wrote:

    
    
    On 27/04/18 15:32, Ian Jackson wrote:
    > George Dunlap writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
    >> How would you apply this directive to the particular situation we
    >> found ourselves in this time?
    >>
    >> As a reminder:
    >>
    >> * Around 3 December, we didn't think we'd be ready to release until 11 December
    >> * The security team had already set an embargo for 12 December
    >> * Our PR people advised us that 13 or 14 December would be the last
    >> suitable day to announce a release in order to have an impact before
    >> Christmas
    
    If we had missed the 13 or 14 December, we would had to push the release 
    beginning of January because in fine we do rely on PR for promoting Xen.
    
    However, we also had Spectre/Meltdown scheduled for beginning of January 
    (I was aware of it when we took the decision). It would have been quite 
    difficult to release Xen 4.10 without that security fixes. This would 
    have meant another push back of the release.
    
    > 
    > Well, we could put off the release.
    > 
    > I guess I'm being quite selfish here.  I'm usually the release
    > technician.  Doing release preparation at the last minute and in
    > strange ways means lots of opportunity for me to make mistakes.
    > 
    > I would like to put something in the release checklist that stops
    > people putting me in a difficult position where I am (a) likely to
    > make mistakes (b) those mistakes will be embarrassing.
    > 
    > Putting this in the release checklist doesn't mean that it always has
    > to be followed, of course.  Checklists are not rules; they are
    > guidelines.
    > 
    > But if this guideline is violated, and as a result I mess something up
    > due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
    > all in a hurry, then it would be nice if it were obvious that the
    > cause of the trouble was the decision to take this risk, rather than
    > my carelessness or lack of attention to detail.
    > 
    > As it happens this last December we lucked out.
    
    While I understand this was not ideal, I still think it was the best 
    solution we had at that time. I would be interested to know what you 
    would have chosen with the situation we had.
    	- Would you have released without the XSAs?
    	- When would you have released Xen?

Another way how to maybe fix this type of issue, is to move towards predictable XSA disclosure. We already release in batches and mostly once a month, if we released XSAs in batches on a pre-determined date once per month, we would not get into the situation we got into. Of course, there are cases where we don't control the publication date, but these cases are fairly rare. And of course, we would need to change the security process. I have been doing some work in this area and will pick up again next week. 
 
That way a release manager can work around race conditions which are caused by XSAs and the RC schedule/RC quality issues.

Regards
Lars
 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH] docs/process/xen-release-management: Lesson to learn

[Julien Grall]

Hi Ian,

On 27/04/18 16:34, Ian Jackson wrote:
> Julien Grall writes ("Re: [Xen-devel] [PATCH] docs/process/xen-release-management: Lesson to learn"):
>>> I guess I'm being quite selfish here.  I'm usually the release
>>> technician.  Doing release preparation at the last minute and in
>>> strange ways means lots of opportunity for me to make mistakes.
>>>
>>> I would like to put something in the release checklist that stops
>>> people putting me in a difficult position where I am (a) likely to
>>> make mistakes (b) those mistakes will be embarrassing.
>>>
>>> Putting this in the release checklist doesn't mean that it always has
>>> to be followed, of course.  Checklists are not rules; they are
>>> guidelines.

The way docs/process/xen-release-management has been written gives more 
an impression of rules/process to follow rather than guidelines. The "Do 
not commit..." seems to enforce it. It might be good to clarify that 
this is a guideline.

>>>
>>> But if this guideline is violated, and as a result I mess something up
>>> due to having to do a lot of complicated, ad-hoc, un-qa-able, work,
>>> all in a hurry, then it would be nice if it were obvious that the
>>> cause of the trouble was the decision to take this risk, rather than
>>> my carelessness or lack of attention to detail.

I think that was obvious for me :). But I understand were you are coming 
from now. And yes, I agree such guidelines would be a useful to have in 
the document.

Going forward, we probably want to find a way to minimize problem around 
XSAs coming up before the releases. This seems to happen at every 
release nowadays.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel