Better anticipation for minor releases

Better anticipation for minor releases

[Alfredo Deza]

Hi Abhishek,

For almost every minor release we have, there is a cloud of
uncertainty where nobody really knows when the release will be cut.

Most of the time there is an email (or series of emails) that trigger
the conversation of "are we now ready to cut a release?", where leads
respond what backports are still pending, or if there are test suites
that aren't really in the clear.

For minor releases a mix of a time-based schedule with a set of each
lead's group of passing tests, would be a nice way to predict when we
can start having the conversation of cutting a release.

It isn't a one-shot fix, but would certainly improve everyone who
wants/needs a new release and can't get any clues as to what that may
be.

Something like:

* Minor releases every 6 weeks
* subset of test suites passing from each lead

Would be a good way to start. If we can get that initial release path,
it would be possible to start thinking about what "gating" really
means for our releases (since we don't really have any sort of gating
today)

What do you think?

Re: Better anticipation for minor releases

[Abhishek]

On 2018-01-31 18:46, Alfredo Deza wrote:
> Hi Abhishek,
> 
> For almost every minor release we have, there is a cloud of
> uncertainty where nobody really knows when the release will be cut.
> 
> Most of the time there is an email (or series of emails) that trigger
> the conversation of "are we now ready to cut a release?", where leads
> respond what backports are still pending, or if there are test suites
> that aren't really in the clear.
> 
> For minor releases a mix of a time-based schedule with a set of each
> lead's group of passing tests, would be a nice way to predict when we
> can start having the conversation of cutting a release.

Agree, we wanted to introduce a better process for 12.2.3 but this 
mostly took a backseat with vacations and all. Yuri has been kind enough 
to spend time on running the QE suites and we've a lot backports raised 
from Shinobu and the rest of the ceph community which was great. We've 
mostly had a relatively reddish run on QE suites for 12.2.2 and this has 
relatively changed a lot in 12.2.3 with smaller greener runs thanks to 
Yuri.

For a ~6 week schedule we need to close all the backports in 4 weeks 
since the final qe run takes ~2weeks. Which gives 4 weeks for adding PRs 
as well as going through integration suites etc, we'll have to come up 
with a soft stop in 3 weeks so that we've sufficient time in the 4th 
week to triage and merge the pending queue + any critical PRs we really 
want in the timeframe.

Quoting Nathan from an earlier email; there are roughly two parts to 
this process: - triaging, backporting and integration testing of PRs
- Asking leads for approval, release notes, working with QE for the 
final suite approval.

If we approximately stop accepting new PRs at the third week, ask leads 
at beginning of week 4 and go through only critical PRs for week 4; and 
go to QE in week5,6 we might be able to somewhat get to a more scheduled 
minor releases.

Brickbats and suggestions ?:)
> It isn't a one-shot fix, but would certainly improve everyone who
> wants/needs a new release and can't get any clues as to what that may
> be.
> 
> Something like:
> 
> * Minor releases every 6 weeks
> * subset of test suites passing from each lead
> 
> Would be a good way to start. If we can get that initial release path,
> it would be possible to start thinking about what "gating" really
> means for our releases (since we don't really have any sort of gating
> today)


> 
> What do you think?
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Better anticipation for minor releases

[Ken Dreyer]

On Wed, Jan 31, 2018 at 12:25 PM, Abhishek <abhishek@suse.com> wrote:
> If we approximately stop accepting new PRs at the third week, ask leads at
> beginning of week 4 and go through only critical PRs for week 4; and go to
> QE in week5,6 we might be able to somewhat get to a more scheduled minor
> releases.
>
> Brickbats and suggestions ?:)

That sounds great to me!

Re: Better anticipation for minor releases

[Alfredo Deza]

On Wed, Jan 31, 2018 at 2:25 PM, Abhishek <abhishek@suse.com> wrote:
> On 2018-01-31 18:46, Alfredo Deza wrote:
>>
>> Hi Abhishek,
>>
>> For almost every minor release we have, there is a cloud of
>> uncertainty where nobody really knows when the release will be cut.
>>
>> Most of the time there is an email (or series of emails) that trigger
>> the conversation of "are we now ready to cut a release?", where leads
>> respond what backports are still pending, or if there are test suites
>> that aren't really in the clear.
>>
>> For minor releases a mix of a time-based schedule with a set of each
>> lead's group of passing tests, would be a nice way to predict when we
>> can start having the conversation of cutting a release.
>
>
> Agree, we wanted to introduce a better process for 12.2.3 but this mostly
> took a backseat with vacations and all. Yuri has been kind enough to spend
> time on running the QE suites and we've a lot backports raised from Shinobu
> and the rest of the ceph community which was great. We've mostly had a
> relatively reddish run on QE suites for 12.2.2 and this has relatively
> changed a lot in 12.2.3 with smaller greener runs thanks to Yuri.
>
> For a ~6 week schedule we need to close all the backports in 4 weeks since
> the final qe run takes ~2weeks. Which gives 4 weeks for adding PRs as well
> as going through integration suites etc, we'll have to come up with a soft
> stop in 3 weeks so that we've sufficient time in the 4th week to triage and
> merge the pending queue + any critical PRs we really want in the timeframe.
>
> Quoting Nathan from an earlier email; there are roughly two parts to this
> process: - triaging, backporting and integration testing of PRs
> - Asking leads for approval, release notes, working with QE for the final
> suite approval.
>
> If we approximately stop accepting new PRs at the third week, ask leads at
> beginning of week 4 and go through only critical PRs for week 4; and go to
> QE in week5,6 we might be able to somewhat get to a more scheduled minor
> releases.

This might be a bit hard to deal with, how do you prevent a PR from
getting merged? I've seen several times PRs getting merged at the last
minute,
with little to no testing, and breaking a release :( It hasn't
happened much lately, but I don't know of a way to block all PRs for a
period of time

One way that sounded interesting to me from the Chef Release Team is
that regardless of the state of the branch, they would agree what SHA1
would end up
being the actual release, and then going through the different phases
of QA up until release, when something needed a fix, the pipeline
would start from the beginning.


>
> Brickbats and suggestions ?:)
>>
>> It isn't a one-shot fix, but would certainly improve everyone who
>> wants/needs a new release and can't get any clues as to what that may
>> be.
>>
>> Something like:
>>
>> * Minor releases every 6 weeks
>> * subset of test suites passing from each lead
>>
>> Would be a good way to start. If we can get that initial release path,
>> it would be possible to start thinking about what "gating" really
>> means for our releases (since we don't really have any sort of gating
>> today)
>
>
>
>>
>> What do you think?
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Better anticipation for minor releases

[Nathan Cutler]

On 02/02/2018 01:05 PM, Alfredo Deza wrote:
>> If we approximately stop accepting new PRs at the third week, ask leads at
>> beginning of week 4 and go through only critical PRs for week 4; and go to
>> QE in week5,6 we might be able to somewhat get to a more scheduled minor
>> releases.
> 
> This might be a bit hard to deal with, how do you prevent a PR from
> getting merged? I've seen several times PRs getting merged at the last
> minute,
> with little to no testing, and breaking a release :( It hasn't
> happened much lately, but I don't know of a way to block all PRs for a
> period of time

Do we have a way of releasing a given SHA1 (which has passed QE) even 
when additional PRs have been merged on top of it?

IIRC in the past we could only release the tip of the named (jewel, 
luminous) branch. That does indeed create a risk of PRs getting merged 
post-QE and inadvertently finding their way into a release.

Nathan

Re: Better anticipation for minor releases

[Ken Dreyer]

This is just my opinion: we should have a quick "bump commit + tag"
Jenkins job that pushes the Git tag live as soon as possible. In
particular we should not wait for packages to be build + signed +
published on download.ceph.com before pushing the bump commit + tag
live.

Bumping and tagging in Git should take a minute or two, whereas
building, signing, and rsync'ing all the packages for that tag takes
hours. I think this time-intensive process causes the Git history to
diverge for too long.

- Ken

On Fri, Feb 2, 2018 at 5:28 AM, Nathan Cutler <ncutler@suse.cz> wrote:
> On 02/02/2018 01:05 PM, Alfredo Deza wrote:
>>>
>>> If we approximately stop accepting new PRs at the third week, ask leads
>>> at
>>> beginning of week 4 and go through only critical PRs for week 4; and go
>>> to
>>> QE in week5,6 we might be able to somewhat get to a more scheduled minor
>>> releases.
>>
>>
>> This might be a bit hard to deal with, how do you prevent a PR from
>> getting merged? I've seen several times PRs getting merged at the last
>> minute,
>> with little to no testing, and breaking a release :( It hasn't
>> happened much lately, but I don't know of a way to block all PRs for a
>> period of time
>
>
> Do we have a way of releasing a given SHA1 (which has passed QE) even when
> additional PRs have been merged on top of it?
>
> IIRC in the past we could only release the tip of the named (jewel,
> luminous) branch. That does indeed create a risk of PRs getting merged
> post-QE and inadvertently finding their way into a release.
>
> Nathan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Better anticipation for minor releases

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On Fri, Feb 02, 2018 at 10:15:14AM -0700, Ken Dreyer wrote:
> This is just my opinion: we should have a quick "bump commit + tag"
> Jenkins job that pushes the Git tag live as soon as possible. In
> particular we should not wait for packages to be build + signed +
> published on download.ceph.com before pushing the bump commit + tag
> live.
> 
> Bumping and tagging in Git should take a minute or two, whereas
> building, signing, and rsync'ing all the packages for that tag takes
> hours. I think this time-intensive process causes the Git history to
> diverge for too long.
Thought:
- Tag or branch a candidate commit for going INTO the QE phase, say
  v12.2.3-rcX.
- Run QE on that tag.
- If QE passes entirely, create final tag at same commit as rc tag.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: Better anticipation for minor releases

[Vasu Kulkarni]

On Fri, Feb 2, 2018 at 11:55 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Fri, Feb 02, 2018 at 10:15:14AM -0700, Ken Dreyer wrote:
>> This is just my opinion: we should have a quick "bump commit + tag"
>> Jenkins job that pushes the Git tag live as soon as possible. In
>> particular we should not wait for packages to be build + signed +
>> published on download.ceph.com before pushing the bump commit + tag
>> live.
>>
>> Bumping and tagging in Git should take a minute or two, whereas
>> building, signing, and rsync'ing all the packages for that tag takes
>> hours. I think this time-intensive process causes the Git history to
>> diverge for too long.
> Thought:
> - Tag or branch a candidate commit for going INTO the QE phase, say
>   v12.2.3-rcX.
> - Run QE on that tag.
> - If QE passes entirely, create final tag at same commit as rc tag.

+1 on this, we already have test suites that need to pass for release,
tagging/branch for specific
release would make it easier to test/release in controlled way, then
working on tip of the stable branch
where unexpected prs can get merged. Also we should freeze other
branches so that commit
goes through only one branch during the dot release cycle until its out.



>
> --
> Robin Hugh Johnson
> Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
> E-Mail   : robbat2@gentoo.org
> GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
> GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: Better anticipation for minor releases

[Nathan Cutler]

On 02/02/2018 08:55 PM, Robin H. Johnson wrote:
> On Fri, Feb 02, 2018 at 10:15:14AM -0700, Ken Dreyer wrote:
>> This is just my opinion: we should have a quick "bump commit + tag"
>> Jenkins job that pushes the Git tag live as soon as possible. In
>> particular we should not wait for packages to be build + signed +
>> published on download.ceph.com before pushing the bump commit + tag
>> live.
>>
>> Bumping and tagging in Git should take a minute or two, whereas
>> building, signing, and rsync'ing all the packages for that tag takes
>> hours. I think this time-intensive process causes the Git history to
>> diverge for too long.
> Thought:
> - Tag or branch a candidate commit for going INTO the QE phase, say
>    v12.2.3-rcX.
> - Run QE on that tag.
> - If QE passes entirely, create final tag at same commit as rc tag.

That assumes it's possible the release tooling supports releasing 
something other than the tip of a branch (i.e. a SHA1 or tag), though, 
right?

Nathan

Re: Better anticipation for minor releases

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On Fri, Feb 02, 2018 at 10:35:20PM +0100, Nathan Cutler wrote:
> > Thought:
> > - Tag or branch a candidate commit for going INTO the QE phase, say
> >    v12.2.3-rcX.
> > - Run QE on that tag.
> > - If QE passes entirely, create final tag at same commit as rc tag.
> That assumes it's possible the release tooling supports releasing 
> something other than the tip of a branch (i.e. a SHA1 or tag), though, 
> right?
If the public tooling doesn't, I'd volunteer a little time to fix it.
As a bonus, the rc tag is available to everybody ELSE for their testing
much earlier.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: Better anticipation for minor releases

[Andrew Schoen]

>>
>> Thought:
>> - Tag or branch a candidate commit for going INTO the QE phase, say
>>    v12.2.3-rcX.
>> - Run QE on that tag.
>> - If QE passes entirely, create final tag at same commit as rc tag.
>
>
> That assumes it's possible the release tooling supports releasing something
> other than the tip of a branch (i.e. a SHA1 or tag), though, right?

I'm pretty sure we have support to build from a tag. The build system
might need a few tweaks, but I think it'd be doable and worth the
effort.

Andrew

Re: Better anticipation for minor releases

[Sage Weil]

On Mon, 5 Feb 2018, Andrew Schoen wrote:
> >>
> >> Thought:
> >> - Tag or branch a candidate commit for going INTO the QE phase, say
> >>    v12.2.3-rcX.
> >> - Run QE on that tag.
> >> - If QE passes entirely, create final tag at same commit as rc tag.
> >
> >
> > That assumes it's possible the release tooling supports releasing something
> > other than the tip of a branch (i.e. a SHA1 or tag), though, right?
> 
> I'm pretty sure we have support to build from a tag. The build system
> might need a few tweaks, but I think it'd be doable and worth the
> effort.

Building from a tag seems like the wrong approach.  What if the build 
fails for some reason (e.g., minor error in cmake files that affects 
release build but not test build)?  Or, perhaps for important, what if we 
get to the point where we do some testing on the final build 
artifact before finalizing/blessing the release?

I don't understand why racing updates is a problem.  git branching/merge 
solves exactly this problem:

 .
 .
 o  foo
 o  bar ... luminous branch target at time we start build
 |\
 | o baz (a racing change to luminous branch)
 | |
 o | v12.2.3 commit bumping release, with signed tag
  \|
   o merge commit (luminous branch after release is published)
   .
   .

If there is something wrong with the branch commit to matching the commit 
to be built, then *that* seems like the thing to fix in the toolchain 
(e.g., check branch once at the start, get sha1, then build *that*.)

sage

Re: Better anticipation for minor releases

[Ken Dreyer]

On Mon, Feb 5, 2018 at 8:50 AM, Sage Weil <sage@newdream.net> wrote:
> Building from a tag seems like the wrong approach.  What if the build
> fails for some reason (e.g., minor error in cmake files that affects
> release build but not test build)?

Do you have a specific example of this type of failure?

In general I want our test builds to be as close to our release builds
as possible. Some of the reasons for the differences are mainly
historical rather than technical. If there are particular differences
that could cause build failures, let's identify those and fix them!

> Or, perhaps for important, what if we
> get to the point where we do some testing on the final build
> artifact before finalizing/blessing the release?

This is an interesting idea. How would you picture someone doing this
testing, and who would that person be?

What happens when the testing fails and people have already merged
more racing PRs to luminous? We would have to back those out right?

v10.2.5 and v10.2.9 were quick releases to fix problems in the
immediate prior releases. If we merged a bunch of PRs at the same time
there, those quick versions would not have been as quick/safe.

> I don't understand why racing updates is a problem.  git branching/merge
> solves exactly this problem:

Again just my opinion, I think this makes the history harder to read.
For example, it's already a challenge to remember to run "git
describe" on the merge commit rather than the code change commit.

Re: Better anticipation for minor releases

[Sage Weil]

On Mon, 5 Feb 2018, Ken Dreyer wrote:
> On Mon, Feb 5, 2018 at 8:50 AM, Sage Weil <sage@newdream.net> wrote:
> > Building from a tag seems like the wrong approach.  What if the build
> > fails for some reason (e.g., minor error in cmake files that affects
> > release build but not test build)?
> 
> Do you have a specific example of this type of failure?
>
> In general I want our test builds to be as close to our release builds
> as possible. Some of the reasons for the differences are mainly
> historical rather than technical. If there are particular differences
> that could cause build failures, let's identify those and fix them!

I don't have an example, but I suspect Alfredo has half a dozen.  Totally 
agree that we should keep the build processes identical, but things 
happen, and even if they don't, it might be that the tip commit hasn't had 
a regular shaman build yet (or that the release person forgot to verify 
that build was successful before kicking off the release build).

> > Or, perhaps for important, what if we
> > get to the point where we do some testing on the final build
> > artifact before finalizing/blessing the release?
> 
> This is an interesting idea. How would you picture someone doing this
> testing, and who would that person be?

Maybe we stage the rc build, run it through qa, and only after that 
passes do we publish the tag and build artifacts.
 
> What happens when the testing fails and people have already merged
> more racing PRs to luminous? We would have to back those out right?

No... this is exactly the point.  Git branches remove any concern for 
"races".  If someone pushed 100 commits to luminous, just create a 
different branch with exactly the commit(s) you *do* want and build from 
that.  After everything is confirmed built, good, and tagged, push and 
merge it back into the luminous stream.

> v10.2.5 and v10.2.9 were quick releases to fix problems in the
> immediate prior releases. If we merged a bunch of PRs at the same time
> there, those quick versions would not have been as quick/safe.
> 
> > I don't understand why racing updates is a problem.  git branching/merge
> > solves exactly this problem:
> 
> Again just my opinion, I think this makes the history harder to read.
> For example, it's already a challenge to remember to run "git
> describe" on the merge commit rather than the code change commit.

It seems like the history complexity is there because history *was* 
complex: work didn't stop for the day(s) that the release was being built 
and finalized.  A simple fork+merge pattern like in my previous email 
isn't that complicated to understand, and it's how pretty every other 
change in the repo appears.

sage

Re: Better anticipation for minor releases

[Nathan Cutler]

> Git branches remove any concern for
> "races".  If someone pushed 100 commits to luminous, just create a
> different branch with exactly the commit(s) you *do* want and build from
> that.  After everything is confirmed built, good, and tagged, push and
> merge it back into the luminous stream.

Sage makes a good point here, but I don't agree with the "remove any 
concern for races" part. In practice, I'm envisioning that it would work 
like this (substitute "jewel", "mimic" etc. for luminous):

1. release preparation phase begins
2. lots of PRs/commits are merged into luminous
3. release preparation phase ends with declaration of a "release 
candidate" SHA1
4. a luminous-rc branch is created from this SHA1
5. QE phase begins
6. all QE work takes place in luminous-rc
7. any QE-related fixes are merged into luminous-rc
8. QE phase ends with declaration of a "release" SHA1 which is (almost - 
see below) guaranteed to be the tip of luminous-rc
9. tag/build phase begins
10. tag is applied to, and packages built from, tip of luminous-rc (as 
usual, except luminous-rc instead of luminous)
11. tag/build phase ends - point release is out
12. luminous-rc is merged into luminous

Granted, this does eliminate racing PRs in luminous, but it's still 
possible for something to get merged into luminous-rc after step 8 and 
before step 11, so the problem doesn't go away - it just gets less likely.

Nathan

Re: Better anticipation for minor releases

[Yuri Weinstein]

That looks good, Nathan!

I would add one more point after 1:
1. release preparation phase begins, PRs cut off date communicated.
1.5.  PRs cut off date reached, no more PRs accepted


On Mon, Feb 5, 2018 at 1:20 PM, Nathan Cutler <ncutler@suse.cz> wrote:
>> Git branches remove any concern for
>> "races".  If someone pushed 100 commits to luminous, just create a
>> different branch with exactly the commit(s) you *do* want and build from
>> that.  After everything is confirmed built, good, and tagged, push and
>> merge it back into the luminous stream.
>
>
> Sage makes a good point here, but I don't agree with the "remove any concern
> for races" part. In practice, I'm envisioning that it would work like this
> (substitute "jewel", "mimic" etc. for luminous):
>
> 1. release preparation phase begins
> 2. lots of PRs/commits are merged into luminous
> 3. release preparation phase ends with declaration of a "release candidate"
> SHA1
> 4. a luminous-rc branch is created from this SHA1
> 5. QE phase begins
> 6. all QE work takes place in luminous-rc
> 7. any QE-related fixes are merged into luminous-rc
> 8. QE phase ends with declaration of a "release" SHA1 which is (almost - see
> below) guaranteed to be the tip of luminous-rc
> 9. tag/build phase begins
> 10. tag is applied to, and packages built from, tip of luminous-rc (as
> usual, except luminous-rc instead of luminous)
> 11. tag/build phase ends - point release is out
> 12. luminous-rc is merged into luminous
>
> Granted, this does eliminate racing PRs in luminous, but it's still possible
> for something to get merged into luminous-rc after step 8 and before step
> 11, so the problem doesn't go away - it just gets less likely.
>
> Nathan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Better anticipation for minor releases

[Vasu Kulkarni]

> On Feb 5, 2018, at 1:20 PM, Nathan Cutler <ncutler@suse.cz> wrote:
> 
>> Git branches remove any concern for
>> "races".  If someone pushed 100 commits to luminous, just create a
>> different branch with exactly the commit(s) you *do* want and build from
>> that.  After everything is confirmed built, good, and tagged, push and
>> merge it back into the luminous stream.
> 
> Sage makes a good point here, but I don't agree with the "remove any concern for races" part. In practice, I'm envisioning that it would work like this (substitute "jewel", "mimic" etc. for luminous):
> 
> 1. release preparation phase begins
> 2. lots of PRs/commits are merged into luminous
> 3. release preparation phase ends with declaration of a "release candidate" SHA1
> 4. a luminous-rc branch is created from this SHA1
> 5. QE phase begins
> 6. all QE work takes place in luminous-rc
> 7. any QE-related fixes are merged into luminous-rc
> 8. QE phase ends with declaration of a "release" SHA1 which is (almost - see below) guaranteed to be the tip of luminous-rc
> 9. tag/build phase begins
> 10. tag is applied to, and packages built from, tip of luminous-rc (as usual, except luminous-rc instead of luminous)
> 11. tag/build phase ends - point release is out
> 12. luminous-rc is merged into luminous
> 
> Granted, this does eliminate racing PRs in luminous, but it's still possible for something to get merged into luminous-rc after step 8 and before step 11, so the problem doesn't go away - it just gets less likely.
For me this just looks like a communication issue or someone not fully understanding the process is trying to push to rc during end phase, the release lead can ensure this doesn’t happen or one can freeze/lock the branch during step 8 so that no further commits are possible.


> 
> Nathan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html