Ceph release cadence

Ceph release cadence

[Sage Weil]

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year, 
and every other such release has been an "LTS" release, with fixes 
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of 
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
kraken).  This limits the value of actually making them.  It also means 
that those who *do* run them are running riskier code (fewer users -> more 
bugs).

- The more recent requirement that upgrading clusters must make a stop at 
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel 
-> lumninous) has been hugely helpful on the development side by reducing 
the amount of cross-version compatibility code to maintain and reducing 
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always 
seems to be some "must-have" thing that delays the release a bit.  This 
doesn't happen as much with the odd releases, but it definitely happens 
with the LTS releases.  When the next LTS is a year away, it is hard to 
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular 
cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release 
cadence)

  + eliminate the confusing odd releases with dubious value
 
* Drop the odd releases, and aim for a ~9 month cadence. This splits the 
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to 
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> 
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage

Re: Ceph release cadence

[Bryan Banister]

Very new to Ceph but long time but long time sys admin who is jaded/opinionated.

My 2 cents:
1) This sounds like a perfect thing to put in a poll and ask/beg people to vote.  Hopefully that will get you more of a response from a larger number of users.
2) Given that the value of the odd releases is "dubious", maybe those that are using these releases can give reason why they feel they need/want them?

I, personally, think having the even releases on a shorter cadence with the most users on each would be best, but I'm still new to this game,
-Bryan

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On Behalf Of Sage Weil
Sent: Wednesday, September 06, 2017 10:24 AM
To: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ceph-maintainers-Qp0mS5GaXlQ@public.gmane.org; ceph-users-Qp0mS5GaXlQ@public.gmane.org
Subject: [ceph-users] Ceph release cadence

Note: External Email
-------------------------------------------------

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year,
and every other such release has been an "LTS" release, with fixes
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis,
kraken).  This limits the value of actually making them.  It also means
that those who *do* run them are running riskier code (fewer users -> more
bugs).

- The more recent requirement that upgrading clusters must make a stop at
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
-> lumninous) has been hugely helpful on the development side by reducing
the amount of cross-version compatibility code to maintain and reducing
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always
seems to be some "must-have" thing that delays the release a bit.  This
doesn't happen as much with the odd releases, but it definitely happens
with the LTS releases.  When the next LTS is a year away, it is hard to
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular
cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

  + eliminate the confusing odd releases with dubious value

* Drop the odd releases, and aim for a ~9 month cadence. This splits the
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

Re: Ceph release cadence

[Kingsley Tart]

On Wed, 2017-09-06 at 15:23 +0000, Sage Weil wrote:
> Hi everyone,
> 
> Traditionally, we have done a major named "stable" release twice a year, 
> and every other such release has been an "LTS" release, with fixes 
> backported for 1-2 years.
> 
> With kraken and luminous we missed our schedule by a lot: instead of 
> releasing in October and April we released in January and August.
> 
> A few observations:
[snip]

Firstly, I'd like to qualify my comments by saying that I haven't yet
tried Ceph[1], though I have been loosely following its progress. This
is partly because I've been busy doing other things.

[1] OK, this is a slight fib - I had a very brief play with it a few
years back but didn't really get anywhere with it and then got diverted
onto other things.

Unless I absolutely have to deploy now, I find myself doing this:

10 not long for new release, wait a bit
20 new release is here, but there's talk of a new one
30 goto 10

Having frequent minor updates and fixes is reassuring, but having
frequent major update changes with the "L" in "LTS" not being
particularly long tends to put me off a bit, largely because I find the
thought of upgrading something so mission critical quite daunting. I
can't speak from any Ceph experience on this one, obviously, but if
there's an easy rollback (even if it's never needed) without having to
rebuild the entire cluster then that would make me more willing to do
it.

-- 
Cheers,
Kingsley.

Re: Ceph release cadence

[Alex Gorbachev]

[-- Attachment #1.1: Type: text/plain, Size: 3313 bytes --]

On Wed, Sep 6, 2017 at 11:23 AM Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?


As a mission critical system user, I am in favor of dropping odd releases
and going to a 9 month cycle.  We never run the odd releases as too risky.
A good deal if functionality comes in updates, and usually the Ceph team
brings them in gently, with the more experimental features off by default.

I suspect the 9 month even cycle will also make it easier to perform more
incremental upgrades, i.e. small jumps rather than big leaps.



>
> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-- 
--
Alex Gorbachev
Storcium

[-- Attachment #1.2: Type: text/html, Size: 4446 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Ken Dreyer]

On Wed, Sep 6, 2017 at 9:23 AM, Sage Weil <sweil@redhat.com> wrote:
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year

This one (#2, regular release cadence) is the one I will value the most.

- Ken

Re: Ceph release cadence

[Joao Eduardo Luis]

On 09/06/2017 04:23 PM, Sage Weil wrote:
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
> 
>    + predictable schedule
>    - some features will miss the target and be delayed a year

Personally, I think a predictable schedule is the way to go. Two major 
reasons come to mind:

1. Developers are actually aware of what the cut-off date is, and will 
plan accordingly; and,

2. Downstreams will have a better notion of when the next release is to 
be expected, and plan accordingly.

However, a one year wait for a release may be a can of worms waiting to 
be opened. Even though it would ideally provide us a lot more time to 
merge stuff and test it, there's also the downside that some stuff may 
be pushed further and further down the line, and eventually merged just 
before the window closes.

For that, I'd argue having an intermediate (staging?) release could be 
helpful, but I fear it would not be anything more than any other 
dev-release. Therefore, if we stick to a one-year cadence, let's have 
frequent dev-releases.

I would also like to argue for a hard cut-off date considerably before 
major holiday seasons. Because no one really wants to be dealing with 
bugs or releasing software while a considerable portion of developers 
are away.

Additionally,

> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).

I can be on board with this too. As long as we have a very clear cut-off 
date regardless.

   -Joao

Re: [ceph-users] Ceph release cadence

[Eric Eastman]

I have been working with Ceph for the last several years and I help
support multiple Ceph clusters. I would like to have the team drop the
Even/Odd release schedule, and go to an all production release
schedule.  I would like releases on no more then a 9 month schedule,
with smaller incremental changes and predictable dates.  It would be
nice to be able to upgrade from at least the last 2 releases.

I would also like to see the RC schedule be more like the Linux kernel
or Samba releases, where we have an idea on how many RCs to expect and
how often they would come out, so we can schedule our testing, and
provide more helpful feedback during the RC period.

Eric

On Wed, Sep 6, 2017 at 9:23 AM, Sage Weil <sweil@redhat.com> wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Deepak Naidu]

Hope collective feedback helps. So here's one.

>>- Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).  
I think the more obvious reason companies/users wanting to use CEPH will stick with LTS versions as it models the 3yr  support cycle.

>>* Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.
Yes, provided an easy upgrade process.


--
Deepak




-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On Behalf Of Sage Weil
Sent: Wednesday, September 06, 2017 8:24 AM
To: ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; ceph-maintainers-Qp0mS5GaXlQ@public.gmane.org; ceph-users-Qp0mS5GaXlQ@public.gmane.org
Subject: [ceph-users] Ceph release cadence

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year, and every other such release has been an "LTS" release, with fixes backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).  This limits the value of actually making them.  It also means that those who *do* run them are running riskier code (fewer users -> more bugs).

- The more recent requirement that upgrading clusters must make a stop at each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel 
-> lumninous) has been hugely helpful on the development side by 
-> reducing
the amount of cross-version compatibility code to maintain and reducing the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always seems to be some "must-have" thing that delays the release a bit.  This doesn't happen as much with the odd releases, but it definitely happens with the LTS releases.  When the next LTS is a year away, it is hard to suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

  + eliminate the confusing odd releases with dubious value
 
* Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: [ceph-users] Ceph release cadence

[Henrik Korkuc]

On 17-09-07 02:42, Deepak Naidu wrote:
> Hope collective feedback helps. So here's one.
>
>>> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).
> I think the more obvious reason companies/users wanting to use CEPH will stick with LTS versions as it models the 3yr  support cycle.
Maybe I missed something, but I think Ceph does not support LTS releases 
for 3 years.

>>> * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.
> Yes, provided an easy upgrade process.
>
>
> --
> Deepak
>
>
>
>
> -----Original Message-----
> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of Sage Weil
> Sent: Wednesday, September 06, 2017 8:24 AM
> To: ceph-devel@vger.kernel.org; ceph-maintainers@ceph.com; ceph-users@ceph.com
> Subject: [ceph-users] Ceph release cadence
>
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year, and every other such release has been an "LTS" release, with fixes backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).  This limits the value of actually making them.  It also means that those who *do* run them are running riskier code (fewer users -> more bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by
> -> reducing
> the amount of cross-version compatibility code to maintain and reducing the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always seems to be some "must-have" thing that delays the release a bit.  This doesn't happen as much with the odd releases, but it definitely happens with the LTS releases.  When the next LTS is a year away, it is hard to suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>    + flexible
>    - unpredictable
>    - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular cadence
>
>    + predictable schedule
>    - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>    + eliminate the confusing odd releases with dubious value
>   
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.
>
>    + eliminate the confusing odd releases with dubious value
>    + waiting for the next release isn't quite as bad
>    - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> nautilus).  Shorten release cycle (~6-9 months).
>
>    + more flexibility for users
>    + downstreams have greater choice in adopting an upstrema release
>    - more LTS branches to maintain
>    - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information.  Any unauthorized review, use, disclosure or distribution
> is prohibited.  If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph release cadence

[Adrian Saul]

> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months

As a user is probably closest to the ideal, although a production deployment might slip out of the LTS view in 18 months given once deployed they tend to stay static.

From a testing perspective it would be good to know you could deploy the "early access" version of a release and test with that rather than having to switch release to productionise when that release is blessed.

Also, and this might be harder to achieve, but could krbd support for new releases be more aligned with kernel versions?  Or at the least a definitive map of what kernels and backports support which release.


Confidentiality: This email and any attachments are confidential and may be subject to copyright, legal or some other professional privilege. They are intended solely for the attention and use of the named addressee(s). They may only be copied, distributed or disclosed with the consent of the copyright owner. If you have received this email by mistake or by breach of the confidentiality clause, please notify the sender immediately by return email and delete or destroy all copies of the email. Any confidentiality, privilege or copyright is not waived or lost because this email has been sent to you by mistake.

Re: [ceph-users] Ceph release cadence

[Henrik Korkuc]

On 17-09-06 18:23, Sage Weil wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>    + flexible
>    - unpredictable
>    - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>    + predictable schedule
>    - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>    + eliminate the confusing odd releases with dubious value
>   
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>    + eliminate the confusing odd releases with dubious value
>    + waiting for the next release isn't quite as bad
>    - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>    + more flexibility for users
>    + downstreams have greater choice in adopting an upstrema release
>    - more LTS branches to maintain
>    - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
What about this:
* drop odd releases
* have ~9 months release schedule
* no version jumping
* bugfix support for 2 release cycles
* list of major incoming features with their status, disabled by feature 
flag.
* have more QA passed dev releases so that people waiting for new 
features would be able to try them out

This way we trade shorter release cycle with longer bugfix support but 
no version jumping. This way stable folks could upgrade from 
"legacy-stable" to "old-stable" having already multiple minor fixes in 
both releases.

And bleading-edge people waiting for some features would know current 
status of new features (e.g. multi-active MDS going stable in L was a 
surprise for me). With dev releases to run in dev/staging env for testing.

Potentially shorter releases would have less features in, so it should 
be less risky to use them and of course shorter wait before new release.

> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [Ceph-maintainers] Ceph release cadence

[Lars Marowsky-Bree]

On 2017-09-06T15:23:34, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

Hi Sage,

thanks for kicking off this discussion - after the L experience, it was
on my hot list to talk about too.

I do agree that we need predictable releases more than feature-rich
releases. Distributors like to plan, but that's not a reason. However,
we like to plan because *users* like to plan their schedules and
upgrades, and I think that matters more.

> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
> kraken).  This limits the value of actually making them.  It also means 
> that those who *do* run them are running riskier code (fewer users -> more 
> bugs).

Yes. Odd releases never really make it to user systems. They're on the
previous LTS release. In the devel releases, the code is often too
unstable, and developers seem to cram everything in. Basically, the odd
releases are long periods working up to the next stable release.

(And they get all the cool names, which I find personally sad. I want my
users to run Infernalis, Kraken, and Mimic. ;-)

> - The more recent requirement that upgrading clusters must make a stop at 
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel 
> -> lumninous) has been hugely helpful on the development side by reducing 
> the amount of cross-version compatibility code to maintain and reducing 
> the number of upgrade combinations to test.

On this, I feel that it might make more sense to phrase this so that
such cross version compatibility is not tied to major releases (which
doesn't really help them plan lifecycles if those releases aren't
reliable), but to time periods.

> - When we try to do a time-based "train" release cadence, there always 
> seems to be some "must-have" thing that delays the release a bit.  This 
> doesn't happen as much with the odd releases, but it definitely happens 
> with the LTS releases.  When the next LTS is a year away, it is hard to 
> suck it up and wait that long.

Yes, I can see that. This is clearly something we'd want to avoid.

> A couple of options:
> 
> * Keep even/odd pattern, and continue being flexible with release dates

I admit I'm not a fan of this one.

> * Drop the odd releases but change nothing else (i.e., 12-month release 
> cadence)
>   + eliminate the confusing odd releases with dubious value

Periods too long for regular users. Admittedly, I suspect for RH and
SUSE with RHCS or SES respectively, this doesn't matter much - but it's
not good for the community as a whole. Also, this means not enough
community / end-user testing will happen for 11 out of those 12 months,
implying such long cycles make it hard to release n+1.0 in high
quality.

I've been doing software development for almost two decades, and no user
really touches betas before one calls it an RC, and even then ...

> * Drop the odd releases, and aim for a ~9 month cadence. This splits the 
> difference between the current even/odd pattern we've been doing.

It's a step up, but the period is still both too long, and unaligned.
This makes lifecycle management for everyone annoying.

> * Drop the odd releases, but relax the "must upgrade through every LTS" to 
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> 
> nautilus).  Shorten release cycle (~6-9 months).
> 
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider

From the list of options you provide, I like this one the best; the ~6
month release cycle means there should be one about once per year as
well, which makes cycling easier to plan.

> Other options we should consider?  Other thoughts?

With about 20-odd years in software development, I've become a big
believer in schedule-driven releases. If it's feature-based, you never
know when they'll get done.

If the schedule intervals are too long though, the urge to press too
much in (so as not to miss the next merge window) is just too high,
meaning the train gets derailed. (Which cascades into the future,
because the next time the pressure will be even higher based on the
previous experience.) This requires strictness.

We've had a few Linux kernel releases that were effectively feature
driven and never quite made it. 1.3.x? 1.5.x? My memory is bad, but they
were a disaster than eventually led Linus to evolve to the current
model.

That serves them really well, and I believe it might be worth
considering for us.

I'd try to move away from the major milestones. Features get integrated
into the next schedule-driven release when they deemed ready and stable;
when they're not, not a big deal, the next one is coming up "soonish".

(This effectively decouples feature development slightly from the
release schedule.)

We could even go for "a release every 3 months, sharp", merge window for
the first month, stabilization the second, release clean up the third,
ship.

Interoperability hacks for the cluster/server side are maintained for 2
years, and then dropped.  Sharp. (Speaking as one of those folks
affected, we should not burden the community with this.) Client interop
is a different story, a bit.

Basically, effectively edging towards continuous integration of features
and bugfixes both. Nobody has to wait for anything much, and can
schedule reasonably independently.

There is a single LTS release: Ceph. Keep on rolling.

Also, Mimic is a good release to pick for a change like this, because it
can be everything to everyone ;-)


Regards,
    Lars

-- 
Architect SDS, Distinguished Engineer
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

RE: [ceph-users] Ceph release cadence

[Deepak Naidu]

>> Maybe I missed something, but I think Ceph does not support LTS releases for 3 years.
Yes, you are correct but it averages to 18mths sometime I see 20mths(Hammer). But anything with 1yr release cycle is not worth the time and having near 3yr support model is best for PROD.

http://docs.ceph.com/docs/master/releases/

--
Deepak

-----Original Message-----
From: Henrik Korkuc [mailto:lists@kirneh.eu] 
Sent: Wednesday, September 06, 2017 10:50 PM
To: Deepak Naidu; Sage Weil; ceph-devel@vger.kernel.org; ceph-maintainers@ceph.com; ceph-users@ceph.com
Subject: Re: [ceph-users] Ceph release cadence

On 17-09-07 02:42, Deepak Naidu wrote:
> Hope collective feedback helps. So here's one.
>
>>> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).
> I think the more obvious reason companies/users wanting to use CEPH will stick with LTS versions as it models the 3yr  support cycle.
Maybe I missed something, but I think Ceph does not support LTS releases for 3 years.

>>> * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.
> Yes, provided an easy upgrade process.
>
>
> --
> Deepak
>
>
>
>
> -----Original Message-----
> From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf 
> Of Sage Weil
> Sent: Wednesday, September 06, 2017 8:24 AM
> To: ceph-devel@vger.kernel.org; ceph-maintainers@ceph.com; 
> ceph-users@ceph.com
> Subject: [ceph-users] Ceph release cadence
>
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year, and every other such release has been an "LTS" release, with fixes backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).  This limits the value of actually making them.  It also means that those who *do* run them are running riskier code (fewer users -> more bugs).
>
> - The more recent requirement that upgrading clusters must make a stop 
> at each LTS (e.g., hammer -> luminous not supported, must go hammer -> 
> jewel
> -> lumninous) has been hugely helpful on the development side by 
> -> reducing
> the amount of cross-version compatibility code to maintain and reducing the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always seems to be some "must-have" thing that delays the release a bit.  This doesn't happen as much with the odd releases, but it definitely happens with the LTS releases.  When the next LTS is a year away, it is hard to suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release 
> dates
>
>    + flexible
>    - unpredictable
>    - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular 
> cadence
>
>    + predictable schedule
>    - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month 
> release
> cadence)
>
>    + eliminate the confusing odd releases with dubious value
>   
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.
>
>    + eliminate the confusing odd releases with dubious value
>    + waiting for the next release isn't quite as bad
>    - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> nautilus).  Shorten release cycle (~6-9 months).
>
>    + more flexibility for users
>    + downstreams have greater choice in adopting an upstrema release
>    - more LTS branches to maintain
>    - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ----------------------------------------------------------------------
> ------------- This email message is for the sole use of the intended 
> recipient(s) and may contain confidential information.  Any 
> unauthorized review, use, disclosure or distribution is prohibited.  
> If you are not the intended recipient, please contact the sender by 
> reply email and destroy all copies of the original message.
> ----------------------------------------------------------------------
> -------------
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph release cadence

[Matthew Vernon]

Hi,

On 06/09/17 16:23, Sage Weil wrote:

> Traditionally, we have done a major named "stable" release twice a year, 
> and every other such release has been an "LTS" release, with fixes 
> backported for 1-2 years.

We use the ceph version that comes with our distribution (Ubuntu LTS);
those come out every 2 years (though we won't move to a brand-new
distribution until we've done some testing!). So from my POV, LTS ceph
releases that come out such that adjacent ceph LTSs fit neatly into
adjacent Ubuntu LTSs is the ideal outcome. We're unlikely to ever try
putting a non-LTS ceph version into production.

I hope this isn't an unusual requirement :)

Matthew


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 

Re: [ceph-users] Ceph release cadence

[Xiaoxi Chen]

It is not because user want to skip odd release, it just we usually
land major and important feature to even release:)

We have several large ceph cluster starting from Firefly Although in
practice we only use LTS release but it is not because we bias LTS
release, just the timeline is a little bit misleading...  Lets use
kraken as an example , it is out on late jan , when I deciding whether
to upgrade all cluster to kraken,  the timeline said Luminous will be
out on April, although from the community activity it doesn't looks
like it, but in all public presentation we(the community) were
releasing the same information.

Considering the effort of  pre-production test and rolling upgrade
usually take half a month, we decide to skip kraken and wait for
another 3 month(but actually 8 months).

Rethinking about it now,  if  either

1) community re-adjust the release timeline in Jun, postpone Luminous
to August , we will do the upgrade.
2) some important features first ready in Kraken( Multi-mds,
Bluestore, EC overwrite), we will also do the upgrade.

But both are not true.




2017-09-08 17:20 GMT+08:00 Matthew Vernon <mv3@sanger.ac.uk>:
> Hi,
>
> On 06/09/17 16:23, Sage Weil wrote:
>
>> Traditionally, we have done a major named "stable" release twice a year,
>> and every other such release has been an "LTS" release, with fixes
>> backported for 1-2 years.
>
> We use the ceph version that comes with our distribution (Ubuntu LTS);
> those come out every 2 years (though we won't move to a brand-new
> distribution until we've done some testing!). So from my POV, LTS ceph
> releases that come out such that adjacent ceph LTSs fit neatly into
> adjacent Ubuntu LTSs is the ideal outcome. We're unlikely to ever try
> putting a non-LTS ceph version into production.
>
> I hope this isn't an unusual requirement :)
>
> Matthew
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph release cadence

[Scottix]

[-- Attachment #1.1: Type: text/plain, Size: 2305 bytes --]

Personally I kind of like the current format and fundamentally we are
talking about Data storage which should be the most tested and scrutinized
piece of software on your computer. Having XYZ feature later than sooner
compared to oh I lost all my data. I am thinking of a recent FS that had a
feature they shouldn't have released. I appreciate the extra time it takes
to release to make it resilient.

Having a LTS version to rely on provides a good assurance the upgrade
process wil be thoroughly tested.
Having a version to do more experimental features keeps the new features at
bay, it follows the Ubuntu model basically.

I feel there were a lot of underpinning features in Luminous that
checkmarked a lot of checkboxes you have been wanting for a while. One
thing to consider possibly a lot of the core features become more
incremental.

I guess from my use case Ceph actually does everything I need it to do atm.
Yes new features and better processes make it better, but more or less I am
pretty content. Maybe I am a small minority in this logic.

On Fri, Sep 8, 2017 at 2:20 AM Matthew Vernon <mv3-5fLPn3lgkryFxr2TtlUqVg@public.gmane.org> wrote:

> Hi,
>
> On 06/09/17 16:23, Sage Weil wrote:
>
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
>
> We use the ceph version that comes with our distribution (Ubuntu LTS);
> those come out every 2 years (though we won't move to a brand-new
> distribution until we've done some testing!). So from my POV, LTS ceph
> releases that come out such that adjacent ceph LTSs fit neatly into
> adjacent Ubuntu LTSs is the ideal outcome. We're unlikely to ever try
> putting a non-LTS ceph version into production.
>
> I hope this isn't an unusual requirement :)
>
> Matthew
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 3190 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [Ceph-maintainers] Ceph release cadence

[Sage Weil]

I'm going to pick on Lars a bit here...

On Thu, 7 Sep 2017, Lars Marowsky-Bree wrote:
> On 2017-09-06T15:23:34, Sage Weil <sweil@redhat.com> wrote:
> > Other options we should consider?  Other thoughts?
> 
> With about 20-odd years in software development, I've become a big
> believer in schedule-driven releases. If it's feature-based, you never
> know when they'll get done.
> 
> If the schedule intervals are too long though, the urge to press too
> much in (so as not to miss the next merge window) is just too high,
> meaning the train gets derailed. (Which cascades into the future,
> because the next time the pressure will be even higher based on the
> previous experience.) This requires strictness.
> 
> We've had a few Linux kernel releases that were effectively feature
> driven and never quite made it. 1.3.x? 1.5.x? My memory is bad, but they
> were a disaster than eventually led Linus to evolve to the current
> model.
> 
> That serves them really well, and I believe it might be worth
> considering for us.

This model is very appealing.  The problem with it that I see is that the 
upstream kernel community doesn't really do stable releases.  Mainline 
developers are just getting their stuff upstream, and entire separate 
organizations and teams are doing the stable distro kernels.  (There are 
upstream stable kernels too, yes, but they don't get much testing AFAICS 
and I'm not sure who uses them.)

More importantly, upgrade and on-disk format issues are present for almost 
everything that we change in Ceph.  Those things rarely come up for the 
kernel.  Even the local file systems (a small piece of the kernel) have 
comparatively fewer format changes that we do, it seems.

These make the upgrade testing a huge concern and burden for the 
Ceph development community.

> I'd try to move away from the major milestones. Features get integrated
> into the next schedule-driven release when they deemed ready and stable;
> when they're not, not a big deal, the next one is coming up "soonish".
> 
> (This effectively decouples feature development slightly from the
> release schedule.)
> 
> We could even go for "a release every 3 months, sharp", merge window for
> the first month, stabilization the second, release clean up the third,
> ship.
> 
> Interoperability hacks for the cluster/server side are maintained for 2
> years, and then dropped.  Sharp. (Speaking as one of those folks
> affected, we should not burden the community with this.) Client interop
> is a different story, a bit.
> 
> Basically, effectively edging towards continuous integration of features
> and bugfixes both. Nobody has to wait for anything much, and can
> schedule reasonably independently.

If I read between the lines a bit here, but this sounds like is:

 - keep the frequently major releases (but possibly shorten the 6mo 
   cadence)
 - do backports for all of them, not just the even ones
 - test upgrades between all of them within a 2 year horizon, instead 
   of just the last major one

Is that accurate?

Unfortunately it sounds to me like that would significantly increase the 
maintenance burden (double it even?) and slow development down.  The user 
base will also end up fragmented across a broader range of versions, which 
means we'll see a wider variety of bugs and each release will be less 
stable.

This is full of trade-offs... time we spend backporting or testing 
upgrades is time we don't spend fixing bugs or improving performance or 
adding features.

sage

Re: [Ceph-maintainers] Ceph release cadence

[Bassam Tabbara]

A big +1 to Lars proposal.

Having the train run on time (3 months) has many benefits. Take a look at the Kuberenetes release cycle for reference (https://github.com/kubernetes/community/wiki/Life-of-a-Kubernetes-Release) and its one of the highest velocity projects and they do stateful, breaking and protocol upgrades too.

> These make the upgrade testing a huge concern and burden for the 
> Ceph development community.
> 

Sage, do you think an investment in more automation around upgrade testing would help this? Limiting the horizon to a ~1 year could help too.

> - test upgrades between all of them within a 2 year horizon, instead 
>   of just the last major one
> 

If you limit the upgrade horizon to 2 previous versions, a customer on n-3 could upgrade to n-2 and then to n. Also I think by declaring the policy most admins will start upgrading more frequently.

My 2 cents.

Re: [Ceph-maintainers] Ceph release cadence

[Gregory Farnum]

I think I'm the resident train release advocate so I'm sure my
advocating that model will surprise nobody. I'm not sure I'd go all
the way to Lars' multi-release maintenance model (although it's
definitely something I'm interested in), but there are two big reasons
I wish we were on a train with more frequent real releases:

1) It reduces the cost of features missing a release. Right now if
something misses an LTS release, that's it for a year. And nobody
likes releasing an LTS without a bunch of big new features, so each
LTS is later than the one before as we scramble to get features merged
in.

...and then we deal with the fact that we scrambled to get a bunch of
features merged in and they weren't quite baked. (Luminous so far
seems to have gone much better in this regard! Hurray! But I think
that has a lot to do with our feature-release-scramble this year being
mostly peripheral stuff around user interfaces that got tacked on
about the time we'd initially planned the release to occur.)

2) Train releases increase predictability for downstreams, partners,
and users around when releases will happen. Right now, the release
process and schedule is entirely opaque to anybody who's not involved
in every single upstream meeting we have; and it's unpredictable even
to those who are. That makes things difficult, as Xiaoxi said.

There are other peripheral but serious benefits I'd expect to see from
fully-validated train releases as well. It would be *awesome* to have
more frequent known-stable points to do new development against. If
you're an external developer and you want a new feature, you have to
either keep it rebased against a fast-changing master branch, or you
need to settle for writing it against a long-out-of-date LTS and then
forward-porting it for merge. If you're an FS developer writing a very
small new OSD feature and you try to validate it against RADOS, you've
no idea if bugs that pop up and look random are because you really did
something wrong or if there's currently an intermittent issue in RADOS
master. I would have *loved* to be able to maintain CephFS integration
branches for features that didn't touch RADOS and were built on top of
the latest release instead of master, but it was utterly infeasible
because there were too many missing features with the long delays.

On Fri, Sep 8, 2017 at 9:16 AM, Sage Weil <sweil@redhat.com> wrote:
> I'm going to pick on Lars a bit here...
>
> On Thu, 7 Sep 2017, Lars Marowsky-Bree wrote:
>> On 2017-09-06T15:23:34, Sage Weil <sweil@redhat.com> wrote:
>> > Other options we should consider?  Other thoughts?
>>
>> With about 20-odd years in software development, I've become a big
>> believer in schedule-driven releases. If it's feature-based, you never
>> know when they'll get done.
>>
>> If the schedule intervals are too long though, the urge to press too
>> much in (so as not to miss the next merge window) is just too high,
>> meaning the train gets derailed. (Which cascades into the future,
>> because the next time the pressure will be even higher based on the
>> previous experience.) This requires strictness.
>>
>> We've had a few Linux kernel releases that were effectively feature
>> driven and never quite made it. 1.3.x? 1.5.x? My memory is bad, but they
>> were a disaster than eventually led Linus to evolve to the current
>> model.
>>
>> That serves them really well, and I believe it might be worth
>> considering for us.
>
> This model is very appealing.  The problem with it that I see is that the
> upstream kernel community doesn't really do stable releases.  Mainline
> developers are just getting their stuff upstream, and entire separate
> organizations and teams are doing the stable distro kernels.  (There are
> upstream stable kernels too, yes, but they don't get much testing AFAICS
> and I'm not sure who uses them.)
>
> More importantly, upgrade and on-disk format issues are present for almost
> everything that we change in Ceph.  Those things rarely come up for the
> kernel.  Even the local file systems (a small piece of the kernel) have
> comparatively fewer format changes that we do, it seems.
>
> These make the upgrade testing a huge concern and burden for the
> Ceph development community.
>
>> I'd try to move away from the major milestones. Features get integrated
>> into the next schedule-driven release when they deemed ready and stable;
>> when they're not, not a big deal, the next one is coming up "soonish".
>>
>> (This effectively decouples feature development slightly from the
>> release schedule.)
>>
>> We could even go for "a release every 3 months, sharp", merge window for
>> the first month, stabilization the second, release clean up the third,
>> ship.
>>
>> Interoperability hacks for the cluster/server side are maintained for 2
>> years, and then dropped.  Sharp. (Speaking as one of those folks
>> affected, we should not burden the community with this.) Client interop
>> is a different story, a bit.
>>
>> Basically, effectively edging towards continuous integration of features
>> and bugfixes both. Nobody has to wait for anything much, and can
>> schedule reasonably independently.
>
> If I read between the lines a bit here, but this sounds like is:
>
>  - keep the frequently major releases (but possibly shorten the 6mo
>    cadence)
>  - do backports for all of them, not just the even ones
>  - test upgrades between all of them within a 2 year horizon, instead
>    of just the last major one
>
> Is that accurate?
>
> Unfortunately it sounds to me like that would significantly increase the
> maintenance burden (double it even?) and slow development down.  The user
> base will also end up fragmented across a broader range of versions, which
> means we'll see a wider variety of bugs and each release will be less
> stable.
>
> This is full of trade-offs... time we spend backporting or testing
> upgrades is time we don't spend fixing bugs or improving performance or
> adding features.

Not all newly-allocated effort for doing maintenance and testing
necessarily means reducing effort available for new feature
development. Sometimes it just makes that development easier and more
efficient!    I think we'd find that more tested and stable releases
would spread joy and stability throughout our development process and
make life much easier on prospective contributors.
-Greg

Re: Ceph release cadence

[Christian Theune]

[-- Attachment #1.1: Type: text/plain, Size: 5036 bytes --]

Hi,

have been using Ceph for multiple years now. It’s unclear to me which of your options fits best, but here are my preferences:

* Updates are risky in a way that we tend to rather not do them every year. Also, having seen jewel, we’ve been well off to avoid two
  major issues what would have bitten us and will upgrade from hammer in the next month or so.

* Non-production releases are of not much value to me, as I have to keep our dev/staging/prod clusters in sync to work on our stuff.
  As you can never downgrade, there’s no value in it for me to try non-production releases (without frying dev for everyone).

* I’d prefer stability over new features. *Specifically* that new features can be properly recombined with existing features (and each
  other) without leading to surprises. (E.g. cache tiering breaking with snapshots and then no way going back and a general notion of
  “that combination wasn’t really well tested).

* I’d prefer versions that I have to be maintained for production-critical issues maybe 2 years, so I can have some time after a new
  production release that overlaps with the new production release receiving important bug fixes until I switch.

Maybe this is close to what your "Drop the odd releases, and aim for a ~9 month cadence.” would say. Waiting for a feature for a year is a pain, but my personal goal for Ceph is that it first has to work properly, meaning: not loose your data, not "stopping the show”, and not drawing you into a corner you can’t get out.

That’s my perspective as a user. As a fellow developer I feel your pain about wanting to release faster and reducing maintenance load, so thanks for asking!

Hope this helps,
Christian

> On Sep 6, 2017, at 5:23 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> 
> Hi everyone,
> 
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
> 
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
> 
> A few observations:
> 
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
> 
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
> 
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
> 
> A couple of options:
> 
> * Keep even/odd pattern, and continue being flexible with release dates
> 
>  + flexible
>  - unpredictable
>  - odd releases of dubious value
> 
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
> 
>  + predictable schedule
>  - some features will miss the target and be delayed a year
> 
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
> 
>  + eliminate the confusing odd releases with dubious value
> 
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
> 
>  + eliminate the confusing odd releases with dubious value
>  + waiting for the next release isn't quite as bad
>  - required upgrades every 9 months instead of ever 12 months
> 
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
> 
>  + more flexibility for users
>  + downstreams have greater choice in adopting an upstrema release
>  - more LTS branches to maintain
>  - more upgrade paths to consider
> 
> Other options we should consider?  Other thoughts?
> 
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Liebe Grüße,
Christian Theune

--
Christian Theune · ct-4ImjS3E6Ur3YxmODkEiZ1Q@public.gmane.org · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick


[-- Attachment #1.2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Sasha Litvak]

[-- Attachment #1.1: Type: text/plain, Size: 6462 bytes --]

As a user,  I woul like to add,  I would like to see a real 2 year support
for LTS releases.  Hammer releases were sketchy at best in 2017.  When
luminous was released The outstanding bugs were auto closed, good buy and
good readance.

 Also the decision to drop certain OS support created a barrier to upgrade
and looking at jewel and luminous upgrade path where you cannot easily go
back after upgrade is completed doesn't add the confidence.

So making upgrades less radical may help production support to be more
consistent and update process less dangerous.

I would say 9 month is a good reference point but for me  it is ready when
it is really ready and tested.

Keeping development release may be better for devs and early adopters.  I
don't believe production admins would go for intermediate one's as they
being released now.

This is only MHO and may be wrong.

On Sep 9, 2017 15:32, "Christian Theune" <ct-4ImjS3E6Ur3YxmODkEiZ1Q@public.gmane.org> wrote:

> Hi,
>
> have been using Ceph for multiple years now. It’s unclear to me which of
> your options fits best, but here are my preferences:
>
> * Updates are risky in a way that we tend to rather not do them every
> year. Also, having seen jewel, we’ve been well off to avoid two
>   major issues what would have bitten us and will upgrade from hammer in
> the next month or so.
>
> * Non-production releases are of not much value to me, as I have to keep
> our dev/staging/prod clusters in sync to work on our stuff.
>   As you can never downgrade, there’s no value in it for me to try
> non-production releases (without frying dev for everyone).
>
> * I’d prefer stability over new features. *Specifically* that new features
> can be properly recombined with existing features (and each
>   other) without leading to surprises. (E.g. cache tiering breaking with
> snapshots and then no way going back and a general notion of
>   “that combination wasn’t really well tested).
>
> * I’d prefer versions that I have to be maintained for production-critical
> issues maybe 2 years, so I can have some time after a new
>   production release that overlaps with the new production release
> receiving important bug fixes until I switch.
>
> Maybe this is close to what your "Drop the odd releases, and aim for a ~9
> month cadence.” would say. Waiting for a feature for a year is a pain, but
> my personal goal for Ceph is that it first has to work properly, meaning:
> not loose your data, not "stopping the show”, and not drawing you into a
> corner you can’t get out.
>
> That’s my perspective as a user. As a fellow developer I feel your pain
> about wanting to release faster and reducing maintenance load, so thanks
> for asking!
>
> Hope this helps,
> Christian
>
> > On Sep 6, 2017, at 5:23 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >  + flexible
> >  - unpredictable
> >  - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >  + predictable schedule
> >  - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >  + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >  + eliminate the confusing odd releases with dubious value
> >  + waiting for the next release isn't quite as bad
> >  - required upgrades every 9 months instead of ever 12 months
> >
> > * Drop the odd releases, but relax the "must upgrade through every LTS"
> to
> > allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> > nautilus).  Shorten release cycle (~6-9 months).
> >
> >  + more flexibility for users
> >  + downstreams have greater choice in adopting an upstrema release
> >  - more LTS branches to maintain
> >  - more upgrade paths to consider
> >
> > Other options we should consider?  Other thoughts?
> >
> > Thanks!
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Liebe Grüße,
> Christian Theune
>
> --
> Christian Theune · ct-4ImjS3E6Ur3YxmODkEiZ1Q@public.gmane.org · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> Zagrodnick
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>

[-- Attachment #1.2: Type: text/html, Size: 8109 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Sasha Litvak]

[-- Attachment #1.1: Type: text/plain, Size: 6321 bytes --]

As a user,  I woul like to add,  I would like to see a real 2 year support
for LTS releases.  Hammer releases were sketchy at best in 2017.  When
luminous was released The outstanding bugs were auto closed, good buy and
good readance.

 Also the decision to drop certain OS support created a barrier to upgrade
and looking at jewel and luminous upgrade path where you cannot easily go
back after upgrade is completed doesn't add the confidence.

So making upgrades less radical may help production support to be more
consistent and update process less dangerous.

I would say 9 month is a good reference point but for me  it is ready when
it is really ready and tested.

Keeping development release may be better for devs and early adopters.  I
don't believe production admins would go for intermediate one's as they
being released now.

This is only MHO and may be wrong

On Saturday, September 9, 2017, Christian Theune <ct-4ImjS3E6Ur3YxmODkEiZ1Q@public.gmane.org> wrote:

> Hi,
>
> have been using Ceph for multiple years now. It’s unclear to me which of
> your options fits best, but here are my preferences:
>
> * Updates are risky in a way that we tend to rather not do them every
> year. Also, having seen jewel, we’ve been well off to avoid two
>   major issues what would have bitten us and will upgrade from hammer in
> the next month or so.
>
> * Non-production releases are of not much value to me, as I have to keep
> our dev/staging/prod clusters in sync to work on our stuff.
>   As you can never downgrade, there’s no value in it for me to try
> non-production releases (without frying dev for everyone).
>
> * I’d prefer stability over new features. *Specifically* that new features
> can be properly recombined with existing features (and each
>   other) without leading to surprises. (E.g. cache tiering breaking with
> snapshots and then no way going back and a general notion of
>   “that combination wasn’t really well tested).
>
> * I’d prefer versions that I have to be maintained for production-critical
> issues maybe 2 years, so I can have some time after a new
>   production release that overlaps with the new production release
> receiving important bug fixes until I switch.
>
> Maybe this is close to what your "Drop the odd releases, and aim for a ~9
> month cadence.” would say. Waiting for a feature for a year is a pain, but
> my personal goal for Ceph is that it first has to work properly, meaning:
> not loose your data, not "stopping the show”, and not drawing you into a
> corner you can’t get out.
>
> That’s my perspective as a user. As a fellow developer I feel your pain
> about wanting to release faster and reducing maintenance load, so thanks
> for asking!
>
> Hope this helps,
> Christian
>
> > On Sep 6, 2017, at 5:23 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org <javascript:;>>
> wrote:
> >
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >  + flexible
> >  - unpredictable
> >  - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >  + predictable schedule
> >  - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >  + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >  + eliminate the confusing odd releases with dubious value
> >  + waiting for the next release isn't quite as bad
> >  - required upgrades every 9 months instead of ever 12 months
> >
> > * Drop the odd releases, but relax the "must upgrade through every LTS"
> to
> > allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> > nautilus).  Shorten release cycle (~6-9 months).
> >
> >  + more flexibility for users
> >  + downstreams have greater choice in adopting an upstrema release
> >  - more LTS branches to maintain
> >  - more upgrade paths to consider
> >
> > Other options we should consider?  Other thoughts?
> >
> > Thanks!
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org <javascript:;>
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Liebe Grüße,
> Christian Theune
>
> --
> Christian Theune · ct-4ImjS3E6Ur3YxmODkEiZ1Q@public.gmane.org <javascript:;> · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> Zagrodnick
>
>

[-- Attachment #1.2: Type: text/html, Size: 7673 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Alexander Litvak]

As a user,  I woul like to add,  I would like to see a real 2 year support for LTS releases.  Hammer releases were sketchy at best in 2017.  When luminous was released The outstanding bugs were auto closed, good buy and good readance.


 Also the decision to drop certain OS support created a barrier to upgrade and looking at jewel and luminous upgrade path where you cannot easily go back after upgrade is completed doesn't add the confidence.


So making upgrades less radical may help production support to be more consistent and update process less dangerous.


I would say 9 month is a good reference point but for me  it is ready when it is really ready and tested.


Keeping development release may be better for devs and early adopters.  I don't believe production admins would go for intermediate one's as they being released now. 


This is only MHO and may be wrong.





⁣

On Sep 9, 2017, 15:32, at 15:32, Christian Theune <ct@flyingcircus.io> wrote:
>Hi,
>
>have been using Ceph for multiple years now. It’s unclear to me which
>of your options fits best, but here are my preferences:
>
>* Updates are risky in a way that we tend to rather not do them every
>year. Also, having seen jewel, we’ve been well off to avoid two
>major issues what would have bitten us and will upgrade from hammer in
>the next month or so.
>
>* Non-production releases are of not much value to me, as I have to
>keep our dev/staging/prod clusters in sync to work on our stuff.
>As you can never downgrade, there’s no value in it for me to try
>non-production releases (without frying dev for everyone).
>
>* I’d prefer stability over new features. *Specifically* that new
>features can be properly recombined with existing features (and each
>other) without leading to surprises. (E.g. cache tiering breaking with
>snapshots and then no way going back and a general notion of
>  “that combination wasn’t really well tested).
>
>* I’d prefer versions that I have to be maintained for
>production-critical issues maybe 2 years, so I can have some time after
>a new
>production release that overlaps with the new production release
>receiving important bug fixes until I switch.
>
>Maybe this is close to what your "Drop the odd releases, and aim for a
>~9 month cadence.” would say. Waiting for a feature for a year is a
>pain, but my personal goal for Ceph is that it first has to work
>properly, meaning: not loose your data, not "stopping the show”, and
>not drawing you into a corner you can’t get out.
>
>That’s my perspective as a user. As a fellow developer I feel your pain
>about wanting to release faster and reducing maintenance load, so
>thanks for asking!
>
>Hope this helps,
>Christian
>
>> On Sep 6, 2017, at 5:23 PM, Sage Weil <sweil@redhat.com> wrote:
>> 
>> Hi everyone,
>> 
>> Traditionally, we have done a major named "stable" release twice a
>year,
>> and every other such release has been an "LTS" release, with fixes
>> backported for 1-2 years.
>> 
>> With kraken and luminous we missed our schedule by a lot: instead of
>> releasing in October and April we released in January and August.
>> 
>> A few observations:
>> 
>> - Not a lot of people seem to run the "odd" releases (e.g.,
>infernalis,
>> kraken).  This limits the value of actually making them.  It also
>means
>> that those who *do* run them are running riskier code (fewer users ->
>more
>> bugs).
>> 
>> - The more recent requirement that upgrading clusters must make a
>stop at
>> each LTS (e.g., hammer -> luminous not supported, must go hammer ->
>jewel
>> -> lumninous) has been hugely helpful on the development side by
>reducing
>> the amount of cross-version compatibility code to maintain and
>reducing
>> the number of upgrade combinations to test.
>> 
>> - When we try to do a time-based "train" release cadence, there
>always
>> seems to be some "must-have" thing that delays the release a bit. 
>This
>> doesn't happen as much with the odd releases, but it definitely
>happens
>> with the LTS releases.  When the next LTS is a year away, it is hard
>to
>> suck it up and wait that long.
>> 
>> A couple of options:
>> 
>> * Keep even/odd pattern, and continue being flexible with release
>dates
>> 
>>  + flexible
>>  - unpredictable
>>  - odd releases of dubious value
>> 
>> * Keep even/odd pattern, but force a 'train' model with a more
>regular
>> cadence
>> 
>>  + predictable schedule
>>  - some features will miss the target and be delayed a year
>> 
>> * Drop the odd releases but change nothing else (i.e., 12-month
>release
>> cadence)
>> 
>>  + eliminate the confusing odd releases with dubious value
>> 
>> * Drop the odd releases, and aim for a ~9 month cadence. This splits
>the
>> difference between the current even/odd pattern we've been doing.
>> 
>>  + eliminate the confusing odd releases with dubious value
>>  + waiting for the next release isn't quite as bad
>>  - required upgrades every 9 months instead of ever 12 months
>> 
>> * Drop the odd releases, but relax the "must upgrade through every
>LTS" to
>> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous
>->
>> nautilus).  Shorten release cycle (~6-9 months).
>> 
>>  + more flexibility for users
>>  + downstreams have greater choice in adopting an upstrema release
>>  - more LTS branches to maintain
>>  - more upgrade paths to consider
>> 
>> Other options we should consider?  Other thoughts?
>> 
>> Thanks!
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>Liebe Grüße,
>Christian Theune
>
>--
>Christian Theune · ct@flyingcircus.io · +49 345 219401 0
>Flying Circus Internet Operations GmbH · http://flyingcircus.io
>Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
>HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
>Zagrodnick
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [Ceph-maintainers] Ceph release cadence

[Yehuda Sadeh-Weinraub]

I'm not a huge fan of train releases, as they tend to never quite make
it on time and it always feels a bit artificial timeline anyway. OTOH,
I do see and understand the need of a predictable schedule with a
roadmap attached to it. There are many that need to have at least a
vague idea on what we're going to ship and when, so that they can plan
ahead. We need it ourselves, as sometimes the schedule can dictate the
engineering decision that we're going to make.
On the other hand, I think that working towards a release that come
out after 9 or 12 months is a bit too long. This is a recipe for more
delays as the penalty for missing a feature is painful. Maybe we can
consider returning to shorter iterations for *dev* releases. These are
checkpoints that need to happen after a short period (2-3 weeks),
where we end up with a minimally tested release that passes some smoke
test. Features are incrementally added to the dev release. The idea
behind a short term dev release is that it minimizes the window where
master is completely unusable, thus reduces the time to stabilization.
Then it's easier to enforce a train schedule if we want to. It might
be easier to let go of a feature that doesn't make it, as it will be
there soon, and maybe if really needed we (or the downstream
maintainer) can make the decision to backport it. This makes me think
that we could revisit the our backport policy/procedure/tooling, so
that we can do it in a sane and safe way when needed and possible.

Yehuda

On Fri, Sep 8, 2017 at 7:59 PM, Gregory Farnum <gfarnum-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> I think I'm the resident train release advocate so I'm sure my
> advocating that model will surprise nobody. I'm not sure I'd go all
> the way to Lars' multi-release maintenance model (although it's
> definitely something I'm interested in), but there are two big reasons
> I wish we were on a train with more frequent real releases:
>
> 1) It reduces the cost of features missing a release. Right now if
> something misses an LTS release, that's it for a year. And nobody
> likes releasing an LTS without a bunch of big new features, so each
> LTS is later than the one before as we scramble to get features merged
> in.
>
> ...and then we deal with the fact that we scrambled to get a bunch of
> features merged in and they weren't quite baked. (Luminous so far
> seems to have gone much better in this regard! Hurray! But I think
> that has a lot to do with our feature-release-scramble this year being
> mostly peripheral stuff around user interfaces that got tacked on
> about the time we'd initially planned the release to occur.)
>
> 2) Train releases increase predictability for downstreams, partners,
> and users around when releases will happen. Right now, the release
> process and schedule is entirely opaque to anybody who's not involved
> in every single upstream meeting we have; and it's unpredictable even
> to those who are. That makes things difficult, as Xiaoxi said.
>
> There are other peripheral but serious benefits I'd expect to see from
> fully-validated train releases as well. It would be *awesome* to have
> more frequent known-stable points to do new development against. If
> you're an external developer and you want a new feature, you have to
> either keep it rebased against a fast-changing master branch, or you
> need to settle for writing it against a long-out-of-date LTS and then
> forward-porting it for merge. If you're an FS developer writing a very
> small new OSD feature and you try to validate it against RADOS, you've
> no idea if bugs that pop up and look random are because you really did
> something wrong or if there's currently an intermittent issue in RADOS
> master. I would have *loved* to be able to maintain CephFS integration
> branches for features that didn't touch RADOS and were built on top of
> the latest release instead of master, but it was utterly infeasible
> because there were too many missing features with the long delays.
>
> On Fri, Sep 8, 2017 at 9:16 AM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> I'm going to pick on Lars a bit here...
>>
>> On Thu, 7 Sep 2017, Lars Marowsky-Bree wrote:
>>> On 2017-09-06T15:23:34, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>> > Other options we should consider?  Other thoughts?
>>>
>>> With about 20-odd years in software development, I've become a big
>>> believer in schedule-driven releases. If it's feature-based, you never
>>> know when they'll get done.
>>>
>>> If the schedule intervals are too long though, the urge to press too
>>> much in (so as not to miss the next merge window) is just too high,
>>> meaning the train gets derailed. (Which cascades into the future,
>>> because the next time the pressure will be even higher based on the
>>> previous experience.) This requires strictness.
>>>
>>> We've had a few Linux kernel releases that were effectively feature
>>> driven and never quite made it. 1.3.x? 1.5.x? My memory is bad, but they
>>> were a disaster than eventually led Linus to evolve to the current
>>> model.
>>>
>>> That serves them really well, and I believe it might be worth
>>> considering for us.
>>
>> This model is very appealing.  The problem with it that I see is that the
>> upstream kernel community doesn't really do stable releases.  Mainline
>> developers are just getting their stuff upstream, and entire separate
>> organizations and teams are doing the stable distro kernels.  (There are
>> upstream stable kernels too, yes, but they don't get much testing AFAICS
>> and I'm not sure who uses them.)
>>
>> More importantly, upgrade and on-disk format issues are present for almost
>> everything that we change in Ceph.  Those things rarely come up for the
>> kernel.  Even the local file systems (a small piece of the kernel) have
>> comparatively fewer format changes that we do, it seems.
>>
>> These make the upgrade testing a huge concern and burden for the
>> Ceph development community.
>>
>>> I'd try to move away from the major milestones. Features get integrated
>>> into the next schedule-driven release when they deemed ready and stable;
>>> when they're not, not a big deal, the next one is coming up "soonish".
>>>
>>> (This effectively decouples feature development slightly from the
>>> release schedule.)
>>>
>>> We could even go for "a release every 3 months, sharp", merge window for
>>> the first month, stabilization the second, release clean up the third,
>>> ship.
>>>
>>> Interoperability hacks for the cluster/server side are maintained for 2
>>> years, and then dropped.  Sharp. (Speaking as one of those folks
>>> affected, we should not burden the community with this.) Client interop
>>> is a different story, a bit.
>>>
>>> Basically, effectively edging towards continuous integration of features
>>> and bugfixes both. Nobody has to wait for anything much, and can
>>> schedule reasonably independently.
>>
>> If I read between the lines a bit here, but this sounds like is:
>>
>>  - keep the frequently major releases (but possibly shorten the 6mo
>>    cadence)
>>  - do backports for all of them, not just the even ones
>>  - test upgrades between all of them within a 2 year horizon, instead
>>    of just the last major one
>>
>> Is that accurate?
>>
>> Unfortunately it sounds to me like that would significantly increase the
>> maintenance burden (double it even?) and slow development down.  The user
>> base will also end up fragmented across a broader range of versions, which
>> means we'll see a wider variety of bugs and each release will be less
>> stable.
>>
>> This is full of trade-offs... time we spend backporting or testing
>> upgrades is time we don't spend fixing bugs or improving performance or
>> adding features.
>
> Not all newly-allocated effort for doing maintenance and testing
> necessarily means reducing effort available for new feature
> development. Sometimes it just makes that development easier and more
> efficient!    I think we'd find that more tested and stable releases
> would spread joy and stability throughout our development process and
> make life much easier on prospective contributors.
> -Greg
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Nathan Cutler]

 From a backporter's perspective, the appealing options are the ones 
that reduce the number of stable releases in maintenance at any 
particular time.

In the current practice, there are always at least two LTS releases, and 
sometimes a non-LTS release as well, that are "live" and supposed to be 
getting backports. For example:

* when kraken was released, hammer and jewel were "live LTS" and kraken 
was "live non-LTS", for a total of three live releases.

* when luminous was released, hammer and kraken were declared EoL and 
there are now only two "live LTS" releases and no "live non-LTS".

During the period when there are three live releases, almost every 
bugfix seen as warranting a backport gets marked for backport to the two 
most recent stable releases. (For example, from January to August 2017 
with very few exceptions tracker issues got marked "Backport: jewel, 
kraken", not just "Backport: jewel".) This, of course, doubled the 
backporting workload, simply because if a bug is severe enough to 
backport to the most recent non-LTS release, it must be severe enough to 
be backported to the most recent LTS release as well. Unfortunately, 
there aren't enough developers working on backports to cover this double 
workload, so in practice the non-LTS release gets insufficient attention.

A "train" model could lower this backporting workload if it was 
accompanied by a declaration that the n-1 release gets backports for all 
important bugfixes and n-2 gets backports for critical bugfixes only 
(and n-3 gets EOLed).

Nathan

Re: Ceph release cadence

[John Spray]

On Wed, Sep 6, 2017 at 4:23 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months

This is my preferred option (second choice would be the next one up,
i.e. same thing but annually).

Our focus should be on delivering solid stuff, but not necessarily
bending over backwards to enable people to run old stuff.  Our
commitment to releases should be that there are either fixes for that
release, or a newer (better) release to upgrade to.  Either way there
is a solution on offer (and any user/vendor who wants to independently
maintain other stable branches is free to do so).

John

> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph release cadence

[Ben Hines]

[-- Attachment #1.1: Type: text/plain, Size: 4688 bytes --]

We have generally been running the latest non LTS 'stable' release since my
cluster is slightly less mission critical than others, and there were
important features to us added in both Infernalis and Kraken. But i really
only care about RGW. If the rgw component could be split out of ceph into a
plugin and independently updated, it'd be awesome for us.

A minor bugfix to radosgw shouldn't be blocked by issues with RBD, for
example. I don't care at all.
Could have packages like:

ceph-core
ceph-radosgw
ceph-rbd ...
ceph-mgr..

Might increase the testing workload, but automation should take care of
that...

ceph-mgr is also similar. Minor (or even major) updates to the GUI
dashboard shouldn't be blocked rolling out to users because we're waiting
on a new RBD feature or critical RGW fix.

radosgw and mgr are really 'clients', after all.

-Ben

On Mon, Sep 11, 2017 at 3:30 PM, John Spray <jspray-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Wed, Sep 6, 2017 at 4:23 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >   + flexible
> >   - unpredictable
> >   - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >   + predictable schedule
> >   - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >   + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >   + eliminate the confusing odd releases with dubious value
> >   + waiting for the next release isn't quite as bad
> >   - required upgrades every 9 months instead of ever 12 months
>
> This is my preferred option (second choice would be the next one up,
> i.e. same thing but annually).
>
> Our focus should be on delivering solid stuff, but not necessarily
> bending over backwards to enable people to run old stuff.  Our
> commitment to releases should be that there are either fixes for that
> release, or a newer (better) release to upgrade to.  Either way there
> is a solution on offer (and any user/vendor who wants to independently
> maintain other stable branches is free to do so).
>
> John
>
> > * Drop the odd releases, but relax the "must upgrade through every LTS"
> to
> > allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> > nautilus).  Shorten release cycle (~6-9 months).
> >
> >   + more flexibility for users
> >   + downstreams have greater choice in adopting an upstrema release
> >   - more LTS branches to maintain
> >   - more upgrade paths to consider
> >
> > Other options we should consider?  Other thoughts?
> >
> > Thanks!
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 6485 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[John Spray]

(this is a little off topic, but it's a fair point that deserves a
response, so here goes)

On Mon, Sep 11, 2017 at 11:38 PM, Ben Hines <bhines@gmail.com> wrote:
> We have generally been running the latest non LTS 'stable' release since my
> cluster is slightly less mission critical than others, and there were
> important features to us added in both Infernalis and Kraken. But i really
> only care about RGW. If the rgw component could be split out of ceph into a
> plugin and independently updated, it'd be awesome for us.
>
> A minor bugfix to radosgw shouldn't be blocked by issues with RBD, for
> example. I don't care at all.
> Could have packages like:
>
> ceph-core
> ceph-radosgw
> ceph-rbd ...
> ceph-mgr..
>
> Might increase the testing workload, but automation should take care of
> that...

I think you might be underestimating the level of effort and expense
involved in the Ceph automated testing!

> ceph-mgr is also similar. Minor (or even major) updates to the GUI dashboard
> shouldn't be blocked rolling out to users because we're waiting on a new RBD
> feature or critical RGW fix.

In general, the solution to this is not to spawn a load of extra work
by breaking things out into separately versioned entities (and dealing
with resulting inter-version compatibility issues, which is a HUGE
overhead), but rather to ensure that the main Ceph are sufficiently
frequent that it's not too much of an issue.  Folks always have the
option of building their own packages, or downloading pre-release
stuff if they have a super-urgent need for a particular bugfix.

Taking the dashboard as an example -- it's under active development,
as is the ceph-mgr module interface it consumes.  In the current
model, I can add a dashboard feature and its mgr interface dependency
in one step.  There is no stable/static/versioned interface that
isolates these components, and creating one would itself be a
significant amount of work.  The monolithic versioning is not without
its downsides, but they are a conscious cost/benefit decision rather
than an oversight.

The suggestion about RBD and RGW specifically is more reasonable (as
they are librados clients and librados is a versioned interface), but
in practice they do have hooks outside of librados (OSD object
classes, ceph-mgr functionality), so it's far from a no-brainer.
Ultimately up to the folks who develop those bits though.

> radosgw and mgr are really 'clients', after all.

mgr is a required part of RADOS as of luminous (some key functionality
lives there now), it's not really a client in the way that RBD/RGW
are.  It has some client-ish stuff in it, but the mon is also a client
to the mgr for some functionality, and some CLI commands go straight
to the mgr.

John

>
> -Ben
>
> On Mon, Sep 11, 2017 at 3:30 PM, John Spray <jspray@redhat.com> wrote:
>>
>> On Wed, Sep 6, 2017 at 4:23 PM, Sage Weil <sweil@redhat.com> wrote:
>> > Hi everyone,
>> >
>> > Traditionally, we have done a major named "stable" release twice a year,
>> > and every other such release has been an "LTS" release, with fixes
>> > backported for 1-2 years.
>> >
>> > With kraken and luminous we missed our schedule by a lot: instead of
>> > releasing in October and April we released in January and August.
>> >
>> > A few observations:
>> >
>> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
>> > kraken).  This limits the value of actually making them.  It also means
>> > that those who *do* run them are running riskier code (fewer users ->
>> > more
>> > bugs).
>> >
>> > - The more recent requirement that upgrading clusters must make a stop
>> > at
>> > each LTS (e.g., hammer -> luminous not supported, must go hammer ->
>> > jewel
>> > -> lumninous) has been hugely helpful on the development side by
>> > reducing
>> > the amount of cross-version compatibility code to maintain and reducing
>> > the number of upgrade combinations to test.
>> >
>> > - When we try to do a time-based "train" release cadence, there always
>> > seems to be some "must-have" thing that delays the release a bit.  This
>> > doesn't happen as much with the odd releases, but it definitely happens
>> > with the LTS releases.  When the next LTS is a year away, it is hard to
>> > suck it up and wait that long.
>> >
>> > A couple of options:
>> >
>> > * Keep even/odd pattern, and continue being flexible with release dates
>> >
>> >   + flexible
>> >   - unpredictable
>> >   - odd releases of dubious value
>> >
>> > * Keep even/odd pattern, but force a 'train' model with a more regular
>> > cadence
>> >
>> >   + predictable schedule
>> >   - some features will miss the target and be delayed a year
>> >
>> > * Drop the odd releases but change nothing else (i.e., 12-month release
>> > cadence)
>> >
>> >   + eliminate the confusing odd releases with dubious value
>> >
>> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
>> > difference between the current even/odd pattern we've been doing.
>> >
>> >   + eliminate the confusing odd releases with dubious value
>> >   + waiting for the next release isn't quite as bad
>> >   - required upgrades every 9 months instead of ever 12 months
>>
>> This is my preferred option (second choice would be the next one up,
>> i.e. same thing but annually).
>>
>> Our focus should be on delivering solid stuff, but not necessarily
>> bending over backwards to enable people to run old stuff.  Our
>> commitment to releases should be that there are either fixes for that
>> release, or a newer (better) release to upgrade to.  Either way there
>> is a solution on offer (and any user/vendor who wants to independently
>> maintain other stable branches is free to do so).
>>
>> John
>>
>> > * Drop the odd releases, but relax the "must upgrade through every LTS"
>> > to
>> > allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
>> > nautilus).  Shorten release cycle (~6-9 months).
>> >
>> >   + more flexibility for users
>> >   + downstreams have greater choice in adopting an upstrema release
>> >   - more LTS branches to maintain
>> >   - more upgrade paths to consider
>> >
>> > Other options we should consider?  Other thoughts?
>> >
>> > Thanks!
>> > sage
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>

Re: Ceph release cadence

[Blair Bethwaite]

[-- Attachment #1.1: Type: text/plain, Size: 1751 bytes --]

On 7 September 2017 at 01:23, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
> + eliminate the confusing odd releases with dubious value
> + waiting for the next release isn't quite as bad
> - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus). Shorten release cycle (~6-9 months).
>
> + more flexibility for users
> + downstreams have greater choice in adopting an upstrema release
> - more LTS branches to maintain
> - more upgrade paths to consider
>
> Other options we should consider? Other thoughts?

We currently use both upstream and distro (RHCS) versions on different
clusters. Downstream releases are still free to apply their own models.

I like the idea of a predictable (and more regular) train model (and I'd
point out that trains can run a little late - there just needs to be some
process around that.

If you were to keep the even/odd release model then I'd suggest considering
a 4 monthly train cadence on these. If the even/odd model is dropped (which
would be my preference) then I like the sound of ~9 monthly releases
splitting the difference between today's model. OpenStack does 6 monthly
cycles, and I think the data shows this is too frequent for most operators
(of course OpenStack is much more complex an multifaceted as well) - many
deployments are well behind the stable deprecation schedule and it's the
norm to hear people talking about doing "skip upgrades".

-- 
Cheers,
~Blairo

[-- Attachment #1.2: Type: text/html, Size: 2063 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Blair Bethwaite]

(Apologies if this is a double post - I think my phone turned it into
HTML and so bounced from ceph-devel)...

We currently use both upstream and distro (RHCS) versions on different
clusters. Downstream releases are still free to apply their own
models.

I like the idea of a predictable (and more regular) train model (and
I'd point out that trains can run a little late - there just needs to
be some process around that.

If you were to keep the even/odd release model then I'd suggest
considering a 4 monthly train cadence on these. If the even/odd model
is dropped (which would be my preference) then I like the sound of ~9
monthly releases splitting the difference between today's model.
OpenStack does 6 monthly cycles, and I think the data shows this is
too frequent for most operators (of course OpenStack is much more
complex and multifaceted as well) - many deployments are well behind
the stable deprecation schedule and it's the norm to hear people
talking about doing "skip upgrades".

On 7 September 2017 at 01:23, Sage Weil <sweil@redhat.com> wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
~Blairo

Re: Ceph release cadence

[Sage Weil]

Here is a concrete proposal for everyone to summarily shoot down (or 
heartily endorse, depending on how your friday is going):

- 9 month cycle
- enforce a predictable release schedule with a freeze date and 
  a release date.  (The actual .0 release of course depends on no blocker 
  bugs being open; not sure how zealous 'train' style projects do 
  this.)
- no more even/odd pattern; all stable releases are created equal.
- support upgrades from up to 3 releases back.

This shortens the cycle a bit to relieve the "this feature must go in" 
stress, without making it so short as to make the release pointless (e.g., 
infernalis, kraken).  (I also think that the feature pressure is much 
lower now than it has been in the past.)

This creates more work for the developers because there are more upgrade 
paths to consider: we no longer have strict "choke points" (like all 
upgrades must go through luminous).  We could reserve the option to pick 
specific choke point releases in the future, perhaps taking care to make 
sure these are the releases that go into downstream distros.  We'll need 
to be more systematic about the upgrade testing.

Somewhat separately, several people expressed concern about having stable 
releases to develop against.  This is somewhat orthogonal to what users 
need.  To that end, we can do a dev checkpoint every 1 or 2 months 
(preferences?), where we fork a 'next' branch and stabilize all of the 
tests before moving on.  This is good practice anyway to avoid 
accumulating low-frequency failures in the test suite that have to be 
squashed at the end.

The timing of the 9 mo cycle could be chosen to strategically avoid 
holiday periods.  E.g, Feb 1, May 1, Aug 1, Nov 1 (hard to miss both Dec 
and Aug, unfortunately).

Thoughts?  Counter-proposals?

sage

Re: Ceph release cadence

[Gregory Farnum]

On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil <sage@newdream.net> wrote:
> Here is a concrete proposal for everyone to summarily shoot down (or
> heartily endorse, depending on how your friday is going):
>
> - 9 month cycle
> - enforce a predictable release schedule with a freeze date and
>   a release date.  (The actual .0 release of course depends on no blocker
>   bugs being open; not sure how zealous 'train' style projects do
>   this.)

Train projects basically commit to a feature freeze enough in advance
of the expected release date that it's feasible, and don't let people
fake it by rushing in stuff they "finished" the day before. I'm not
sure if every-9-month LTSes will be more conducive to that or not — if
we do scheduled releases, we still fundamentally need to be able to
say "nope, that feature we've been saying for 9 months we hope to have
out in this LTS won't make it until the next one". And we seem pretty
bad at that.

> - no more even/odd pattern; all stable releases are created equal.
> - support upgrades from up to 3 releases back.
>
> This shortens the cycle a bit to relieve the "this feature must go in"
> stress, without making it so short as to make the release pointless (e.g.,
> infernalis, kraken).  (I also think that the feature pressure is much
> lower now than it has been in the past.)
>
> This creates more work for the developers because there are more upgrade
> paths to consider: we no longer have strict "choke points" (like all
> upgrades must go through luminous).  We could reserve the option to pick
> specific choke point releases in the future, perhaps taking care to make
> sure these are the releases that go into downstream distros.  We'll need
> to be more systematic about the upgrade testing.

This sounds generally good to me — we did multiple-release upgrades
for a long time, and stuff is probably more complicated now but I
don't think it will actually be that big a deal.

3 releases back might be a bit much though — that's 27 months! (For
luminous, the beginning of 2015. Hammer.)

> Somewhat separately, several people expressed concern about having stable
> releases to develop against.  This is somewhat orthogonal to what users
> need.  To that end, we can do a dev checkpoint every 1 or 2 months
> (preferences?), where we fork a 'next' branch and stabilize all of the
> tests before moving on.  This is good practice anyway to avoid
> accumulating low-frequency failures in the test suite that have to be
> squashed at the end.

So this sounds like a fine idea to me, but how do we distinguish this
from the intermediate stable releases?

By which I mean, are we *really* going to do a stabilization branch
that will never get seen by users? What kind of testing and bug fixing
are we going to commit to doing against it, and how do we balance that
effort with feature work?

It seems like the same conflict we have now, only since the dev
checkpoints are less important they'll lose more often. Then we'll end
up having 9 months of scheduled work to debug for a user release
instead of 5 months that slipped to 7 or 8...

-Greg
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4147 bytes --]

On Fri, 22 Sep 2017, Gregory Farnum wrote:
> On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil <sage@newdream.net> wrote:
> > Here is a concrete proposal for everyone to summarily shoot down (or
> > heartily endorse, depending on how your friday is going):
> >
> > - 9 month cycle
> > - enforce a predictable release schedule with a freeze date and
> >   a release date.  (The actual .0 release of course depends on no blocker
> >   bugs being open; not sure how zealous 'train' style projects do
> >   this.)
> 
> Train projects basically commit to a feature freeze enough in advance
> of the expected release date that it's feasible, and don't let people
> fake it by rushing in stuff they "finished" the day before. I'm not
> sure if every-9-month LTSes will be more conducive to that or not — if
> we do scheduled releases, we still fundamentally need to be able to
> say "nope, that feature we've been saying for 9 months we hope to have
> out in this LTS won't make it until the next one". And we seem pretty
> bad at that.

I'll be the first to say I'm no small part of the "we" there.  But I'm 
also suggesting that's not a reason not to try to do better.  As I 
said I think this will be easier than in the past because we don't 
have as many headline features we're trying to wedge in.

In any case, is there an alternative way to get to the much-desired 
regular cadence?

> > - no more even/odd pattern; all stable releases are created equal.
> > - support upgrades from up to 3 releases back.
> >
> > This shortens the cycle a bit to relieve the "this feature must go in"
> > stress, without making it so short as to make the release pointless (e.g.,
> > infernalis, kraken).  (I also think that the feature pressure is much
> > lower now than it has been in the past.)
> >
> > This creates more work for the developers because there are more upgrade
> > paths to consider: we no longer have strict "choke points" (like all
> > upgrades must go through luminous).  We could reserve the option to pick
> > specific choke point releases in the future, perhaps taking care to make
> > sure these are the releases that go into downstream distros.  We'll need
> > to be more systematic about the upgrade testing.
> 
> This sounds generally good to me — we did multiple-release upgrades
> for a long time, and stuff is probably more complicated now but I
> don't think it will actually be that big a deal.
> 
> 3 releases back might be a bit much though — that's 27 months! (For
> luminous, the beginning of 2015. Hammer.)

I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot 
of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe 
it's best to start with that, though?  It's still an improvement over the 
current ~12 months.

> > Somewhat separately, several people expressed concern about having stable
> > releases to develop against.  This is somewhat orthogonal to what users
> > need.  To that end, we can do a dev checkpoint every 1 or 2 months
> > (preferences?), where we fork a 'next' branch and stabilize all of the
> > tests before moving on.  This is good practice anyway to avoid
> > accumulating low-frequency failures in the test suite that have to be
> > squashed at the end.
> 
> So this sounds like a fine idea to me, but how do we distinguish this
> from the intermediate stable releases?
> 
> By which I mean, are we *really* going to do a stabilization branch
> that will never get seen by users? What kind of testing and bug fixing
> are we going to commit to doing against it, and how do we balance that
> effort with feature work?
> 
> It seems like the same conflict we have now, only since the dev
> checkpoints are less important they'll lose more often. Then we'll end
> up having 9 months of scheduled work to debug for a user release
> instead of 5 months that slipped to 7 or 8...

What if we frame this stabilization period in terms of stability of the 
test suite.  That gives us something concrete to aim for, lets us move on 
when we reach some threshold, and aligns perfectly with the thing that 
makes it hard to safely land new code (noisy test results)...

sage

Re: [ceph-users] Ceph release cadence

[Blair Bethwaite]

On 23 September 2017 at 11:58, Sage Weil <sage@newdream.net> wrote:
> I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
> of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
> it's best to start with that, though?  It's still an improvement over the
> current ~12 months.

FWIW, I'd be happy with this if it means higher confidence in an
18-monthly skip-upgrade being a feasible operational approach.

-- 
Cheers,
~Blairo

Re: Ceph release cadence

[Brady Deetz]

[-- Attachment #1.1: Type: text/plain, Size: 7645 bytes --]

I'll be first to admit that most of my comments are anecdotal. But, I
suspect when it comes to storage many of us don't require a lot to get
scared back into our dark corners. In short it seems that the dev team
should get better at selecting features and delivering on the existing
scheduled cadence before shortening it. To me, the odd releases represent
feature previews for the next even release. If that's a fair way to look at
them, they could play a very important role in the stability of the even
release.

On Sep 22, 2017 8:59 PM, "Sage Weil" <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:

On Fri, 22 Sep 2017, Gregory Farnum wrote:
> On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> > Here is a concrete proposal for everyone to summarily shoot down (or
> > heartily endorse, depending on how your friday is going):
> >
> > - 9 month cycle
> > - enforce a predictable release schedule with a freeze date and
> >   a release date.  (The actual .0 release of course depends on no
blocker
> >   bugs being open; not sure how zealous 'train' style projects do
> >   this.)
>
> Train projects basically commit to a feature freeze enough in advance
> of the expected release date that it's feasible, and don't let people
> fake it by rushing in stuff they "finished" the day before. I'm not
> sure if every-9-month LTSes will be more conducive to that or not — if
> we do scheduled releases, we still fundamentally need to be able to
> say "nope, that feature we've been saying for 9 months we hope to have
> out in this LTS won't make it until the next one". And we seem pretty
> bad at that.

I'll be the first to say I'm no small part of the "we" there.  But I'm
also suggesting that's not a reason not to try to do better.  As I
said I think this will be easier than in the past because we don't
have as many headline features we're trying to wedge in.


That's excellent as long as it actually happens. Otherwise the collective
you may end up pushing worse code on a 9mo cycle than the current
theoretical 12mo cycle that is delayed when necessary. We all know that
software development never happens on time or on budget.


In any case, is there an alternative way to get to the much-desired
regular cadence?

> > - no more even/odd pattern; all stable releases are created equal.
> > - support upgrades from up to 3 releases back.
> >
> > This shortens the cycle a bit to relieve the "this feature must go in"
> > stress, without making it so short as to make the release pointless
(e.g.,
> > infernalis, kraken).  (I also think that the feature pressure is much
> > lower now than it has been in the past.)
> >
> > This creates more work for the developers because there are more upgrade
> > paths to consider: we no longer have strict "choke points" (like all
> > upgrades must go through luminous).  We could reserve the option to pick
> > specific choke point releases in the future, perhaps taking care to make
> > sure these are the releases that go into downstream distros.  We'll need
> > to be more systematic about the upgrade testing.
>
> This sounds generally good to me — we did multiple-release upgrades
> for a long time, and stuff is probably more complicated now but I
> don't think it will actually be that big a deal.
>
> 3 releases back might be a bit much though — that's 27 months! (For
> luminous, the beginning of 2015. Hammer.)

I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
it's best to start with that, though?  It's still an improvement over the
current ~12 months.


A lot of vulnerabilities and bugs can come out in one year. As such, I
upgrade anything in my environment, at minimum, once a year. The "if it
ain't broke don't fix it" mentality is usually more dangerous than an
upgrade between minor releases. But... I will say that as my Ceph
environment grows, upgrades become increasingly difficult to manage and
anxiety increases with every node I add to my growing 2PB cluster.


> > Somewhat separately, several people expressed concern about having
stable
> > releases to develop against.  This is somewhat orthogonal to what users
> > need.  To that end, we can do a dev checkpoint every 1 or 2 months
> > (preferences?), where we fork a 'next' branch and stabilize all of the
> > tests before moving on.  This is good practice anyway to avoid
> > accumulating low-frequency failures in the test suite that have to be
> > squashed at the end.
>
> So this sounds like a fine idea to me, but how do we distinguish this
> from the intermediate stable releases?
>
> By which I mean, are we *really* going to do a stabilization branch
> that will never get seen by users? What kind of testing and bug fixing
> are we going to commit to doing against it, and how do we balance that
> effort with feature work?
>
> It seems like the same conflict we have now, only since the dev
> checkpoints are less important they'll lose more often. Then we'll end
> up having 9 months of scheduled work to debug for a user release
> instead of 5 months that slipped to 7 or 8...

What if we frame this stabilization period in terms of stability of the
test suite.  That gives us something concrete to aim for, lets us move on
when we reach some threshold, and aligns perfectly with the thing that
makes it hard to safely land new code (noisy test results)...


All of the text below can be summarized as me saying a 9mo release cycle is
reasonable as long as a parallel focus of delivering rock solid code is as
important as meeting an arbitrary deadline. The fact that uncertainty plays
a significant role in people holding back on upgrades from hammer should
raise a flag.

Time for a story from a cephfs jewel operator:
About a year ago, I watched a web cast you did demonstrating the dev
process for ceph where you patched a bug, ran the test suite, and packaged
the patch live. I was relieved to see that the Ceph process was organized
and not a complete wild-west after I had just deployed a little over 1PB of
cephfs.

Not long after that webcast, I very easily triggered a bug in Jewel that
crashed both of my MDS servers (active/standby) because I made a very
simple mistake by deleting a pool defined in the ceph.dir.layout xattr of a
cephfs directory. A stupid mistake, yes. But a very very easy mistake to
make. John Spray managed to track down the bug; but it still resulted in a
12 hour outage and there was absolutely nothing I could have done to
resolve the issue other than reverse time or patch the MDS code myself
(unlikely to happen quickly). I'm forever in debt to John for his work that
day.

It's precisely that experience that emotionally makes me want to wait for
an upgrade to luminous. I desperately want Bluestore, but I also like my
data and don't feel like restoring my entire cluster from tape because of a
bug some other poor soul could trigger before me.

As such, features are excellent; I'd do many things to have cephfs ec-coded
direct writes arrive on my cluster tomorrow. But, I'm willing to wait in
the name of stability and confidence. As such, bumping the release schedule
from 12 to 9 months seems like it will just rush devs to "finalize" code
for the major releases.



sage
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[-- Attachment #1.2: Type: text/html, Size: 10520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Ceph release cadence

[Joao Eduardo Luis]

I am happy with this branch of the thread!

I'm guessing this would start post-Mimic though, if no one objects and 
if we want to target a March release?

   -Joao

On 09/23/2017 02:58 AM, Sage Weil wrote:
> On Fri, 22 Sep 2017, Gregory Farnum wrote:
>> On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil <sage@newdream.net> wrote:
>>> Here is a concrete proposal for everyone to summarily shoot down (or
>>> heartily endorse, depending on how your friday is going):
>>>
>>> - 9 month cycle
>>> - enforce a predictable release schedule with a freeze date and
>>>    a release date.  (The actual .0 release of course depends on no blocker
>>>    bugs being open; not sure how zealous 'train' style projects do
>>>    this.)
>>
>> Train projects basically commit to a feature freeze enough in advance
>> of the expected release date that it's feasible, and don't let people
>> fake it by rushing in stuff they "finished" the day before. I'm not
>> sure if every-9-month LTSes will be more conducive to that or not — if
>> we do scheduled releases, we still fundamentally need to be able to
>> say "nope, that feature we've been saying for 9 months we hope to have
>> out in this LTS won't make it until the next one". And we seem pretty
>> bad at that.
> 
> I'll be the first to say I'm no small part of the "we" there.  But I'm
> also suggesting that's not a reason not to try to do better.  As I
> said I think this will be easier than in the past because we don't
> have as many headline features we're trying to wedge in.
> 
> In any case, is there an alternative way to get to the much-desired
> regular cadence?
> 
>>> - no more even/odd pattern; all stable releases are created equal.
>>> - support upgrades from up to 3 releases back.
>>>
>>> This shortens the cycle a bit to relieve the "this feature must go in"
>>> stress, without making it so short as to make the release pointless (e.g.,
>>> infernalis, kraken).  (I also think that the feature pressure is much
>>> lower now than it has been in the past.)
>>>
>>> This creates more work for the developers because there are more upgrade
>>> paths to consider: we no longer have strict "choke points" (like all
>>> upgrades must go through luminous).  We could reserve the option to pick
>>> specific choke point releases in the future, perhaps taking care to make
>>> sure these are the releases that go into downstream distros.  We'll need
>>> to be more systematic about the upgrade testing.
>>
>> This sounds generally good to me — we did multiple-release upgrades
>> for a long time, and stuff is probably more complicated now but I
>> don't think it will actually be that big a deal.
>>
>> 3 releases back might be a bit much though — that's 27 months! (For
>> luminous, the beginning of 2015. Hammer.)
> 
> I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
> of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
> it's best to start with that, though?  It's still an improvement over the
> current ~12 months.
> 
>>> Somewhat separately, several people expressed concern about having stable
>>> releases to develop against.  This is somewhat orthogonal to what users
>>> need.  To that end, we can do a dev checkpoint every 1 or 2 months
>>> (preferences?), where we fork a 'next' branch and stabilize all of the
>>> tests before moving on.  This is good practice anyway to avoid
>>> accumulating low-frequency failures in the test suite that have to be
>>> squashed at the end.
>>
>> So this sounds like a fine idea to me, but how do we distinguish this
>> from the intermediate stable releases?
>>
>> By which I mean, are we *really* going to do a stabilization branch
>> that will never get seen by users? What kind of testing and bug fixing
>> are we going to commit to doing against it, and how do we balance that
>> effort with feature work?
>>
>> It seems like the same conflict we have now, only since the dev
>> checkpoints are less important they'll lose more often. Then we'll end
>> up having 9 months of scheduled work to debug for a user release
>> instead of 5 months that slipped to 7 or 8...
> 
> What if we frame this stabilization period in terms of stability of the
> test suite.  That gives us something concrete to aim for, lets us move on
> when we reach some threshold, and aligns perfectly with the thing that
> makes it hard to safely land new code (noisy test results)...
> 
> sage
> 

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com