download.ceph.com repository changes

download.ceph.com repository changes

[Alfredo Deza]

Hi all,

After the 12.2.6 release went out, we've been thinking on better ways
to remove a version from our repositories to prevent users from
upgrading/installing a known bad release.

The way our repos are structured today means every single version of
the release is included in the repository. That is, for Luminous,
every 12.x.x version of the binaries is in the same repo. This is true
for both RPM and DEB repositories.

However, the DEB repos don't allow pinning to a given version because
our tooling (namely reprepro) doesn't construct the repositories in a
way that this is allowed. For RPM repos this is fine, and version
pinning works.

To remove a bad version we have to proposals (and would like to hear
ideas on other possibilities), one that would involve symlinks and the
other one which purges the known bad version from our repos.

*Symlinking*
When releasing we would have a "previous" and "latest" symlink that
would get updated as versions move forward. It would require
separation of versions at the URL level (all versions would no longer
be available in one repo).

The URL structure would then look like:

    debian/luminous/12.2.3/
    debian/luminous/previous/  (points to 12.2.5)
    debian/luminous/latest/   (points to 12.2.7)

Caveats: the url structure would change from debian-luminous/ to
prevent breakage, and the versions would be split. For RPMs it would
mean a regression if someone is used to pinning, for example pinning
to 12.2.2 wouldn't be possible using the same url.

Pros: Faster release times, less need to move packages around, and
easier to remove a bad version


*Single version removal*
Our tooling would need to go and remove the known bad version from the
repository, which would require to rebuild the repository again, so
that the metadata is updated with the difference in the binaries.

Caveats: time intensive process, almost like cutting a new release
which takes about a day (and sometimes longer). Error prone since the
process wouldn't be the same (one off, just when a version needs to be
removed)

Pros: all urls for download.ceph.com and its structure are kept the same.

Re: download.ceph.com repository changes

[Dan van der Ster]

On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> Hi all,
>
> After the 12.2.6 release went out, we've been thinking on better ways
> to remove a version from our repositories to prevent users from
> upgrading/installing a known bad release.
>
> The way our repos are structured today means every single version of
> the release is included in the repository. That is, for Luminous,
> every 12.x.x version of the binaries is in the same repo. This is true
> for both RPM and DEB repositories.
>
> However, the DEB repos don't allow pinning to a given version because
> our tooling (namely reprepro) doesn't construct the repositories in a
> way that this is allowed. For RPM repos this is fine, and version
> pinning works.
>
> To remove a bad version we have to proposals (and would like to hear
> ideas on other possibilities), one that would involve symlinks and the
> other one which purges the known bad version from our repos.

What we did with our mirror was: `rm -f *12.2.6*; createrepo --update
.` Took a few seconds. Then disabled the mirror cron.

-- Dan

>
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that
> would get updated as versions move forward. It would require
> separation of versions at the URL level (all versions would no longer
> be available in one repo).
>
> The URL structure would then look like:
>
>     debian/luminous/12.2.3/
>     debian/luminous/previous/  (points to 12.2.5)
>     debian/luminous/latest/   (points to 12.2.7)
>
> Caveats: the url structure would change from debian-luminous/ to
> prevent breakage, and the versions would be split. For RPMs it would
> mean a regression if someone is used to pinning, for example pinning
> to 12.2.2 wouldn't be possible using the same url.
>
> Pros: Faster release times, less need to move packages around, and
> easier to remove a bad version
>
>
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the
> repository, which would require to rebuild the repository again, so
> that the metadata is updated with the difference in the binaries.
>
> Caveats: time intensive process, almost like cutting a new release
> which takes about a day (and sometimes longer). Error prone since the
> process wouldn't be the same (one off, just when a version needs to be
> removed)
>
> Pros: all urls for download.ceph.com and its structure are kept the same.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: download.ceph.com repository changes

[Alfredo Deza]

On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>
> What we did with our mirror was: `rm -f *12.2.6*; createrepo --update
> .` Took a few seconds. Then disabled the mirror cron.

Up until next time when we cut another release and you have to
re-enable the mirror with 12.2.6 in it :(

This is also fast for RPM repos, but not quite fast for DEB repos.
Finally, *if* you are doing this, the metadata changes, and the repos
need to
be signed again. I am curious how that --update operation didn't make
installations complain

>
> -- Dan
>
>>
>> *Symlinking*
>> When releasing we would have a "previous" and "latest" symlink that
>> would get updated as versions move forward. It would require
>> separation of versions at the URL level (all versions would no longer
>> be available in one repo).
>>
>> The URL structure would then look like:
>>
>>     debian/luminous/12.2.3/
>>     debian/luminous/previous/  (points to 12.2.5)
>>     debian/luminous/latest/   (points to 12.2.7)
>>
>> Caveats: the url structure would change from debian-luminous/ to
>> prevent breakage, and the versions would be split. For RPMs it would
>> mean a regression if someone is used to pinning, for example pinning
>> to 12.2.2 wouldn't be possible using the same url.
>>
>> Pros: Faster release times, less need to move packages around, and
>> easier to remove a bad version
>>
>>
>> *Single version removal*
>> Our tooling would need to go and remove the known bad version from the
>> repository, which would require to rebuild the repository again, so
>> that the metadata is updated with the difference in the binaries.
>>
>> Caveats: time intensive process, almost like cutting a new release
>> which takes about a day (and sometimes longer). Error prone since the
>> process wouldn't be the same (one off, just when a version needs to be
>> removed)
>>
>> Pros: all urls for download.ceph.com and its structure are kept the same.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: download.ceph.com repository changes

[Ken Dreyer]

On Tue, Jul 24, 2018 at 8:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>
> What we did with our mirror was: `rm -f *12.2.6*; createrepo --update
> .` Took a few seconds. Then disabled the mirror cron.

Unfortunately with Debian repositories, reprepro is a lot more
complicated, and then we have to re-sign the new repository metadata,
so it's a little more involved there.

BUT perfect is the enemy of the good so maybe we should have just done
your suggestion for RPMs at least.

- Ken

Re: download.ceph.com repository changes

[Dan van der Ster]

On Tue, Jul 24, 2018 at 4:59 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >>
> >> Hi all,
> >>
> >> After the 12.2.6 release went out, we've been thinking on better ways
> >> to remove a version from our repositories to prevent users from
> >> upgrading/installing a known bad release.
> >>
> >> The way our repos are structured today means every single version of
> >> the release is included in the repository. That is, for Luminous,
> >> every 12.x.x version of the binaries is in the same repo. This is true
> >> for both RPM and DEB repositories.
> >>
> >> However, the DEB repos don't allow pinning to a given version because
> >> our tooling (namely reprepro) doesn't construct the repositories in a
> >> way that this is allowed. For RPM repos this is fine, and version
> >> pinning works.
> >>
> >> To remove a bad version we have to proposals (and would like to hear
> >> ideas on other possibilities), one that would involve symlinks and the
> >> other one which purges the known bad version from our repos.
> >
> > What we did with our mirror was: `rm -f *12.2.6*; createrepo --update
> > .` Took a few seconds. Then disabled the mirror cron.
>
> Up until next time when we cut another release and you have to
> re-enable the mirror with 12.2.6 in it :(
>

Right... we re-sync'd 12.2.6 along with 12.2.7 -- but people here
mostly grab the highest version.

> This is also fast for RPM repos, but not quite fast for DEB repos.
> Finally, *if* you are doing this, the metadata changes, and the repos
> need to
> be signed again. I am curious how that --update operation didn't make
> installations complain

Good question.. I don't know enough about the repo signatures to
comment on this.
I do know that all clients who had distro-sync'd up to 12.2.6
successfully distro-sync'd back to 12.2.5.
(Our client machines yum distro-sync daily).

-- Dan

>
> >
> > -- Dan
> >
> >>
> >> *Symlinking*
> >> When releasing we would have a "previous" and "latest" symlink that
> >> would get updated as versions move forward. It would require
> >> separation of versions at the URL level (all versions would no longer
> >> be available in one repo).
> >>
> >> The URL structure would then look like:
> >>
> >>     debian/luminous/12.2.3/
> >>     debian/luminous/previous/  (points to 12.2.5)
> >>     debian/luminous/latest/   (points to 12.2.7)
> >>
> >> Caveats: the url structure would change from debian-luminous/ to
> >> prevent breakage, and the versions would be split. For RPMs it would
> >> mean a regression if someone is used to pinning, for example pinning
> >> to 12.2.2 wouldn't be possible using the same url.
> >>
> >> Pros: Faster release times, less need to move packages around, and
> >> easier to remove a bad version
> >>
> >>
> >> *Single version removal*
> >> Our tooling would need to go and remove the known bad version from the
> >> repository, which would require to rebuild the repository again, so
> >> that the metadata is updated with the difference in the binaries.
> >>
> >> Caveats: time intensive process, almost like cutting a new release
> >> which takes about a day (and sometimes longer). Error prone since the
> >> process wouldn't be the same (one off, just when a version needs to be
> >> removed)
> >>
> >> Pros: all urls for download.ceph.com and its structure are kept the same.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: download.ceph.com repository changes

[Dan van der Ster]

On Tue, Jul 24, 2018 at 5:08 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> On Tue, Jul 24, 2018 at 4:59 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >
> > On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
> > > On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> After the 12.2.6 release went out, we've been thinking on better ways
> > >> to remove a version from our repositories to prevent users from
> > >> upgrading/installing a known bad release.
> > >>
> > >> The way our repos are structured today means every single version of
> > >> the release is included in the repository. That is, for Luminous,
> > >> every 12.x.x version of the binaries is in the same repo. This is true
> > >> for both RPM and DEB repositories.
> > >>
> > >> However, the DEB repos don't allow pinning to a given version because
> > >> our tooling (namely reprepro) doesn't construct the repositories in a
> > >> way that this is allowed. For RPM repos this is fine, and version
> > >> pinning works.
> > >>
> > >> To remove a bad version we have to proposals (and would like to hear
> > >> ideas on other possibilities), one that would involve symlinks and the
> > >> other one which purges the known bad version from our repos.
> > >
> > > What we did with our mirror was: `rm -f *12.2.6*; createrepo --update
> > > .` Took a few seconds. Then disabled the mirror cron.
> >
> > Up until next time when we cut another release and you have to
> > re-enable the mirror with 12.2.6 in it :(
> >
>
> Right... we re-sync'd 12.2.6 along with 12.2.7 -- but people here
> mostly grab the highest version.
>
> > This is also fast for RPM repos, but not quite fast for DEB repos.
> > Finally, *if* you are doing this, the metadata changes, and the repos
> > need to
> > be signed again. I am curious how that --update operation didn't make
> > installations complain
>
> Good question.. I don't know enough about the repo signatures to
> comment on this.

I asked our mirror man. Apparently we don't sign the repo, only the
rpms. So not applicable in general I suppose.

Another completely different (and not my) idea, how about we retag the
last good release with z+1. In this case we had 12.2.5 as the last
good, and 12.2.6 broken, so we add the v12.2.7 tag on v12.2.5,
effectively re-pushing 12.2.5 to the top.

-- dan

> I do know that all clients who had distro-sync'd up to 12.2.6
> successfully distro-sync'd back to 12.2.5.
> (Our client machines yum distro-sync daily).
>
> -- Dan
>
> >
> > >
> > > -- Dan
> > >
> > >>
> > >> *Symlinking*
> > >> When releasing we would have a "previous" and "latest" symlink that
> > >> would get updated as versions move forward. It would require
> > >> separation of versions at the URL level (all versions would no longer
> > >> be available in one repo).
> > >>
> > >> The URL structure would then look like:
> > >>
> > >>     debian/luminous/12.2.3/
> > >>     debian/luminous/previous/  (points to 12.2.5)
> > >>     debian/luminous/latest/   (points to 12.2.7)
> > >>
> > >> Caveats: the url structure would change from debian-luminous/ to
> > >> prevent breakage, and the versions would be split. For RPMs it would
> > >> mean a regression if someone is used to pinning, for example pinning
> > >> to 12.2.2 wouldn't be possible using the same url.
> > >>
> > >> Pros: Faster release times, less need to move packages around, and
> > >> easier to remove a bad version
> > >>
> > >>
> > >> *Single version removal*
> > >> Our tooling would need to go and remove the known bad version from the
> > >> repository, which would require to rebuild the repository again, so
> > >> that the metadata is updated with the difference in the binaries.
> > >>
> > >> Caveats: time intensive process, almost like cutting a new release
> > >> which takes about a day (and sometimes longer). Error prone since the
> > >> process wouldn't be the same (one off, just when a version needs to be
> > >> removed)
> > >>
> > >> Pros: all urls for download.ceph.com and its structure are kept the same.
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: download.ceph.com repository changes

[Brent Kennedy]

It would be nice if ceph-deploy could select the version as well as the
release.  E.G:  --release luminous --version 12.2.7   

Otherwise, I deploy a newest release to a new OSD server, then have to
upgrade the rest of the cluster ( unless the cluster is on a previous
release at the highest level )

Not sure if this adds to this particular discussion though :)

-Brent

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On Behalf Of Dan
van der Ster
Sent: Tuesday, July 24, 2018 11:26 AM
To: Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: ceph-users <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>; ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
ceph-maintainers-Qp0mS5GaXlQ@public.gmane.org
Subject: Re: [ceph-users] download.ceph.com repository changes

On Tue, Jul 24, 2018 at 5:08 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>
> On Tue, Jul 24, 2018 at 4:59 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >
> > On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>
wrote:
> > > On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> After the 12.2.6 release went out, we've been thinking on better 
> > >> ways to remove a version from our repositories to prevent users 
> > >> from upgrading/installing a known bad release.
> > >>
> > >> The way our repos are structured today means every single version 
> > >> of the release is included in the repository. That is, for 
> > >> Luminous, every 12.x.x version of the binaries is in the same 
> > >> repo. This is true for both RPM and DEB repositories.
> > >>
> > >> However, the DEB repos don't allow pinning to a given version 
> > >> because our tooling (namely reprepro) doesn't construct the 
> > >> repositories in a way that this is allowed. For RPM repos this is 
> > >> fine, and version pinning works.
> > >>
> > >> To remove a bad version we have to proposals (and would like to 
> > >> hear ideas on other possibilities), one that would involve 
> > >> symlinks and the other one which purges the known bad version from
our repos.
> > >
> > > What we did with our mirror was: `rm -f *12.2.6*; createrepo 
> > > --update .` Took a few seconds. Then disabled the mirror cron.
> >
> > Up until next time when we cut another release and you have to 
> > re-enable the mirror with 12.2.6 in it :(
> >
>
> Right... we re-sync'd 12.2.6 along with 12.2.7 -- but people here 
> mostly grab the highest version.
>
> > This is also fast for RPM repos, but not quite fast for DEB repos.
> > Finally, *if* you are doing this, the metadata changes, and the 
> > repos need to be signed again. I am curious how that --update 
> > operation didn't make installations complain
>
> Good question.. I don't know enough about the repo signatures to 
> comment on this.

I asked our mirror man. Apparently we don't sign the repo, only the rpms. So
not applicable in general I suppose.

Another completely different (and not my) idea, how about we retag the last
good release with z+1. In this case we had 12.2.5 as the last good, and
12.2.6 broken, so we add the v12.2.7 tag on v12.2.5, effectively re-pushing
12.2.5 to the top.

-- dan

> I do know that all clients who had distro-sync'd up to 12.2.6 
> successfully distro-sync'd back to 12.2.5.
> (Our client machines yum distro-sync daily).
>
> -- Dan
>
> >
> > >
> > > -- Dan
> > >
> > >>
> > >> *Symlinking*
> > >> When releasing we would have a "previous" and "latest" symlink 
> > >> that would get updated as versions move forward. It would require 
> > >> separation of versions at the URL level (all versions would no 
> > >> longer be available in one repo).
> > >>
> > >> The URL structure would then look like:
> > >>
> > >>     debian/luminous/12.2.3/
> > >>     debian/luminous/previous/  (points to 12.2.5)
> > >>     debian/luminous/latest/   (points to 12.2.7)
> > >>
> > >> Caveats: the url structure would change from debian-luminous/ to 
> > >> prevent breakage, and the versions would be split. For RPMs it 
> > >> would mean a regression if someone is used to pinning, for 
> > >> example pinning to 12.2.2 wouldn't be possible using the same url.
> > >>
> > >> Pros: Faster release times, less need to move packages around, 
> > >> and easier to remove a bad version
> > >>
> > >>
> > >> *Single version removal*
> > >> Our tooling would need to go and remove the known bad version 
> > >> from the repository, which would require to rebuild the 
> > >> repository again, so that the metadata is updated with the difference
in the binaries.
> > >>
> > >> Caveats: time intensive process, almost like cutting a new 
> > >> release which takes about a day (and sometimes longer). Error 
> > >> prone since the process wouldn't be the same (one off, just when 
> > >> a version needs to be
> > >> removed)
> > >>
> > >> Pros: all urls for download.ceph.com and its structure are kept the
same.
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe 
> > >> ceph-devel" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org 
> > >> More majordomo info at  
> > >> http://vger.kernel.org/majordomo-info.html
_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: download.ceph.com repository changes

[Alfredo Deza]

On Tue, Jul 24, 2018 at 1:19 PM, Brent Kennedy <bkennedy-3tLf1voIkJTQT0dZR+AlfA@public.gmane.org> wrote:
> It would be nice if ceph-deploy could select the version as well as the
> release.  E.G:  --release luminous --version 12.2.7
>
> Otherwise, I deploy a newest release to a new OSD server, then have to
> upgrade the rest of the cluster ( unless the cluster is on a previous
> release at the highest level )
>
> Not sure if this adds to this particular discussion though :)

That might work for RPMs, but it doesn't work for DEB repos because
our repos don't support it
>
> -Brent
>
> -----Original Message-----
> From: ceph-users [mailto:ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org] On Behalf Of Dan
> van der Ster
> Sent: Tuesday, July 24, 2018 11:26 AM
> To: Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: ceph-users <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>; ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org;
> ceph-maintainers-Qp0mS5GaXlQ@public.gmane.org
> Subject: Re: [ceph-users] download.ceph.com repository changes
>
> On Tue, Jul 24, 2018 at 5:08 PM Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org> wrote:
>>
>> On Tue, Jul 24, 2018 at 4:59 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> >
>> > On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster <dan-EOCVfBHj35C+XT7JhA+gdA@public.gmane.org>
> wrote:
>> > > On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> After the 12.2.6 release went out, we've been thinking on better
>> > >> ways to remove a version from our repositories to prevent users
>> > >> from upgrading/installing a known bad release.
>> > >>
>> > >> The way our repos are structured today means every single version
>> > >> of the release is included in the repository. That is, for
>> > >> Luminous, every 12.x.x version of the binaries is in the same
>> > >> repo. This is true for both RPM and DEB repositories.
>> > >>
>> > >> However, the DEB repos don't allow pinning to a given version
>> > >> because our tooling (namely reprepro) doesn't construct the
>> > >> repositories in a way that this is allowed. For RPM repos this is
>> > >> fine, and version pinning works.
>> > >>
>> > >> To remove a bad version we have to proposals (and would like to
>> > >> hear ideas on other possibilities), one that would involve
>> > >> symlinks and the other one which purges the known bad version from
> our repos.
>> > >
>> > > What we did with our mirror was: `rm -f *12.2.6*; createrepo
>> > > --update .` Took a few seconds. Then disabled the mirror cron.
>> >
>> > Up until next time when we cut another release and you have to
>> > re-enable the mirror with 12.2.6 in it :(
>> >
>>
>> Right... we re-sync'd 12.2.6 along with 12.2.7 -- but people here
>> mostly grab the highest version.
>>
>> > This is also fast for RPM repos, but not quite fast for DEB repos.
>> > Finally, *if* you are doing this, the metadata changes, and the
>> > repos need to be signed again. I am curious how that --update
>> > operation didn't make installations complain
>>
>> Good question.. I don't know enough about the repo signatures to
>> comment on this.
>
> I asked our mirror man. Apparently we don't sign the repo, only the rpms. So
> not applicable in general I suppose.
>
> Another completely different (and not my) idea, how about we retag the last
> good release with z+1. In this case we had 12.2.5 as the last good, and
> 12.2.6 broken, so we add the v12.2.7 tag on v12.2.5, effectively re-pushing
> 12.2.5 to the top.
>
> -- dan
>
>> I do know that all clients who had distro-sync'd up to 12.2.6
>> successfully distro-sync'd back to 12.2.5.
>> (Our client machines yum distro-sync daily).
>>
>> -- Dan
>>
>> >
>> > >
>> > > -- Dan
>> > >
>> > >>
>> > >> *Symlinking*
>> > >> When releasing we would have a "previous" and "latest" symlink
>> > >> that would get updated as versions move forward. It would require
>> > >> separation of versions at the URL level (all versions would no
>> > >> longer be available in one repo).
>> > >>
>> > >> The URL structure would then look like:
>> > >>
>> > >>     debian/luminous/12.2.3/
>> > >>     debian/luminous/previous/  (points to 12.2.5)
>> > >>     debian/luminous/latest/   (points to 12.2.7)
>> > >>
>> > >> Caveats: the url structure would change from debian-luminous/ to
>> > >> prevent breakage, and the versions would be split. For RPMs it
>> > >> would mean a regression if someone is used to pinning, for
>> > >> example pinning to 12.2.2 wouldn't be possible using the same url.
>> > >>
>> > >> Pros: Faster release times, less need to move packages around,
>> > >> and easier to remove a bad version
>> > >>
>> > >>
>> > >> *Single version removal*
>> > >> Our tooling would need to go and remove the known bad version
>> > >> from the repository, which would require to rebuild the
>> > >> repository again, so that the metadata is updated with the difference
> in the binaries.
>> > >>
>> > >> Caveats: time intensive process, almost like cutting a new
>> > >> release which takes about a day (and sometimes longer). Error
>> > >> prone since the process wouldn't be the same (one off, just when
>> > >> a version needs to be
>> > >> removed)
>> > >>
>> > >> Pros: all urls for download.ceph.com and its structure are kept the
> same.
>> > >> --
>> > >> To unsubscribe from this list: send the line "unsubscribe
>> > >> ceph-devel" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> > >> More majordomo info at
>> > >> http://vger.kernel.org/majordomo-info.html
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

Re: download.ceph.com repository changes

[Sage Weil]

On Tue, 24 Jul 2018, Alfredo Deza wrote:
> Hi all,
> 
> After the 12.2.6 release went out, we've been thinking on better ways
> to remove a version from our repositories to prevent users from
> upgrading/installing a known bad release.
> 
> The way our repos are structured today means every single version of
> the release is included in the repository. That is, for Luminous,
> every 12.x.x version of the binaries is in the same repo. This is true
> for both RPM and DEB repositories.
> 
> However, the DEB repos don't allow pinning to a given version because
> our tooling (namely reprepro) doesn't construct the repositories in a
> way that this is allowed. For RPM repos this is fine, and version
> pinning works.
> 
> To remove a bad version we have to proposals (and would like to hear
> ideas on other possibilities), one that would involve symlinks and the
> other one which purges the known bad version from our repos.
> 
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that
> would get updated as versions move forward. It would require
> separation of versions at the URL level (all versions would no longer
> be available in one repo).
> 
> The URL structure would then look like:
> 
>     debian/luminous/12.2.3/
>     debian/luminous/previous/  (points to 12.2.5)
>     debian/luminous/latest/   (points to 12.2.7)
> 
> Caveats: the url structure would change from debian-luminous/ to
> prevent breakage, and the versions would be split. For RPMs it would
> mean a regression if someone is used to pinning, for example pinning
> to 12.2.2 wouldn't be possible using the same url.
> 
> Pros: Faster release times, less need to move packages around, and
> easier to remove a bad version

I think the core question is: how many users use or depend on the 
version-pinning features with RPMs.  The symlinking path is easy to 
implement but breaks that ability... you have to change the repo URL to 
pin to a release.

(Nobody is doing version pinning with debs because our reprepro repos 
don't support it.)

In the future we can always move to the multi-version repos and maintain 
the per-version repos in parallel, right?

Thanks!
sage



> 
> 
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the
> repository, which would require to rebuild the repository again, so
> that the metadata is updated with the difference in the binaries.
> 
> Caveats: time intensive process, almost like cutting a new release
> which takes about a day (and sometimes longer). Error prone since the
> process wouldn't be the same (one off, just when a version needs to be
> removed)
> 
> Pros: all urls for download.ceph.com and its structure are kept the same.
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 

Re: [Ceph-maintainers] download.ceph.com repository changes

[Fabian Grünbichler]

On Tue, Jul 24, 2018 at 10:38:43AM -0400, Alfredo Deza wrote:
> Hi all,
> 
> After the 12.2.6 release went out, we've been thinking on better ways
> to remove a version from our repositories to prevent users from
> upgrading/installing a known bad release.
> 
> The way our repos are structured today means every single version of
> the release is included in the repository. That is, for Luminous,
> every 12.x.x version of the binaries is in the same repo. This is true
> for both RPM and DEB repositories.
> 
> However, the DEB repos don't allow pinning to a given version because
> our tooling (namely reprepro) doesn't construct the repositories in a
> way that this is allowed. For RPM repos this is fine, and version
> pinning works.

If you mean that reprepo does not support referencing multiple versions
of packages in the Packages file, there is a patched fork that does
(that seems well-supported):

https://github.com/profitbricks/reprepro

> 
> To remove a bad version we have to proposals (and would like to hear
> ideas on other possibilities), one that would involve symlinks and the
> other one which purges the known bad version from our repos.
> 
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that
> would get updated as versions move forward. It would require
> separation of versions at the URL level (all versions would no longer
> be available in one repo).
> 
> The URL structure would then look like:
> 
>     debian/luminous/12.2.3/
>     debian/luminous/previous/  (points to 12.2.5)
>     debian/luminous/latest/   (points to 12.2.7)
> 
> Caveats: the url structure would change from debian-luminous/ to
> prevent breakage, and the versions would be split. For RPMs it would
> mean a regression if someone is used to pinning, for example pinning
> to 12.2.2 wouldn't be possible using the same url.
> 
> Pros: Faster release times, less need to move packages around, and
> easier to remove a bad version
> 
> 
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the
> repository, which would require to rebuild the repository again, so
> that the metadata is updated with the difference in the binaries.
> 
> Caveats: time intensive process, almost like cutting a new release
> which takes about a day (and sometimes longer). Error prone since the
> process wouldn't be the same (one off, just when a version needs to be
> removed)

I am not involved in this process, but that seems like something is
wrong somewhere. You keep all the binary debs on the public mirror, so
"retracting" a broken latest one should just consist of:

- deleting the .deb files of the broken release
- regenerating the Packages*, Content* and *Release* metadata files

The former should be quasi-instant, the latter takes a bit (ceph
packages are quite big, especially the ones containing debug symbols,
and they need to be hashed multiple times), but nowhere near a day.

If you keep the "old" metadata files around, both steps should be almost
instant:
- delete broken .deb files
- revert (expensive) metadata files to previous snapshot

> Pros: all urls for download.ceph.com and its structure are kept the same.

that is quite a big pro, IMHO.

Re: [Ceph-maintainers] download.ceph.com repository changes

[Alfredo Deza]

On Fri, Jul 27, 2018 at 3:28 AM, Fabian Grünbichler
<f.gruenbichler@proxmox.com> wrote:
> On Tue, Jul 24, 2018 at 10:38:43AM -0400, Alfredo Deza wrote:
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>
> If you mean that reprepo does not support referencing multiple versions
> of packages in the Packages file, there is a patched fork that does
> (that seems well-supported):
>
> https://github.com/profitbricks/reprepro
>
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>>
>> *Symlinking*
>> When releasing we would have a "previous" and "latest" symlink that
>> would get updated as versions move forward. It would require
>> separation of versions at the URL level (all versions would no longer
>> be available in one repo).
>>
>> The URL structure would then look like:
>>
>>     debian/luminous/12.2.3/
>>     debian/luminous/previous/  (points to 12.2.5)
>>     debian/luminous/latest/   (points to 12.2.7)
>>
>> Caveats: the url structure would change from debian-luminous/ to
>> prevent breakage, and the versions would be split. For RPMs it would
>> mean a regression if someone is used to pinning, for example pinning
>> to 12.2.2 wouldn't be possible using the same url.
>>
>> Pros: Faster release times, less need to move packages around, and
>> easier to remove a bad version
>>
>>
>> *Single version removal*
>> Our tooling would need to go and remove the known bad version from the
>> repository, which would require to rebuild the repository again, so
>> that the metadata is updated with the difference in the binaries.
>>
>> Caveats: time intensive process, almost like cutting a new release
>> which takes about a day (and sometimes longer). Error prone since the
>> process wouldn't be the same (one off, just when a version needs to be
>> removed)
>
> I am not involved in this process, but that seems like something is
> wrong somewhere. You keep all the binary debs on the public mirror, so
> "retracting" a broken latest one should just consist of:
>
> - deleting the .deb files of the broken release
> - regenerating the Packages*, Content* and *Release* metadata files
>
> The former should be quasi-instant, the latter takes a bit (ceph
> packages are quite big, especially the ones containing debug symbols,
> and they need to be hashed multiple times), but nowhere near a day.

The release process involves a bit more than this. Our repositories
are created from scratch from a set of binaries. "Removing" a binary
needs to be done at the level of the service that stores them, and
then the repositories need to be recreated. They then need to be
pulled in
for signing, and then pushed out to download.ceph.com.

That part doesn't take a day, but it does take a few hours.

>
> If you keep the "old" metadata files around, both steps should be almost
> instant:
> - delete broken .deb files
> - revert (expensive) metadata files to previous snapshot
>
>> Pros: all urls for download.ceph.com and its structure are kept the same.
>
> that is quite a big pro, IMHO.
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [Ceph-maintainers] download.ceph.com repository changes

[Ken Dreyer]

On Fri, Jul 27, 2018 at 1:28 AM, Fabian Grünbichler
<f.gruenbichler@proxmox.com> wrote:
> On Tue, Jul 24, 2018 at 10:38:43AM -0400, Alfredo Deza wrote:
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>
> If you mean that reprepo does not support referencing multiple versions
> of packages in the Packages file, there is a patched fork that does
> (that seems well-supported):
>
> https://github.com/profitbricks/reprepro

Thanks for this link. That's great to know someone's working on this.

What's the status of merging that back into the main reprepro code, or
else shipping that fork as the new reprepro package in Debian /
Ubuntu? The Ceph project could end up responsible for maintaining that
reprepro fork if the main Ubuntu community does not pick it up :) The
fork is several years old, and the latest update on
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570623 was over a
year ago.

- Ken
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [Ceph-maintainers] download.ceph.com repository changes

[Fabian Grünbichler]

On Mon, Jul 30, 2018 at 11:36:55AM -0600, Ken Dreyer wrote:
> On Fri, Jul 27, 2018 at 1:28 AM, Fabian Grünbichler
> <f.gruenbichler-YTcQvvOqK21BDgjK7y7TUQ@public.gmane.org> wrote:
> > On Tue, Jul 24, 2018 at 10:38:43AM -0400, Alfredo Deza wrote:
> >> Hi all,
> >>
> >> After the 12.2.6 release went out, we've been thinking on better ways
> >> to remove a version from our repositories to prevent users from
> >> upgrading/installing a known bad release.
> >>
> >> The way our repos are structured today means every single version of
> >> the release is included in the repository. That is, for Luminous,
> >> every 12.x.x version of the binaries is in the same repo. This is true
> >> for both RPM and DEB repositories.
> >>
> >> However, the DEB repos don't allow pinning to a given version because
> >> our tooling (namely reprepro) doesn't construct the repositories in a
> >> way that this is allowed. For RPM repos this is fine, and version
> >> pinning works.
> >
> > If you mean that reprepo does not support referencing multiple versions
> > of packages in the Packages file, there is a patched fork that does
> > (that seems well-supported):
> >
> > https://github.com/profitbricks/reprepro
> 
> Thanks for this link. That's great to know someone's working on this.
> 
> What's the status of merging that back into the main reprepro code, or
> else shipping that fork as the new reprepro package in Debian /
> Ubuntu? The Ceph project could end up responsible for maintaining that
> reprepro fork if the main Ubuntu community does not pick it up :) The
> fork is several years old, and the latest update on
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570623 was over a
> year ago.

I don't know anything more than what is publicly available about either
merging back to the original reprepo or shipping in Debian/Ubuntu. We
are using our own custom repo software built around lower level tools, I
was just aware of the fork for unrelated reasons :)

Re: download.ceph.com repository changes

[Alfredo Deza]

Reviving this old thread.

I still think this is something we should consider as users still
experience problems:

* Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
later they add other nodes but version moved to 14.2.2
* Impossible to use a version that is not what the latest is (e.g. if
someone doesn't need the release from Monday, but wants the one from 6
months ago), similar to the above
* When a release is underway, the repository breaks because syncing
packages takes hours. The operation is not atomic.
* It is not currently possible to "remove" a bad release, in the past,
this means cutting a new release as soon as possible, which can take
days

The latest issue (my fault!) was to cut a release and get the packages
out without communicating with the release manager, which caused users
to note there is a new version *as soon as it was up* vs, a
process that could've not touched the 'latest' url until the
announcement goes out.

If you have been affected by any of these issues (or others I didn't
come up with), please let us know in this thread so that we can find
some common ground and try to improve the process.

Thanks!

On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> Hi all,
>
> After the 12.2.6 release went out, we've been thinking on better ways
> to remove a version from our repositories to prevent users from
> upgrading/installing a known bad release.
>
> The way our repos are structured today means every single version of
> the release is included in the repository. That is, for Luminous,
> every 12.x.x version of the binaries is in the same repo. This is true
> for both RPM and DEB repositories.
>
> However, the DEB repos don't allow pinning to a given version because
> our tooling (namely reprepro) doesn't construct the repositories in a
> way that this is allowed. For RPM repos this is fine, and version
> pinning works.
>
> To remove a bad version we have to proposals (and would like to hear
> ideas on other possibilities), one that would involve symlinks and the
> other one which purges the known bad version from our repos.
>
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that
> would get updated as versions move forward. It would require
> separation of versions at the URL level (all versions would no longer
> be available in one repo).
>
> The URL structure would then look like:
>
>     debian/luminous/12.2.3/
>     debian/luminous/previous/  (points to 12.2.5)
>     debian/luminous/latest/   (points to 12.2.7)
>
> Caveats: the url structure would change from debian-luminous/ to
> prevent breakage, and the versions would be split. For RPMs it would
> mean a regression if someone is used to pinning, for example pinning
> to 12.2.2 wouldn't be possible using the same url.
>
> Pros: Faster release times, less need to move packages around, and
> easier to remove a bad version
>
>
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the
> repository, which would require to rebuild the repository again, so
> that the metadata is updated with the difference in the binaries.
>
> Caveats: time intensive process, almost like cutting a new release
> which takes about a day (and sometimes longer). Error prone since the
> process wouldn't be the same (one off, just when a version needs to be
> removed)
>
> Pros: all urls for download.ceph.com and its structure are kept the same.
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Marc Roos]

 
Just wanted to say we do not have any problems with current/past setup. 
Our ceph nodes are not even connected to the internet and we relay 
everything via 'our own local mirror'. 






-----Original Message-----
From: Alfredo Deza [mailto:adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org] 
Sent: dinsdag 17 september 2019 15:15
To: ceph-maintainers-Qp0mS5GaXlQ@public.gmane.org; ceph-users; ceph-devel
Subject: [ceph-users] Re: download.ceph.com repository changes

Reviving this old thread.

I still think this is something we should consider as users still 
experience problems:

* Impossible to 'pin' to a version. User installs 14.2.0 and 4 months 
later they add other nodes but version moved to 14.2.2
* Impossible to use a version that is not what the latest is (e.g. if 
someone doesn't need the release from Monday, but wants the one from 6 
months ago), similar to the above
* When a release is underway, the repository breaks because syncing 
packages takes hours. The operation is not atomic.
* It is not currently possible to "remove" a bad release, in the past, 
this means cutting a new release as soon as possible, which can take 
days

The latest issue (my fault!) was to cut a release and get the packages 
out without communicating with the release manager, which caused users 
to note there is a new version *as soon as it was up* vs, a process that 
could've not touched the 'latest' url until the announcement goes out.

If you have been affected by any of these issues (or others I didn't 
come up with), please let us know in this thread so that we can find 
some common ground and try to improve the process.

Thanks!

On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> Hi all,
>
> After the 12.2.6 release went out, we've been thinking on better ways 
> to remove a version from our repositories to prevent users from 
> upgrading/installing a known bad release.
>
> The way our repos are structured today means every single version of 
> the release is included in the repository. That is, for Luminous, 
> every 12.x.x version of the binaries is in the same repo. This is true 

> for both RPM and DEB repositories.
>
> However, the DEB repos don't allow pinning to a given version because 
> our tooling (namely reprepro) doesn't construct the repositories in a 
> way that this is allowed. For RPM repos this is fine, and version 
> pinning works.
>
> To remove a bad version we have to proposals (and would like to hear 
> ideas on other possibilities), one that would involve symlinks and the 

> other one which purges the known bad version from our repos.
>
> *Symlinking*
> When releasing we would have a "previous" and "latest" symlink that 
> would get updated as versions move forward. It would require 
> separation of versions at the URL level (all versions would no longer 
> be available in one repo).
>
> The URL structure would then look like:
>
>     debian/luminous/12.2.3/
>     debian/luminous/previous/  (points to 12.2.5)
>     debian/luminous/latest/   (points to 12.2.7)
>
> Caveats: the url structure would change from debian-luminous/ to 
> prevent breakage, and the versions would be split. For RPMs it would 
> mean a regression if someone is used to pinning, for example pinning 
> to 12.2.2 wouldn't be possible using the same url.
>
> Pros: Faster release times, less need to move packages around, and 
> easier to remove a bad version
>
>
> *Single version removal*
> Our tooling would need to go and remove the known bad version from the 

> repository, which would require to rebuild the repository again, so 
> that the metadata is updated with the difference in the binaries.
>
> Caveats: time intensive process, almost like cutting a new release 
> which takes about a day (and sometimes longer). Error prone since the 
> process wouldn't be the same (one off, just when a version needs to be
> removed)
>
> Pros: all urls for download.ceph.com and its structure are kept the 
same.
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org To unsubscribe send an 
email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Abhishek Lekshmanan]

"Alfredo Deza" <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> writes:

> Reviving this old thread.
>
> I still think this is something we should consider as users still
> experience problems:
>
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2
> * Impossible to use a version that is not what the latest is (e.g. if
> someone doesn't need the release from Monday, but wants the one from 6
> months ago), similar to the above
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.

One of the main problems is the non atomicity here, so one way would be
that we announce we're building and syncing packages and the release
should be out soon, the main problem then is that this process can vary
between hours to a couple of days (or if it's a fri. longer).

> * It is not currently possible to "remove" a bad release, in the past,
> this means cutting a new release as soon as possible, which can take
> days
>
> The latest issue (my fault!) was to cut a release and get the packages
> out without communicating with the release manager, which caused users
> to note there is a new version *as soon as it was up* vs, a
> process that could've not touched the 'latest' url until the
> announcement goes out.
>
> If you have been affected by any of these issues (or others I didn't
> come up with), please let us know in this thread so that we can find
> some common ground and try to improve the process.
>
> Thanks!
>
> On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>>
>> *Symlinking*
>> When releasing we would have a "previous" and "latest" symlink that
>> would get updated as versions move forward. It would require
>> separation of versions at the URL level (all versions would no longer
>> be available in one repo).
>>
>> The URL structure would then look like:
>>
>>     debian/luminous/12.2.3/
>>     debian/luminous/previous/  (points to 12.2.5)
>>     debian/luminous/latest/   (points to 12.2.7)
>>
>> Caveats: the url structure would change from debian-luminous/ to
>> prevent breakage, and the versions would be split. For RPMs it would
>> mean a regression if someone is used to pinning, for example pinning
>> to 12.2.2 wouldn't be possible using the same url.
>>
>> Pros: Faster release times, less need to move packages around, and
>> easier to remove a bad version
>>
>>
>> *Single version removal*
>> Our tooling would need to go and remove the known bad version from the
>> repository, which would require to rebuild the repository again, so
>> that the metadata is updated with the difference in the binaries.
>>
>> Caveats: time intensive process, almost like cutting a new release
>> which takes about a day (and sometimes longer). Error prone since the
>> process wouldn't be the same (one off, just when a version needs to be
>> removed)
>>
>> Pros: all urls for download.ceph.com and its structure are kept the same.
> _______________________________________________
> ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
> To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
>

-- 
Abhishek 
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Janne Johansson]

[-- Attachment #1.1: Type: text/plain, Size: 818 bytes --]

Den tis 17 sep. 2019 kl 15:15 skrev Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>:

> Reviving this old thread.
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.
>

Couldn't they be almost atomic?
I believe both "yum" and "apt" would only consider rpms/debs if they are
listed in the repo index files*, so you should be able to copy out rpms and
debs and at the final step "mv" the index files into place, making the
packages visible in a super short time?

*) .xml.gz for rpms, db/ files for debs.

Obviously this is a very small part of the list of problems, but I think
anyone mirroring huge sets of rpms/debs would have faced and solved this
particular nit by now?


-- 
May the most significant bit of your life be positive.

[-- Attachment #1.2: Type: text/html, Size: 1298 bytes --]

[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Sasha Litvak]

[-- Attachment #1.1: Type: text/plain, Size: 5065 bytes --]

I have been affected by few issues mentioned by Alfredo.

* Version Pinning:  Had to install several debs of specific version to be
able to pull dependencies of the correct version.  I believe that other
projects resolving it by creating a virtual package that pulls all of the
proper dependencies in.  Not sure if the same done by RPM / Yum.

* Unannounced releases.  I believe it is more of a procedural issue and
unfortunately something will need to be done to enforce the compliance once
rules of packages release are finalized.

* I am bothered with a quality of the releases of a very complex system
that can bring down a whole house and keep it down for a while.  While I
wish the QA would be perfect, I wonder if it would be practical to release
new packages to a testing repo before moving it to a main one.  There is a
chance then someone will detect a problem before it becomes a
production issue.   Let it seat for a couple days or weeks in testing.
People who need new update right away or just want to test will install it
and report the problems.  Others will not be affected.

Just my 2c,

On Tue, Sep 17, 2019, 8:15 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Reviving this old thread.
>
> I still think this is something we should consider as users still
> experience problems:
>
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2
> * Impossible to use a version that is not what the latest is (e.g. if
> someone doesn't need the release from Monday, but wants the one from 6
> months ago), similar to the above
> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.
> * It is not currently possible to "remove" a bad release, in the past,
> this means cutting a new release as soon as possible, which can take
> days
>
> The latest issue (my fault!) was to cut a release and get the packages
> out without communicating with the release manager, which caused users
> to note there is a new version *as soon as it was up* vs, a
> process that could've not touched the 'latest' url until the
> announcement goes out.
>
> If you have been affected by any of these issues (or others I didn't
> come up with), please let us know in this thread so that we can find
> some common ground and try to improve the process.
>
> Thanks!
>
> On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >
> > Hi all,
> >
> > After the 12.2.6 release went out, we've been thinking on better ways
> > to remove a version from our repositories to prevent users from
> > upgrading/installing a known bad release.
> >
> > The way our repos are structured today means every single version of
> > the release is included in the repository. That is, for Luminous,
> > every 12.x.x version of the binaries is in the same repo. This is true
> > for both RPM and DEB repositories.
> >
> > However, the DEB repos don't allow pinning to a given version because
> > our tooling (namely reprepro) doesn't construct the repositories in a
> > way that this is allowed. For RPM repos this is fine, and version
> > pinning works.
> >
> > To remove a bad version we have to proposals (and would like to hear
> > ideas on other possibilities), one that would involve symlinks and the
> > other one which purges the known bad version from our repos.
> >
> > *Symlinking*
> > When releasing we would have a "previous" and "latest" symlink that
> > would get updated as versions move forward. It would require
> > separation of versions at the URL level (all versions would no longer
> > be available in one repo).
> >
> > The URL structure would then look like:
> >
> >     debian/luminous/12.2.3/
> >     debian/luminous/previous/  (points to 12.2.5)
> >     debian/luminous/latest/   (points to 12.2.7)
> >
> > Caveats: the url structure would change from debian-luminous/ to
> > prevent breakage, and the versions would be split. For RPMs it would
> > mean a regression if someone is used to pinning, for example pinning
> > to 12.2.2 wouldn't be possible using the same url.
> >
> > Pros: Faster release times, less need to move packages around, and
> > easier to remove a bad version
> >
> >
> > *Single version removal*
> > Our tooling would need to go and remove the known bad version from the
> > repository, which would require to rebuild the repository again, so
> > that the metadata is updated with the difference in the binaries.
> >
> > Caveats: time intensive process, almost like cutting a new release
> > which takes about a day (and sometimes longer). Error prone since the
> > process wouldn't be the same (one off, just when a version needs to be
> > removed)
> >
> > Pros: all urls for download.ceph.com and its structure are kept the
> same.
> _______________________________________________
> ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
> To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
>

[-- Attachment #1.2: Type: text/html, Size: 6478 bytes --]

[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Yoann Moulin]

Hello,

> I still think this is something we should consider as users still
> experience problems:
> 
> * Impossible to 'pin' to a version. User installs 14.2.0 and 4 months
> later they add other nodes but version moved to 14.2.2
> * Impossible to use a version that is not what the latest is (e.g. if
> someone doesn't need the release from Monday, but wants the one from 6
> months ago), similar to the above

I think this could be a good idea, even if there is bugfix in next release, sometimes you don't want to fix a version in a stable cluster. I use
CEPH-ansible but I don't know if it's possible to choose a release that is not the latest.

> * When a release is underway, the repository breaks because syncing
> packages takes hours. The operation is not atomic.

Maybe build package on another server or tl least on another repo and when it's done, move all of them in one atomic operation after last tests?

So we can imagine having a prerelease channel where volunteers can use it and test and let the stable official repo untouched until it's ready.

Asking CEPH users to maintain a local repo just to keep a specific version installabled for himself is a bit hard and not what I expect from a
machine critical storage solution.

> * It is not currently possible to "remove" a bad release, in the past,
> this means cutting a new release as soon as possible, which can take
> days

This is also an important process to setup, Ceph is used for critical infrastructure, there are already plenty of tests and is really stable but
we all know how a bug can be malicious and destructive and how quickly users can lose confidence in it. Be able to remove, quickly, a buggy
package is a must I think. And this will be a guarantee of quality to your users.

> The latest issue (my fault!) was to cut a release and get the packages
> out without communicating with the release manager, which caused users
> to note there is a new version *as soon as it was up* vs, a
> process that could've not touched the 'latest' url until the
> announcement goes out.

In French, we say something like "There are two kinds of administrators, those who have made mistakes under root, and those who will do it."
(I'm a member of the first group.) But processes are here to avoid or at least reduce the impact of those mistakes. And this is why we use CEPH,
its  design can avoid some mistakes as root. Invest a few time to improve the release process may increase even more confidence in Ceph.

> If you have been affected by any of these issues (or others I didn't
> come up with), please let us know in this thread so that we can find
> some common ground and try to improve the process.

my 2 cents,

Best,

> On Tue, Jul 24, 2018 at 10:38 AM Alfredo Deza <adeza-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>>
>> Hi all,
>>
>> After the 12.2.6 release went out, we've been thinking on better ways
>> to remove a version from our repositories to prevent users from
>> upgrading/installing a known bad release.
>>
>> The way our repos are structured today means every single version of
>> the release is included in the repository. That is, for Luminous,
>> every 12.x.x version of the binaries is in the same repo. This is true
>> for both RPM and DEB repositories.
>>
>> However, the DEB repos don't allow pinning to a given version because
>> our tooling (namely reprepro) doesn't construct the repositories in a
>> way that this is allowed. For RPM repos this is fine, and version
>> pinning works.
>>
>> To remove a bad version we have to proposals (and would like to hear
>> ideas on other possibilities), one that would involve symlinks and the
>> other one which purges the known bad version from our repos.
>>
>> *Symlinking*
>> When releasing we would have a "previous" and "latest" symlink that
>> would get updated as versions move forward. It would require
>> separation of versions at the URL level (all versions would no longer
>> be available in one repo).
>>
>> The URL structure would then look like:
>>
>>     debian/luminous/12.2.3/
>>     debian/luminous/previous/  (points to 12.2.5)
>>     debian/luminous/latest/   (points to 12.2.7)
>>
>> Caveats: the url structure would change from debian-luminous/ to
>> prevent breakage, and the versions would be split. For RPMs it would
>> mean a regression if someone is used to pinning, for example pinning
>> to 12.2.2 wouldn't be possible using the same url.
>>
>> Pros: Faster release times, less need to move packages around, and
>> easier to remove a bad version
>>
>>
>> *Single version removal*
>> Our tooling would need to go and remove the known bad version from the
>> repository, which would require to rebuild the repository again, so
>> that the metadata is updated with the difference in the binaries.
>>
>> Caveats: time intensive process, almost like cutting a new release
>> which takes about a day (and sometimes longer). Error prone since the
>> process wouldn't be the same (one off, just when a version needs to be
>> removed)
>>
>> Pros: all urls for download.ceph.com and its structure are kept the same.
> _______________________________________________
> ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
> To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
> 


-- 
Yoann Moulin
EPFL IC-IT
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Ken Dreyer]

On Tue, Sep 17, 2019 at 8:03 AM Sasha Litvak
<alexander.v.litvak-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> * I am bothered with a quality of the releases of a very complex system that
> can bring down a whole house and keep it down for a while.  While I wish the
> QA would be perfect, I wonder if it would be practical to release new
> packages to a testing repo before moving it to a main one.  There is a
> chance then someone will detect a problem before it becomes a production
> issue.  Let it seat for a couple days or weeks in testing.  People who need
> new update right away or just want to test will install it and report the
> problems.  Others will not be affected.

I think it would be a good step forward to have a separate "testing"
repository. This repository would be a little more cutting-edge, and we'd copy
all the binaries over to the "main" repository location after 48 hours or
something.

This would let us all publicly test the candidate GPG-signed packages, for
example.

- Ken
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[David Turner]

[-- Attachment #1.1: Type: text/plain, Size: 3006 bytes --]

IRT a testing/cutting edge repo, the non-LTS versions of Ceph have been
removed because very few people ever used them and tested them. The
majority of people that would be using the testing repo would be people
needing a bug fix ASAP. Very few people would actually use this regularly
and its effectiveness would be almost zero in preventing problems slipping
through.

At work I haven't had a problem with which version of Ceph is being
installed because we always have local mirrors of the repo that we only
update with the upstream repos when we're ready to test a new version in
our QA environments long before we promote the version for production use.
That said, I've been bit by this multiple times in my home environment
where I've accidentally updated a server or reinstalled a server and needed
to upgrade my Ceph cluster before I could finish because it installed a
newer version of Ceph. I have had to download the entire copy of a version
from online, put it into a folder on disk, and set up a repo feeding from
that local folder to install a specific version. This would be very handy
to just use the ability in apt or yum to just specify a different version
of a package in the repo.

Problem releases have become more problematic than needed because the
packages were left the default packages after a bug was known because there
was no way to remove them from the repo. People continue to see the upgrade
and grabbing it not realizing it's a busted release. I've only seen that
happen on the ML here, but I personally will not touch a new release for at
least 2 weeks after it's been released even in my testing clusters.

On Tue, Sep 24, 2019 at 4:06 PM Ken Dreyer <kdreyer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On Tue, Sep 17, 2019 at 8:03 AM Sasha Litvak
> <alexander.v.litvak-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > * I am bothered with a quality of the releases of a very complex system
> that
> > can bring down a whole house and keep it down for a while.  While I wish
> the
> > QA would be perfect, I wonder if it would be practical to release new
> > packages to a testing repo before moving it to a main one.  There is a
> > chance then someone will detect a problem before it becomes a production
> > issue.  Let it seat for a couple days or weeks in testing.  People who
> need
> > new update right away or just want to test will install it and report the
> > problems.  Others will not be affected.
>
> I think it would be a good step forward to have a separate "testing"
> repository. This repository would be a little more cutting-edge, and we'd
> copy
> all the binaries over to the "main" repository location after 48 hours or
> something.
>
> This would let us all publicly test the candidate GPG-signed packages, for
> example.
>
> - Ken
> _______________________________________________
> ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
> To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
>

[-- Attachment #1.2: Type: text/html, Size: 3772 bytes --]

[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Janne Johansson]

[-- Attachment #1.1: Type: text/plain, Size: 1402 bytes --]

Den tis 24 sep. 2019 kl 23:35 skrev David Turner <drakonstein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:

>
> At work I haven't had a problem with which version of Ceph is being
> installed because we always have local mirrors of the repo that we only
> update with the upstream repos when we're ready to test a new version in
> our QA environments long before we promote the version for production use.
>
> Problem releases have become more problematic than needed because the
> packages were left the default packages after a bug was known because there
> was no way to remove them from the repo. People continue to see the upgrade
> and grabbing it not realizing it's a busted release. I've only seen that
> happen on the ML here, but I personally will not touch a new release for at
> least 2 weeks after it's been released even in my testing clusters.
>

So this solution (having a mirror of your own) then becomes "when should I
run mirror sync" instead of "when should I make an install that pulls
whatever is deemed current in the repo" which might help you but still
would be prone to falling into the same trap, "bad" pgks gets listed as the
latest and hence get installed, regardless of if bad means "has bugs" or
just "installed before announcement is out or before announcement+2 weeks".
The window gets smaller but not zero. 8-/

-- 
May the most significant bit of your life be positive.

[-- Attachment #1.2: Type: text/html, Size: 1901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Sasha Litvak]

[-- Attachment #1.1: Type: text/plain, Size: 2472 bytes --]

I guess for me the more crucial  questions should be answered:

1.  How can a busted release be taken out of repos (some metadata update I
hope)?
2.  Can some fix(es) be added into a test release so they can be accessed
by community and tested  / used before next general release is avaialble.
I was thinking about trying to pick some critical backports for internal
build / testing but it is not very simple for everyone.  It seems that
shaman has some test builds going on, so may be if a track issue will
mention that build xxxyz contains fixes for ticket 9999 "for testing
purposes only use at your peril", it will allow the community to test the
build and for someone who in need to get to the higher ground quickly.



On Wed, Sep 25, 2019 at 12:41 AM Janne Johansson <icepic.dz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:

> Den tis 24 sep. 2019 kl 23:35 skrev David Turner <drakonstein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>
>>
>> At work I haven't had a problem with which version of Ceph is being
>> installed because we always have local mirrors of the repo that we only
>> update with the upstream repos when we're ready to test a new version in
>> our QA environments long before we promote the version for production use.
>>
>> Problem releases have become more problematic than needed because the
>> packages were left the default packages after a bug was known because there
>> was no way to remove them from the repo. People continue to see the upgrade
>> and grabbing it not realizing it's a busted release. I've only seen that
>> happen on the ML here, but I personally will not touch a new release for at
>> least 2 weeks after it's been released even in my testing clusters.
>>
>
> So this solution (having a mirror of your own) then becomes "when should I
> run mirror sync" instead of "when should I make an install that pulls
> whatever is deemed current in the repo" which might help you but still
> would be prone to falling into the same trap, "bad" pgks gets listed as the
> latest and hence get installed, regardless of if bad means "has bugs" or
> just "installed before announcement is out or before announcement+2 weeks".
> The window gets smaller but not zero. 8-/
>
> --
> May the most significant bit of your life be positive.
> _______________________________________________
> ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
> To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org
>

[-- Attachment #1.2: Type: text/html, Size: 3483 bytes --]

[-- Attachment #2: Type: text/plain, Size: 193 bytes --]

_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[Ken Dreyer]

On Wed, Sep 25, 2019 at 1:56 PM Sasha Litvak
<alexander.v.litvak-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> I guess for me the more crucial  questions should be answered:
>

> 1.  How can a busted release be taken out of repos (some metadata
> update I hope)?

It's hard to define the word "busted" in a way that satisfies everyone.

For example, in the past, we've heard rumors that the releases are
"busted" and we need to pull them, and in reality the impact turned out
to be way smaller than expected. If we let panic reign, we would have
builds appearing and disappearing quite often.

Or to borrow a real example, what happens when the new (buggy) release
fixes a CVE, and we coordinated the disclosure of the CVE to happen on
release day? Then we're hurting some users to help others.

> 2.  Can some fix(es) be added into a test release so they can be
> accessed by community and tested  / used before next general release
> is avaialble.  I was thinking about trying to pick some critical
> backports for internal build / testing but it is not very simple for
> everyone.  It seems that shaman has some test builds going on, so may
> be if a track issue will mention that build xxxyz contains fixes for
> ticket 9999 "for testing purposes only use at your peril", it will
> allow the community to test the build and for someone who in need to
> get to the higher ground quickly.

There's certainly more to explore here. Maybe we need a defined release
criteria so things are less subjective. Or maybe increasing the
frequency and decreasing the size of releases will help. I'm interested
in lowering the barriers for contributors to test early and often so
our cutting-edge users get their builds fast and our more conservative
users get something more stable.
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org

Re: download.ceph.com repository changes

[vitalif-Y5Dpw2iYaIgvJsYlp49lxw]

The worst part about the official repository is that it lacks Debian 
packages

Also of course it would be very convenient to be able to install any 
version from the repos, not just the latest one. It's certainly possible 
with debian repos...
_______________________________________________
ceph-users mailing list -- ceph-users-a8pt6IJUokc@public.gmane.org
To unsubscribe send an email to ceph-users-leave-a8pt6IJUokc@public.gmane.org