Fixed URL for latest branch shaman build?

Fixed URL for latest branch shaman build?

[Dan van der Ster]

Hi all,

tl;dr: Is there a fixed URL which points to the latest jewel build on shaman?

We used to use gitbuilder.ceph.com (via a caching proxy) to
occasionally run the latest build for a given branch. This was
convenient -- we'd just set a yum repo with url

   http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/x86_64/

And it would always point to the build for the latest commit.

Now with shaman, unless I'm mistaken, we have a unique repo for each
build. E.g. here's the latest build's repo:

https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/x86_64/

So I think what I'm asking for is a symlink 'latest' which we can use like this:

  https://1.chacra.ceph.com/r/ceph/jewel/latest/centos/7/flavors/default/x86_64/

Is there someway to do this with shaman?

Cheers, Dan

Re: Fixed URL for latest branch shaman build?

[Gregory Farnum]

On Mon, Feb 27, 2017 at 6:21 AM, Dan van der Ster <dan@vanderster.com> wrote:
> Hi all,
>
> tl;dr: Is there a fixed URL which points to the latest jewel build on shaman?
>
> We used to use gitbuilder.ceph.com (via a caching proxy) to
> occasionally run the latest build for a given branch. This was
> convenient -- we'd just set a yum repo with url
>
>    http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/x86_64/
>
> And it would always point to the build for the latest commit.
>
> Now with shaman, unless I'm mistaken, we have a unique repo for each
> build. E.g. here's the latest build's repo:
>
> https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/x86_64/
>
> So I think what I'm asking for is a symlink 'latest' which we can use like this:
>
>   https://1.chacra.ceph.com/r/ceph/jewel/latest/centos/7/flavors/default/x86_64/
>
> Is there someway to do this with shaman?

+Alfredo,Andrew

Re: Fixed URL for latest branch shaman build?

[Andrew Schoen]

On Mon, Feb 27, 2017 at 8:21 AM, Dan van der Ster <dan@vanderster.com> wrote:
> Hi all,
>
> tl;dr: Is there a fixed URL which points to the latest jewel build on shaman?

There is, but it doesn't work exactly like how this has worked in the
past with the gitbuilders. The new system uses multiple repo builder
nodes (chacra) and reports those back to shaman.ceph.com. Because of
this, the latest repo for jewel might not always exist on the same
chacra node.

We've setup an API endpoint on shaman.ceph.com that will redirect you
to a repo file definition for the 'latest' built repo of your choice.

For example, this url will always redirect to an endpoint that will
print out plain text that looks like either a deb or rpm repo.

$ curl -L https://shaman.ceph.com/api/repos/ceph/jewel/latest/centos/7/repo/
[ceph]
name=ceph packages for \$basearch
baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/\$basearch
enabled=1
gpgcheck=0
type=rpm-md

[ceph-noarch]
name=ceph noarch packages
baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md

[ceph-source]
name=ceph source packages
baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md

You can modify that url to change the branch, release, distro and
distro version values for the repo you want. Using 'latest' in the
sha1 position will always give you the latest sha1 built that shaman
knows about.

I realize this is a bit more work for any CI system that used to pull
down the latest from gitbuilder, but it's the best we have currently
to deal with a variable amount of chacra builders for these repos.

Thanks,
Andrew

Re: Fixed URL for latest branch shaman build?

[Dan van der Ster]

On Mon, Feb 27, 2017 at 5:32 PM, Andrew Schoen <aschoen@redhat.com> wrote:
> On Mon, Feb 27, 2017 at 8:21 AM, Dan van der Ster <dan@vanderster.com> wrote:
>> Hi all,
>>
>> tl;dr: Is there a fixed URL which points to the latest jewel build on shaman?
>
> There is, but it doesn't work exactly like how this has worked in the
> past with the gitbuilders. The new system uses multiple repo builder
> nodes (chacra) and reports those back to shaman.ceph.com. Because of
> this, the latest repo for jewel might not always exist on the same
> chacra node.
>
> We've setup an API endpoint on shaman.ceph.com that will redirect you
> to a repo file definition for the 'latest' built repo of your choice.
>
> For example, this url will always redirect to an endpoint that will
> print out plain text that looks like either a deb or rpm repo.
>
> $ curl -L https://shaman.ceph.com/api/repos/ceph/jewel/latest/centos/7/repo/
> [ceph]
> name=ceph packages for \$basearch
> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/\$basearch
> enabled=1
> gpgcheck=0
> type=rpm-md
>
> [ceph-noarch]
> name=ceph noarch packages
> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/noarch
> enabled=1
> gpgcheck=0
> type=rpm-md
>
> [ceph-source]
> name=ceph source packages
> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/SRPMS
> enabled=1
> gpgcheck=0
> type=rpm-md
>
> You can modify that url to change the branch, release, distro and
> distro version values for the repo you want. Using 'latest' in the
> sha1 position will always give you the latest sha1 built that shaman
> knows about.
>
> I realize this is a bit more work for any CI system that used to pull
> down the latest from gitbuilder, but it's the best we have currently
> to deal with a variable amount of chacra builders for these repos.

Thanks -- this should work for us.

Is there a way to be specific about which flavor? We don't want the
notcmalloc jewel builds, for example.

--
Dan


>
> Thanks,
> Andrew

Re: Fixed URL for latest branch shaman build?

[Andrew Schoen]

> Is there a way to be specific about which flavor? We don't want the
> notcmalloc jewel builds, for example.

Yes, you can also include the flavor in the url. If you do not include
it, you'll get the 'default' or unflavored build.

https://shaman.ceph.com/api/repos/ceph/jewel/latest/centos/7/flavors/notcmalloc/repo/

vs.

https://shaman.ceph.com/api/repos/ceph/jewel/latest/centos/7/flavors/default/repo/

- Andrew

Re: Fixed URL for latest branch shaman build?

[Ken Dreyer]

Dan if you have ideas for how we could best expose this in the UI,
please comment in https://github.com/ceph/shaman/issues/88 !

- Ken

On Mon, Feb 27, 2017 at 9:42 AM, Dan van der Ster <dan@vanderster.com> wrote:
> On Mon, Feb 27, 2017 at 5:32 PM, Andrew Schoen <aschoen@redhat.com> wrote:
>> On Mon, Feb 27, 2017 at 8:21 AM, Dan van der Ster <dan@vanderster.com> wrote:
>>> Hi all,
>>>
>>> tl;dr: Is there a fixed URL which points to the latest jewel build on shaman?
>>
>> There is, but it doesn't work exactly like how this has worked in the
>> past with the gitbuilders. The new system uses multiple repo builder
>> nodes (chacra) and reports those back to shaman.ceph.com. Because of
>> this, the latest repo for jewel might not always exist on the same
>> chacra node.
>>
>> We've setup an API endpoint on shaman.ceph.com that will redirect you
>> to a repo file definition for the 'latest' built repo of your choice.
>>
>> For example, this url will always redirect to an endpoint that will
>> print out plain text that looks like either a deb or rpm repo.
>>
>> $ curl -L https://shaman.ceph.com/api/repos/ceph/jewel/latest/centos/7/repo/
>> [ceph]
>> name=ceph packages for \$basearch
>> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/\$basearch
>> enabled=1
>> gpgcheck=0
>> type=rpm-md
>>
>> [ceph-noarch]
>> name=ceph noarch packages
>> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/noarch
>> enabled=1
>> gpgcheck=0
>> type=rpm-md
>>
>> [ceph-source]
>> name=ceph source packages
>> baseurl=https://1.chacra.ceph.com/r/ceph/jewel/ac3ba2adcd21ac011ad556ac4506623e61fbe696/centos/7/flavors/default/SRPMS
>> enabled=1
>> gpgcheck=0
>> type=rpm-md
>>
>> You can modify that url to change the branch, release, distro and
>> distro version values for the repo you want. Using 'latest' in the
>> sha1 position will always give you the latest sha1 built that shaman
>> knows about.
>>
>> I realize this is a bit more work for any CI system that used to pull
>> down the latest from gitbuilder, but it's the best we have currently
>> to deal with a variable amount of chacra builders for these repos.
>
> Thanks -- this should work for us.
>
> Is there a way to be specific about which flavor? We don't want the
> notcmalloc jewel builds, for example.
>
> --
> Dan
>
>
>>
>> Thanks,
>> Andrew
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html