new naming convention for building repos and binaries from branches

new naming convention for building repos and binaries from branches

[Alfredo Deza]

Hi all,

The (current) behavior with gitbuilders is that it builds every commit
of every branch in Github.

With the new CI system, although we can cope with the demand as the
Ceph project grows in contributions, we are planning on only building
branches that really need to be built.

This keeps the whole system going as fast as possible (we are
currently getting full repositories in about 1 hour) and very
efficient.

We need to come up with a naming convention for branches to be built.
At the moment we are building everything and that is preventing us
from enabling other features in the system like all the other
"flavors" (e.g. tcmalloc).

Branch names for builds will need certain restrictions. Because the
system is based on HTTP components, no HTTP reserved chars can be used
(e.g. historic/libradosgw would be a no-go). Other invalid chars are:
& , ; ? = + $ :

We could keep the implicit convention for `wip-*` or maybe something similar?

Ideas welcomed :)

Re: new naming convention for building repos and binaries from branches

[John Spray]

On Thu, Oct 6, 2016 at 5:43 PM, Alfredo Deza <adeza@redhat.com> wrote:
> Hi all,
>
> The (current) behavior with gitbuilders is that it builds every commit
> of every branch in Github.
>
> With the new CI system, although we can cope with the demand as the
> Ceph project grows in contributions, we are planning on only building
> branches that really need to be built.
>
> This keeps the whole system going as fast as possible (we are
> currently getting full repositories in about 1 hour) and very
> efficient.
>
> We need to come up with a naming convention for branches to be built.
> At the moment we are building everything and that is preventing us
> from enabling other features in the system like all the other
> "flavors" (e.g. tcmalloc).
>
> Branch names for builds will need certain restrictions. Because the
> system is based on HTTP components, no HTTP reserved chars can be used
> (e.g. historic/libradosgw would be a no-go). Other invalid chars are:
> & , ; ? = + $ :
>
> We could keep the implicit convention for `wip-*` or maybe something similar?

Building everything that starts with wip-* sounds fine to me.  But
isn't that usually everything that we push to the ceph/ceph
repository?  What is is that we're trying to avoid building?

John

> Ideas welcomed :)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new naming convention for building repos and binaries from branches

[Andrew Schoen]

>> We could keep the implicit convention for `wip-*` or maybe something similar?
>
> Building everything that starts with wip-* sounds fine to me.  But
> isn't that usually everything that we push to the ceph/ceph
> repository?  What is is that we're trying to avoid building?
>
> John

I think the idea is that we only want to build branches that we plan
to run tests against. As we add more flavors, distros and
architectures to build for the workload increases exponentially. The
system is designed to scale, but let's not build things we don't
really need.

Andrew

Re: new naming convention for building repos and binaries from branches

[Ken Dreyer]

On Thu, Oct 6, 2016 at 10:30 AM, Andrew Schoen <aschoen@redhat.com> wrote:
>>> We could keep the implicit convention for `wip-*` or maybe something similar?
>>
>> Building everything that starts with wip-* sounds fine to me.  But
>> isn't that usually everything that we push to the ceph/ceph
>> repository?  What is is that we're trying to avoid building?
>>
>> John
>
> I think the idea is that we only want to build branches that we plan
> to run tests against. As we add more flavors, distros and
> architectures to build for the workload increases exponentially. The
> system is designed to scale, but let's not build things we don't
> really need.

What branches in https://github.com/ceph/ceph are being built that do
not need to be built?

- Ken

Re: new naming convention for building repos and binaries from branches

[Sage Weil]

On Thu, 6 Oct 2016, Andrew Schoen wrote:
> >> We could keep the implicit convention for `wip-*` or maybe something similar?
> >
> > Building everything that starts with wip-* sounds fine to me.  But
> > isn't that usually everything that we push to the ceph/ceph
> > repository?  What is is that we're trying to avoid building?
> >
> > John
> 
> I think the idea is that we only want to build branches that we plan
> to run tests against. As we add more flavors, distros and
> architectures to build for the workload increases exponentially. The
> system is designed to scale, but let's not build things we don't
> really need.

I think we need a "do not build" prefix and a "build this asap" prefix. 
Maybe nobuild-* and asap-*?

sage

Re: new naming convention for building repos and binaries from branches

[Gregory Farnum]

On Thu, Oct 6, 2016 at 10:34 AM, Sage Weil <sweil@redhat.com> wrote:
> On Thu, 6 Oct 2016, Andrew Schoen wrote:
>> >> We could keep the implicit convention for `wip-*` or maybe something similar?
>> >
>> > Building everything that starts with wip-* sounds fine to me.  But
>> > isn't that usually everything that we push to the ceph/ceph
>> > repository?  What is is that we're trying to avoid building?
>> >
>> > John
>>
>> I think the idea is that we only want to build branches that we plan
>> to run tests against. As we add more flavors, distros and
>> architectures to build for the workload increases exponentially. The
>> system is designed to scale, but let's not build things we don't
>> really need.
>
> I think we need a "do not build" prefix and a "build this asap" prefix.
> Maybe nobuild-* and asap-*?

I can't imagine anybody's going to remember to stick "nobuild" in
front of things. If we're really concerned about not building
everything, it needs to be proactive. I'd really like two interfaces:
1) append "-build" to the end of a branch. This would have to be a
postfix so we don't pick up every branch that fixes some build issue
or rearranges the makefiles.
2) Have a web page or github integration we can use to say "include
this branch in the builds".

In particular what I'm thinking is, we want a way to build stuff right
away from the push for quick turnaround times. But we'll want to push
branches for RFC etc before we really care if they're widely built,
and renaming them or trying to update two refs would be a hassle.
-Greg

Re: new naming convention for building repos and binaries from branches

[Sage Weil]

On Thu, 6 Oct 2016, Gregory Farnum wrote:
> On Thu, Oct 6, 2016 at 10:34 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Thu, 6 Oct 2016, Andrew Schoen wrote:
> >> >> We could keep the implicit convention for `wip-*` or maybe something similar?
> >> >
> >> > Building everything that starts with wip-* sounds fine to me.  But
> >> > isn't that usually everything that we push to the ceph/ceph
> >> > repository?  What is is that we're trying to avoid building?
> >> >
> >> > John
> >>
> >> I think the idea is that we only want to build branches that we plan
> >> to run tests against. As we add more flavors, distros and
> >> architectures to build for the workload increases exponentially. The
> >> system is designed to scale, but let's not build things we don't
> >> really need.
> >
> > I think we need a "do not build" prefix and a "build this asap" prefix.
> > Maybe nobuild-* and asap-*?
> 
> I can't imagine anybody's going to remember to stick "nobuild" in
> front of things. If we're really concerned about not building
> everything, it needs to be proactive. I'd really like two interfaces:
> 1) append "-build" to the end of a branch. This would have to be a
> postfix so we don't pick up every branch that fixes some build issue
> or rearranges the makefiles.
> 2) Have a web page or github integration we can use to say "include
> this branch in the builds".
> 
> In particular what I'm thinking is, we want a way to build stuff right
> away from the push for quick turnaround times. But we'll want to push
> branches for RFC etc before we really care if they're widely built,
> and renaming them or trying to update two refs would be a hassle.

It might be simpler to just avoid pushing things to ceph.git that aren't 
meant to be built; that's what our personal github clone repos are usually 
for.  Which means (IMO) we only really need a "please build this branch 
before the others".  And that doesn't need to be done as part of the 
branch name.  In fact, it's probably better if it isn't, so that you can 
prioritize a branch even after it has been pushed.

But this reminds me... a while back I created a ceph-ci.git repo, the idea 
being that we could move *all* of the wip-* and random branches from 
ceph.git over there and have ceph.git only contain the "official" branches 
(like master, jewel, hammer, etc.).  CI would pull from both, and we could 
allow developers access to ceph-ci.git without giving them access to 
ceph.git.

We talked about this ages ago but never actually made any switch.  Maybe 
this is a good time to do it?

Original thread here: http://www.spinics.net/lists/ceph-devel/msg24251.html

sage

Re: new naming convention for building repos and binaries from branches

[Alfredo Deza]

On Thu, Oct 6, 2016 at 1:56 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Thu, Oct 6, 2016 at 10:34 AM, Sage Weil <sweil@redhat.com> wrote:
>> On Thu, 6 Oct 2016, Andrew Schoen wrote:
>>> >> We could keep the implicit convention for `wip-*` or maybe something similar?
>>> >
>>> > Building everything that starts with wip-* sounds fine to me.  But
>>> > isn't that usually everything that we push to the ceph/ceph
>>> > repository?

It isn't. There are a bunch of active branches that are not wip-*

>> What is is that we're trying to avoid building?

We should "build" (create binaries and repos for multiple distros) for
the branches that need to be tested out, either by
teuthology or by consuming the repo directly (e.g. ceph-deploy install
--dev=my-branch)

If you believe that every single active branch in
https://github.com/ceph/ceph/branches/active meets those requirements,
then
we should just not try to get a naming convention. I don't know if
that is the case.

>>> >
>>> > John
>>>
>>> I think the idea is that we only want to build branches that we plan
>>> to run tests against. As we add more flavors, distros and
>>> architectures to build for the workload increases exponentially. The
>>> system is designed to scale, but let's not build things we don't
>>> really need.
>>
>> I think we need a "do not build" prefix and a "build this asap" prefix.
>> Maybe nobuild-* and asap-*?
>
> I can't imagine anybody's going to remember to stick "nobuild" in
> front of things.

Yep, I agree, we can't do it that way.

>If we're really concerned about not building
> everything, it needs to be proactive. I'd really like two interfaces:
> 1) append "-build" to the end of a branch. This would have to be a
> postfix so we don't pick up every branch that fixes some build issue
> or rearranges the makefiles.
> 2) Have a web page or github integration we can use to say "include
> this branch in the builds".
>
> In particular what I'm thinking is, we want a way to build stuff right
> away from the push for quick turnaround times.

That is the case *today* and will not change later. A dev might
experience a slight delay but the whole idea
is that even under heavy load the waiting time will not be 12 hours
(unless everything is burning down?)

>But we'll want to push
> branches for RFC etc before we really care if they're widely built,
> and renaming them or trying to update two refs would be a hassle.
> -Greg

The current caveat we have today is that the machines that produce the
repos are at about 60% full (they use 500gb drives each, we have 4
currently). Turning on 1 extra flavor would get us
out of space on at least two pretty quickly.

*We can add more to the pool*. So this is not really a make-or-break
situation, we can scale. But if there is a way to be more efficient by
building stuff that we really care, then lets
at least try.

Re: new naming convention for building repos and binaries from branches

[Marcus Watts]

Someone in the middle said:
...
> >If we're really concerned about not building
...

I think that's key.  We build a lot more stuff, a lot
more often, than I think most people actually need or use.
Which means we wind up waiting for builds to complete
that nobody will ever use.

I think the natural progression of things is:

stage 1.	coded up, ready to share, not ready to build or test.
		(might be being built out of stream,  etc.)
		early stage of collaboration, code review, etc.
		[ I think people currently do this in various
		different ways... ]
stage 2.	ready to test / build / for a LIMITED set of
		architectures (often just one.)
		start of testing, don't need all architectures yet.
		OR, failed on one architecture, want to just test *that*
		architecture.  [ We don't currently have a way to
		do this, but I bet at least half of our builds could
		work this way. ]
stage 3.	ready to test / build / for ALL architectures.
		about to commit fix to master / general distribution,
		known to work on one architecture, but want to actually
		do the thorough test on everything.

Doing this by special branch names seems a bit clumsy to me.  I suppose
we could have "nobuild-FOO", "ubuntuonly-FOO", "buildeverywhere-FOO".
But it seems more natural to me to have some kind of prefix
matching/configuration thing that the builder could use to the same
effect.  Then progressing wip-FOO from stage 2 to stage 3 could be just
a matter of the programmer adjusting the filter for the "wip-foo" from
"ubuntuonly" to "buildeverywhere", then the automation building the
identical source on the other architectures (and *not* rebuilding on
the original architecture for which perfectly good builds don't need to
be repeated.)

					-Marcus Watts

Re: new naming convention for building repos and binaries from branches

[Andrew Schoen]

>
> I think the natural progression of things is:
>
> stage 1.        coded up, ready to share, not ready to build or test.
>                 (might be being built out of stream,  etc.)
>                 early stage of collaboration, code review, etc.
>                 [ I think people currently do this in various
>                 different ways... ]
> stage 2.        ready to test / build / for a LIMITED set of
>                 architectures (often just one.)
>                 start of testing, don't need all architectures yet.
>                 OR, failed on one architecture, want to just test *that*
>                 architecture.  [ We don't currently have a way to
>                 do this, but I bet at least half of our builds could
>                 work this way. ]
> stage 3.        ready to test / build / for ALL architectures.
>                 about to commit fix to master / general distribution,
>                 known to work on one architecture, but want to actually
>                 do the thorough test on everything.
>

Stage 1 and 2 could be accomplished by started a build with kraken in
IRC or by using the jenkins web UI (I'd recommend kraken though). By
default we are currently building for xenial, trusty and centos7 on
x86_64 for every push to ceph.git. That's look something like this:

!ci build ceph-dev BRANCH=$branch

You could also limit the distros:

!ci build ceph-dev BRANCH=$branch DISTROS="xenial"

It's stage 3 that I think we need the naming convention for.

Re: new naming convention for building repos and binaries from branches

[Andrew Schoen]

> I think we need a "do not build" prefix and a "build this asap" prefix.
> Maybe nobuild-* and asap-*?
>

Is the 'asap-*' want here because of the typical long wait on the gitbuilders?

The way we're configured now we can build 10 branches at a time, all
of those branches building for 3 distros. I haven't seen a build take
longer than an hour or so. It'd be easy to scale that up as well, but
I don't see the queue ever being backed up at our current load.

Re: new naming convention for building repos and binaries from branches

[Kamble, Nitin A]

> On Oct 7, 2016, at 7:27 AM, Andrew Schoen <aschoen@redhat.com> wrote:
> 
>> I think we need a "do not build" prefix and a "build this asap" prefix.
>> Maybe nobuild-* and asap-*?
>> 
> 
> Is the 'asap-*' want here because of the typical long wait on the gitbuilders?
> 
> The way we're configured now we can build 10 branches at a time, all
> of those branches building for 3 distros. I haven't seen a build take
> longer than an hour or so. It'd be easy to scale that up as well, but
> I don't see the queue ever being backed up at our current load.

Most of the builds will be building almost the same files, I hope you guys are
using build caching mechanisms such as ccache to accelerate the build time.

Nitin

Re: new naming convention for building repos and binaries from branches

[Alfredo Deza]

On Fri, Oct 7, 2016 at 3:16 PM, Kamble, Nitin A
<Nitin.Kamble@teradata.com> wrote:
>
>> On Oct 7, 2016, at 7:27 AM, Andrew Schoen <aschoen@redhat.com> wrote:
>>
>>> I think we need a "do not build" prefix and a "build this asap" prefix.
>>> Maybe nobuild-* and asap-*?

What we are going to start with *today* is to build every branch that
is prefixed with "wip-" since
that is the implicit/historical naming convention for branches.

We will include all the release branches (e.g. hammer, infernalis,
jewel, kraken) as well
as master.

This will allow the "do not build this" branches to just be anything
not prefixed with "wip-" (there are already a few of those
today, like dnm- and testing-)

>>>
>>
>> Is the 'asap-*' want here because of the typical long wait on the gitbuilders?
>>
>> The way we're configured now we can build 10 branches at a time, all
>> of those branches building for 3 distros. I haven't seen a build take
>> longer than an hour or so. It'd be easy to scale that up as well, but
>> I don't see the queue ever being backed up at our current load.
>
> Most of the builds will be building almost the same files, I hope you guys are
> using build caching mechanisms such as ccache to accelerate the build time.
>
> Nitin
>
>

Re: new naming convention for building repos and binaries from branches

[Sage Weil]

On Thu, 13 Oct 2016, Alfredo Deza wrote:
> On Fri, Oct 7, 2016 at 3:16 PM, Kamble, Nitin A
> <Nitin.Kamble@teradata.com> wrote:
> >
> >> On Oct 7, 2016, at 7:27 AM, Andrew Schoen <aschoen@redhat.com> wrote:
> >>
> >>> I think we need a "do not build" prefix and a "build this asap" prefix.
> >>> Maybe nobuild-* and asap-*?
> 
> What we are going to start with *today* is to build every branch that
> is prefixed with "wip-" since
> that is the implicit/historical naming convention for branches.
> 
> We will include all the release branches (e.g. hammer, infernalis,
> jewel, kraken) as well
> as master.
> 
> This will allow the "do not build this" branches to just be anything
> not prefixed with "wip-" (there are already a few of those
> today, like dnm- and testing-)

That seems okay for now.  I think this might have gotten lost in the 
earlier thread, though:

Why don't we take this opportunity to create a ceph-ci.git and lock down 
ceph.git?

 - ceph.git would contain only official branches (master, jewel, etc.)
 - ceph-ci.git would have all the branches that need to get build for 
   ci purposes.
 - ci would build everything in ceph.git and ceph-ci.git that doesn't have 
   a / in it
 - if you don't want a branch built, don't push it to either one; push it 
   to your private clone (which is what you should be doing anyway!)
 - restrict access to ceph.git to core developers (and trusted robots)
 - grant access to ceph-ci to a broader set of developers (and robots)

Right now, I think this requires that

1) shaman and/or jenkins build branches in both ceph.git and ceph-ci.git

and then at our leisure,

2) we delete or move all non-official branches over to ceph-gi.git 
(prefixing historical branches with old/ or similar), and
3) we restrict ceph.git access.

?
sage

Re: new naming convention for building repos and binaries from branches

[Gregory Farnum]

+1 this idea

On Fri, Oct 14, 2016 at 12:03 PM, Sage Weil <sage@newdream.net> wrote:
> That seems okay for now.  I think this might have gotten lost in the
> earlier thread, though:
>
> Why don't we take this opportunity to create a ceph-ci.git and lock down
> ceph.git?
>
>  - ceph.git would contain only official branches (master, jewel, etc.)
>  - ceph-ci.git would have all the branches that need to get build for
>    ci purposes.
>  - ci would build everything in ceph.git and ceph-ci.git that doesn't have
>    a / in it
>  - if you don't want a branch built, don't push it to either one; push it
>    to your private clone (which is what you should be doing anyway!)
>  - restrict access to ceph.git to core developers (and trusted robots)
>  - grant access to ceph-ci to a broader set of developers (and robots)

Except for this. Until/unless we find something requiring ceph.git
access, we should lock down push access and only let code go in via
Github PRs. Give ceph-ci access to everybody who's allowed in to
sepia. Why open any gates preemptively if we don't need to? :)
-Greg

>
> Right now, I think this requires that
>
> 1) shaman and/or jenkins build branches in both ceph.git and ceph-ci.git
>
> and then at our leisure,
>
> 2) we delete or move all non-official branches over to ceph-gi.git
> (prefixing historical branches with old/ or similar), and
> 3) we restrict ceph.git access.
>
> ?
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new naming convention for building repos and binaries from branches

[Alfredo Deza]

On Fri, Oct 14, 2016 at 3:53 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> +1 this idea
>
> On Fri, Oct 14, 2016 at 12:03 PM, Sage Weil <sage@newdream.net> wrote:
>> That seems okay for now.  I think this might have gotten lost in the
>> earlier thread, though:
>>
>> Why don't we take this opportunity to create a ceph-ci.git and lock down
>> ceph.git?
>>
>>  - ceph.git would contain only official branches (master, jewel, etc.)
>>  - ceph-ci.git would have all the branches that need to get build for
>>    ci purposes.
>>  - ci would build everything in ceph.git and ceph-ci.git that doesn't have
>>    a / in it
>>  - if you don't want a branch built, don't push it to either one; push it
>>    to your private clone (which is what you should be doing anyway!)

But nobody does and it is not enforceable :(

>>  - restrict access to ceph.git to core developers (and trusted robots)
>>  - grant access to ceph-ci to a broader set of developers (and robots)
>
> Except for this. Until/unless we find something requiring ceph.git
> access, we should lock down push access and only let code go in via
> Github PRs. Give ceph-ci access to everybody who's allowed in to
> sepia. Why open any gates preemptively if we don't need to? :)

This sounds very similar to the problem we have today. Building
everything in ceph.git and ceph-ci.git would
get us right where we are right now (although not from the start).

Not that we can't adapt these rules and settings as we move forward though.

> -Greg
>
>>
>> Right now, I think this requires that
>>
>> 1) shaman and/or jenkins build branches in both ceph.git and ceph-ci.git
>>
>> and then at our leisure,
>>
>> 2) we delete or move all non-official branches over to ceph-gi.git
>> (prefixing historical branches with old/ or similar), and
>> 3) we restrict ceph.git access.
>>
>> ?
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new naming convention for building repos and binaries from branches

[Sage Weil]

On Mon, 17 Oct 2016, Alfredo Deza wrote:
> On Fri, Oct 14, 2016 at 3:53 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> > +1 this idea
> >
> > On Fri, Oct 14, 2016 at 12:03 PM, Sage Weil <sage@newdream.net> wrote:
> >> That seems okay for now.  I think this might have gotten lost in the
> >> earlier thread, though:
> >>
> >> Why don't we take this opportunity to create a ceph-ci.git and lock down
> >> ceph.git?
> >>
> >>  - ceph.git would contain only official branches (master, jewel, etc.)
> >>  - ceph-ci.git would have all the branches that need to get build for
> >>    ci purposes.
> >>  - ci would build everything in ceph.git and ceph-ci.git that doesn't have
> >>    a / in it
> >>  - if you don't want a branch built, don't push it to either one; push it
> >>    to your private clone (which is what you should be doing anyway!)
> 
> But nobody does and it is not enforceable :(

We can restrict access to ceph.git, which will force people to change 
their habits... and hope that they do so for the better.

> >>  - restrict access to ceph.git to core developers (and trusted robots)
> >>  - grant access to ceph-ci to a broader set of developers (and robots)
> >
> > Except for this. Until/unless we find something requiring ceph.git
> > access, we should lock down push access and only let code go in via
> > Github PRs. Give ceph-ci access to everybody who's allowed in to
> > sepia. Why open any gates preemptively if we don't need to? :)
> 
> This sounds very similar to the problem we have today. Building
> everything in ceph.git and ceph-ci.git would
> get us right where we are right now (although not from the start).
> 
> Not that we can't adapt these rules and settings as we move forward though.

Yeah.

One other thought I had: we can move all the current extraneous branches 
in ceph.git to ceph-archive.git or something so that ceph-ci.git starts 
out fresh and clean...

sage