Some gitbuilders not working

Some gitbuilders not working

[David Zafman]

We are seeing gitbuilder failures.  This is what I saw on one.

error: Failed build dependencies:
xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64

Re: Some gitbuilders not working

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]


Hi David,

On 09/01/2015 07:55, David Zafman wrote:
> 
> We are seeing gitbuilder failures.  This is what I saw on one.
>
> error: Failed build dependencies:
> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64

This is because https://github.com/ceph/ceph/pull/3036 was recently merged. However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet about a month ago. Maybe some gitbuilder machines were not updated accordingly. If I'm not mistaken this is a manual process but I'm not sure exactly how to do it myself.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Some gitbuilders not working

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

I manually installed xmlstarlet on gitbuilder-ceph-rpm-rhel7-amd64-basic. That's hopefully going to help until it's properly installed.

On 09/01/2015 11:33, Loic Dachary wrote:
> 
> Hi David,
> 
> On 09/01/2015 07:55, David Zafman wrote:
>>
>> We are seeing gitbuilder failures.  This is what I saw on one.
>>
>> error: Failed build dependencies:
>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
> 
> This is because https://github.com/ceph/ceph/pull/3036 was recently merged. However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet about a month ago. Maybe some gitbuilder machines were not updated accordingly. If I'm not mistaken this is a manual process but I'm not sure exactly how to do it myself.
> 
> Cheers
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Some gitbuilders not working

[Ken Dreyer]

On 01/09/2015 03:33 AM, Loic Dachary wrote:
> On 09/01/2015 07:55, David Zafman wrote:
>>
>> We are seeing gitbuilder failures.  This is what I saw on one.
>>
>> error: Failed build dependencies:
>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
> 
> This is because https://github.com/ceph/ceph/pull/3036 was recently merged.
> However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet
> about a month ago. Maybe some gitbuilder machines were not updated
> accordingly. If I'm not mistaken this is a manual process but I'm not sure
> exactly how to do it myself.

I'm not entirely sure of the SOP to apply fabfile changes across our
infrastructure, either. Is there a simple "push this fabfile change
everywhere" command for gitbuilder?

In the long term we'll want to get to a point where we're building
Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can
resolve these sort of things automatically without having to deal with
installing new BuildRequires by hand.

- Ken

[1] http://fedoraproject.org/wiki/Projects/Mock

Re: Some gitbuilders not working

[Sage Weil]

On Fri, 9 Jan 2015, Ken Dreyer wrote:
> On 01/09/2015 03:33 AM, Loic Dachary wrote:
> > On 09/01/2015 07:55, David Zafman wrote:
> >>
> >> We are seeing gitbuilder failures.  This is what I saw on one.
> >>
> >> error: Failed build dependencies:
> >> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
> > 
> > This is because https://github.com/ceph/ceph/pull/3036 was recently merged.
> > However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet
> > about a month ago. Maybe some gitbuilder machines were not updated
> > accordingly. If I'm not mistaken this is a manual process but I'm not sure
> > exactly how to do it myself.
> 
> I'm not entirely sure of the SOP to apply fabfile changes across our
> infrastructure, either. Is there a simple "push this fabfile change
> everywhere" command for gitbuilder?

Not really, but it is high time we established one.

At the top of the fabfile all of the targets are defined, so that if you 
run bin/fab w/o arguments it should run on *all* of them. Unfortunately 
IIRC it's a bit disruptive as it restarts the autobuild service which will 
interrupt and restart the current build.

But, that list isn't well-maintained, so it doesn't map to the current set 
of gitbuilders.  That's probably pretty easy to fix.  Even if we 
do, though, I think fab isn't super great about skipping targets that are 
down?  Not sure.

In any case, the way I've run it in practice is on specific targets taht 
need updating, like so

 bin/fab gitbuilder_auto:host=user@foo

(or something like that, I forget the exact syntax).  Often on a 
host that isn't in the master list.

More frequently the procedure has been:

 - add a dependency to the fabfile for future builders
 - manually install the package on the existing ones to avoid the above 
pain

which mostly works but doesn't regularly verify that the fabfile is 
sufficient and correct.
 
> In the long term we'll want to get to a point where we're building
> Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can
> resolve these sort of things automatically without having to deal with
> installing new BuildRequires by hand.

That sounds great, although I wonder if it is slow?  We used to do 
all of our debian builds with pbuilder which does a similar thing, 
but it is way slow because it reinstalls deps on every run.  We moved 
to doing native dpkg builds because time is of the essence for test 
builds and use pbuilder only for releases...

sage

Re: Some gitbuilders not working

[Ken Dreyer]

On 01/09/2015 08:16 AM, Sage Weil wrote:
> On Fri, 9 Jan 2015, Ken Dreyer wrote:
>> On 01/09/2015 03:33 AM, Loic Dachary wrote:
>>> On 09/01/2015 07:55, David Zafman wrote:
>>>>
>>>> We are seeing gitbuilder failures.  This is what I saw on one.
>>>>
>>>> error: Failed build dependencies:
>>>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
>>>
>>> This is because https://github.com/ceph/ceph/pull/3036 was recently merged.
>>> However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet
>>> about a month ago. Maybe some gitbuilder machines were not updated
>>> accordingly. If I'm not mistaken this is a manual process but I'm not sure
>>> exactly how to do it myself.
>>
>> I'm not entirely sure of the SOP to apply fabfile changes across our
>> infrastructure, either. Is there a simple "push this fabfile change
>> everywhere" command for gitbuilder?
> 
> Not really, but it is high time we established one.
> 
> At the top of the fabfile all of the targets are defined, so that if you 
> run bin/fab w/o arguments it should run on *all* of them. Unfortunately 
> IIRC it's a bit disruptive as it restarts the autobuild service which will 
> interrupt and restart the current build.
> 
> But, that list isn't well-maintained, so it doesn't map to the current set 
> of gitbuilders.  That's probably pretty easy to fix.  Even if we 
> do, though, I think fab isn't super great about skipping targets that are 
> down?  Not sure.
> 
> In any case, the way I've run it in practice is on specific targets taht 
> need updating, like so
> 
>  bin/fab gitbuilder_auto:host=user@foo
> 
> (or something like that, I forget the exact syntax).  Often on a 
> host that isn't in the master list.
> 
> More frequently the procedure has been:
> 
>  - add a dependency to the fabfile for future builders
>  - manually install the package on the existing ones to avoid the above 
> pain
> 
> which mostly works but doesn't regularly verify that the fabfile is 
> sufficient and correct.
>  
>> In the long term we'll want to get to a point where we're building
>> Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can
>> resolve these sort of things automatically without having to deal with
>> installing new BuildRequires by hand.
> 
> That sounds great, although I wonder if it is slow?  We used to do 
> all of our debian builds with pbuilder which does a similar thing, 
> but it is way slow because it reinstalls deps on every run.  We moved 
> to doing native dpkg builds because time is of the essence for test 
> builds and use pbuilder only for releases...

It sounds like we should research exactly how much time mock and
pbuilder would add. Mock implements caching at a lot of levels, though
it's true that it won't be as fast as just running plain rpmbuild. We
should get some hard numbers so we know how painful the speed hit would be.

I'm weighing the time that mock would slow down the builds against the
time and energy that it takes to
 - Duplicate the BuildRequires in autobuild-ceph.git
 - Review the change for correctness and merge the autobuild-ceph.git
   pull request
 - Deploy the autobuild-ceph.git changes out to the appropriate
   gitbuilders

And then there's the "people stuff", like comprehending that all of the
above is even necessary, or communicating about this on mailing lists
like this thread, etc.

The fact that pbuilders are introduced very late (only during releases)
means that we encounter errors during a release day when Alfredo or I
have tried to get a release out the door. The pbuilder stuff isn't
touched until we try to do a release, and in the mean time, it breaks
because it's not getting continually tested. We're paying the "slow"
penalty in other areas down the road.

I'm not looking to purposefully slow down other people's work, so I'll
continue to mull over how we can balance speed with correctness.

- Ken

Re: Some gitbuilders not working

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4969 bytes --]

Hi,

On 14/01/2015 17:56, Ken Dreyer wrote:
> On 01/09/2015 08:16 AM, Sage Weil wrote:
>> On Fri, 9 Jan 2015, Ken Dreyer wrote:
>>> On 01/09/2015 03:33 AM, Loic Dachary wrote:
>>>> On 09/01/2015 07:55, David Zafman wrote:
>>>>>
>>>>> We are seeing gitbuilder failures.  This is what I saw on one.
>>>>>
>>>>> error: Failed build dependencies:
>>>>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
>>>>
>>>> This is because https://github.com/ceph/ceph/pull/3036 was recently merged.
>>>> However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet
>>>> about a month ago. Maybe some gitbuilder machines were not updated
>>>> accordingly. If I'm not mistaken this is a manual process but I'm not sure
>>>> exactly how to do it myself.
>>>
>>> I'm not entirely sure of the SOP to apply fabfile changes across our
>>> infrastructure, either. Is there a simple "push this fabfile change
>>> everywhere" command for gitbuilder?
>>
>> Not really, but it is high time we established one.
>>
>> At the top of the fabfile all of the targets are defined, so that if you 
>> run bin/fab w/o arguments it should run on *all* of them. Unfortunately 
>> IIRC it's a bit disruptive as it restarts the autobuild service which will 
>> interrupt and restart the current build.
>>
>> But, that list isn't well-maintained, so it doesn't map to the current set 
>> of gitbuilders.  That's probably pretty easy to fix.  Even if we 
>> do, though, I think fab isn't super great about skipping targets that are 
>> down?  Not sure.
>>
>> In any case, the way I've run it in practice is on specific targets taht 
>> need updating, like so
>>
>>  bin/fab gitbuilder_auto:host=user@foo
>>
>> (or something like that, I forget the exact syntax).  Often on a 
>> host that isn't in the master list.
>>
>> More frequently the procedure has been:
>>
>>  - add a dependency to the fabfile for future builders
>>  - manually install the package on the existing ones to avoid the above 
>> pain
>>
>> which mostly works but doesn't regularly verify that the fabfile is 
>> sufficient and correct.
>>  
>>> In the long term we'll want to get to a point where we're building
>>> Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can
>>> resolve these sort of things automatically without having to deal with
>>> installing new BuildRequires by hand.
>>
>> That sounds great, although I wonder if it is slow?  We used to do 
>> all of our debian builds with pbuilder which does a similar thing, 
>> but it is way slow because it reinstalls deps on every run.  We moved 
>> to doing native dpkg builds because time is of the essence for test 
>> builds and use pbuilder only for releases...
> 
> It sounds like we should research exactly how much time mock and
> pbuilder would add. Mock implements caching at a lot of levels, though
> it's true that it won't be as fast as just running plain rpmbuild. We
> should get some hard numbers so we know how painful the speed hit would be.
> 
> I'm weighing the time that mock would slow down the builds against the
> time and energy that it takes to
>  - Duplicate the BuildRequires in autobuild-ceph.git
>  - Review the change for correctness and merge the autobuild-ceph.git
>    pull request
>  - Deploy the autobuild-ceph.git changes out to the appropriate
>    gitbuilders

It could be changed to use https://github.com/ceph/ceph/blob/master/install-deps.sh instead.

> And then there's the "people stuff", like comprehending that all of the
> above is even necessary, or communicating about this on mailing lists
> like this thread, etc.
> 
> The fact that pbuilders are introduced very late (only during releases)
> means that we encounter errors during a release day when Alfredo or I
> have tried to get a release out the door. The pbuilder stuff isn't
> touched until we try to do a release, and in the mean time, it breaks
> because it's not getting continually tested. We're paying the "slow"
> penalty in other areas down the road.
> 
> I'm not looking to purposefully slow down other people's work, so I'll
> continue to mull over how we can balance speed with correctness.

IMHO the overhead of mock is so small that it can be ignored. pbuilder is much more expensive (because of the tarbal / tree removal) and there is no way to reduce that overhead, IIRC.

There also is a need to run make check on each platform instead of a small subset (ubuntu 12.04 + 14.04 on amd64 + i386 only). That can only be conveniently done within a container (the pbuilder / mock chroot isolation is not good enough). 

We could combine the two: move the current packaging process to use containers instead of mock / pbuilder. It has the advantage of being a single method cross distributions and allowing more comprehensive checks (make check) during packaging.

My 2cts ;-)

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Some gitbuilders not working

[Sage Weil]

On Wed, 14 Jan 2015, Ken Dreyer wrote:
> On 01/09/2015 08:16 AM, Sage Weil wrote:
> > On Fri, 9 Jan 2015, Ken Dreyer wrote:
> >> On 01/09/2015 03:33 AM, Loic Dachary wrote:
> >>> On 09/01/2015 07:55, David Zafman wrote:
> >>>>
> >>>> We are seeing gitbuilder failures.  This is what I saw on one.
> >>>>
> >>>> error: Failed build dependencies:
> >>>> xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64
> >>>
> >>> This is because https://github.com/ceph/ceph/pull/3036 was recently merged.
> >>> However, https://github.com/ceph/autobuild-ceph/pull/19 added xmlstarlet
> >>> about a month ago. Maybe some gitbuilder machines were not updated
> >>> accordingly. If I'm not mistaken this is a manual process but I'm not sure
> >>> exactly how to do it myself.
> >>
> >> I'm not entirely sure of the SOP to apply fabfile changes across our
> >> infrastructure, either. Is there a simple "push this fabfile change
> >> everywhere" command for gitbuilder?
> > 
> > Not really, but it is high time we established one.
> > 
> > At the top of the fabfile all of the targets are defined, so that if you 
> > run bin/fab w/o arguments it should run on *all* of them. Unfortunately 
> > IIRC it's a bit disruptive as it restarts the autobuild service which will 
> > interrupt and restart the current build.
> > 
> > But, that list isn't well-maintained, so it doesn't map to the current set 
> > of gitbuilders.  That's probably pretty easy to fix.  Even if we 
> > do, though, I think fab isn't super great about skipping targets that are 
> > down?  Not sure.
> > 
> > In any case, the way I've run it in practice is on specific targets taht 
> > need updating, like so
> > 
> >  bin/fab gitbuilder_auto:host=user@foo
> > 
> > (or something like that, I forget the exact syntax).  Often on a 
> > host that isn't in the master list.
> > 
> > More frequently the procedure has been:
> > 
> >  - add a dependency to the fabfile for future builders
> >  - manually install the package on the existing ones to avoid the above 
> > pain
> > 
> > which mostly works but doesn't regularly verify that the fabfile is 
> > sufficient and correct.
> >  
> >> In the long term we'll want to get to a point where we're building
> >> Ceph's RPM packages using mock [1] instead of rpmbuild so that mock can
> >> resolve these sort of things automatically without having to deal with
> >> installing new BuildRequires by hand.
> > 
> > That sounds great, although I wonder if it is slow?  We used to do 
> > all of our debian builds with pbuilder which does a similar thing, 
> > but it is way slow because it reinstalls deps on every run.  We moved 
> > to doing native dpkg builds because time is of the essence for test 
> > builds and use pbuilder only for releases...
> 
> It sounds like we should research exactly how much time mock and
> pbuilder would add. Mock implements caching at a lot of levels, though
> it's true that it won't be as fast as just running plain rpmbuild. We
> should get some hard numbers so we know how painful the speed hit would be.
> 
> I'm weighing the time that mock would slow down the builds against the
> time and energy that it takes to
>  - Duplicate the BuildRequires in autobuild-ceph.git
>  - Review the change for correctness and merge the autobuild-ceph.git
>    pull request
>  - Deploy the autobuild-ceph.git changes out to the appropriate
>    gitbuilders
> 
> And then there's the "people stuff", like comprehending that all of the
> above is even necessary, or communicating about this on mailing lists
> like this thread, etc.
> 
> The fact that pbuilders are introduced very late (only during releases)
> means that we encounter errors during a release day when Alfredo or I
> have tried to get a release out the door. The pbuilder stuff isn't
> touched until we try to do a release, and in the mean time, it breaks
> because it's not getting continually tested. We're paying the "slow"
> penalty in other areas down the road.
> 
> I'm not looking to purposefully slow down other people's work, so I'll
> continue to mull over how we can balance speed with correctness.

I agree this woudl vastly simplify things.  Let's see what the performancd 
delta is on the new build boxes (which have lots of modern cores, RAM, and 
SSDs).

Also/alternatively, a step that installs all the BuildRequires on the host 
(even during each run) should be quick in the usual case but similarly 
avoid breakage... Even just making make an install-deps.sh that will Do 
The Right Thing might do the trick?  (Also usefulf or devs checking out 
the source who want to get their environment going quickly.)

sag

Some gitbuilders not working

[David Zafman]

We are seeing gitbuilder failures.  This is what I saw on one.

error: Failed build dependencies:
xmlstarlet is needed by ceph-1:0.90-821.g680fe3c.el7.x86_64

David Zafman
Senior Developer
http://www.redhat.com