Little update on the build platform

Little update on the build platform

[Erwan Velu]

Hi folks,

Just a short email to mention several updates that have been pushed to the build platfrorm 
- build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
- centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
- xfs is becoming the default rootfs of those VMs
- build requires are already installed reducing the time required by install-deps.sh
- An diagnosis helper is now in place to spot known build failures  :
A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
If you find new build failures, feel free to ping me to add them.
This helps at finding build issues in a second while the build logs are pretty long to read.

All that improvements are targeting a more reliable build system.

Please note that we have a lot of segfaults during the builds, if some can have a look around that would be lovely.

Erwan,

Re: Little update on the build platform

[Piotr Dałek]

On Tue, May 31, 2016 at 01:49:12PM -0400, Erwan Velu wrote:
> Hi folks,
> 
> Just a short email to mention several updates that have been pushed to the build platfrorm 
> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
> - xfs is becoming the default rootfs of those VMs
> - build requires are already installed reducing the time required by install-deps.sh
> - An diagnosis helper is now in place to spot known build failures  :
> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
> If you find new build failures, feel free to ping me to add them.
> This helps at finding build issues in a second while the build logs are pretty long to read.
> 
> All that improvements are targeting a more reliable build system.

I suggested that on #ceph-devel some time ago: how about not doing actual build 
when it is known that pull request in question does not alter code (like doc changes, 
for example https://github.com/ceph/ceph/pull/9389)? 

-- 
Piotr Dałek
branch@predictor.org.pl
http://blog.predictor.org.pl
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Little update on the build platform

[Erwan Velu]

That's pretty easy for a human to determine if a PR does affect the code.
If we'd like to automate that, that would mean having a list of files known not to need triggering a rebuild and updating that list everytime a new file is added/retired.

----- Mail original -----
De: "Piotr Dałek" <branch@predictor.org.pl>
À: ceph-devel@vger.kernel.org
Envoyé: Mardi 31 Mai 2016 19:58:56
Objet: Re: Little update on the build platform

On Tue, May 31, 2016 at 01:49:12PM -0400, Erwan Velu wrote:
> Hi folks,
> 
> Just a short email to mention several updates that have been pushed to the build platfrorm 
> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
> - xfs is becoming the default rootfs of those VMs
> - build requires are already installed reducing the time required by install-deps.sh
> - An diagnosis helper is now in place to spot known build failures  :
> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
> If you find new build failures, feel free to ping me to add them.
> This helps at finding build issues in a second while the build logs are pretty long to read.
> 
> All that improvements are targeting a more reliable build system.

I suggested that on #ceph-devel some time ago: how about not doing actual build 
when it is known that pull request in question does not alter code (like doc changes, 
for example https://github.com/ceph/ceph/pull/9389)? 

-- 
Piotr Dałek
branch@predictor.org.pl
http://blog.predictor.org.pl
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Little update on the build platform

[Piotr Dałek]

On Tue, May 31, 2016 at 02:02:49PM -0400, Erwan Velu wrote:
> That's pretty easy for a human to determine if a PR does affect the code.
> If we'd like to automate that, that would mean having a list of files known not to need triggering a rebuild and updating that list everytime a new file is added/retired.

How about ignoring everything that is under /doc/ and not under /doc/man/?
Seems like it would do the trick most of the time...

-- 
Piotr Dałek
branch@predictor.org.pl
http://blog.predictor.org.pl
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Little update on the build platform

[John Spray]

On Tue, May 31, 2016 at 6:49 PM, Erwan Velu <evelu@redhat.com> wrote:
> Hi folks,
>
> Just a short email to mention several updates that have been pushed to the build platfrorm
> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
> - xfs is becoming the default rootfs of those VMs
> - build requires are already installed reducing the time required by install-deps.sh
> - An diagnosis helper is now in place to spot known build failures  :
> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
> If you find new build failures, feel free to ping me to add them.
> This helps at finding build issues in a second while the build logs are pretty long to read.

This is awesome, thank you.

John

> All that improvements are targeting a more reliable build system.
>
> Please note that we have a lot of segfaults during the builds, if some can have a look around that would be lovely.
>
> Erwan,
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Little update on the build platform

[Ken Dreyer]

On Tue, May 31, 2016 at 12:44 PM, John Spray <jspray@redhat.com> wrote:
> On Tue, May 31, 2016 at 6:49 PM, Erwan Velu <evelu@redhat.com> wrote:
>> Hi folks,
>>
>> Just a short email to mention several updates that have been pushed to the build platfrorm
>> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
>> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
>> - xfs is becoming the default rootfs of those VMs
>> - build requires are already installed reducing the time required by install-deps.sh
>> - An diagnosis helper is now in place to spot known build failures  :
>> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
>> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
>> If you find new build failures, feel free to ping me to add them.
>> This helps at finding build issues in a second while the build logs are pretty long to read.
>
> This is awesome, thank you.
>

Agreed, way to go.

- Ken

Re: Little update on the build platform

[Erwan Velu]

Please note also that we the average build time for a PR have dropped to 20mn which is great !


----- Mail original -----
De: "Ken Dreyer" <kdreyer@redhat.com>
À: "John Spray" <jspray@redhat.com>
Cc: "Erwan Velu" <evelu@redhat.com>, "Ceph Development" <ceph-devel@vger.kernel.org>
Envoyé: Mardi 31 Mai 2016 21:14:21
Objet: Re: Little update on the build platform

On Tue, May 31, 2016 at 12:44 PM, John Spray <jspray@redhat.com> wrote:
> On Tue, May 31, 2016 at 6:49 PM, Erwan Velu <evelu@redhat.com> wrote:
>> Hi folks,
>>
>> Just a short email to mention several updates that have been pushed to the build platfrorm
>> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
>> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
>> - xfs is becoming the default rootfs of those VMs
>> - build requires are already installed reducing the time required by install-deps.sh
>> - An diagnosis helper is now in place to spot known build failures  :
>> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
>> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
>> If you find new build failures, feel free to ping me to add them.
>> This helps at finding build issues in a second while the build logs are pretty long to read.
>
> This is awesome, thank you.
>

Agreed, way to go.

- Ken
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Little update on the build platform

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1964 bytes --]

On Wed, 1 Jun 2016, Erwan Velu wrote:
> Please note also that we the average build time for a PR have dropped to 
> 20mn which is great !

This is great!  :)

sage

> 
> 
> ----- Mail original -----
> De: "Ken Dreyer" <kdreyer@redhat.com>
> À: "John Spray" <jspray@redhat.com>
> Cc: "Erwan Velu" <evelu@redhat.com>, "Ceph Development" <ceph-devel@vger.kernel.org>
> Envoyé: Mardi 31 Mai 2016 21:14:21
> Objet: Re: Little update on the build platform
> 
> On Tue, May 31, 2016 at 12:44 PM, John Spray <jspray@redhat.com> wrote:
> > On Tue, May 31, 2016 at 6:49 PM, Erwan Velu <evelu@redhat.com> wrote:
> >> Hi folks,
> >>
> >> Just a short email to mention several updates that have been pushed to the build platfrorm
> >> - build VMs are now generated by the edeploy (https://github.com/redhat-cip/edeploy) project
> >> - centos 7, ubuntu {trusty|xenial} are now built with the same automated procedure
> >> - xfs is becoming the default rootfs of those VMs
> >> - build requires are already installed reducing the time required by install-deps.sh
> >> - An diagnosis helper is now in place to spot known build failures  :
> >> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
> >> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/
> >> If you find new build failures, feel free to ping me to add them.
> >> This helps at finding build issues in a second while the build logs are pretty long to read.
> >
> > This is awesome, thank you.
> >
> 
> Agreed, way to go.
> 
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Little update on the build platform

[Nathan Cutler]

> - An diagnosis helper is now in place to spot known build failures  :
> A list of common issues have been entered (like gcc or jenkins failures, segfaults, etc...)
> A typical output looks like : https://jenkins.ceph.com/job/ceph-pull-requests/6448/

Thanks, guys, this looks like an excellent improvement.

Now I have a PR with failing make check and it appears that Jenkins is 
aware of the failures:

https://jenkins.ceph.com/job/ceph-pull-requests/6550/consoleFull#17782142537ba86caa-bd6c-4071-8005-3f6d80f92e07

The two recognized failures are highlighted in red:

FATAL: java.io.IOException: Unexpected termination of the channel

java.lang.NullPointerException: no workspace from node 
hudson.slaves.DumbSlave[158.69.84.81+trusty_huge__c69f129f-3dff-46c2-9bd4-d14364fcb62f] 
which is computer hudson.slaves.SlaveComputer@1957dd58 and has channel null

But I don't know what to do about them? Do we have documentation of the 
common failure cases and what can be done to fix them?

Nathan

Re: Little update on the build platform

[Ken Dreyer]

On Thu, Jun 2, 2016 at 2:11 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> - An diagnosis helper is now in place to spot known build failures  :
>> A list of common issues have been entered (like gcc or jenkins failures,
>> segfaults, etc...)
>> A typical output looks like :
>> https://jenkins.ceph.com/job/ceph-pull-requests/6448/
>
>
> Thanks, guys, this looks like an excellent improvement.
>
> Now I have a PR with failing make check and it appears that Jenkins is aware
> of the failures:
>
> https://jenkins.ceph.com/job/ceph-pull-requests/6550/consoleFull#17782142537ba86caa-bd6c-4071-8005-3f6d80f92e07
>
> The two recognized failures are highlighted in red:
>
> FATAL: java.io.IOException: Unexpected termination of the channel
>
> java.lang.NullPointerException: no workspace from node
> hudson.slaves.DumbSlave[158.69.84.81+trusty_huge__c69f129f-3dff-46c2-9bd4-d14364fcb62f]
> which is computer hudson.slaves.SlaveComputer@1957dd58 and has channel null
>
> But I don't know what to do about them? Do we have documentation of the
> common failure cases and what can be done to fix them?

This sounds like a network issue in OVH (cloud provider where Jenkins
is hosted)?