Transitioning Ceph from Autotools to CMake

Transitioning Ceph from Autotools to CMake

[Ali Maredia]

After discussing with several other Ceph developers and Sage, I wanted
to start a discussion about making CMake the primary build system for Ceph.

CMake works just fine as it is (make -j4 on master with CMake builds
350% faster than autotools!), but there's more work needed to make it 
into a first-class build system.

Short term (1-2 weeks):
 - Making sure CMake works on all supported platforms: Centos7, RHEL7,
   Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
   target platforms I have noted to test on already.
 - Adding a target similar to make check
 - Creating CMake targets that build packages (such as for rpm or debian)
 - Writing documentation for those who haven't used CMake before to smooth the
   transition over
 - Making sure no targets or dependencies are missing from the
   current CMake build, and that CMake supports all current 
   targets, configurations and options
 - Replacing the integration autotools has with any automated build/test
   systems such as the gitbuilder

Longer term (2-4 weeks):
 - Removing the current autotools files, to avoid doubling build system
   workload
 - Adding more but shorter CMakeLists.txt files to a tree like structure
   where a CMakeLists.txt would be in every subdirectory
 
I'm already working on a target similar to the make check target, and plan
on working on the other short term goals over the next weeks and beyond.

I wanted to get feedback from the community any reasons why someone started using 
CMake but stopped (ex: lack of functionality), and more broadly, on what other 
obstacles there might be for the transition.

-Ali

Re: Transitioning Ceph from Autotools to CMake

[Mark Nelson]

Just here to provide moral support. Go CMake go! :)

Mark

On 07/30/2015 02:01 PM, Ali Maredia wrote:
> After discussing with several other Ceph developers and Sage, I wanted
> to start a discussion about making CMake the primary build system for Ceph.
>
> CMake works just fine as it is (make -j4 on master with CMake builds
> 350% faster than autotools!), but there's more work needed to make it
> into a first-class build system.
>
> Short term (1-2 weeks):
>   - Making sure CMake works on all supported platforms: Centos7, RHEL7,
>     Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
>     target platforms I have noted to test on already.
>   - Adding a target similar to make check
>   - Creating CMake targets that build packages (such as for rpm or debian)
>   - Writing documentation for those who haven't used CMake before to smooth the
>     transition over
>   - Making sure no targets or dependencies are missing from the
>     current CMake build, and that CMake supports all current
>     targets, configurations and options
>   - Replacing the integration autotools has with any automated build/test
>     systems such as the gitbuilder
>
> Longer term (2-4 weeks):
>   - Removing the current autotools files, to avoid doubling build system
>     workload
>   - Adding more but shorter CMakeLists.txt files to a tree like structure
>     where a CMakeLists.txt would be in every subdirectory
>
> I'm already working on a target similar to the make check target, and plan
> on working on the other short term goals over the next weeks and beyond.
>
> I wanted to get feedback from the community any reasons why someone started using
> CMake but stopped (ex: lack of functionality), and more broadly, on what other
> obstacles there might be for the transition.
>
> -Ali
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Transitioning Ceph from Autotools to CMake

[Ken Dreyer]

On 07/30/2015 01:01 PM, Ali Maredia wrote:
>  - Creating CMake targets that build packages (such as for rpm or debian)

There was some discussion on the list a while back about how we don't
really need to go through the full autoconf + ./configure + "make rpm"
routine simply to generate packages with the right version information.
My conclusion was that we should be able to generate a release tarball
(and even SRPM or RPM) with a shell script, rather than trying to do it
in autotools with a make target.

The reasoning was speed and efficiency: there's no point to running
"./configure" to generate an SRPM when the RPM itself runs "./configure"
during its build.

I don't know how well this maps to CMake, and whether CMake gives you
some sort of "skip everything and immediately do <foo>"... clearly I
need to learn more about CMake :)

>  - Replacing the integration autotools has with any automated build/test
>    systems such as the gitbuilder

We hope to eventually replace gitbuilder with something more scalable
like Jenkins, but in the meantime we're maintaining both systems :(

gitbuilder is https://github.com/ceph/autobuild-ceph
Jenkins is https://github.com/ceph/ceph-build

And then Loic also runs a make check bot in his own Jenkins instance.

Re: Transitioning Ceph from Autotools to CMake

[Orit Wasserman]

On Thu, 2015-07-30 at 15:01 -0400, Ali Maredia wrote:
> After discussing with several other Ceph developers and Sage, I wanted
> to start a discussion about making CMake the primary build system for Ceph.
> 
> CMake works just fine as it is (make -j4 on master with CMake builds
> 350% faster than autotools!), but there's more work needed to make it 
> into a first-class build system.

I been using cmake on fedora for awhile because it is much much faster. 
I encountered several issues:
1. It breaks for time to time
2. no make check , I started working on it
(https://github.com/oritwas/ceph/tree/cmake_add_make_check) the main
issue is that part of the unit tests fail when run in the cmake build
location. We will need to fix those tests.
3. no vstart.sh , starting working on this too but have less progress
here. At the moment in order to use vstart I copy the exe and libs to
src dir.

Orit
> 
> Short term (1-2 weeks):
>  - Making sure CMake works on all supported platforms: Centos7, RHEL7,
>    Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
>    target platforms I have noted to test on already.
>  - Adding a target similar to make check
>  - Creating CMake targets that build packages (such as for rpm or debian)
>  - Writing documentation for those who haven't used CMake before to smooth the
>    transition over
>  - Making sure no targets or dependencies are missing from the
>    current CMake build, and that CMake supports all current 
>    targets, configurations and options
>  - Replacing the integration autotools has with any automated build/test
>    systems such as the gitbuilder
> 

> Longer term (2-4 weeks):
>  - Removing the current autotools files, to avoid doubling build system
>    workload
>  - Adding more but shorter CMakeLists.txt files to a tree like structure
>    where a CMakeLists.txt would be in every subdirectory
>  
> I'm already working on a target similar to the make check target, and plan
> on working on the other short term goals over the next weeks and beyond.
> 
> I wanted to get feedback from the community any reasons why someone started using 
> CMake but stopped (ex: lack of functionality), and more broadly, on what other 
> obstacles there might be for the transition.
> 
> -Ali
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Transitioning Ceph from Autotools to CMake

[John Spray]

On Sat, Aug 1, 2015 at 8:24 PM, Orit Wasserman <owasserm@redhat.com> wrote:
>
>
> 3. no vstart.sh , starting working on this too but have less progress
> here. At the moment in order to use vstart I copy the exe and libs to
> src dir.

I just started playing with CMake on Friday, adding some missing cephfs
bits.  I was going to fix (3) as well, but I don't want to duplicate work
-- do you have an existing branch at all?  Presumably this will mostly be a
case of adding appropriate prefixes to commands.

Cheers,
John

Re: Transitioning Ceph from Autotools to CMake

[John Spray]

OK, here are vstart+ceph.in changes that work well enough in my out of
tree build:
https://github.com/ceph/ceph/pull/5457

John

On Mon, Aug 3, 2015 at 11:09 AM, John Spray <jspray@redhat.com> wrote:
> On Sat, Aug 1, 2015 at 8:24 PM, Orit Wasserman <owasserm@redhat.com> wrote:
>>
>>
>> 3. no vstart.sh , starting working on this too but have less progress
>> here. At the moment in order to use vstart I copy the exe and libs to
>> src dir.
>
> I just started playing with CMake on Friday, adding some missing cephfs
> bits.  I was going to fix (3) as well, but I don't want to duplicate work
> -- do you have an existing branch at all?  Presumably this will mostly be a
> case of adding appropriate prefixes to commands.
>
> Cheers,
> John

Re: Transitioning Ceph from Autotools to CMake

[Owen Synge]

Dear Ali,

I am glad you are making progress.

Sadly I don't yet know cmake.

Please consider the systemd wip branch. It might be wise to leave
autotools around a little longer, until all functionality is in the cmake.

Best regards

Owen


On 07/30/2015 09:01 PM, Ali Maredia wrote:
> After discussing with several other Ceph developers and Sage, I wanted
> to start a discussion about making CMake the primary build system for Ceph.
> 
> CMake works just fine as it is (make -j4 on master with CMake builds
> 350% faster than autotools!), but there's more work needed to make it 
> into a first-class build system.
> 
> Short term (1-2 weeks):
>  - Making sure CMake works on all supported platforms: Centos7, RHEL7,
>    Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
>    target platforms I have noted to test on already.
>  - Adding a target similar to make check
>  - Creating CMake targets that build packages (such as for rpm or debian)
>  - Writing documentation for those who haven't used CMake before to smooth the
>    transition over
>  - Making sure no targets or dependencies are missing from the
>    current CMake build, and that CMake supports all current 
>    targets, configurations and options
>  - Replacing the integration autotools has with any automated build/test
>    systems such as the gitbuilder
> 
> Longer term (2-4 weeks):
>  - Removing the current autotools files, to avoid doubling build system
>    workload
>  - Adding more but shorter CMakeLists.txt files to a tree like structure
>    where a CMakeLists.txt would be in every subdirectory
>  
> I'm already working on a target similar to the make check target, and plan
> on working on the other short term goals over the next weeks and beyond.
> 
> I wanted to get feedback from the community any reasons why someone started using 
> CMake but stopped (ex: lack of functionality), and more broadly, on what other 
> obstacles there might be for the transition.
> 
> -Ali
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Transitioning Ceph from Autotools to CMake

[Ali Maredia]

Owen,

I understand your concern, and don't think any transition will be made to
CMake untill all the functionality is in it and until it has been thoroughly 
vetted by the entire community to ensure a smooth transition.

I pushed a branch earlier today called wip-cmake 
(https://github.com/ceph/ceph/tree/wip-cmake) and plan to continue Orit's 
make check work, and coordinate with John on the vstart work he's done
already as my very first action items.

-Ali

----- Original Message -----
From: "Owen Synge" <osynge@suse.com>
To: "Ali Maredia" <amaredia@redhat.com>, ceph-devel@vger.kernel.org
Sent: Tuesday, August 4, 2015 6:42:31 AM
Subject: Re: Transitioning Ceph from Autotools to CMake

Dear Ali,

I am glad you are making progress.

Sadly I don't yet know cmake.

Please consider the systemd wip branch. It might be wise to leave
autotools around a little longer, until all functionality is in the cmake.

Best regards

Owen


On 07/30/2015 09:01 PM, Ali Maredia wrote:
> After discussing with several other Ceph developers and Sage, I wanted
> to start a discussion about making CMake the primary build system for Ceph.
> 
> CMake works just fine as it is (make -j4 on master with CMake builds
> 350% faster than autotools!), but there's more work needed to make it 
> into a first-class build system.
> 
> Short term (1-2 weeks):
>  - Making sure CMake works on all supported platforms: Centos7, RHEL7,
>    Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
>    target platforms I have noted to test on already.
>  - Adding a target similar to make check
>  - Creating CMake targets that build packages (such as for rpm or debian)
>  - Writing documentation for those who haven't used CMake before to smooth the
>    transition over
>  - Making sure no targets or dependencies are missing from the
>    current CMake build, and that CMake supports all current 
>    targets, configurations and options
>  - Replacing the integration autotools has with any automated build/test
>    systems such as the gitbuilder
> 
> Longer term (2-4 weeks):
>  - Removing the current autotools files, to avoid doubling build system
>    workload
>  - Adding more but shorter CMakeLists.txt files to a tree like structure
>    where a CMakeLists.txt would be in every subdirectory
>  
> I'm already working on a target similar to the make check target, and plan
> on working on the other short term goals over the next weeks and beyond.
> 
> I wanted to get feedback from the community any reasons why someone started using 
> CMake but stopped (ex: lack of functionality), and more broadly, on what other 
> obstacles there might be for the transition.
> 
> -Ali
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Transitioning Ceph from Autotools to CMake

[Orit Wasserman]

On Tue, 2015-08-04 at 00:38 +0100, John Spray wrote:
> OK, here are vstart+ceph.in changes that work well enough in my out of
> tree build:
> https://github.com/ceph/ceph/pull/5457

Great!
> 
> John
> 
> On Mon, Aug 3, 2015 at 11:09 AM, John Spray <jspray@redhat.com> wrote:
> > On Sat, Aug 1, 2015 at 8:24 PM, Orit Wasserman <owasserm@redhat.com> wrote:
> >>
> >>
> >> 3. no vstart.sh , starting working on this too but have less progress
> >> here. At the moment in order to use vstart I copy the exe and libs to
> >> src dir.
> >
> > I just started playing with CMake on Friday, adding some missing cephfs
> > bits.  I was going to fix (3) as well, but I don't want to duplicate work
> > -- do you have an existing branch at all?  Presumably this will mostly be a
> > case of adding appropriate prefixes to commands.
> >
> > Cheers,
> > John

Re: Transitioning Ceph from Autotools to CMake

[Owen Synge]

Dear Ali,

my point is no longer relevant, but your reassurances is still very
relevant.

Thanks

Owen

On 08/04/2015 08:26 PM, Ali Maredia wrote:
> Owen,
> 
> I understand your concern, and don't think any transition will be made to
> CMake untill all the functionality is in it and until it has been thoroughly 
> vetted by the entire community to ensure a smooth transition.
> 
> I pushed a branch earlier today called wip-cmake 
> (https://github.com/ceph/ceph/tree/wip-cmake) and plan to continue Orit's 
> make check work, and coordinate with John on the vstart work he's done
> already as my very first action items.
> 
> -Ali
> 
> ----- Original Message -----
> From: "Owen Synge" <osynge@suse.com>
> To: "Ali Maredia" <amaredia@redhat.com>, ceph-devel@vger.kernel.org
> Sent: Tuesday, August 4, 2015 6:42:31 AM
> Subject: Re: Transitioning Ceph from Autotools to CMake
> 
> Dear Ali,
> 
> I am glad you are making progress.
> 
> Sadly I don't yet know cmake.
> 
> Please consider the systemd wip branch. It might be wise to leave
> autotools around a little longer, until all functionality is in the cmake.
> 
> Best regards
> 
> Owen
> 
> 
> On 07/30/2015 09:01 PM, Ali Maredia wrote:
>> After discussing with several other Ceph developers and Sage, I wanted
>> to start a discussion about making CMake the primary build system for Ceph.
>>
>> CMake works just fine as it is (make -j4 on master with CMake builds
>> 350% faster than autotools!), but there's more work needed to make it 
>> into a first-class build system.
>>
>> Short term (1-2 weeks):
>>  - Making sure CMake works on all supported platforms: Centos7, RHEL7,
>>    Ubuntu 14.04 & 12.04, Fedora 22, Debian Jessie, Debian Wheezy are the
>>    target platforms I have noted to test on already.
>>  - Adding a target similar to make check
>>  - Creating CMake targets that build packages (such as for rpm or debian)
>>  - Writing documentation for those who haven't used CMake before to smooth the
>>    transition over
>>  - Making sure no targets or dependencies are missing from the
>>    current CMake build, and that CMake supports all current 
>>    targets, configurations and options
>>  - Replacing the integration autotools has with any automated build/test
>>    systems such as the gitbuilder
>>
>> Longer term (2-4 weeks):
>>  - Removing the current autotools files, to avoid doubling build system
>>    workload
>>  - Adding more but shorter CMakeLists.txt files to a tree like structure
>>    where a CMakeLists.txt would be in every subdirectory
>>  
>> I'm already working on a target similar to the make check target, and plan
>> on working on the other short term goals over the next weeks and beyond.
>>
>> I wanted to get feedback from the community any reasons why someone started using 
>> CMake but stopped (ex: lack of functionality), and more broadly, on what other 
>> obstacles there might be for the transition.
>>
>> -Ali
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html