Configure dependencies can be the same as make dependencies

Configure dependencies can be the same as make dependencies

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2660 bytes --]

Hi Kefu,

In the context of https://github.com/ceph/ceph/pull/4449 and https://github.com/ceph/ceph/pull/4544 I see you're going out of your way to implement mechanisms that make it possible to require less dependencies when running 

   ./configure

than what is required by

   ./make check

I think the right solution is to require the same set of dependencies, regardless. It can easily be done by running ./install-deps.sh[1]. 

This script is already used in jenkins.ceph.com and saved us from the recurring pain of manually updating the jenkins slaves every time a dependency was added to either ceph.spec.in or debian/control. 

Although it is possible to run ./configure with a subset of the dependencies that are required to run make check, it trades automatic installation of packages for significant manual maintenance. It saves a little disk and bandwidth every time a dependency is modified in the ceph.spec.in or debian/control files, which happens a few times a months at most. The work items to manually maintain this difference:

* the set of dependencies that ./configure requires needs to be manually maintained, it is not listed anywhere at the moment. It changes less often than ceph.spec.in or debian/control but it does change from time to time
* the configure script has to be engineered to only require dependencies (assuming these dependencies are listed somewhere). In other word, every time you change the configure script you have to be extra careful to not introduce a new dependency, even when it would help implement what you're after in a simpler way.
* the configure script dependencies in the context of CI actually change every time you consider using --enable-something because it modifies the set of files your tarbal is made of. If the corresponding something is not installed, the configure will likely not do what you want and you'll have to add something manually on the CI machine (or use install-deps.sh but that would defeat the purpose of separating configure dependencies from make dependencies)
* to avoid adding dependencies to configure, it is tempting to forbid file generation when preparing the tarbal (I'm thinking .8 generation specifically), although it is legitimate for the tarbal generation to involve some processing and transformation of the sources that require tools to run
* it is very unlikely a new developer will remember configure dependencies and make dependencies are different and mistakes will be done all the time, creating needless frustration

Cheers

[1] https://github.com/ceph/ceph/blob/master/install-deps.sh
-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Configure dependencies can be the same as make dependencies

[Gregory Farnum]

On Tue, May 5, 2015 at 1:34 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Kefu,
>
> In the context of https://github.com/ceph/ceph/pull/4449 and https://github.com/ceph/ceph/pull/4544 I see you're going out of your way to implement mechanisms that make it possible to require less dependencies when running
>
>    ./configure
>
> than what is required by
>
>    ./make check
>
> I think the right solution is to require the same set of dependencies, regardless. It can easily be done by running ./install-deps.sh[1].
>
> This script is already used in jenkins.ceph.com and saved us from the recurring pain of manually updating the jenkins slaves every time a dependency was added to either ceph.spec.in or debian/control.
>
> Although it is possible to run ./configure with a subset of the dependencies that are required to run make check, it trades automatic installation of packages for significant manual maintenance. It saves a little disk and bandwidth every time a dependency is modified in the ceph.spec.in or debian/control files, which happens a few times a months at most. The work items to manually maintain this difference:
>
> * the set of dependencies that ./configure requires needs to be manually maintained, it is not listed anywhere at the moment. It changes less often than ceph.spec.in or debian/control but it does change from time to time
> * the configure script has to be engineered to only require dependencies (assuming these dependencies are listed somewhere). In other word, every time you change the configure script you have to be extra careful to not introduce a new dependency, even when it would help implement what you're after in a simpler way.
> * the configure script dependencies in the context of CI actually change every time you consider using --enable-something because it modifies the set of files your tarbal is made of. If the corresponding something is not installed, the configure will likely not do what you want and you'll have to add something manually on the CI machine (or use install-deps.sh but that would defeat the purpose of separating configure dependencies from make dependencies)
> * to avoid adding dependencies to configure, it is tempting to forbid file generation when preparing the tarbal (I'm thinking .8 generation specifically), although it is legitimate for the tarbal generation to involve some processing and transformation of the sources that require tools to run
> * it is very unlikely a new developer will remember configure dependencies and make dependencies are different and mistakes will be done all the time, creating needless frustration

Maybe I'm missing something, but...

configure's purpose in life is to set up the build system so that Ceph
will successfully build on the current system. If configure assumes
that you have stuff installed that you don't need to have installed to
build, then it is not doing its job.
Similarly, if "make check" won't run on a system that otherwise builds
Ceph, that's a problem with "make check", not a problem with configure
or the rest of Ceph. If we require *extra* dependencies to support
make check that's very bad — it's becoming more popular for vendors to
build Ceph on more exotic architectures and systems, and the chances
are good that they'll do the minimal porting job, and simply skip
running "make check" if it requires extra bits.

Now, there are times when we decide something is important so we have
configure default to one configuration and require a flag to change
it. But in general that's not what we want to do, and IIRC there was
an issue where configure was actually failing despite it being
perfectly fine to build without some library or other.

Etc etc.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Configure dependencies can be the same as make dependencies

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4237 bytes --]



On 05/05/2015 16:31, Gregory Farnum wrote:
> On Tue, May 5, 2015 at 1:34 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Kefu,
>>
>> In the context of https://github.com/ceph/ceph/pull/4449 and https://github.com/ceph/ceph/pull/4544 I see you're going out of your way to implement mechanisms that make it possible to require less dependencies when running
>>
>>    ./configure
>>
>> than what is required by
>>
>>    ./make check
>>
>> I think the right solution is to require the same set of dependencies, regardless. It can easily be done by running ./install-deps.sh[1].
>>
>> This script is already used in jenkins.ceph.com and saved us from the recurring pain of manually updating the jenkins slaves every time a dependency was added to either ceph.spec.in or debian/control.
>>
>> Although it is possible to run ./configure with a subset of the dependencies that are required to run make check, it trades automatic installation of packages for significant manual maintenance. It saves a little disk and bandwidth every time a dependency is modified in the ceph.spec.in or debian/control files, which happens a few times a months at most. The work items to manually maintain this difference:
>>
>> * the set of dependencies that ./configure requires needs to be manually maintained, it is not listed anywhere at the moment. It changes less often than ceph.spec.in or debian/control but it does change from time to time
>> * the configure script has to be engineered to only require dependencies (assuming these dependencies are listed somewhere). In other word, every time you change the configure script you have to be extra careful to not introduce a new dependency, even when it would help implement what you're after in a simpler way.
>> * the configure script dependencies in the context of CI actually change every time you consider using --enable-something because it modifies the set of files your tarbal is made of. If the corresponding something is not installed, the configure will likely not do what you want and you'll have to add something manually on the CI machine (or use install-deps.sh but that would defeat the purpose of separating configure dependencies from make dependencies)
>> * to avoid adding dependencies to configure, it is tempting to forbid file generation when preparing the tarbal (I'm thinking .8 generation specifically), although it is legitimate for the tarbal generation to involve some processing and transformation of the sources that require tools to run
>> * it is very unlikely a new developer will remember configure dependencies and make dependencies are different and mistakes will be done all the time, creating needless frustration
> 
> Maybe I'm missing something, but...

The bit of context I forgot to mention is that ./configure is often used for the sole purpose of running make dist (with the risk of bundling stuff that does not actually build or run but that's not too much of a concern if the build is done immediately afterward, using the tarbal created by make dist).

> 
> configure's purpose in life is to set up the build system so that Ceph
> will successfully build on the current system. If configure assumes
> that you have stuff installed that you don't need to have installed to
> build, then it is not doing its job.
> Similarly, if "make check" won't run on a system that otherwise builds
> Ceph, that's a problem with "make check", not a problem with configure
> or the rest of Ceph. If we require *extra* dependencies to support
> make check that's very bad — it's becoming more popular for vendors to
> build Ceph on more exotic architectures and systems, and the chances
> are good that they'll do the minimal porting job, and simply skip
> running "make check" if it requires extra bits.
> 
> Now, there are times when we decide something is important so we have
> configure default to one configuration and require a flag to change
> it. But in general that's not what we want to do, and IIRC there was
> an issue where configure was actually failing despite it being
> perfectly fine to build without some library or other.
> 
> Etc etc.
> -Greg
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Configure dependencies can be the same as make dependencies

[Kefu Chai]

----- Original Message -----
> From: "Gregory Farnum" <greg@gregs42.com>
> To: "Loic Dachary" <loic@dachary.org>
> Cc: "Kefu Chai" <kchai@redhat.com>, "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Tuesday, May 5, 2015 10:31:11 PM
> Subject: Re: Configure dependencies can be the same as make dependencies
> 
> On Tue, May 5, 2015 at 1:34 AM, Loic Dachary <loic@dachary.org> wrote:
> > Hi Kefu,
> >
> > In the context of https://github.com/ceph/ceph/pull/4449 and
> > https://github.com/ceph/ceph/pull/4544 I see you're going out of your way
> > to implement mechanisms that make it possible to require less dependencies
> > when running
> >
> >    ./configure
> >
> > than what is required by
> >
> >    ./make check
> >
> > I think the right solution is to require the same set of dependencies,
> > regardless. It can easily be done by running ./install-deps.sh[1].
> >
> > This script is already used in jenkins.ceph.com and saved us from the
> > recurring pain of manually updating the jenkins slaves every time a
> > dependency was added to either ceph.spec.in or debian/control.
> >
> > Although it is possible to run ./configure with a subset of the
> > dependencies that are required to run make check, it trades automatic
> > installation of packages for significant manual maintenance. It saves a
> > little disk and bandwidth every time a dependency is modified in the
> > ceph.spec.in or debian/control files, which happens a few times a months
> > at most. The work items to manually maintain this difference:
> >
> > * the set of dependencies that ./configure requires needs to be manually
> > maintained, it is not listed anywhere at the moment. It changes less often
> > than ceph.spec.in or debian/control but it does change from time to time
> > * the configure script has to be engineered to only require dependencies
> > (assuming these dependencies are listed somewhere). In other word, every
> > time you change the configure script you have to be extra careful to not
> > introduce a new dependency, even when it would help implement what you're
> > after in a simpler way.
> > * the configure script dependencies in the context of CI actually change
> > every time you consider using --enable-something because it modifies the
> > set of files your tarbal is made of. If the corresponding something is not
> > installed, the configure will likely not do what you want and you'll have
> > to add something manually on the CI machine (or use install-deps.sh but
> > that would defeat the purpose of separating configure dependencies from
> > make dependencies)
> > * to avoid adding dependencies to configure, it is tempting to forbid file
> > generation when preparing the tarbal (I'm thinking .8 generation
> > specifically), although it is legitimate for the tarbal generation to
> > involve some processing and transformation of the sources that require
> > tools to run
> > * it is very unlikely a new developer will remember configure dependencies
> > and make dependencies are different and mistakes will be done all the
> > time, creating needless frustration

yeah. we should rely on ./install-deps.sh to prepare the environment for
all slaves. that's a more consistent and manageable way in the long term.
@alfredo, what do you think?

as to the change in https://github.com/ceph/ceph/pull/4544, my original
intention was to port my quick fix in next branch back to the master, and
to minimize the dependency for "configure; make dist".

but now, i think it's but a way to please the developer who does not want
to install sphinx-build. so it's just a nice-to-have now.

and if "--without-man-pages", the configure dependencies and make dependencies 
are the same.

i just updated https://github.com/ceph/ceph/pull/4449, to reference this
mail thread.

> 
> Maybe I'm missing something, but...
> 
> configure's purpose in life is to set up the build system so that Ceph
> will successfully build on the current system. If configure assumes
> that you have stuff installed that you don't need to have installed to
> build, then it is not doing its job.

agreed. but build includes "make check", IMHO. unless cross compiling.
i think that's one of reasons why debhelper has dh_auto_test. debhelper
is a set of tools helping building debian packages. i am not suggesting
that "it is correct since it's working that way", but there is probably
some rationale behind this, i think.

> Similarly, if "make check" won't run on a system that otherwise builds
> Ceph, that's a problem with "make check", not a problem with configure
> or the rest of Ceph. If we require *extra* dependencies to support

one task of configure is to figure out the settings for Makefile. so,
yes, it's a problem of "make check", but configure is responsible to
help it.


> make check that's very bad — it's becoming more popular for vendors to
> build Ceph on more exotic architectures and systems, and the chances
> are good that they'll do the minimal porting job, and simply skip
> running "make check" if it requires extra bits.

IIUC, this requires more fine-grained control to disable the check for 
dependencies only used in "make check". I am not sure if this is expected.
for the good of the vendor who wants to port ceph to an exotic platform,
a workable "make check" is always expected. at least "make check" for the
part of ceph the venders want to port/build. for example, if the vendor
just want to run ceph-osd on a certain platform, it would be nice to 
pass "--with-osd --without-mon --without-mds ..." to configure, and make
sure that "make check" passes. and in this case, "make check" should only
exercise the function of OSD and its dependencies.


> 
> Now, there are times when we decide something is important so we have
> configure default to one configuration and require a flag to change
> it. But in general that's not what we want to do, and IIRC there was
> an issue where configure was actually failing despite it being
> perfectly fine to build without some library or other.



Greg


> 
> Etc etc.
> -Greg
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Configure dependencies can be the same as make dependencies

[Ken Dreyer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 05/05/2015 08:36 AM, Loic Dachary wrote:
> 
> 
> On 05/05/2015 16:31, Gregory Farnum wrote:
>> On Tue, May 5, 2015 at 1:34 AM, Loic Dachary <loic@dachary.org>
>> wrote:
>>> Hi Kefu,
>>> 
>>> In the context of https://github.com/ceph/ceph/pull/4449 and
>>> https://github.com/ceph/ceph/pull/4544 I see you're going out
>>> of your way to implement mechanisms that make it possible to
>>> require less dependencies when running
>>> 
>>> ./configure
>>> 
>>> than what is required by
>>> 
>>> ./make check
>>> 
>>> I think the right solution is to require the same set of
>>> dependencies, regardless. It can easily be done by running
>>> ./install-deps.sh[1].
>>> 
>>> This script is already used in jenkins.ceph.com and saved us
>>> from the recurring pain of manually updating the jenkins slaves
>>> every time a dependency was added to either ceph.spec.in or
>>> debian/control.
>>> 
>>> Although it is possible to run ./configure with a subset of the
>>> dependencies that are required to run make check, it trades
>>> automatic installation of packages for significant manual
>>> maintenance. It saves a little disk and bandwidth every time a
>>> dependency is modified in the ceph.spec.in or debian/control
>>> files, which happens a few times a months at most. The work
>>> items to manually maintain this difference:
>>> 
>>> * the set of dependencies that ./configure requires needs to be
>>> manually maintained, it is not listed anywhere at the moment.
>>> It changes less often than ceph.spec.in or debian/control but
>>> it does change from time to time * the configure script has to
>>> be engineered to only require dependencies (assuming these
>>> dependencies are listed somewhere). In other word, every time
>>> you change the configure script you have to be extra careful to
>>> not introduce a new dependency, even when it would help
>>> implement what you're after in a simpler way. * the configure
>>> script dependencies in the context of CI actually change every
>>> time you consider using --enable-something because it modifies
>>> the set of files your tarbal is made of. If the corresponding
>>> something is not installed, the configure will likely not do
>>> what you want and you'll have to add something manually on the
>>> CI machine (or use install-deps.sh but that would defeat the
>>> purpose of separating configure dependencies from make
>>> dependencies) * to avoid adding dependencies to configure, it
>>> is tempting to forbid file generation when preparing the tarbal
>>> (I'm thinking .8 generation specifically), although it is
>>> legitimate for the tarbal generation to involve some processing
>>> and transformation of the sources that require tools to run *
>>> it is very unlikely a new developer will remember configure
>>> dependencies and make dependencies are different and mistakes
>>> will be done all the time, creating needless frustration
>> 
>> Maybe I'm missing something, but...
> 
> The bit of context I forgot to mention is that ./configure is often
> used for the sole purpose of running make dist (with the risk of
> bundling stuff that does not actually build or run but that's not
> too much of a concern if the build is done immediately afterward,
> using the tarbal created by make dist).
> 

I think the long-term solution to Kefu's issue is that we need to
remove the requirement to run through a full "./configure" invocation
just to get a tarball. All the RPM and Debian packages internally run
./configure, so running it a second time slows things down. I think it
makes sense to implement the tarball-generation functionality using a
simpler script at the root of the ceph.git tree. The operation should
be about as fast as "git archive".

The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
takes a full "./configure" run to get to a point where Make can write
the proper version numbers into that file. Ideally we could skip all
of that and simply do the variable interpolation with sed or something.

- - Ken

Re: Configure dependencies can be the same as make dependencies

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4391 bytes --]



On 05/05/2015 20:59, Ken Dreyer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 05/05/2015 08:36 AM, Loic Dachary wrote:
>>
>>
>> On 05/05/2015 16:31, Gregory Farnum wrote:
>>> On Tue, May 5, 2015 at 1:34 AM, Loic Dachary <loic@dachary.org>
>>> wrote:
>>>> Hi Kefu,
>>>>
>>>> In the context of https://github.com/ceph/ceph/pull/4449 and
>>>> https://github.com/ceph/ceph/pull/4544 I see you're going out
>>>> of your way to implement mechanisms that make it possible to
>>>> require less dependencies when running
>>>>
>>>> ./configure
>>>>
>>>> than what is required by
>>>>
>>>> ./make check
>>>>
>>>> I think the right solution is to require the same set of
>>>> dependencies, regardless. It can easily be done by running
>>>> ./install-deps.sh[1].
>>>>
>>>> This script is already used in jenkins.ceph.com and saved us
>>>> from the recurring pain of manually updating the jenkins slaves
>>>> every time a dependency was added to either ceph.spec.in or
>>>> debian/control.
>>>>
>>>> Although it is possible to run ./configure with a subset of the
>>>> dependencies that are required to run make check, it trades
>>>> automatic installation of packages for significant manual
>>>> maintenance. It saves a little disk and bandwidth every time a
>>>> dependency is modified in the ceph.spec.in or debian/control
>>>> files, which happens a few times a months at most. The work
>>>> items to manually maintain this difference:
>>>>
>>>> * the set of dependencies that ./configure requires needs to be
>>>> manually maintained, it is not listed anywhere at the moment.
>>>> It changes less often than ceph.spec.in or debian/control but
>>>> it does change from time to time * the configure script has to
>>>> be engineered to only require dependencies (assuming these
>>>> dependencies are listed somewhere). In other word, every time
>>>> you change the configure script you have to be extra careful to
>>>> not introduce a new dependency, even when it would help
>>>> implement what you're after in a simpler way. * the configure
>>>> script dependencies in the context of CI actually change every
>>>> time you consider using --enable-something because it modifies
>>>> the set of files your tarbal is made of. If the corresponding
>>>> something is not installed, the configure will likely not do
>>>> what you want and you'll have to add something manually on the
>>>> CI machine (or use install-deps.sh but that would defeat the
>>>> purpose of separating configure dependencies from make
>>>> dependencies) * to avoid adding dependencies to configure, it
>>>> is tempting to forbid file generation when preparing the tarbal
>>>> (I'm thinking .8 generation specifically), although it is
>>>> legitimate for the tarbal generation to involve some processing
>>>> and transformation of the sources that require tools to run *
>>>> it is very unlikely a new developer will remember configure
>>>> dependencies and make dependencies are different and mistakes
>>>> will be done all the time, creating needless frustration
>>>
>>> Maybe I'm missing something, but...
>>
>> The bit of context I forgot to mention is that ./configure is often
>> used for the sole purpose of running make dist (with the risk of
>> bundling stuff that does not actually build or run but that's not
>> too much of a concern if the build is done immediately afterward,
>> using the tarbal created by make dist).
>>
> 
> I think the long-term solution to Kefu's issue is that we need to
> remove the requirement to run through a full "./configure" invocation
> just to get a tarball. All the RPM and Debian packages internally run
> ./configure, so running it a second time slows things down. I think it
> makes sense to implement the tarball-generation functionality using a
> simpler script at the root of the ceph.git tree. The operation should
> be about as fast as "git archive".

I agree. It's going to be significant work but it's worth it.

> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
> takes a full "./configure" run to get to a point where Make can write
> the proper version numbers into that file. Ideally we could skip all
> of that and simply do the variable interpolation with sed or something.
> 
> - - Ken
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Configure dependencies can be the same as make dependencies

[Sage Weil]

On Tue, 5 May 2015, Loic Dachary wrote:
> > I think the long-term solution to Kefu's issue is that we need to
> > remove the requirement to run through a full "./configure" invocation
> > just to get a tarball. All the RPM and Debian packages internally run
> > ./configure, so running it a second time slows things down. I think it
> > makes sense to implement the tarball-generation functionality using a
> > simpler script at the root of the ceph.git tree. The operation should
> > be about as fast as "git archive".
> 
> I agree. It's going to be significant work but it's worth it.

Yep!

> > The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
> > takes a full "./configure" run to get to a point where Make can write
> > the proper version numbers into that file. Ideally we could skip all
> > of that and simply do the variable interpolation with sed or something.

Yep!

sage

Re: Configure dependencies can be the same as make dependencies

[Owen Synge]

Sorry to catch this thread late.

I come here via patch

https://github.com/ceph/ceph/pull/4911#issuecomment-110422312

I think you guys are missing that configure is doing some thing here.

(1) Configure is generating the spec file.
(2) It could also generate the deb files.

What no one has done is to:

(A) Use configure to eliminate the differences between OS's.

If any one had done (A) you would not be considering this I hope.

This impact is particularly felt in the ceph.spec.in file where because
of the lack of (A) in the process we are plagued with OS specific
conditionals.

If the path of say:

	src/ceph-osd-prestart.sh

is ether:

	/usr/libexec/ceph

or:

	/usr/lib/ceph


Is purely distribution specific.

Having conditionals in

	ceph.sepc.in

defeats the point of generating it via autotools.

we can be remove many hard coded values replaced with variable and that
probably will only grow in number for example

	%if 0%{?rhel} || 0%{?fedora}
		--with-systemd-libexec-dir=/usr/libexec/ceph \
	%endif
	%if 0%{?opensuse} || 0%{?suse_version}
		--with-systemd-libexec-dir=/usr/lib/ceph/ \
	%endif

some wont need distribution specific locations like:

	--with-systemd-unit-dir=%_unitdir

In the long term replaced all these path variables could be replaced by
single parameter.

	./configure --with-distro-defaults=redhat
	make rpm
	./configure --with-distro-defaults=suse
	make rpm
	./configure --with-distro-defaults=debian
	make deb
	./configure --with-distro-defaults=ubuntu
	make deb

Very easily.

In summary, please reconsider this decision as if you follow this policy
we are left with conditionals all over the ceph.spec.in file.


Best regards

Owen






On 05/05/2015 09:31 PM, Sage Weil wrote:
> On Tue, 5 May 2015, Loic Dachary wrote:
>>> I think the long-term solution to Kefu's issue is that we need to
>>> remove the requirement to run through a full "./configure" invocation
>>> just to get a tarball. All the RPM and Debian packages internally run
>>> ./configure, so running it a second time slows things down. I think it
>>> makes sense to implement the tarball-generation functionality using a
>>> simpler script at the root of the ceph.git tree. The operation should
>>> be about as fast as "git archive".
>>
>> I agree. It's going to be significant work but it's worth it.
> 
> Yep!
> 
>>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
>>> takes a full "./configure" run to get to a point where Make can write
>>> the proper version numbers into that file. Ideally we could skip all
>>> of that and simply do the variable interpolation with sed or something.
> 
> Yep!
> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

what this would look like if the system is with templates.

[Owen Synge]

> we can be remove many hard coded values replaced with variable and that
> probably will only grow in number for example
> 
> 	%if 0%{?rhel} || 0%{?fedora}
> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
> 	%endif
> 	%if 0%{?opensuse} || 0%{?suse_version}
> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
> 	%endif

	--with-systemd-libexec-dir=@systemd_libexec_dir@ \

No OS distribution specific rubbish needed :)

Best regards

Owen

Re: Configure dependencies can be the same as make dependencies

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4525 bytes --]

On Tue, 9 Jun 2015, Owen Synge wrote:
> Sorry to catch this thread late.

There were two goals here:

 1- make the generated tarball deterministic and independent of 
configure options.  Currently we have horrible hackery to include rocksdb 
source in the tarball when configure doesn't get the rocksdb arguments, 
and were hitting similar roadblocks/pain with libxio.  This is avoided 
with the make_dist.sh script (although we're not using it yet because of 
the specfile generation piece).

 2- avoid multiple passes through ./configure for 'make dist' and the 
subsequent build.  We currently do it for the 'make dist' step (which also 
generates ceph.spec) and then for the eventual build.  Using autotools for 
generating the spec means we still do it twice.

I'm less concerned about build speed and it may be worth consolidating all 
of the distro logic in configure.ac.  Honestly I don't really know if 
that is better or not (vs conditionals in the specfile)... I have no real 
love for autotools.

But hopefully the tarball step can remain a fast, deterministic step that 
doesn't require a configure pass?

sage




> 
> I come here via patch
> 
> https://github.com/ceph/ceph/pull/4911#issuecomment-110422312
> 
> I think you guys are missing that configure is doing some thing here.
> 
> (1) Configure is generating the spec file.
> (2) It could also generate the deb files.
> 
> What no one has done is to:
> 
> (A) Use configure to eliminate the differences between OS's.
> 
> If any one had done (A) you would not be considering this I hope.
> 
> This impact is particularly felt in the ceph.spec.in file where because
> of the lack of (A) in the process we are plagued with OS specific
> conditionals.
> 
> If the path of say:
> 
> 	src/ceph-osd-prestart.sh
> 
> is ether:
> 
> 	/usr/libexec/ceph
> 
> or:
> 
> 	/usr/lib/ceph
> 
> 
> Is purely distribution specific.
> 
> Having conditionals in
> 
> 	ceph.sepc.in
> 
> defeats the point of generating it via autotools.
> 
> we can be remove many hard coded values replaced with variable and that
> probably will only grow in number for example
> 
> 	%if 0%{?rhel} || 0%{?fedora}
> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
> 	%endif
> 	%if 0%{?opensuse} || 0%{?suse_version}
> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
> 	%endif
> 
> some wont need distribution specific locations like:
> 
> 	--with-systemd-unit-dir=%_unitdir
> 
> In the long term replaced all these path variables could be replaced by
> single parameter.
> 
> 	./configure --with-distro-defaults=redhat
> 	make rpm
> 	./configure --with-distro-defaults=suse
> 	make rpm
> 	./configure --with-distro-defaults=debian
> 	make deb
> 	./configure --with-distro-defaults=ubuntu
> 	make deb
> 
> Very easily.
> 
> In summary, please reconsider this decision as if you follow this policy
> we are left with conditionals all over the ceph.spec.in file.
> 
> 
> Best regards
> 
> Owen
> 
> 
> 
> 
> 
> 
> On 05/05/2015 09:31 PM, Sage Weil wrote:
> > On Tue, 5 May 2015, Loic Dachary wrote:
> >>> I think the long-term solution to Kefu's issue is that we need to
> >>> remove the requirement to run through a full "./configure" invocation
> >>> just to get a tarball. All the RPM and Debian packages internally run
> >>> ./configure, so running it a second time slows things down. I think it
> >>> makes sense to implement the tarball-generation functionality using a
> >>> simpler script at the root of the ceph.git tree. The operation should
> >>> be about as fast as "git archive".
> >>
> >> I agree. It's going to be significant work but it's worth it.
> > 
> > Yep!
> > 
> >>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
> >>> takes a full "./configure" run to get to a point where Make can write
> >>> the proper version numbers into that file. Ideally we could skip all
> >>> of that and simply do the variable interpolation with sed or something.
> > 
> > Yep!
> > 
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -- 
> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
> 21284 (AG
> Nürnberg)
> 
> Maxfeldstraße 5
> 
> 90409 Nürnberg
> 
> Germany
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Configure dependencies can be the same as make dependencies

[Owen Synge]

On 06/09/2015 07:22 PM, Sage Weil wrote:
> On Tue, 9 Jun 2015, Owen Synge wrote:
>> Sorry to catch this thread late.
> 
> There were two goals here:
> 
>  1- make the generated tarball deterministic and independent of 
> configure options.  Currently we have horrible hackery to include rocksdb 
> source in the tarball when configure doesn't get the rocksdb arguments, 
> and were hitting similar roadblocks/pain with libxio.  This is avoided 
> with the make_dist.sh script (although we're not using it yet because of 
> the specfile generation piece).

I think deterministic is a reasonable goal.

I think independent of configure options, is not really a goal, its a
solution to a goal that misses a couple of observations. The goal is
deterministic operation.

The important observations are:

(a) The same source will need to be released multiple times.
(b) It is nice to release with some differences to each environment.

If you combine (a) and (b) you quickly see why the "--configure" step
exists before:

	make dist-bzip2

as say on release you iterate between distributions to produce each
setup deterministically.

To illustrate I have put a little of my proposals for a replacement
model of "make_dist.sh" to illustrate this in my opinion improved process.

But if this step exists imagining a couple of patches later:

 	$cat make_dist.sh
	#!/bin/sh -e
	# Common code
	autogen.sh
	# Redhat
	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
		--with-rpm-dir=/export/redhat/
 	make srpm
	# Suse
 	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
		--with-rpm-dir=/export/suse/
 	make srpm
	# Debian
 	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
		--with-sdeb-dir=/export/debian/
 	make sdeb
	# Ubuntu
 	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
		--with-sdeb-dir=/export/ubuntu
 	make sdeb

This will also solve the problems (1) and (2) with ceph.spec.in to
ceph.spec transformation. And not just solve it once but make it clearer
for each distribution.

Problems.

These are problems with your planned approach I don't have good answers
for, and this approach solves now.

(1) having autotools and another thing template ceph.spec.in could be
complex.

(2) This code need to be documented as its not standard autotools.

(3) How much faster must the template tool be to be acceptably faster
than ./configure

>  2- avoid multiple passes through ./configure for 'make dist' and the 
> subsequent build.  We currently do it for the 'make dist' step (which also 
> generates ceph.spec) and then for the eventual build.  Using autotools for 
> generating the spec means we still do it twice.
> 
> I'm less concerned about build speed and it may be worth consolidating all 
> of the distro logic in configure.ac.  Honestly I don't really know if 
> that is better or not (vs conditionals in the specfile)... I have no real 
> love for autotools.

autotools is not some thing I love, but I find it very useful, within
the scope of building C/C++, I have found it has a solution with _every_
use case already catered for. You just need some time to find it, and it
may look different from what you thought you wanted and how you would
make it, but I find the pre made autotools solution is a better solution
than making your own from experience.

I must say though, autotools is much nicer than rpm for macros and
variables and template expansion.

> But hopefully the tarball step can remain a fast, deterministic step that 
> doesn't require a configure pass?

I really support you desire for deterministic.

I commend your desire that each step remain fast.

I believe you are missing a step if you follow the path of skipping the
configure pass, and you loose some important functionality.

I point out now what functionality in the hope that we can use it to
make other work easier, and autoools can be used to template any files
it needs to, including at least, spec files and deb files by putting
paths into autotools.


Owen



> sage
> 
> 
> 
> 
>>
>> I come here via patch
>>
>> https://github.com/ceph/ceph/pull/4911#issuecomment-110422312
>>
>> I think you guys are missing that configure is doing some thing here.
>>
>> (1) Configure is generating the spec file.
>> (2) It could also generate the deb files.
>>
>> What no one has done is to:
>>
>> (A) Use configure to eliminate the differences between OS's.
>>
>> If any one had done (A) you would not be considering this I hope.
>>
>> This impact is particularly felt in the ceph.spec.in file where because
>> of the lack of (A) in the process we are plagued with OS specific
>> conditionals.
>>
>> If the path of say:
>>
>> 	src/ceph-osd-prestart.sh
>>
>> is ether:
>>
>> 	/usr/libexec/ceph
>>
>> or:
>>
>> 	/usr/lib/ceph
>>
>>
>> Is purely distribution specific.
>>
>> Having conditionals in
>>
>> 	ceph.sepc.in
>>
>> defeats the point of generating it via autotools.
>>
>> we can be remove many hard coded values replaced with variable and that
>> probably will only grow in number for example
>>
>> 	%if 0%{?rhel} || 0%{?fedora}
>> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
>> 	%endif
>> 	%if 0%{?opensuse} || 0%{?suse_version}
>> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
>> 	%endif
>>
>> some wont need distribution specific locations like:
>>
>> 	--with-systemd-unit-dir=%_unitdir
>>
>> In the long term replaced all these path variables could be replaced by
>> single parameter.
>>
>> 	./configure --with-distro-defaults=redhat
>> 	make rpm
>> 	./configure --with-distro-defaults=suse
>> 	make rpm
>> 	./configure --with-distro-defaults=debian
>> 	make deb
>> 	./configure --with-distro-defaults=ubuntu
>> 	make deb
>>
>> Very easily.
>>
>> In summary, please reconsider this decision as if you follow this policy
>> we are left with conditionals all over the ceph.spec.in file.
>>
>>
>> Best regards
>>
>> Owen
>>
>>
>>
>>
>>
>>
>> On 05/05/2015 09:31 PM, Sage Weil wrote:
>>> On Tue, 5 May 2015, Loic Dachary wrote:
>>>>> I think the long-term solution to Kefu's issue is that we need to
>>>>> remove the requirement to run through a full "./configure" invocation
>>>>> just to get a tarball. All the RPM and Debian packages internally run
>>>>> ./configure, so running it a second time slows things down. I think it
>>>>> makes sense to implement the tarball-generation functionality using a
>>>>> simpler script at the root of the ceph.git tree. The operation should
>>>>> be about as fast as "git archive".
>>>>
>>>> I agree. It's going to be significant work but it's worth it.
>>>
>>> Yep!
>>>
>>>>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
>>>>> takes a full "./configure" run to get to a point where Make can write
>>>>> the proper version numbers into that file. Ideally we could skip all
>>>>> of that and simply do the variable interpolation with sed or something.
>>>
>>> Yep!
>>>
>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> -- 
>> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
>> 21284 (AG
>> Nürnberg)
>>
>> Maxfeldstraße 5
>>
>> 90409 Nürnberg
>>
>> Germany
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Configure dependencies can be the same as make dependencies

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 8978 bytes --]

On Tue, 9 Jun 2015, Owen Synge wrote:
> On 06/09/2015 07:22 PM, Sage Weil wrote:
> > On Tue, 9 Jun 2015, Owen Synge wrote:
> >> Sorry to catch this thread late.
> > 
> > There were two goals here:
> > 
> >  1- make the generated tarball deterministic and independent of 
> > configure options.  Currently we have horrible hackery to include rocksdb 
> > source in the tarball when configure doesn't get the rocksdb arguments, 
> > and were hitting similar roadblocks/pain with libxio.  This is avoided 
> > with the make_dist.sh script (although we're not using it yet because of 
> > the specfile generation piece).
> 
> I think deterministic is a reasonable goal.
> 
> I think independent of configure options, is not really a goal, its a
> solution to a goal that misses a couple of observations. The goal is
> deterministic operation.
> 
> The important observations are:
> 
> (a) The same source will need to be released multiple times.
> (b) It is nice to release with some differences to each environment.

This doesn't make sense to me.  There should be *one* upstream .tar.gz (or 
whatever) that is the canonical source associated with a release.  That is 
entirely independent of which distro you eventually build that source on, 
and should (more or less) correspond to a snapshot of the git tree for the 
release tag.

Currently the only reason ceph.spec is generated from autotools is to fill 
in the version string.  I can see why filling in all the other stuff makes 
sense, but that points toward a process like

 tar zxvf ceph-9.2.1.tar.gz    # that's the canonical release tarball
 cd ceph-9.2.1
 ./autogen.sh
 ./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
               --with-rpm-dir=/export/redhat/
 make srpm

right?

Making different release tarball flavors for each distro seems crazy to 
me...

sage



> 
> If you combine (a) and (b) you quickly see why the "--configure" step
> exists before:
> 
> 	make dist-bzip2
> 
> as say on release you iterate between distributions to produce each
> setup deterministically.
> 
> To illustrate I have put a little of my proposals for a replacement
> model of "make_dist.sh" to illustrate this in my opinion improved process.
> 
> But if this step exists imagining a couple of patches later:
> 
>  	$cat make_dist.sh
> 	#!/bin/sh -e
> 	# Common code
> 	autogen.sh
> 	# Redhat
> 	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
> 		--with-rpm-dir=/export/redhat/
>  	make srpm
> 	# Suse
>  	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
> 		--with-rpm-dir=/export/suse/
>  	make srpm
> 	# Debian
>  	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
> 		--with-sdeb-dir=/export/debian/
>  	make sdeb
> 	# Ubuntu
>  	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
> 		--with-sdeb-dir=/export/ubuntu
>  	make sdeb
> 
> This will also solve the problems (1) and (2) with ceph.spec.in to
> ceph.spec transformation. And not just solve it once but make it clearer
> for each distribution.
> 
> Problems.
> 
> These are problems with your planned approach I don't have good answers
> for, and this approach solves now.
> 
> (1) having autotools and another thing template ceph.spec.in could be
> complex.
> 
> (2) This code need to be documented as its not standard autotools.
> 
> (3) How much faster must the template tool be to be acceptably faster
> than ./configure
> 
> >  2- avoid multiple passes through ./configure for 'make dist' and the 
> > subsequent build.  We currently do it for the 'make dist' step (which also 
> > generates ceph.spec) and then for the eventual build.  Using autotools for 
> > generating the spec means we still do it twice.
> > 
> > I'm less concerned about build speed and it may be worth consolidating all 
> > of the distro logic in configure.ac.  Honestly I don't really know if 
> > that is better or not (vs conditionals in the specfile)... I have no real 
> > love for autotools.
> 
> autotools is not some thing I love, but I find it very useful, within
> the scope of building C/C++, I have found it has a solution with _every_
> use case already catered for. You just need some time to find it, and it
> may look different from what you thought you wanted and how you would
> make it, but I find the pre made autotools solution is a better solution
> than making your own from experience.
> 
> I must say though, autotools is much nicer than rpm for macros and
> variables and template expansion.
> 
> > But hopefully the tarball step can remain a fast, deterministic step that 
> > doesn't require a configure pass?
> 
> I really support you desire for deterministic.
> 
> I commend your desire that each step remain fast.
> 
> I believe you are missing a step if you follow the path of skipping the
> configure pass, and you loose some important functionality.
> 
> I point out now what functionality in the hope that we can use it to
> make other work easier, and autoools can be used to template any files
> it needs to, including at least, spec files and deb files by putting
> paths into autotools.
> 
> 
> Owen
> 
> 
> 
> > sage
> > 
> > 
> > 
> > 
> >>
> >> I come here via patch
> >>
> >> https://github.com/ceph/ceph/pull/4911#issuecomment-110422312
> >>
> >> I think you guys are missing that configure is doing some thing here.
> >>
> >> (1) Configure is generating the spec file.
> >> (2) It could also generate the deb files.
> >>
> >> What no one has done is to:
> >>
> >> (A) Use configure to eliminate the differences between OS's.
> >>
> >> If any one had done (A) you would not be considering this I hope.
> >>
> >> This impact is particularly felt in the ceph.spec.in file where because
> >> of the lack of (A) in the process we are plagued with OS specific
> >> conditionals.
> >>
> >> If the path of say:
> >>
> >> 	src/ceph-osd-prestart.sh
> >>
> >> is ether:
> >>
> >> 	/usr/libexec/ceph
> >>
> >> or:
> >>
> >> 	/usr/lib/ceph
> >>
> >>
> >> Is purely distribution specific.
> >>
> >> Having conditionals in
> >>
> >> 	ceph.sepc.in
> >>
> >> defeats the point of generating it via autotools.
> >>
> >> we can be remove many hard coded values replaced with variable and that
> >> probably will only grow in number for example
> >>
> >> 	%if 0%{?rhel} || 0%{?fedora}
> >> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
> >> 	%endif
> >> 	%if 0%{?opensuse} || 0%{?suse_version}
> >> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
> >> 	%endif
> >>
> >> some wont need distribution specific locations like:
> >>
> >> 	--with-systemd-unit-dir=%_unitdir
> >>
> >> In the long term replaced all these path variables could be replaced by
> >> single parameter.
> >>
> >> 	./configure --with-distro-defaults=redhat
> >> 	make rpm
> >> 	./configure --with-distro-defaults=suse
> >> 	make rpm
> >> 	./configure --with-distro-defaults=debian
> >> 	make deb
> >> 	./configure --with-distro-defaults=ubuntu
> >> 	make deb
> >>
> >> Very easily.
> >>
> >> In summary, please reconsider this decision as if you follow this policy
> >> we are left with conditionals all over the ceph.spec.in file.
> >>
> >>
> >> Best regards
> >>
> >> Owen
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 05/05/2015 09:31 PM, Sage Weil wrote:
> >>> On Tue, 5 May 2015, Loic Dachary wrote:
> >>>>> I think the long-term solution to Kefu's issue is that we need to
> >>>>> remove the requirement to run through a full "./configure" invocation
> >>>>> just to get a tarball. All the RPM and Debian packages internally run
> >>>>> ./configure, so running it a second time slows things down. I think it
> >>>>> makes sense to implement the tarball-generation functionality using a
> >>>>> simpler script at the root of the ceph.git tree. The operation should
> >>>>> be about as fast as "git archive".
> >>>>
> >>>> I agree. It's going to be significant work but it's worth it.
> >>>
> >>> Yep!
> >>>
> >>>>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
> >>>>> takes a full "./configure" run to get to a point where Make can write
> >>>>> the proper version numbers into that file. Ideally we could skip all
> >>>>> of that and simply do the variable interpolation with sed or something.
> >>>
> >>> Yep!
> >>>
> >>> sage
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >>
> >> -- 
> >> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
> >> 21284 (AG
> >> Nürnberg)
> >>
> >> Maxfeldstraße 5
> >>
> >> 90409 Nürnberg
> >>
> >> Germany
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> 
> -- 
> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
> 21284 (AG
> Nürnberg)
> 
> Maxfeldstraße 5
> 
> 90409 Nürnberg
> 
> Germany
> 
> 

Re: Configure dependencies can be the same as make dependencies

[Owen Synge]

On 06/09/2015 08:44 PM, Sage Weil wrote:
> On Tue, 9 Jun 2015, Owen Synge wrote:
>> On 06/09/2015 07:22 PM, Sage Weil wrote:
>>> On Tue, 9 Jun 2015, Owen Synge wrote:
>>>> Sorry to catch this thread late.
>>>
>>> There were two goals here:
>>>
>>>  1- make the generated tarball deterministic and independent of 
>>> configure options.  Currently we have horrible hackery to include rocksdb 
>>> source in the tarball when configure doesn't get the rocksdb arguments, 
>>> and were hitting similar roadblocks/pain with libxio.  This is avoided 
>>> with the make_dist.sh script (although we're not using it yet because of 
>>> the specfile generation piece).
>>
>> I think deterministic is a reasonable goal.
>>
>> I think independent of configure options, is not really a goal, its a
>> solution to a goal that misses a couple of observations. The goal is
>> deterministic operation.
>>
>> The important observations are:
>>
>> (a) The same source will need to be released multiple times.
>> (b) It is nice to release with some differences to each environment.
> 
> This doesn't make sense to me. 

I guess this is a matter of my poor wording.

> There should be *one* upstream .tar.gz (or 
> whatever) that is the canonical source associated with a release.  

I agree, git at git hub provides this with a tag.

I then think it is presented, on a web site, at this stage the upstream
.tar.gz can be post processed if needed.

The source is taken by distributions such as redhat, suse, debian,
ubuntu, and presented to users both as source and as binaries.

Packaging files including ceph.spec being made from ceph.spec.in are
part of that post release presentation work that have (for reasons of
pragmatism) been placed in the same git repository.

While this packaging code/config has to be traceable to a canonical
source, they do not have to be the canonical source, and may be
generated from templates, provided the values are stored.

> That is
> entirely independent of which distro you eventually build that source on, 
> and should (more or less) correspond to a snapshot of the git tree for the 
> release tag.

To clarify; the bits where it does not correspond to a fixed value  in
git that this discussion is based on.

I suggest all such values can be stored together and placed in a release
script.

> Currently the only reason ceph.spec is generated from autotools is to fill 
> in the version string.  I can see why filling in all the other stuff makes 
> sense, but that points toward a process like
> 
>  tar zxvf ceph-9.2.1.tar.gz    # that's the canonical release tarball
>  cd ceph-9.2.1
>  ./autogen.sh
>  ./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
>                --with-rpm-dir=/export/redhat/
>  make srpm
> 
> right?

right,

> Making different release tarball flavors for each distro seems crazy to 
> me...

Thankfully I cant think of anything in ceph that it would needed be
needed for apart from the version information.

If you had more variables like the version information that was
distribution specific then per distribution tar balls might be needed.

Should their be a "support contract" variable, this might make the need
for different source releases having different values in templated code.
I am glad ceph does not have or need such a variable.

Owen


>> If you combine (a) and (b) you quickly see why the "--configure" step
>> exists before:
>>
>> 	make dist-bzip2
>>
>> as say on release you iterate between distributions to produce each
>> setup deterministically.
>>
>> To illustrate I have put a little of my proposals for a replacement
>> model of "make_dist.sh" to illustrate this in my opinion improved process.
>>
>> But if this step exists imagining a couple of patches later:
>>
>>  	$cat make_dist.sh
>> 	#!/bin/sh -e
>> 	# Common code
>> 	autogen.sh
>> 	# Redhat
>> 	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
>> 		--with-rpm-dir=/export/redhat/
>>  	make srpm
>> 	# Suse
>>  	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
>> 		--with-rpm-dir=/export/suse/
>>  	make srpm
>> 	# Debian
>>  	./configure --with-systemd-libexec-dir=/usr/lib/ceph/ \
>> 		--with-sdeb-dir=/export/debian/
>>  	make sdeb
>> 	# Ubuntu
>>  	./configure --with-systemd-libexec-dir=/usr/libexec/ceph \
>> 		--with-sdeb-dir=/export/ubuntu
>>  	make sdeb
>>
>> This will also solve the problems (1) and (2) with ceph.spec.in to
>> ceph.spec transformation. And not just solve it once but make it clearer
>> for each distribution.
>>
>> Problems.
>>
>> These are problems with your planned approach I don't have good answers
>> for, and this approach solves now.
>>
>> (1) having autotools and another thing template ceph.spec.in could be
>> complex.
>>
>> (2) This code need to be documented as its not standard autotools.
>>
>> (3) How much faster must the template tool be to be acceptably faster
>> than ./configure
>>
>>>  2- avoid multiple passes through ./configure for 'make dist' and the 
>>> subsequent build.  We currently do it for the 'make dist' step (which also 
>>> generates ceph.spec) and then for the eventual build.  Using autotools for 
>>> generating the spec means we still do it twice.
>>>
>>> I'm less concerned about build speed and it may be worth consolidating all 
>>> of the distro logic in configure.ac.  Honestly I don't really know if 
>>> that is better or not (vs conditionals in the specfile)... I have no real 
>>> love for autotools.
>>
>> autotools is not some thing I love, but I find it very useful, within
>> the scope of building C/C++, I have found it has a solution with _every_
>> use case already catered for. You just need some time to find it, and it
>> may look different from what you thought you wanted and how you would
>> make it, but I find the pre made autotools solution is a better solution
>> than making your own from experience.
>>
>> I must say though, autotools is much nicer than rpm for macros and
>> variables and template expansion.
>>
>>> But hopefully the tarball step can remain a fast, deterministic step that 
>>> doesn't require a configure pass?
>>
>> I really support you desire for deterministic.
>>
>> I commend your desire that each step remain fast.
>>
>> I believe you are missing a step if you follow the path of skipping the
>> configure pass, and you loose some important functionality.
>>
>> I point out now what functionality in the hope that we can use it to
>> make other work easier, and autoools can be used to template any files
>> it needs to, including at least, spec files and deb files by putting
>> paths into autotools.
>>
>>
>> Owen
>>
>>
>>
>>> sage
>>>
>>>
>>>
>>>
>>>>
>>>> I come here via patch
>>>>
>>>> https://github.com/ceph/ceph/pull/4911#issuecomment-110422312
>>>>
>>>> I think you guys are missing that configure is doing some thing here.
>>>>
>>>> (1) Configure is generating the spec file.
>>>> (2) It could also generate the deb files.
>>>>
>>>> What no one has done is to:
>>>>
>>>> (A) Use configure to eliminate the differences between OS's.
>>>>
>>>> If any one had done (A) you would not be considering this I hope.
>>>>
>>>> This impact is particularly felt in the ceph.spec.in file where because
>>>> of the lack of (A) in the process we are plagued with OS specific
>>>> conditionals.
>>>>
>>>> If the path of say:
>>>>
>>>> 	src/ceph-osd-prestart.sh
>>>>
>>>> is ether:
>>>>
>>>> 	/usr/libexec/ceph
>>>>
>>>> or:
>>>>
>>>> 	/usr/lib/ceph
>>>>
>>>>
>>>> Is purely distribution specific.
>>>>
>>>> Having conditionals in
>>>>
>>>> 	ceph.sepc.in
>>>>
>>>> defeats the point of generating it via autotools.
>>>>
>>>> we can be remove many hard coded values replaced with variable and that
>>>> probably will only grow in number for example
>>>>
>>>> 	%if 0%{?rhel} || 0%{?fedora}
>>>> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
>>>> 	%endif
>>>> 	%if 0%{?opensuse} || 0%{?suse_version}
>>>> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
>>>> 	%endif
>>>>
>>>> some wont need distribution specific locations like:
>>>>
>>>> 	--with-systemd-unit-dir=%_unitdir
>>>>
>>>> In the long term replaced all these path variables could be replaced by
>>>> single parameter.
>>>>
>>>> 	./configure --with-distro-defaults=redhat
>>>> 	make rpm
>>>> 	./configure --with-distro-defaults=suse
>>>> 	make rpm
>>>> 	./configure --with-distro-defaults=debian
>>>> 	make deb
>>>> 	./configure --with-distro-defaults=ubuntu
>>>> 	make deb
>>>>
>>>> Very easily.
>>>>
>>>> In summary, please reconsider this decision as if you follow this policy
>>>> we are left with conditionals all over the ceph.spec.in file.
>>>>
>>>>
>>>> Best regards
>>>>
>>>> Owen
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 05/05/2015 09:31 PM, Sage Weil wrote:
>>>>> On Tue, 5 May 2015, Loic Dachary wrote:
>>>>>>> I think the long-term solution to Kefu's issue is that we need to
>>>>>>> remove the requirement to run through a full "./configure" invocation
>>>>>>> just to get a tarball. All the RPM and Debian packages internally run
>>>>>>> ./configure, so running it a second time slows things down. I think it
>>>>>>> makes sense to implement the tarball-generation functionality using a
>>>>>>> simpler script at the root of the ceph.git tree. The operation should
>>>>>>> be about as fast as "git archive".
>>>>>>
>>>>>> I agree. It's going to be significant work but it's worth it.
>>>>>
>>>>> Yep!
>>>>>
>>>>>>> The "ceph.spec.in" -> "ceph.spec" suffers from a similar issue. It
>>>>>>> takes a full "./configure" run to get to a point where Make can write
>>>>>>> the proper version numbers into that file. Ideally we could skip all
>>>>>>> of that and simply do the variable interpolation with sed or something.
>>>>>
>>>>> Yep!
>>>>>
>>>>> sage
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>
>>>> -- 
>>>> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
>>>> 21284 (AG
>>>> Nürnberg)
>>>>
>>>> Maxfeldstraße 5
>>>>
>>>> 90409 Nürnberg
>>>>
>>>> Germany
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>
>> -- 
>> SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
>> 21284 (AG
>> Nürnberg)
>>
>> Maxfeldstraße 5
>>
>> 90409 Nürnberg
>>
>> Germany
>>

-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: what this would look like if the system is with templates.

[Ken Dreyer]

On 06/09/2015 11:19 AM, Owen Synge wrote:
> 
>> we can be remove many hard coded values replaced with variable and that
>> probably will only grow in number for example
>>
>> 	%if 0%{?rhel} || 0%{?fedora}
>> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
>> 	%endif
>> 	%if 0%{?opensuse} || 0%{?suse_version}
>> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
>> 	%endif
> 
> 	--with-systemd-libexec-dir=@systemd_libexec_dir@ \
> 
> No OS distribution specific rubbish needed :)

Passing an autoconf variable (@systemd_libexec_dir@) to an autoconf
argument (--with-systemd-libexec-dir) seems really over-complicated to me.

I don't see the issues with putting os-specific things in the .spec
file; that's how many other (non-Ceph) projects do it.

- Ken

Re: what this would look like if the system is with templates.

[Owen Synge]

On 06/10/2015 01:06 AM, Ken Dreyer wrote:
> On 06/09/2015 11:19 AM, Owen Synge wrote:
>>
>>> we can be remove many hard coded values replaced with variable and that
>>> probably will only grow in number for example
>>>
>>> 	%if 0%{?rhel} || 0%{?fedora}
>>> 		--with-systemd-libexec-dir=/usr/libexec/ceph \
>>> 	%endif
>>> 	%if 0%{?opensuse} || 0%{?suse_version}
>>> 		--with-systemd-libexec-dir=/usr/lib/ceph/ \
>>> 	%endif
>>
>> 	--with-systemd-libexec-dir=@systemd_libexec_dir@ \
>>
>> No OS distribution specific rubbish needed :)
> 
> Passing an autoconf variable (@systemd_libexec_dir@) to an autoconf
> argument (--with-systemd-libexec-dir) seems really over-complicated to me.

It makes separating things like version info from function hard.

The %else function is a minefield.

The %if statements around nearly every line in configure give you a hint
of the problem.

Autotools is not the last word in pretty language but it does warn when
you do platform specific stuff very well, and its a lot more flexible
than rpm.

> I don't see the issues with putting os-specific things in the .spec
> file; that's how many other (non-Ceph) projects do it.

These conditionals do not occur just done once if used in the spec.in
file though its worse, you need the same conditionals on each of

	configure
	files
	install

Admittedly if we follow the current process of deprecating direct
platform based installs we can get the conditionals down to two places
in the spec.in file rather than 3.

I think having the conditionals set in one place is superior to 2 or 3
places. To do this you would template the rpm spec file.

Best regards

Owen