* [Buildroot] Design issue with the out-of-tree support
@ 2013-05-23 11:12 Thomas Petazzoni
2013-05-23 12:43 ` Will Wagner
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Thomas Petazzoni @ 2013-05-23 11:12 UTC (permalink / raw)
To: buildroot
Hello,
I've restarted the work on supporting the per-package out-of-tree
support, and I stumbled across a problem about which I'd like to seek
the community opinion.
Currently, the "autoreconf" is executed as a pre-configure hook, i.e it
is part of the configure step, and this is wrong when you move to
out-of-tree build per packages.
When you do out-of-tree build per package, for each package, you have:
* A source tree, output/src/<name>-<version>/
* A target build tree, output/build/<name>-<version>/
* Optionally, a host build tree, output/build/host-<name>-<version>/
Running the configure script is something that should be done in the
build tree, so it's really part of the steps that should be executed
once for the target build, once for the host build.
However, running autoreconf is something that should be done in the
source tree. It's part of the "patching" process of the package, more
than its configuration step.
So, in my out-of-tree branch, I've moved the autoreconf step as a
post-patch hook rather than a pre-configure hook.
The problem is that dependencies of a package are only prepared before
its configure step is executed. This means that host-autoconf,
host-automake and host-libtool are not yet available at the "patch"
step, so we can't autoreconf yet.
You could tell me: let's move the package dependencies as a dependency
of the patch step rather than the configure step. Yes, you could tell
me that. But it would break the RTAI integration that we have in
Buildroot, which relies on the assumption that the patch step of a
given package can be executed without its dependencies being pulled in.
See a473a616d27e46951a23d90249a31b08006098bf (package: change ordering
of steps).
So, what I've done in my prototype of out-of-tree support is something
like this:
$(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)
...
$(1)-configure: $$($(2)_DEPENDENCIES)
The idea is that if the package needs host-autoconf, or host-automake
or host-libtool in its dependencies, then we pull these dependencies
before the "patch" step, and all other dependencies are pulled before
the "configure" step.
This works for most packages, but not all! Some packages, for their
autoreconf step, need more than host-autoconf, host-automake or
host-libtool. They might require host-pkgconf or host-gettext, to
install some .m4 macro files that are needed to make the autoreconf
work. So in fact, discriminating between dependencies that should be
pulled in before the "patch" step and the dependencies that can wait
until the "configure" step is not simple.
Do you see solutions to this?
I see several possibilities:
* In all packages, split in two variables the dependencies needed at
autoreconf time from the dependencies needed at configure time. But
this seems horrible and difficult to maintain and understand for
users.
* Make $(1)-patch depend on all dependencies. This will break RTAI, so
we can exclude 'linux' for the list of $(1)-patch dependencies. So
something like:
$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
...
$(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))
...
* Make autoreconf a step on its own, instead of being either a
pre-patch hook or a post-patch hook. This may also allow to do
something like a 'make <pkg>-reautoreconf' target, like we have
'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this
autoreconf step would be the one that has:
$(1)-autoreconf: $$($(2)_DEPENDENCIES)
which would work ok, since the RTAI/Linux integration depends on
rtai-patch, which wouldn't pull the dependencies of the rtai package.
However, I am not yet sure how to insert this step into the package
logic, since this step is specific to autotools package, and
therefore would normally not belong to the pkg-generic.mk
infrastructure.
* Find a different solution for RTAI, so that we can pull the
dependencies before the "patch" step.
Here is a quick explanation for the RTAI issue, to refresh your minds
if needed:
RTAI is delivered as a tarball, which has a configure script to build
its userspace libraries, so it is a autotools package, see
package/rtai/.
This tarball also contains the kernel patches that must be applied.
In order for the kernel to be built, we need these patches to be
available, so the package/rtai package must be extracted and patched.
This explains why you have LINUX_DEPENDENCIES += rtai-patch in
linux/linux-ext-rtai.mk.
And on the other hand, the 'rtai' package itself can't be
configured/built *before* the Linux kernel has been built, because it
builds kernel modules, and to build kernel modules, you need an
already built kernel tree (and remember, to build the kernel, we must
first apply the RTAI patches).
Ideas?
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-23 11:12 [Buildroot] Design issue with the out-of-tree support Thomas Petazzoni
@ 2013-05-23 12:43 ` Will Wagner
2013-05-23 12:53 ` Thomas Petazzoni
2013-05-23 18:12 ` Yann E. MORIN
2013-05-27 16:50 ` Arnout Vandecappelle
2 siblings, 1 reply; 10+ messages in thread
From: Will Wagner @ 2013-05-23 12:43 UTC (permalink / raw)
To: buildroot
On 23/05/2013 12:12, Thomas Petazzoni wrote:
>
> So, what I've done in my prototype of out-of-tree support is something
> like this:
>
> $(1)-patch: $(filter host-autoconf host-automake host-libtool,$$($(2)_DEPENDENCIES)
> ...
>
> $(1)-configure: $$($(2)_DEPENDENCIES)
>
> The idea is that if the package needs host-autoconf, or host-automake
> or host-libtool in its dependencies, then we pull these dependencies
> before the "patch" step, and all other dependencies are pulled before
> the "configure" step.
>
> This works for most packages, but not all! Some packages, for their
> autoreconf step, need more than host-autoconf, host-automake or
> host-libtool. They might require host-pkgconf or host-gettext, to
> install some .m4 macro files that are needed to make the autoreconf
> work. So in fact, discriminating between dependencies that should be
> pulled in before the "patch" step and the dependencies that can wait
> until the "configure" step is not simple.
>
> Do you see solutions to this?
Thomas, I might be missing something but it seems like all the needed
dependencies are host dependencies. Can we not make all host
dependencies be on the patch step and all others on the configure step.
I think that would work for rtai wouldn't it?
Regards
Will
--
------------------------------------------------------------------------
Will Wagner will_wagner at carallon.com
Development Manager Office Tel: +44 (0)20 7471 9224
Carallon Ltd, Studio G20, Shepherds Building, Rockley Rd, London W14 0DA
------------------------------------------------------------------------
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-23 12:43 ` Will Wagner
@ 2013-05-23 12:53 ` Thomas Petazzoni
2013-05-23 14:49 ` Markos Chandras
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-05-23 12:53 UTC (permalink / raw)
To: buildroot
Dear Will Wagner,
On Thu, 23 May 2013 13:43:24 +0100, Will Wagner wrote:
> > Do you see solutions to this?
>
> Thomas, I might be missing something but it seems like all the needed
> dependencies are host dependencies. Can we not make all host
> dependencies be on the patch step and all others on the configure step.
> I think that would work for rtai wouldn't it?
Wow, thanks for taking the time to look into the problem, and enter the
discussion!
It seems like what you're proposing would work. It is not worst than the
solution of filtering out linux on one side, and including linux on the
other side that I was proposing, but I still don't find this very clean.
While it solves the problem, the solution seems completely disconnected
from it. In a few years: "Why the heck do we have host dependencies
before patch, and other dependencies before configure, instead of
having all of them together before patch? Ah, yes, because there is
this crazy specific need for RTAI".
So certainly, what you're proposing solves the problem, it just doesn't
really feel like a nice and clean solution. I know it's a very
subjective opinion I'm giving here, so if others do agree with that
proposal, I'll be fine with it, but I'd be interested in hearing the
ideas and opinions of others to see if we can come up with something
cleaner.
Thanks!
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-23 12:53 ` Thomas Petazzoni
@ 2013-05-23 14:49 ` Markos Chandras
0 siblings, 0 replies; 10+ messages in thread
From: Markos Chandras @ 2013-05-23 14:49 UTC (permalink / raw)
To: buildroot
Hi Thomas,
> While it solves the problem, the solution seems completely disconnected
> from it. In a few years: "Why the heck do we have host dependencies
> before patch, and other dependencies before configure, instead of
> having all of them together before patch? Ah, yes, because there is
> this crazy specific need for RTAI".
In my opinion, having two kind of dependencies will create headaches
and it will make the current
packaging process more complicated.
>
> So certainly, what you're proposing solves the problem, it just doesn't
> really feel like a nice and clean solution. I know it's a very
> subjective opinion I'm giving here, so if others do agree with that
> proposal, I'll be fine with it, but I'd be interested in hearing the
> ideas and opinions of others to see if we can come up with something
> cleaner.
>
Like you said in your first e-mail, I am in favor of this approach:
* Make $(1)-patch depend on all dependencies. This will break RTAI, so
we can exclude 'linux' for the list of $(1)-patch dependencies. So
something like:
$(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
...
$(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))
--
Regards,
Markos Chandras
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-23 11:12 [Buildroot] Design issue with the out-of-tree support Thomas Petazzoni
2013-05-23 12:43 ` Will Wagner
@ 2013-05-23 18:12 ` Yann E. MORIN
2013-05-27 16:50 ` Arnout Vandecappelle
2 siblings, 0 replies; 10+ messages in thread
From: Yann E. MORIN @ 2013-05-23 18:12 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2013-05-23 13:12 +0200, Thomas Petazzoni spake thusly:
> I've restarted the work on supporting the per-package out-of-tree
> support, and I stumbled across a problem about which I'd like to seek
> the community opinion.
Thank you for this thorough explanations!
> However, running autoreconf is something that should be done in the
> source tree. It's part of the "patching" process of the package, more
> than its configuration step.
Agreed.
> I see several possibilities:
>
> * In all packages, split in two variables the dependencies needed at
> autoreconf time from the dependencies needed at configure time. But
> this seems horrible and difficult to maintain and understand for
> users.
Nope.
> * Make $(1)-patch depend on all dependencies. This will break RTAI, so
> we can exclude 'linux' for the list of $(1)-patch dependencies. So
> something like:
>
> $(1)-patch: $(filter-out linux,$$($(2)_DEPENDENCIES))
> ...
>
> $(1)-configure: $(filter linux,$$($(2)_DEPENDENCIES))
> ...
What when we have another package like that? And a third?
That's not really nice IMHO.
> * Make autoreconf a step on its own, instead of being either a
> pre-patch hook or a post-patch hook. This may also allow to do
> something like a 'make <pkg>-reautoreconf' target, like we have
> 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this
> autoreconf step would be the one that has:
>
> $(1)-autoreconf: $$($(2)_DEPENDENCIES)
I believe that's the sane approach.
> which would work ok, since the RTAI/Linux integration depends on
> rtai-patch, which wouldn't pull the dependencies of the rtai package.
>
> However, I am not yet sure how to insert this step into the package
> logic, since this step is specific to autotools package, and
> therefore would normally not belong to the pkg-generic.mk
> infrastructure.
Maybe have the autotools infra override the dependency chain, by
tweaking $(2)_TARGET_CONFIGURE or something like that?
> * Find a different solution for RTAI, so that we can pull the
> dependencies before the "patch" step.
I'd say we should keep it that way, and wait for another one or two
packages (if they even exist) with similar requirements, before we try
to get a generic solution for that RTAI issue.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-23 11:12 [Buildroot] Design issue with the out-of-tree support Thomas Petazzoni
2013-05-23 12:43 ` Will Wagner
2013-05-23 18:12 ` Yann E. MORIN
@ 2013-05-27 16:50 ` Arnout Vandecappelle
2013-05-27 19:12 ` Thomas Petazzoni
2 siblings, 1 reply; 10+ messages in thread
From: Arnout Vandecappelle @ 2013-05-27 16:50 UTC (permalink / raw)
To: buildroot
On 23/05/13 13:12, Thomas Petazzoni wrote:
> * Make autoreconf a step on its own, instead of being either a
> pre-patch hook or a post-patch hook. This may also allow to do
> something like a 'make <pkg>-reautoreconf' target, like we have
> 'make <pkg>-reconfigure' and 'make <pkg>-rebuild'. Then, this
> autoreconf step would be the one that has:
>
> $(1)-autoreconf: $$($(2)_DEPENDENCIES)
>
> which would work ok, since the RTAI/Linux integration depends on
> rtai-patch, which wouldn't pull the dependencies of the rtai package.
>
> However, I am not yet sure how to insert this step into the package
> logic, since this step is specific to autotools package, and
> therefore would normally not belong to the pkg-generic.mk
> infrastructure.
I haven't read the thread in detail or thought about it very carefully,
but some time ago I already thought that this would be better.
It is not necessarily limited to autoreconf. For example, the way that
xenomai patches the linux sources fits in this as well (xenomai could add
to LINUX_POST_AUTORECONF_HOOKS).
So I would propose to add another step to the generic infrastructure. I
would call it the PREPARE step, and make it a full step with CMDS and HOOKS.
The question is whether the OVERRIDE_SRCDIR infrastructure should apply
this step or not. Conceptually, it shouldn't, but that means that users
of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do
the autoreconf manually. Which is probably for the best anyway, but which
is a change compared to current behaviour.
BTW the libtool fixup should probably be done in this prepare step as
well. But again, I haven't thought this through in detail.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-27 16:50 ` Arnout Vandecappelle
@ 2013-05-27 19:12 ` Thomas Petazzoni
2013-05-27 19:44 ` Arnout Vandecappelle
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-05-27 19:12 UTC (permalink / raw)
To: buildroot
Dear Arnout Vandecappelle,
On Mon, 27 May 2013 18:50:54 +0200, Arnout Vandecappelle wrote:
> I haven't read the thread in detail or thought about it very carefully,
> but some time ago I already thought that this would be better.
>
> It is not necessarily limited to autoreconf. For example, the way that
> xenomai patches the linux sources fits in this as well (xenomai could add
> to LINUX_POST_AUTORECONF_HOOKS).
>
> So I would propose to add another step to the generic infrastructure. I
> would call it the PREPARE step, and make it a full step with CMDS and HOOKS.
Indeed. For now, it's only has a CMDS and no HOOKS, but it could be
added later on.
> The question is whether the OVERRIDE_SRCDIR infrastructure should apply
> this step or not. Conceptually, it shouldn't, but that means that users
> of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do
> the autoreconf manually. Which is probably for the best anyway, but which
> is a change compared to current behaviour.
For now, I consider doing the prepare step even on OVERRIDE_SRCDIR
package, even if that means that we are modifying the source directory.
> BTW the libtool fixup should probably be done in this prepare step as
> well. But again, I haven't thought this through in detail.
This is already done:
http://git.free-electrons.com/users/thomas-petazzoni/buildroot/commit/?h=out-of-tree-v3&id=5cf6166d713a43cdf83cdf794502eb77be36d43f
Thanks for the feedback!
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-27 19:12 ` Thomas Petazzoni
@ 2013-05-27 19:44 ` Arnout Vandecappelle
2013-05-27 19:53 ` Thomas Petazzoni
0 siblings, 1 reply; 10+ messages in thread
From: Arnout Vandecappelle @ 2013-05-27 19:44 UTC (permalink / raw)
To: buildroot
On 27/05/13 21:12, Thomas Petazzoni wrote:
> Dear Arnout Vandecappelle,
>
> On Mon, 27 May 2013 18:50:54 +0200, Arnout Vandecappelle wrote:
[snip]
>> The question is whether the OVERRIDE_SRCDIR infrastructure should apply
>> this step or not. Conceptually, it shouldn't, but that means that users
>> of OVERRIDE_SRCDIR for packages that need an autoreconf would need to do
>> the autoreconf manually. Which is probably for the best anyway, but which
>> is a change compared to current behaviour.
>
> For now, I consider doing the prepare step even on OVERRIDE_SRCDIR
> package, even if that means that we are modifying the source directory.
Given that we've recently accepted some patches that solve problems of
non-writeable source directories, I don't think this is appropriate. I
think it's a valid use case to use OVERRIDE_SRCDIR to define the custom
source of a package, which is located in some shared directory. Having
random builds modifying that source is not appropriate.
And now is the time to define this behaviour - changing it later can be
considered "ABI breakage".
Regards,
Arnout
[snip]
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-27 19:44 ` Arnout Vandecappelle
@ 2013-05-27 19:53 ` Thomas Petazzoni
2013-05-28 19:26 ` Arnout Vandecappelle
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-05-27 19:53 UTC (permalink / raw)
To: buildroot
Dear Arnout Vandecappelle,
On Mon, 27 May 2013 21:44:19 +0200, Arnout Vandecappelle wrote:
> > For now, I consider doing the prepare step even on OVERRIDE_SRCDIR
> > package, even if that means that we are modifying the source directory.
>
> Given that we've recently accepted some patches that solve problems of
> non-writeable source directories, I don't think this is appropriate. I
> think it's a valid use case to use OVERRIDE_SRCDIR to define the custom
> source of a package, which is located in some shared directory. Having
> random builds modifying that source is not appropriate.
>
> And now is the time to define this behaviour - changing it later can be
> considered "ABI breakage".
Right.
What about packages that are using the "local" site method (which
basically, are equivalent to doing OVERRIDE_SRCDIR) ?
I know some people (or, said otherwise, customers) that use the local
SITE_METHOD for autotools-based components. Those are version
controlled, and of course, only configure.ac + Makefile.am are version
controlled. So, when the build starts, "something" should do the
autoreconf.
If Buildroot doesn't do it, who will do it? The user manually, when
there are 30+ software components? Doesn't seem really practical and
realistic.
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] Design issue with the out-of-tree support
2013-05-27 19:53 ` Thomas Petazzoni
@ 2013-05-28 19:26 ` Arnout Vandecappelle
0 siblings, 0 replies; 10+ messages in thread
From: Arnout Vandecappelle @ 2013-05-28 19:26 UTC (permalink / raw)
To: buildroot
On 27/05/13 21:53, Thomas Petazzoni wrote:
> Dear Arnout Vandecappelle,
>
> On Mon, 27 May 2013 21:44:19 +0200, Arnout Vandecappelle wrote:
>
>>> For now, I consider doing the prepare step even on OVERRIDE_SRCDIR
>>> package, even if that means that we are modifying the source directory.
>>
>> Given that we've recently accepted some patches that solve problems of
>> non-writeable source directories, I don't think this is appropriate. I
>> think it's a valid use case to use OVERRIDE_SRCDIR to define the custom
>> source of a package, which is located in some shared directory. Having
>> random builds modifying that source is not appropriate.
>>
>> And now is the time to define this behaviour - changing it later can be
>> considered "ABI breakage".
>
> Right.
>
> What about packages that are using the "local" site method (which
> basically, are equivalent to doing OVERRIDE_SRCDIR) ?
>
> I know some people (or, said otherwise, customers) that use the local
> SITE_METHOD for autotools-based components. Those are version
> controlled, and of course, only configure.ac + Makefile.am are version
> controlled. So, when the build starts, "something" should do the
> autoreconf.
>
> If Buildroot doesn't do it, who will do it? The user manually, when
> there are 30+ software components? Doesn't seem really practical and
> realistic.
Well, actually the autotools-generated makefile will itself detect that
configure.ac has changed and regenerate it in the source directory, so a
non-writeable source directory will be broken anyway.
Okay then. If anyone has an issue with buildroot messing with their
OVERRIDE_SRCDIR, they can still add a OVERRIDE_SKIP_PREPARE variable that
disables the prepare step.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-05-28 19:26 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-23 11:12 [Buildroot] Design issue with the out-of-tree support Thomas Petazzoni
2013-05-23 12:43 ` Will Wagner
2013-05-23 12:53 ` Thomas Petazzoni
2013-05-23 14:49 ` Markos Chandras
2013-05-23 18:12 ` Yann E. MORIN
2013-05-27 16:50 ` Arnout Vandecappelle
2013-05-27 19:12 ` Thomas Petazzoni
2013-05-27 19:44 ` Arnout Vandecappelle
2013-05-27 19:53 ` Thomas Petazzoni
2013-05-28 19:26 ` Arnout Vandecappelle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.