[Buildroot] Installation of package files in staging and target

[Buildroot] Installation of package files in staging and target

[Thomas Petazzoni]

Hello,

Currently, each userspace package is allowed to install files in both
the staging directory and the target directory.

Typically, a "program" package will install its program(s) only in the
target directory, while a "library" package will install its library
and headers in the staging directory and in the target directory (the
headers, static libraries and other useless files are removed using
global mechanisms before creating the target filesystem image).

I have just sent a patch that proposes to enable debugging symbols by
default when building packages. This would currently ensure that all
libraries installed in 'staging' have debugging symbols. However,
programs would be built with debugging symbols, but as they are only
installed in 'target', the final binary would still be stripped.

I'd like to propose the following scheme :

 * The packages no longer install files in staging and target
   differently. They just install files.

 * All files are installed in both the staging and the target
   directory, by the package infrastructure.

 * The files in the staging directory are left untouched.

 * Some automatic cleanup is done in the target directory :
    - Stripping of binaries and libraries
    - Removal of static libraries, pkg-config files, header files,
      documentation, etc.
    - Removal of files listed in a per-package <pkg>_REMOVE_TARGET
      variable.

This would make the target directory be a cleaned up and stripped
*subset* of the staging directory. We would therefore have in the
staging directory the libraries *and* binaries with debugging symbols
for all packages.

What do you think about this ?

Of course, this is not something that is going to be implemented right
now. I'm just opening the discussion to see if this makes sense, and if
so, how we could move forward.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Installation of package files in staging and target

[Yann E. MORIN]

Thomas, All,

On Tuesday 12 July 2011 00:01:17 Thomas Petazzoni wrote:
> I have just sent a patch that proposes to enable debugging symbols by
> default when building packages. This would currently ensure that all
> libraries installed in 'staging' have debugging symbols. However,
> programs would be built with debugging symbols, but as they are only
> installed in 'target', the final binary would still be stripped.
> 
> I'd like to propose the following scheme :
> 
>  * The packages no longer install files in staging and target
>    differently. They just install files.
> 
>  * All files are installed in both the staging and the target
>    directory, by the package infrastructure.
> 
>  * The files in the staging directory are left untouched.
> 
>  * Some automatic cleanup is done in the target directory :
>     - Stripping of binaries and libraries
>     - Removal of static libraries, pkg-config files, header files,
>       documentation, etc.
>     - Removal of files listed in a per-package <pkg>_REMOVE_TARGET
>       variable.
> 
> This would make the target directory be a cleaned up and stripped
> *subset* of the staging directory. We would therefore have in the
> staging directory the libraries *and* binaries with debugging symbols
> for all packages.

In this case, why install in both places? Would something like the
following be part of the solution:

 * packages install in staging, and nowhere else

 * building the image(s) makes a copy of staging -> target
   - leaves staging untouched
   - cleans up target: strip, rm man include pkg-config, and all the
     superfluous stuff you highlighted below...

Of course, I am not saying that this is easier, or even better! ;-)
In this case, the problematic part would be to gather all the
<pkg>_REMOVE_TARGET stuff. Maybe the pkg infra could do smthg
like this when $(eval $(call ....)) is called:
  REMOVE_TARGET+=$($pkg)_REMOVE_TARGET)

And then the image build would do (Yeah, ultra-simplified...):
  cp -a staging target; \
  cd target; \
  rm -rf $(REMOVE_TARGET); \
  rm -rf man doc include blabla...; \
  strip bin/foo; \
  mk-image-or-tarball-or-whatever --from=. --to=output/root.img

I don't know how spaces in REMOVE_TARGET would be handled, though... ;-/
Also, leading '/' should be removed, or beware... :-]

> What do you think about this ?

This is IMHO a sane suggestion. :-)

> Of course, this is not something that is going to be implemented right
> now. I'm just opening the discussion to see if this makes sense, and if
> so, how we could move forward.

Hehe! There are only 24h in day, unfortunately!... :-]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

Hello Thomas, all,

On Tue, 2011-07-12 at 00:01 +0200, Thomas Petazzoni wrote:
> Typically, a "program" package will install its program(s) only in the
> target directory, while a "library" package will install its library
> and headers in the staging directory and in the target directory (the
> headers, static libraries and other useless files are removed using
> global mechanisms before creating the target filesystem image).
> 
<snip>
> This would make the target directory be a cleaned up and stripped
> *subset* of the staging directory. We would therefore have in the
> staging directory the libraries *and* binaries with debugging symbols
> for all packages.
> 
> What do you think about this ?

I'm a little behind in tracking the changes with toolchain movements
from a few months back but I was of the understanding that the current
install scheme still allows the staging directory to be treated as a
kind of mini SDK and so if I build a system using Buildroot I can tar up
the staging dir and give it to application developers as an interface.
Is this still the case?  If so, wouldn't putting executable programs
into the staging dir be a bad thing?

[Buildroot] Installation of package files in staging and target

[Thomas Petazzoni]

Le Tue, 12 Jul 2011 11:55:12 +0100,
Quotient Remainder <quotientvremainder@gmail.com> a ?crit :

> I'm a little behind in tracking the changes with toolchain movements
> from a few months back but I was of the understanding that the current
> install scheme still allows the staging directory to be treated as a
> kind of mini SDK and so if I build a system using Buildroot I can tar
> up the staging dir and give it to application developers as an
> interface. Is this still the case?  If so, wouldn't putting
> executable programs into the staging dir be a bad thing?

Why would it be bad ? The binaries for the target are not inside
host/usr/bin, but inside host/usr/<tuple>/sysroot/usr/bin, so even if
the user adds host/usr/bin to its path to get access to the SDK, the
target binaries won't be visible.

It's true however that this would enlarge the size of the SDK.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

On Tue, 2011-07-12 at 13:51 +0200, Thomas Petazzoni wrote:
> Le Tue, 12 Jul 2011 11:55:12 +0100,
> Quotient Remainder <quotientvremainder@gmail.com> a ?crit :
> 
> > I'm a little behind in tracking the changes with toolchain movements
> > from a few months back but I was of the understanding that the current
> > install scheme still allows the staging directory to be treated as a
> > kind of mini SDK and so if I build a system using Buildroot I can tar
> > up the staging dir and give it to application developers as an
> > interface. Is this still the case?  If so, wouldn't putting
> > executable programs into the staging dir be a bad thing?
> 
> Why would it be bad ? The binaries for the target are not inside
> host/usr/bin, but inside host/usr/<tuple>/sysroot/usr/bin, so even if
> the user adds host/usr/bin to its path to get access to the SDK, the
> target binaries won't be visible.
> 
> It's true however that this would enlarge the size of the SDK.

"Bad" isn't the right word; I shouldn't have used it.
It wasn't that they'd end up in the host's path that was my concern.
My thinking was that the binaries don't belong in an SDK since all we'd
need in it are header files and libraries so applications could be built
and linked with them.
I suppose it comes down to clarifying what the purpose of STAGING_DIR
actually is.

The size increase you mention would be significant.  Just looking at my
current build.  "output/staging/" is 330 MiB (of which 150 MiB is docs
and locale) and the stripped "output/target/" is 61 MiB, which could be
ten times larger with debugging symbols and it is this that would be
included in staging, right?

[Buildroot] Installation of package files in staging and target

[Yann E. MORIN]

Quotient, Thomas, All,

On Tuesday 12 July 2011 17:23:15 Quotient Remainder wrote:
> On Tue, 2011-07-12 at 13:51 +0200, Thomas Petazzoni wrote:
> > Le Tue, 12 Jul 2011 11:55:12 +0100,
> > Quotient Remainder <quotientvremainder@gmail.com> a ?crit :
> > 
> > > I'm a little behind in tracking the changes with toolchain movements
> > > from a few months back but I was of the understanding that the current
> > > install scheme still allows the staging directory to be treated as a
> > > kind of mini SDK and so if I build a system using Buildroot I can tar
> > > up the staging dir and give it to application developers as an
> > > interface. Is this still the case?  If so, wouldn't putting
> > > executable programs into the staging dir be a bad thing?
> > 
> > Why would it be bad ? The binaries for the target are not inside
> > host/usr/bin, but inside host/usr/<tuple>/sysroot/usr/bin, so even if
> > the user adds host/usr/bin to its path to get access to the SDK, the
> > target binaries won't be visible.
> > 
> > It's true however that this would enlarge the size of the SDK.
> 
> "Bad" isn't the right word; I shouldn't have used it.
> It wasn't that they'd end up in the host's path that was my concern.
> My thinking was that the binaries don't belong in an SDK since all we'd
> need in it are header files and libraries so applications could be built
> and linked with them.

I see your point. But as we create the target dir as a cleaned-up copy of
staging, then we could create an sdk dir as a cleaned-up copy of 
staging as well, except using a different way of cleaning up;

 * target/ is a carefuly-crafted copy, minus:
   includes and static libs
   doc and man and whatnot
   pkg-config stuff
   and all other stuff irrelevant to running

 * sdk/ is a carefuly-crafted copy of staging, including:
   includes, shared and static libs
   pkg-config stuff
   ?...

That way, it becomes much easier to install packages. That may even simplify
the package removal later (if we are careful at install time). And as target/
and sdk/ are created at the end, we can more easily ensure what we keep/trash
when doing the copy.

> I suppose it comes down to clarifying what the purpose of STAGING_DIR
> actually is.

Granted. Supposing that we go for a single-step install into staging, and
doing the copy to target/ and sdk/ right at the end, here how it would go.
A bit of pre-requisite are needed here; for example, difference between
--prefix and DESTDIR is expected to be known; ditto $(O):

 * packages are installed with DESTDIR=$(O)/staging
     - exception made for bootloaders, and potentially kernel, which
       install directly in $(O)/images (although I am realy not a big
       fan, but that's personal taste)

 * After all packages are installed:
   + $(O)/staging is copied to $(O)/target, and the copy is cleaned-up
   + $(O)/staging is copied to $(O)/sdk, and the copy is cleaned-up

 * $(O)/target is used as input to generate the images:
   + images are installed in $(O)/images
     - file system image(s)
     - potentially, kernel image


So, we could even have four top-level make rules:
  o images:     populate $(O)/images  (the default, as today)
  o target:     populate $(O)/target
  o sdk:        populate $(O)/sdk
  o staging:    populate $(O)/staging (aka build and install all packages)

Of course, with the required dependencies:
  images -> target -> staging
  sdk -> staging

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

On Tue, 2011-07-12 at 19:39 +0200, Yann E. MORIN wrote:
> Quotient, Thomas, All,
> On Tuesday 12 July 2011 17:23:15 Quotient Remainder wrote:
> > My thinking was that the binaries don't belong in an SDK since all we'd
> > need in it are header files and libraries so applications could be built
> > and linked with them.
> 
> I see your point. But as we create the target dir as a cleaned-up copy of
> staging, then we could create an sdk dir as a cleaned-up copy of 
> staging as well, except using a different way of cleaning up;
> 
>  * target/ is a carefuly-crafted copy, minus:
>    includes and static libs
>    doc and man and whatnot
>    pkg-config stuff
>    and all other stuff irrelevant to running
> 
>  * sdk/ is a carefuly-crafted copy of staging, including:
>    includes, shared and static libs
>    pkg-config stuff
>    ?...
> 
> That way, it becomes much easier to install packages. That may even simplify
> the package removal later (if we are careful at install time). And as target/
> and sdk/ are created at the end, we can more easily ensure what we keep/trash
> when doing the copy.

Probably not directly related but kind of...
We've been doing some stuff involving bundling up individual packages
into installable, signed chunks and to facilitate this we've added an
extra install stage to "package/Makefile.package.in".  Effectively it
adds a ".stamp-pkg-installed" target, which sets a target-specific value
for TARGET_DIR (e.g. ".../output/pkg") and calls the normal target
install macros.  This has the effect of creating an individual directory
tree of all files installed to the target.  A bit of extra processing is
needed for the likes of device drivers that modify existing files
("/lib/modules/.../modules.dep" requires running depmod on target after
installation, installing extra libraries in "/usr/local/lib" requires
ldconfig) but in general.  I know the installing bit isn't relevant for
Buildroot mainline but I'm just wondering if the principle of separating
packages' effects on the RFS is useful to anyone.  Maybe it'll trigger a
moment of inspiration with someone.

> > I suppose it comes down to clarifying what the purpose of STAGING_DIR
> > actually is.
> 
> Granted. Supposing that we go for a single-step install into staging, and
> doing the copy to target/ and sdk/ right at the end, here how it would go.
> A bit of pre-requisite are needed here; for example, difference between
> --prefix and DESTDIR is expected to be known; ditto $(O):
> 
>  * packages are installed with DESTDIR=$(O)/staging
>      - exception made for bootloaders, and potentially kernel, which
>        install directly in $(O)/images (although I am realy not a big
>        fan, but that's personal taste)
> 

Where does the skeleton fs come into play?  Is it copied to "target/"
directly or would it pass through "staging/" first?  Probably the
former, IMHO, since it's only useful on the target, unless it gets
fiddled with during the build process, in which case it should pass
through "staging/".

>  * After all packages are installed:
>    + $(O)/staging is copied to $(O)/target, and the copy is cleaned-up
>    + $(O)/staging is copied to $(O)/sdk, and the copy is cleaned-up
> 
>  * $(O)/target is used as input to generate the images:
>    + images are installed in $(O)/images
>      - file system image(s)
>      - potentially, kernel image
> 
> 
> So, we could even have four top-level make rules:
>   o images:     populate $(O)/images  (the default, as today)
>   o target:     populate $(O)/target
>   o sdk:        populate $(O)/sdk
>   o staging:    populate $(O)/staging (aka build and install all packages)
> 
> Of course, with the required dependencies:
>   images -> target -> staging
>   sdk -> staging

That sounds extremely promising, and probably not all that difficult
coding-wise!

[Buildroot] Installation of package files in staging and target

[Yann E. MORIN]

Quotient, Thomas, All,

On Tuesday 12 July 2011 21:48:38 Quotient Remainder wrote:
> On Tue, 2011-07-12 at 19:39 +0200, Yann E. MORIN wrote:
> > That way, it becomes much easier to install packages. That may even simplify
> > the package removal later (if we are careful at install time). And as target/
> > and sdk/ are created at the end, we can more easily ensure what we keep/trash
> > when doing the copy.
> 
> Probably not directly related but kind of...
> We've been doing some stuff involving bundling up individual packages
> into installable, signed chunks and to facilitate this we've added an
> extra install stage to "package/Makefile.package.in".  Effectively it
> adds a ".stamp-pkg-installed" target, which sets a target-specific value
> for TARGET_DIR (e.g. ".../output/pkg") and calls the normal target
> install macros.  This has the effect of creating an individual directory
> tree of all files installed to the target.  A bit of extra processing is
> needed for the likes of device drivers that modify existing files
> ("/lib/modules/.../modules.dep" requires running depmod on target after
> installation,

depmod can be run prior to building the file system images, once all
kernel modules have been installed. You'd just have to provide a dummy
depmod that does nothing, so that modules that insists on calling depmod
can, but you delegate the real depmod later, when it indeed makes sense
to call it.

> installing extra libraries in "/usr/local/lib" requires
> ldconfig) but in general.  I know the installing bit isn't relevant for
> Buildroot mainline

There has been some thoughts and efforts in doing something similar (IIUC).
. Maybe you could share your changes, so they get mainlined, and every one
profits from it, and you stop having to port them from version to version.
;-)

> but I'm just wondering if the principle of separating
> packages' effects on the RFS is useful to anyone.  Maybe it'll trigger a
> moment of inspiration with someone.

At least, it allows to properly find new files added by the package, files
added that already existed, and files modified (which would fail, as the
modified file would be missing).

All without needing to play tricky tricks with any unionfs mechanism.

> >  * packages are installed with DESTDIR=$(O)/staging
> >      - exception made for bootloaders, and potentially kernel, which
> >        install directly in $(O)/images (although I am realy not a big
> >        fan, but that's personal taste)
> 
> Where does the skeleton fs come into play?  Is it copied to "target/"
> directly or would it pass through "staging/" first?  Probably the
> former, IMHO, since it's only useful on the target, unless it gets
> fiddled with during the build process, in which case it should pass
> through "staging/".

I'd say the skeleton should be handled as if it were a package. staging/ is
created as a copy of the skeleton, then packages are installed in there.

This allows packages to, for example, add users to /etc/passwd, or do
whatever they have to.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

On Tue, 2011-07-12 at 22:46 +0200, Yann E. MORIN wrote:
> Quotient, Thomas, All,
> 
> On Tuesday 12 July 2011 21:48:38 Quotient Remainder wrote:
> > We've been doing some stuff involving bundling up individual packages
> > into installable, signed chunks and to facilitate this we've added an
> > extra install stage to "package/Makefile.package.in".  Effectively it
> > adds a ".stamp-pkg-installed" target, which sets a target-specific value
> > for TARGET_DIR (e.g. ".../output/pkg") and calls the normal target
> > install macros.  This has the effect of creating an individual directory
> > tree of all files installed to the target.  A bit of extra processing is
> > needed for the likes of device drivers that modify existing files
> > ("/lib/modules/.../modules.dep" requires running depmod on target after
> > installation,
> 
> depmod can be run prior to building the file system images, once all
> kernel modules have been installed. You'd just have to provide a dummy
> depmod that does nothing, so that modules that insists on calling depmod
> can, but you delegate the real depmod later, when it indeed makes sense
> to call it.

I didn't explain it very well.  These cryptographically signed chunks
are used for making run-time changes to the RFS so any host-generated
depmod or ldconfig output is not useful.  The packages are generated in
Buildroot but the default installation is only a tiny subset (just
enough to get the unit to a very early state of user-space
"self-awareness").  All extra things are in these chunks.  No sane
person would want to do this but it's a requirement for the kind of unit
we're involved with.
The reason I mentioned it was just to explain the package separation
method.

> . Maybe you could share your changes, so they get mainlined, and every one
> profits from it, and you stop having to port them from version to version.
> ;-)

As it happens I think I have some time tomorrow so will try to post a
patch to illustrate it, if there's interest.

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

On Tue, 2011-07-12 at 22:45 +0100, Quotient Remainder wrote:

> > . Maybe you could share your changes, so they get mainlined, and every one
> > profits from it, and you stop having to port them from version to version.
> > ;-)
> 
> As it happens I think I have some time tomorrow so will try to post a
> patch to illustrate it, if there's interest.

I've posted a patch for this titled: "Generate a package-specific tree
of installed components for all generic targets."

There's also a separate patch from me that has no connection with this
thread, just a suggested modification on the recent RAWNAME change.

[Buildroot] Installation of package files in staging and target

[Quotient Remainder]

On Wed, 2011-07-13 at 13:54 +0100, Quotient Remainder wrote:
> I've posted a patch for this titled: "Generate a package-specific tree
> of installed components for all generic targets."
> 
> There's also a separate patch from me that has no connection with this
> thread, just a suggested modification on the recent RAWNAME change.

These patches haven't appeared on the list, nor have any of the other
unrelated ones I've posted in the last day or two.  Is there a
pre-approval holding-pen that they get stuck in or do I need to
resubmit?  I used:
    git send-email --to='buildroot at busybox.net' *.patch

Also, Yann, I get a delivery failed on your address every time. Is gmail
blocked?  Sorry for going kind of OT but since I can't contact directly,
what choice do I have? :(

[Buildroot] Installation of package files in staging and target

[Alper YILDIRIM]

Quotient Remainder wrote:
> 
> On Tue, 2011-07-12 at 22:45 +0100, Quotient Remainder wrote:
> 
> I've posted a patch for this titled: "Generate a package-specific tree
> of installed components for all generic targets."
> 
> There's also a separate patch from me that has no connection with this
> thread, just a suggested modification on the recent RAWNAME change.
> 
> 

Hello, i am interested with your patch because we are in the stage of
adding such a feature to buildroot.
Maybe your patch can get mainlined in the future
-- 
View this message in context: http://old.nabble.com/Installation-of-package-files-in-staging-and-target-tp32041764p32069154.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.