[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Thomas Petazzoni]

Hello,

On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
wrote:

> I have seen your mail on the buildroot mailing list.
> I download your buildroot branch 'ppsh' and test it with my configurations.

First of all, thanks a lot for taking the time to test my patches and
reporting the issues. It is very, very useful feedback, so many thanks!

Unfortunately, your e-mail was too large to make it to the mailing
list, due to the huge attached file you had. Next time, it would be
better if you could upload the log file somewhere, and just give a link
in your e-mail.

See below my replies for the different issues your report.

> Test environment:
> when using your buildroot, I will execute the parallel build command 'make -j8'.
> when using mainstream buildroot, I will execute 'make -j1'.
> config?
> 1\ qemu_x86_defconfig
> 2\ my_private_configuration(it is ok in mainstream buildroot)
> 
> Q1: Compatibility Issues
> In qemu_x86_defconfig, mainstream and 'ppsh' is all ok because it is simple.
> In my private, if using 'make -j8' in the 'ppsh', there still has the first question as shows in lines 103-116 of the top-level Makefile.
> i.e.
> >>> openssl Configuring  
> checking whether we are using A... yes(A has been compiled)/no(A has not been compiled)
> 
> in mainstream buildroot, the order of package A is before openssl. So openssl is always support A.
> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the mainstream, there will cause software inconsistencies.

I don't understand this problem. Which "package A" are you referring
to ?

With the per-package SDK work, a given package will only see the
dependencies it has explicitly expressed in its .mk file. This is
entirely expected behavior.

Currently (without per-package SDK), a given package sees everything
that is in STAGING_DIR, even if it comes from packages it doesn't
depend on. Even though this means a package can pick up a dependency
that is not explicitly described in its .mk file, it is acceptable
today because the build order is guaranteed to be the same across
rebuilds of the same configuration. However, with top-level parallel
build, this build order guarantee goes away, which is the primary
motivation for moving to per-package SDK, as implemented in my patch
series.

Could you describe in more details the problem you're seeing?

I see two possibilities:

 - What is missing to build openssl is a dependency that is expressed
   in openssl.mk (i.e listed in OPENSSL_DEPENDENCIES). If that is what
   you're seeing, it is a bug in the per-package SDK, and I want to
   understand better to fix the problem.

 - What is missing to build openssl is a dependency that is *NOT*
   expressed in openssl.mk, and in this case, the fact that this
   dependency is no longer visible to the openssl configure script is
   the expected behavior.

> Q2: Space Expension
> in qemu_x86_defconfig, the size of output increased from 3.4G to 3.6G between mainstream with 'ppsh'.
> in my private, the rate of space occupancy growth is higher. it`s crazy from 3.6G to 9G .This obviously can not accept. (A large part of reason is that I use a very large toolchain-external which about 1.5G)
> Before a package only consumes one space, but now a package maybe reoccupied space below per-package.
> BEFORE(mainstream):
> 1.7G    ./build
> 115M    ./images
> 37M    ./target
> 1.8G    ./host
> 3.6G    .
> 
> NOW('ppsh')
> 1.7G    ./build
> 5.5G    ./per-package
> 115M    ./images
> 37M    ./target
> 1.8G    ./host
> 8.9G    .

Which tool (and options) have you used to calculate this size?

Indeed, we are using hardlinks, so files are not duplicated. Only files
that are overwritten get de-duplicated.

Could you investigate this a bit more, checking first if the tool you
use to calculate the size properly accounts for hard links ?

> Q3: ERROR: Could not generate syscalls.(uclibc_error.log)
> I have not solved this problem, I put the error message in the attachment

This is weird. I'll try to do a uClibc toolchain build, since I only
did a glibc toolchain build. Do you have a minimal defconfig to
reproduce this? Is it 100% reproducible?

> Q4: Ccache path needs to be fixed(ccache_error.log)
> When ccache is enabled, it will generate the 'command not found' error like below.
> make: /home/zhulianc/inprogress/orig/tpbuildroot/output/host/bin/ccache: Command not found
> this issue has been fix.

I haven't looked at ccache support to be compatible with per-package
SDK, so I'm not surprised it is broken, so it's glad to see you looked
into this.

> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index afbec44..66bfcad 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -50,6 +50,7 @@ BR_CCACHE_INITIAL_SETUP = $(call qstrip,$(BR2_CCACHE_INITIAL_SETUP))
>  ifneq ($(BR_CCACHE_INITIAL_SETUP),)
>  define HOST_CCACHE_DO_INITIAL_SETUP
>      @$(call MESSAGE,"Applying initial settings")
> +    rsync -au --link-dest=$(HOST_DIR) $(HOST_DIR) $(call qstrip,$(BR2_HOST_DIR))
>      $(CCACHE) $(BR_CCACHE_INITIAL_SETUP)
>      $(CCACHE) -s
>  endef

I don't think that's what we want. The global HOST_DIR (output/host)
should basically be empty during the entire build process. Instead,
what we want is to have the ccache host tools installed in the
per-package SDK. This means that all packages have to have a dependency
on host-ccache, which is not currently the case. So we probably instead
want something like:

ifeq ($(BR2_CCACHE),y)
$(2)_DEPENDENCIES += host-ccache
endif

in package/pkg-generic.mk. But of course, we need to exclude the
host-ccache package from this, otherwise we would have a circular
dependency. Do you want to try something like this?

Again, many thanks for this feedback. Hopefully we can solve the
various problems you have reported and make the per-package SDK
functionality suitable for merging in the future.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Arnout Vandecappelle]

On 07-11-17 18:13, Thomas Petazzoni wrote:
>> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
>> index afbec44..66bfcad 100644
>> --- a/package/ccache/ccache.mk
>> +++ b/package/ccache/ccache.mk
>> @@ -50,6 +50,7 @@ BR_CCACHE_INITIAL_SETUP = $(call qstrip,$(BR2_CCACHE_INITIAL_SETUP))
>>  ifneq ($(BR_CCACHE_INITIAL_SETUP),)
>>  define HOST_CCACHE_DO_INITIAL_SETUP
>>      @$(call MESSAGE,"Applying initial settings")
>> +    rsync -au --link-dest=$(HOST_DIR) $(HOST_DIR) $(call qstrip,$(BR2_HOST_DIR))
>>      $(CCACHE) $(BR_CCACHE_INITIAL_SETUP)
>>      $(CCACHE) -s
>>  endef
> I don't think that's what we want. The global HOST_DIR (output/host)
> should basically be empty during the entire build process. Instead,
> what we want is to have the ccache host tools installed in the
> per-package SDK. This means that all packages have to have a dependency
> on host-ccache, which is not currently the case. So we probably instead
> want something like:
> 
> ifeq ($(BR2_CCACHE),y)
> $(2)_DEPENDENCIES += host-ccache
> endif
> 
> in package/pkg-generic.mk. But of course, we need to exclude the
> host-ccache package from this, otherwise we would have a circular
> dependency. Do you want to try something like this?

 Ha, I didn't understand the original problem but now I see. The issue is that
ccache is part of DEPENDENCIES_HOST_PREREQ, and DEPENDENCIES_HOST_PREREQ is
*not* added to each package's DEPENDENCIES, instead it is added to the
core-dependencies target and each package depends on that. It's an easy way to
avoid circular dependencies, I guess.

 So I think the proper solution would be to add DEPENDENCIES_HOST_PREREQ to the
dependencies of all packages except themselves, similar to how it's done for
toolchain.

 By the way there will also be a problem when host-tar is part of
DEPENDENCIES_HOST_PREREQ: ccache needs host-tar to be able to build, but there
is no explicit dependency between them, it relies purely on left-to-right order.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Thomas Petazzoni]

Hello,

On Tue, 7 Nov 2017 18:55:20 +0100, Arnout Vandecappelle wrote:

> > in package/pkg-generic.mk. But of course, we need to exclude the
> > host-ccache package from this, otherwise we would have a circular
> > dependency. Do you want to try something like this?  
> 
>  Ha, I didn't understand the original problem but now I see. The issue is that
> ccache is part of DEPENDENCIES_HOST_PREREQ, and DEPENDENCIES_HOST_PREREQ is
> *not* added to each package's DEPENDENCIES,

Exactly.

> instead it is added to the
> core-dependencies target and each package depends on that. It's an easy way to
> avoid circular dependencies, I guess.
> 
>  So I think the proper solution would be to add DEPENDENCIES_HOST_PREREQ to the
> dependencies of all packages except themselves, similar to how it's done for
> toolchain.

Isn't this DEPENDENCIES_HOST_PREREQ thing a bit crappy in the first
place? Why isn't host-ccache simply added as a dependency of all
packages, except itself?

Another example is that today, host-lzip is always built at the
beginning of build, even if you're not building any package that needs
the lzip "decompressor".

So perhaps this calls for a little bit of rework in this area, no?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Arnout Vandecappelle]

On 07-11-17 20:55, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 7 Nov 2017 18:55:20 +0100, Arnout Vandecappelle wrote:
> 
>>> in package/pkg-generic.mk. But of course, we need to exclude the
>>> host-ccache package from this, otherwise we would have a circular
>>> dependency. Do you want to try something like this?  
>>
>>  Ha, I didn't understand the original problem but now I see. The issue is that
>> ccache is part of DEPENDENCIES_HOST_PREREQ, and DEPENDENCIES_HOST_PREREQ is
>> *not* added to each package's DEPENDENCIES,
> 
> Exactly.
> 
>> instead it is added to the
>> core-dependencies target and each package depends on that. It's an easy way to
>> avoid circular dependencies, I guess.
>>
>>  So I think the proper solution would be to add DEPENDENCIES_HOST_PREREQ to the
>> dependencies of all packages except themselves, similar to how it's done for
>> toolchain.
> 
> Isn't this DEPENDENCIES_HOST_PREREQ thing a bit crappy in the first
> place?

 Nobody claimed it wasn't :-) It's a relic from 2007...

> Why isn't host-ccache simply added as a dependency of all
> packages, except itself?
> 
> Another example is that today, host-lzip is always built at the
> beginning of build, even if you're not building any package that needs
> the lzip "decompressor".
> 
> So perhaps this calls for a little bit of rework in this area, no?

 Yep. For cmake it is done the right way. The same could be done for the other
dependencies. And it may be useful to extend the mechanism to other system
packages. E.g. m4 is not really sensitive to version changes.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Thomas Petazzoni]

Hello,

On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
wrote:

> Q3: ERROR: Could not generate syscalls.(uclibc_error.log)
> I have not solved this problem, I put the error message in the attachment

So I built a basic uClibc system, toolchain included with the following
defconfig:

BR2_arm=y
BR2_cortex_a8=y

And it built just fine.

Are you able to reproduce this problem? Does it happen every time, or
just once in a while?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Yann E. MORIN]

Hello Chu, Zhuliang; All,

On 2017-11-07 11:48 +0000, Chu, Zhuliang (NSB - CN/Shanghai) spake thusly:
> I have seen your mail on the buildroot mailing list.
> I download your buildroot branch 'ppsh' and test it with my configurations.
> After my test, I will report about 4 problem under below.

Thanks for the feedback on this early series, very valuable indeed! :-)

> Test environment:
> when using your buildroot, I will execute the parallel build command 'make -j8'.
> when using mainstream buildroot, I will execute 'make -j1'.
> config?
> 1\ qemu_x86_defconfig
> 2\ my_private_configuration(it is ok in mainstream buildroot)
> 
> Q1: Compatibility Issues
> In qemu_x86_defconfig, mainstream and 'ppsh' is all ok because it is simple.
> In my private, if using 'make -j8' in the 'ppsh', there still has the first question as shows in lines 103-116 of the top-level
> Makefile.
> i.e.
> >>> openssl Configuring
> checking whether we are using A... yes(A has been compiled)/no(A has not been compiled)

This does not look like this is opensl configure messages...

Here, openssl (really, libopenssl nowadays) configure step says things
like:

    ---8<---
    Configuring for linux-armv4
        no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128 (skip dir)
        no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
        no-jpake        [experimental] OPENSSL_NO_JPAKE (skip dir)
        no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
        no-libunbound   [experimental] OPENSSL_NO_LIBUNBOUND (skip dir)
        no-md2          [default]  OPENSSL_NO_MD2 (skip dir)
        no-rc5          [option]   OPENSSL_NO_RC5 (skip dir)
        no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
        no-sctp         [default]  OPENSSL_NO_SCTP (skip dir)
        no-ssl-trace    [default]  OPENSSL_NO_SSL_TRACE (skip dir)
        no-ssl2         [default]  OPENSSL_NO_SSL2 (skip dir)
        no-store        [experimental] OPENSSL_NO_STORE (skip dir)
        no-unit-test    [default]  OPENSSL_NO_UNIT_TEST (skip dir)
        no-weak-ssl-ciphers [default]  OPENSSL_NO_WEAK_SSL_CIPHERS (skip dir)
    IsMK1MF=0
    CC            =/home/ymorin/dev/buildroot/O/host/bin/arm-linux-gnueabihf-gcc
    CFLAG         =-fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
    EX_LIBS       =-ldl
    CPUID_OBJ     =armcap.o armv4cpuid.o
    BN_ASM        =bn_asm.o armv4-mont.o armv4-gf2m.o
    EC_ASM        =
    DES_ENC       =des_enc.o fcrypt_b.o
    AES_ENC       =aes_cbc.o aes-armv4.o bsaes-armv7.o aesv8-armx.o
    BF_ENC        =bf_enc.o
    CAST_ENC      =c_enc.o
    RC4_ENC       =rc4_enc.o rc4_skey.o
    RC5_ENC       =rc5_enc.o
    MD5_OBJ_ASM   =
    SHA1_OBJ_ASM  =sha1-armv4-large.o sha256-armv4.o sha512-armv4.o
    RMD160_OBJ_ASM=
    CMLL_ENC      =camellia.o cmll_misc.o cmll_cbc.o
    MODES_OBJ     =ghash-armv4.o ghashv8-armx.o
    ENGINES_OBJ   =
    PROCESSOR     =
    RANLIB        =/home/ymorin/dev/buildroot/O/host/bin/arm-linux-gnueabihf-ranlib
    ARFLAGS       =
    PERL          =/usr/bin/perl
    THIRTY_TWO_BIT mode
    DES_UNROLL used
    DES_INT used
    BN_LLONG mode
    RC4 uses uchar
    RC4_CHUNK is unsigned long
    BF_PTR used
    created directory `include/openssl'
    e_os2.h => include/openssl/e_os2.h
    making links in crypto...
    crypto.h => ../include/openssl/crypto.h
    [...]
    ---8<---

So, I wonder what openssl you are using... :-/

> in mainstream buildroot, the order of package A is before openssl. So openssl is always support A.
> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the mainstream, there will cause software
> inconsistencies.

This behavioural change is expected, indeed. Top-level parallel build
can no longer guarantee the build order, by the mere fact of being
parallel. Whatever currently depends on the build order will indeed be
broken once top-level parralel build is merged.

The solution is to correctly express dependencies in the .mk files.

Now, if your 'A' package is not something that is known to buildroot,
but changes the way a package known to buildroot behaves, there is not
much we in Buildroot can do, unless you submit the 'A' package for
inclusion in Buildroot...

> Q2: Space Expension
> in qemu_x86_defconfig, the size of output increased from 3.4G to 3.6G between mainstream with 'ppsh'.
> in my private, the rate of space occupancy growth is higher. it`s crazy from 3.6G to 9G .This obviously can not accept. (A large
> part of reason is that I use a very large toolchain-external which about 1.5G)
> Before a package only consumes one space, but now a package maybe reoccupied space below per-package.
> BEFORE(mainstream):
> 1.7G    ./build
> 115M    ./images
> 37M    ./target
> 1.8G    ./host
> 3.6G    .
> 
> NOW('ppsh')
> 1.7G    ./build
> 5.5G    ./per-package
> 115M    ./images
> 37M    ./target
> 1.8G    ./host
> 8.9G    .

As others already stated, in may depend on the tools you use to report
the disk usage, but can also depend on the type of filesystem.

What filesystem are you using: ext3/4, btrfs, xfs, other?

What command did you run to come up with those numbers?

> Q3: ERROR: Could not generate syscalls.(uclibc_error.log)
> I have not solved this problem, I put the error message in the attachment
> 
> Q4: Ccache path needs to be fixed(ccache_error.log)
> When ccache is enabled, it will generate the 'command not found' error like below.
> make: /home/zhulianc/inprogress/orig/tpbuildroot/output/host/bin/ccache: Command not found
> this issue has been fix.
> 
> patch(0001-ccache-fix-ccache-HOST_DIR.patch):
> -----------------------------------------------------------------------------------
> From 2e60c344a47fe64d92b3d29af8143d73802e71c4 Mon Sep 17 00:00:00 2001
> From: Zhuliang Chu <zhuliang.chu@nokia-sbell.com>
> Date: Tue, 7 Nov 2017 19:23:46 +0800
> Subject: [PATCH 1/1] ccache: fix ccache HOST_DIR
> 
> The ccache binary is located in the per-package directory when ccache is installed.
> But it is set at '$(HOST_DIR)/usr/bin/ccache' in top-level Makefile as below:
> CCACHE := $(HOST_DIR)/usr/bin/ccache

I wonder on what version of Buildroot you applied the patches: we no
longer have the usr directory in $(HOST_DIR) now.

Can you try to reproduce your issues using the current master branch of
Buildroot, please?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Thomas De Schampheleire]

2017-11-08 18:29 GMT+01:00 Yann E. MORIN <yann.morin.1998@free.fr>:
> Hello Chu, Zhuliang; All,
>
> On 2017-11-07 11:48 +0000, Chu, Zhuliang (NSB - CN/Shanghai) spake thusly:
>> I have seen your mail on the buildroot mailing list.
>> I download your buildroot branch 'ppsh' and test it with my configurations.
>> After my test, I will report about 4 problem under below.
>
> Thanks for the feedback on this early series, very valuable indeed! :-)
>
>> Test environment:
>> when using your buildroot, I will execute the parallel build command 'make -j8'.
>> when using mainstream buildroot, I will execute 'make -j1'.
>> config?
>> 1\ qemu_x86_defconfig
>> 2\ my_private_configuration(it is ok in mainstream buildroot)
>>
>> Q1: Compatibility Issues
>> In qemu_x86_defconfig, mainstream and 'ppsh' is all ok because it is simple.
>> In my private, if using 'make -j8' in the 'ppsh', there still has the first question as shows in lines 103-116 of the top-level
>> Makefile.
>> i.e.
>> >>> openssl Configuring
>> checking whether we are using A... yes(A has been compiled)/no(A has not been compiled)
>
> This does not look like this is opensl configure messages...
>
> Here, openssl (really, libopenssl nowadays) configure step says things
> like:
>
>     ---8<---
>     Configuring for linux-armv4
>         no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128 (skip dir)
>         no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
>         no-jpake        [experimental] OPENSSL_NO_JPAKE (skip dir)
>         no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
>         no-libunbound   [experimental] OPENSSL_NO_LIBUNBOUND (skip dir)
>         no-md2          [default]  OPENSSL_NO_MD2 (skip dir)
>         no-rc5          [option]   OPENSSL_NO_RC5 (skip dir)
>         no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
>         no-sctp         [default]  OPENSSL_NO_SCTP (skip dir)
>         no-ssl-trace    [default]  OPENSSL_NO_SSL_TRACE (skip dir)
>         no-ssl2         [default]  OPENSSL_NO_SSL2 (skip dir)
>         no-store        [experimental] OPENSSL_NO_STORE (skip dir)
>         no-unit-test    [default]  OPENSSL_NO_UNIT_TEST (skip dir)
>         no-weak-ssl-ciphers [default]  OPENSSL_NO_WEAK_SSL_CIPHERS (skip dir)
>     IsMK1MF=0
>     CC            =/home/ymorin/dev/buildroot/O/host/bin/arm-linux-gnueabihf-gcc
>     CFLAG         =-fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
>     EX_LIBS       =-ldl
>     CPUID_OBJ     =armcap.o armv4cpuid.o
>     BN_ASM        =bn_asm.o armv4-mont.o armv4-gf2m.o
>     EC_ASM        =
>     DES_ENC       =des_enc.o fcrypt_b.o
>     AES_ENC       =aes_cbc.o aes-armv4.o bsaes-armv7.o aesv8-armx.o
>     BF_ENC        =bf_enc.o
>     CAST_ENC      =c_enc.o
>     RC4_ENC       =rc4_enc.o rc4_skey.o
>     RC5_ENC       =rc5_enc.o
>     MD5_OBJ_ASM   =
>     SHA1_OBJ_ASM  =sha1-armv4-large.o sha256-armv4.o sha512-armv4.o
>     RMD160_OBJ_ASM=
>     CMLL_ENC      =camellia.o cmll_misc.o cmll_cbc.o
>     MODES_OBJ     =ghash-armv4.o ghashv8-armx.o
>     ENGINES_OBJ   =
>     PROCESSOR     =
>     RANLIB        =/home/ymorin/dev/buildroot/O/host/bin/arm-linux-gnueabihf-ranlib
>     ARFLAGS       =
>     PERL          =/usr/bin/perl
>     THIRTY_TWO_BIT mode
>     DES_UNROLL used
>     DES_INT used
>     BN_LLONG mode
>     RC4 uses uchar
>     RC4_CHUNK is unsigned long
>     BF_PTR used
>     created directory `include/openssl'
>     e_os2.h => include/openssl/e_os2.h
>     making links in crypto...
>     crypto.h => ../include/openssl/crypto.h
>     [...]
>     ---8<---
>
> So, I wonder what openssl you are using... :-/
>
>> in mainstream buildroot, the order of package A is before openssl. So openssl is always support A.
>> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the mainstream, there will cause software
>> inconsistencies.
>
> This behavioural change is expected, indeed. Top-level parallel build
> can no longer guarantee the build order, by the mere fact of being
> parallel. Whatever currently depends on the build order will indeed be
> broken once top-level parralel build is merged.
>
> The solution is to correctly express dependencies in the .mk files.
>
> Now, if your 'A' package is not something that is known to buildroot,
> but changes the way a package known to buildroot behaves, there is not
> much we in Buildroot can do, unless you submit the 'A' package for
> inclusion in Buildroot...
>
>> Q2: Space Expension
>> in qemu_x86_defconfig, the size of output increased from 3.4G to 3.6G between mainstream with 'ppsh'.
>> in my private, the rate of space occupancy growth is higher. it`s crazy from 3.6G to 9G .This obviously can not accept. (A large
>> part of reason is that I use a very large toolchain-external which about 1.5G)
>> Before a package only consumes one space, but now a package maybe reoccupied space below per-package.
>> BEFORE(mainstream):
>> 1.7G    ./build
>> 115M    ./images
>> 37M    ./target
>> 1.8G    ./host
>> 3.6G    .
>>
>> NOW('ppsh')
>> 1.7G    ./build
>> 5.5G    ./per-package
>> 115M    ./images
>> 37M    ./target
>> 1.8G    ./host
>> 8.9G    .
>
> As others already stated, in may depend on the tools you use to report
> the disk usage, but can also depend on the type of filesystem.
>
> What filesystem are you using: ext3/4, btrfs, xfs, other?
>
> What command did you run to come up with those numbers?
>
>> Q3: ERROR: Could not generate syscalls.(uclibc_error.log)
>> I have not solved this problem, I put the error message in the attachment
>>
>> Q4: Ccache path needs to be fixed(ccache_error.log)
>> When ccache is enabled, it will generate the 'command not found' error like below.
>> make: /home/zhulianc/inprogress/orig/tpbuildroot/output/host/bin/ccache: Command not found
>> this issue has been fix.
>>
>> patch(0001-ccache-fix-ccache-HOST_DIR.patch):
>> -----------------------------------------------------------------------------------
>> From 2e60c344a47fe64d92b3d29af8143d73802e71c4 Mon Sep 17 00:00:00 2001
>> From: Zhuliang Chu <zhuliang.chu@nokia-sbell.com>
>> Date: Tue, 7 Nov 2017 19:23:46 +0800
>> Subject: [PATCH 1/1] ccache: fix ccache HOST_DIR
>>
>> The ccache binary is located in the per-package directory when ccache is installed.
>> But it is set at '$(HOST_DIR)/usr/bin/ccache' in top-level Makefile as below:
>> CCACHE := $(HOST_DIR)/usr/bin/ccache
>
> I wonder on what version of Buildroot you applied the patches: we no
> longer have the usr directory in $(HOST_DIR) now.

Trying to bridge the timezone gap here: the version we are currently
using is 2017.02.5, so normally Chu Zhuliang worked on top of that.

Best regards,
Thomas

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Chu, Zhuliang]

Hello Thomas Petazzoni, All,

>Hello,
>
>On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
>wrote:
>
>> I have seen your mail on the buildroot mailing list.
>> I download your buildroot branch 'ppsh' and test it with my configurations.
>
>First of all, thanks a lot for taking the time to test my patches and reporting the issues. It is very, very useful feedback, so many thanks!
>
>Unfortunately, your e-mail was too large to make it to the mailing list, due to the huge attached file you had. Next time, it would be better if you could upload the log file somewhere, and just give a link in your e-mail.
>
>See below my replies for the different issues your report.

I am still researching your patch series and testing it in my free time. So my reply may be a little delay.
Today I will report 2 problems to you from my testing.


I very much appreciate your efforts on the `per-package`. I think it will become a good solution to solve the problem of parallel compilation buildroot. Thanks Thomas.

About previous Q1: Compatibility Issues
>> 
>> in mainstream buildroot, the order of package A is before openssl. So openssl is always support A.
>> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the mainstream, there will cause software inconsistencies.
>
>I don't understand this problem. Which "package A" are you referring to ?
> Could you describe in more details the problem you're seeing?

Sorry my introduction is too simple to cause ambiguity in my first email.
This issue is only my worries but don`t be verified in my test.

My worries:
When I am a customer, I always use current buildroot. 
In current buildroot, the build order guarantee is very import and the order is determined after many times of testing by customer.
Example package 'A', it is not explicitly expressed in openssl`s .mk file. But in current build order, 'A' is before openssl. So in the configuring step of openssl,the openssl will check that 'A' has been installed. So I am sure that the final result is the openssl support 'A'.
If per-package gets to the point where it can be merged and it will be pushed into buildroot mainstream. In customers, they will find the buildroot can be parallel compilation now. They will be very happy to use this feature. So they will use `make -j8` instead of 'make -j1'. If they don`t explicitly describe the dependency 'A' in openssl`s .mk file. And now the openssl don`t support 'A'. But this is not expected behavior for customer.
Yes, if the customer know the situation 'A', it is simple. He will explicitly express in openssl .mk file. But if he don`t know, what would he do?
We know that there are a lot of unconspicuous dependencies in current buildroot and this is also why buildroot can't be parallel compilation now.

So the intention I put forward the question is that promoting the community to provide a method to help the current buildroot smooth transition to the per-package when the per-package is pushed into the buildroot mainstream.
So I said this will cause software inconsistencies in future in previous email.

About previous Q2: Space Expension
>Which tool (and options) have you used to calculate this size?
>
Only use linux simple command:  du --max-depth=1 -h

>Indeed, we are using hardlinks, so files are not duplicated. Only files that are overwritten get de-duplicated.
> Could you investigate this a bit more, checking first if the tool you use to calculate the size properly accounts for hard links ?
Sorry Thomas, this problem is likely to be one of my mistake rather than 'per-package'.
This is also associated with the problem which I will report. I will explain there.

About previous Q3/Q4, I will reply in another email.

Now I report 2 problems from my testing or consider.

Q1: rsync conflicts
I want to generate this conflicts in testing but this is difficult. So this is also only my worries.
At the same time in parallel compilation, different threads want to modify the same file of a package in different per-package/package directory and modify the content isn`t the same.
This will generate conflicts. How to solve this problem?
I.E. thread t1 modify file A in per-package/package/A, t1 want to modify A content from 'TEXT' to 'EXIT'. At the same time, thread t2 also modify A content from 'TEXT' to 'TXXT'. And I think rsync has enough power to handle such parallel changes in this situation. but after the successful modify there is still problem. for example, rsync make the content as 'TXXT' and this result is suitable thread t2. But the thread t1 wants to 'EXIT'. 
I know this probability is minimal. But what should we do if such a conflict arises?

Q2: toolchain HOST_DIR
I use  external custom toolchain. 'BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y'
line 66 of pkg-toolchain-external.mk66 in toolchain/toolchain-external/ is 'TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR = $(HOST_DIR)/opt/ext-toolchain '.
this $(HOST_DIR) is also 'output/host/' rather than 'output/per-package/....'. 


Chu,Zhuliang
Best regards,

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Chu, Zhuliang]

Hello Thomas,

I add some ccnfigurationsthis to enable ccache in qemu_x86_defconfig.
BR2_CCACHE_DIR="$(shell readlink -m $(TOPDIR)/../ccache-buildroot)"
BR2_CCACHE_INITIAL_SETUP="--max-size=10G"
BR2_CCACHE_USE_BASEDIR=y

first this will report 'command not found' error as Q4.
I use my Q4 patch to continue compilation and then this error will appeared .

________________________________________
From: Thomas Petazzoni [thomas.petazzoni at free-electrons.com]
Sent: Thursday, November 09, 2017 0:55
To: Chu, Zhuliang (NSB - CN/Shanghai)
Cc: buildroot at uclibc.org; yann.morin.1998 at free.fr; arnout at mind.be; peter at korsgaard.com; patrickdepinguin+buildroot at gmail.com
Subject: Re: [Buildroot] [RFCv1 0/4] Per-package SDK and target directories

Hello,

On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
wrote:

> Q3: ERROR: Could not generate syscalls.(uclibc_error.log)
> I have not solved this problem, I put the error message in the attachment

So I built a basic uClibc system, toolchain included with the following
defconfig:

BR2_arm=y
BR2_cortex_a8=y

And it built just fine.

Are you able to reproduce this problem? Does it happen every time, or
just once in a while?

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Chu, Zhuliang]

Hello,

>By the way there will also be a problem when host-tar is part of
>DEPENDENCIES_HOST_PREREQ: ccache needs host-tar to be able to build, but there
>is no explicit dependency between them, it relies purely on left-to-right order.

the host-ccache mainly because its directory is defined in the top-level Makefile as below.
 453 ifeq ($(BR2_CCACHE),y)
 454 CCACHE := $(HOST_DIR)/bin/ccache
 455 BR_CACHE_DIR ?= $(call qstrip,$(BR2_CCACHE_DIR))
 456 export BR_CACHE_DIR
 457 HOSTCC := $(CCACHE) $(HOSTCC)
 458 HOSTCXX := $(CCACHE) $(HOSTCXX)
 459 else
 460 export BR_NO_CCACHE
 461 endif

In the top Makefile, $(CCACHE) is defined as 'output/host/bin/ccache'. So during the compilation process, the $(CCACHE) will do not be found. 
In fact, its real location is in `output/per-package/host-ccache/host/bin/ccache`.
The path to host-tar is not defined in top Makefile, So host-tar did not report the same error.

Chu,Zhuliang
Best Regards?

________________________________________
From: Arnout Vandecappelle [arnout at mind.be]
Sent: Wednesday, November 08, 2017 1:55
To: Thomas Petazzoni; Chu, Zhuliang (NSB - CN/Shanghai)
Cc: buildroot at uclibc.org; yann.morin.1998 at free.fr; peter at korsgaard.com; patrickdepinguin+buildroot at gmail.com
Subject: Re: [Buildroot] [RFCv1 0/4] Per-package SDK and target directories

On 07-11-17 18:13, Thomas Petazzoni wrote:
>> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
>> index afbec44..66bfcad 100644
>> --- a/package/ccache/ccache.mk
>> +++ b/package/ccache/ccache.mk
>> @@ -50,6 +50,7 @@ BR_CCACHE_INITIAL_SETUP = $(call qstrip,$(BR2_CCACHE_INITIAL_SETUP))
>>  ifneq ($(BR_CCACHE_INITIAL_SETUP),)
>>  define HOST_CCACHE_DO_INITIAL_SETUP
>>      @$(call MESSAGE,"Applying initial settings")
>> +    rsync -au --link-dest=$(HOST_DIR) $(HOST_DIR) $(call qstrip,$(BR2_HOST_DIR))
>>      $(CCACHE) $(BR_CCACHE_INITIAL_SETUP)
>>      $(CCACHE) -s
>>  endef
> I don't think that's what we want. The global HOST_DIR (output/host)
> should basically be empty during the entire build process. Instead,
> what we want is to have the ccache host tools installed in the
> per-package SDK. This means that all packages have to have a dependency
> on host-ccache, which is not currently the case. So we probably instead
> want something like:
>
> ifeq ($(BR2_CCACHE),y)
> $(2)_DEPENDENCIES += host-ccache
> endif
>
> in package/pkg-generic.mk. But of course, we need to exclude the
> host-ccache package from this, otherwise we would have a circular
> dependency. Do you want to try something like this?

 Ha, I didn't understand the original problem but now I see. The issue is that
ccache is part of DEPENDENCIES_HOST_PREREQ, and DEPENDENCIES_HOST_PREREQ is
*not* added to each package's DEPENDENCIES, instead it is added to the
core-dependencies target and each package depends on that. It's an easy way to
avoid circular dependencies, I guess.

 So I think the proper solution would be to add DEPENDENCIES_HOST_PREREQ to the
dependencies of all packages except themselves, similar to how it's done for
toolchain.

 By the way there will also be a problem when host-tar is part of
DEPENDENCIES_HOST_PREREQ: ccache needs host-tar to be able to build, but there
is no explicit dependency between them, it relies purely on left-to-right order.

 Regards,
 Arnout

--
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Chu, Zhuliang]

Hello Yann E,

>
>So, I wonder what openssl you are using... :-/
I am sorry to cause your confusion. This is an imaginary example I give.
I've already explained this issue in the email that responded to Thomas Petazzoni.

>> Q2: Space Expension
>> in qemu_x86_defconfig, the size of output increased from 3.4G to 3.6G between mainstream with 'ppsh'.
>> in my private, the rate of space occupancy growth is higher. it`s crazy from 3.6G to 9G .This obviously can not accept. (A large
>> part of reason is that I use a very large toolchain-external which about 1.5G)
>> Before a package only consumes one space, but now a package maybe reoccupied space below per-package.
>
>As others already stated, in may depend on the tools you use to report
>the disk usage, but can also depend on the type of filesystem.
>
>What filesystem are you using: ext3/4, btrfs, xfs, other?
>
>What command did you run to come up with those numbers?

my filesystem is xfs of centos7.
Sorry.This problem is likely to be one of my mistake rather than 'per-package'.
That day I encountered a compilation error 'toolchain HOST_DIR' which had been reported to Thomas Petazzoni.
The details:
when run the command 'output/per-package/toolchain/host//usr/bin/mips64-octeon-linux-gnu-gcc  --version'.
there will raise a error 'output/per-package/toolchain-external-custom/host/opt/ext-toolchain/bin/mips64-octeon-linux-gnu-gcc: No such file or directory'
When I encountered this error. I made a patch to solve it.
But after I saw thomas reply, I carefully checked my patch and I found out that my patch duplicates the toolchain-tar in the per-package/ package which dependent toolchain.

I am currently testing per-package. I have not finished all the compilation.
I have encountered many problems, of which some are my own environment issues and others are per-package issues.
I am trying to solve these problems to promote its progress.
Hopefully per-package SDK can be pushed into the mainstream as soon as possible.

Chu,Zhuliang
Best Regards,

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Arnout Vandecappelle]

On 09-11-17 06:02, Chu, Zhuliang (NSB - CN/Shanghai) wrote:
> Hello,
> 
>> By the way there will also be a problem when host-tar is part of
>> DEPENDENCIES_HOST_PREREQ: ccache needs host-tar to be able to build, but there
>> is no explicit dependency between them, it relies purely on left-to-right order.
> the host-ccache mainly because its directory is defined in the top-level Makefile as below.

 Good catch!

>  453 ifeq ($(BR2_CCACHE),y)
>  454 CCACHE := $(HOST_DIR)/bin/ccache
>  455 BR_CACHE_DIR ?= $(call qstrip,$(BR2_CCACHE_DIR))
>  456 export BR_CACHE_DIR
>  457 HOSTCC := $(CCACHE) $(HOSTCC)
>  458 HOSTCXX := $(CCACHE) $(HOSTCXX)

 The problem is that these things are assigned with := instead of = so
$(HOST_DIR) will not get expanded to per-package like it should. Putting = here
will lead to a self-referencing recursive variable. The obvious thing would be
to change it into
HOSTCC_CCACHE = $(CCACHE) $(HOSTCC)
and use HOSTCC_CCACHE everywhere. Unfortunately, that is *very* risky because we
export HOSTCC, so we have no full control over all its uses. (Admittedly, the
risk is limited to not using ccache in some cases, which is not that bad.)

 Perhaps the following can work:

HOSTCC_ORIG := $(HOSTCC)
HOSTCC = $(CCACHE) $(HOSTCC_ORIG)

(and we can probably reuse HOSTCC_NOCCACHE instead of HOSTCC_ORIG). This does
seem to work from a very quick check.

>  459 else
>  460 export BR_NO_CCACHE
>  461 endif
> 
> In the top Makefile, $(CCACHE) is defined as 'output/host/bin/ccache'. So during the compilation process, the $(CCACHE) will do not be found. 

 For CCACHE itself there is no real problem, we can use = instead of := there,
it does not reference itself.

 Regards,
 Arnout

> In fact, its real location is in `output/per-package/host-ccache/host/bin/ccache`.
> The path to host-tar is not defined in top Makefile, So host-tar did not report the same error.

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Arnout Vandecappelle]

On 09-11-17 05:32, Chu, Zhuliang (NSB - CN/Shanghai) wrote:
> Hello Thomas Petazzoni, All,
> 
>> Hello,
>>
>> On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
>> wrote:
[snip]
>>> in mainstream buildroot, the order of package A is before openssl. So openssl is always support A.
>>> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the mainstream, there will cause software inconsistencies.
>>
>> I don't understand this problem. Which "package A" are you referring to ?
>> Could you describe in more details the problem you're seeing?
> 
> Sorry my introduction is too simple to cause ambiguity in my first email.
> This issue is only my worries but don`t be verified in my test.
> 
> My worries:
> When I am a customer, I always use current buildroot. 
> In current buildroot, the build order guarantee is very import and the order is determined after many times of testing by customer.

 No, we don't guarantee the build order, it just happens to work that way.

> Example package 'A', it is not explicitly expressed in openssl`s .mk file. But in current build order, 'A' is before openssl. So in the configuring step of openssl,the openssl will check that 'A' has been installed. So I am sure that the final result is the openssl support 'A'.
> If per-package gets to the point where it can be merged and it will be pushed into buildroot mainstream. In customers, they will find the buildroot can be parallel compilation now. They will be very happy to use this feature. So they will use `make -j8` instead of 'make -j1'. If they don`t explicitly describe the dependency 'A' in openssl`s .mk file. And now the openssl don`t support 'A'. But this is not expected behavior for customer.

 If a dependency is not tracked in the .mk file, you can't count on it. Now
already it is possible that the order changes due to other dependencies (e.g.
package C gets an optional dependency on openssl in the new Buildroot version,
and package C comes alphabetically before package A, then the new Buildroot
version will build openssl before package A instead of after.

[snip]
> Q1: rsync conflicts
> I want to generate this conflicts in testing but this is difficult. So this is also only my worries.
> At the same time in parallel compilation, different threads want to modify the same file of a package in different per-package/package directory and modify the content isn`t the same.
> This will generate conflicts. How to solve this problem?
> I.E. thread t1 modify file A in per-package/package/A, t1 want to modify A content from 'TEXT' to 'EXIT'. At the same time, thread t2 also modify A content from 'TEXT' to 'TXXT'.

 Yes, this is a problem and potential showstopper. We discussed it at length at
the last BR developer meeting [1]. The conclusion is that we simply cannot
support packages that edit existing files in top-level parallel build context -
they would have to encode their dependencies explicitly to force a specific
build order. To detect how bad the problem is for in-tree packages, Yann posted
a series [2] that allows to detect it. The idea is that we will turn that
detection into an error when PPS is merged.


> And I think rsync has enough power to handle such parallel changes in this situation. but after the successful modify there is still problem. for example, rsync make the content as 'TXXT' and this result is suitable thread t2. But the thread t1 wants to 'EXIT'. 

 Actually, I checked, and rsync has no protection at all against parallel
execution. E.g. it will first test if a directory exists already and then create
it; if another rsync is at the same time creating that directory, one of the
directory creations will fail. So all the rsync's into the same (per-package)
directory have to be serialised.

> I know this probability is minimal. But what should we do if such a conflict arises?

 As I said: error out.


> Q2: toolchain HOST_DIR
> I use  external custom toolchain. 'BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y'
> line 66 of pkg-toolchain-external.mk66 in toolchain/toolchain-external/ is 'TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR = $(HOST_DIR)/opt/ext-toolchain '.
> this $(HOST_DIR) is also 'output/host/' rather than 'output/per-package/....'. 

 Do you have an actual example of that? The toolchain is just a package, so in
all places where this variable is used, HOST_DIR should expand into
output/per-package/.... The except is the definition of TOOLCHAIN_EXTERNAL_BIN,
but that was fixed in patch 3/4.


 So, for your further testing, can you try without CCACHE and without patches
that attempt to fix it? Thanks to your testing we found out that ccache is
problematic, and it will take a bit of time to fix that. As you have noticed,
fixing it correctly is not trivial :-)


 Regards,
 Arnout


[1] https://elinux.org/Buildroot:DeveloperDaysELCE2017#Top-level_parallel_build
[2] http://patchwork.ozlabs.org/project/buildroot/list/?series=10725

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Chu, Zhuliang]

Hello Arnout, All,

I think it is a good news that I have used per-package to fully compile the real configuration of my company, and after a simple test, the image generated by per-package works well on the real board.

> 
>  Actually, I checked, and rsync has no protection at all against parallel execution.
> E.g. it will first test if a directory exists already and then create it; if another rsync is
> at the same time creating that directory, one of the directory creations will fail. So
> all the rsync's into the same (per-package) directory have to be serialised.
> 
> > I know this probability is minimal. But what should we do if such a conflict arises?
> 
>  As I said: error out.

Ok,I see. You mean that if there is a conflict directly throw an error and then end the compilation.

> > Q2: toolchain HOST_DIR
> > I use  external custom toolchain. 'BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y'
> > line 66 of pkg-toolchain-external.mk66 in toolchain/toolchain-external/ 

>  Do you have an actual example of that? The toolchain is just a package, so in all
> places where this variable is used, HOST_DIR should expand into
> output/per-package/.... The except is the definition of TOOLCHAIN_EXTERNAL_BIN,
> but that was fixed in patch 3/4.
> 
> 
>  So, for your further testing, can you try without CCACHE and without patches that
> attempt to fix it? Thanks to your testing we found out that ccache is problematic,
> and it will take a bit of time to fix that. As you have noticed, fixing it correctly is not
> trivial :-)

Yes. I obey your advice. At compile time, I didn`t enable the CCACHE, and then this error disappeared.
But the toolchain does have another HOST_DIR error in toolchain/toolchain/toolchain.mk of the line 42 like below.
42 toolchain: $(HOST_DIR)/share/buildroot/toolchainfile.cmake
43 toolchain: $(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake
The host here still shows the output/host instead of per-package/xx/host.
I think the definition of '$(eval $(virtual-package))' in toolchain.mk will not affect the $(HOST_DIR).
Because in Top Makefile toolchain/*/*.mk was included before package/*/*.mk like below
499 include toolchain/*.mk
500 include toolchain/*/*.mk   ...
509 include $(sort $(wildcard package/*/*.mk))

I have solved the problems I encountered in my test. Among them, except for the previous HOST_CCACHE and the toolchain $(HOST_DIR) in this mail, and one more I am confirming. I think a total of 3 issues related to per-package.

I am still comparing the performance of the current buildroot and per-package and finishing these patches, and when I finish that, I will post a test report to the community.

> -----Original Message-----
> From: Arnout Vandecappelle [mailto:arnout at mind.be]
> Sent: 2017?11?11? 18:37
> To: Chu, Zhuliang (NSB - CN/Shanghai) <zhuliang.chu@nokia-sbell.com>; Thomas
> Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: buildroot at uclibc.org; yann.morin.1998 at free.fr; peter at korsgaard.com;
> patrickdepinguin+buildroot at gmail.com
> Subject: Re: [Buildroot] [RFCv1 0/4] Per-package SDK and target directories
> 
> 
> 
> On 09-11-17 05:32, Chu, Zhuliang (NSB - CN/Shanghai) wrote:
> > Hello Thomas Petazzoni, All,
> >
> >> Hello,
> >>
> >> On Tue, 7 Nov 2017 11:48:17 +0000, Chu, Zhuliang (NSB - CN/Shanghai)
> >> wrote:
> [snip]
> >>> in mainstream buildroot, the order of package A is before openssl. So openssl is
> always support A.
> >>> but in 'ppsh', openssl is always not support A. if once 'ppsh' is merged into the
> mainstream, there will cause software inconsistencies.
> >>
> >> I don't understand this problem. Which "package A" are you referring to ?
> >> Could you describe in more details the problem you're seeing?
> >
> > Sorry my introduction is too simple to cause ambiguity in my first email.
> > This issue is only my worries but don`t be verified in my test.
> >
> > My worries:
> > When I am a customer, I always use current buildroot.
> > In current buildroot, the build order guarantee is very import and the order is
> determined after many times of testing by customer.
> 
>  No, we don't guarantee the build order, it just happens to work that way.
> 
> > Example package 'A', it is not explicitly expressed in openssl`s .mk file. But in
> current build order, 'A' is before openssl. So in the configuring step of openssl,the
> openssl will check that 'A' has been installed. So I am sure that the final result is the
> openssl support 'A'.
> > If per-package gets to the point where it can be merged and it will be pushed into
> buildroot mainstream. In customers, they will find the buildroot can be parallel
> compilation now. They will be very happy to use this feature. So they will use `make
> -j8` instead of 'make -j1'. If they don`t explicitly describe the dependency 'A' in
> openssl`s .mk file. And now the openssl don`t support 'A'. But this is not expected
> behavior for customer.
> 
>  If a dependency is not tracked in the .mk file, you can't count on it. Now already it
> is possible that the order changes due to other dependencies (e.g.
> package C gets an optional dependency on openssl in the new Buildroot version, and
> package C comes alphabetically before package A, then the new Buildroot version
> will build openssl before package A instead of after.
> 
> [snip]
> > Q1: rsync conflicts
> > I want to generate this conflicts in testing but this is difficult. So this is also only
> my worries.
> > At the same time in parallel compilation, different threads want to modify the
> same file of a package in different per-package/package directory and modify the
> content isn`t the same.
> > This will generate conflicts. How to solve this problem?
> > I.E. thread t1 modify file A in per-package/package/A, t1 want to modify A content
> from 'TEXT' to 'EXIT'. At the same time, thread t2 also modify A content from 'TEXT' to
> 'TXXT'.
> 
>  Yes, this is a problem and potential showstopper. We discussed it at length at the
> last BR developer meeting [1]. The conclusion is that we simply cannot support
> packages that edit existing files in top-level parallel build context - they would have
> to encode their dependencies explicitly to force a specific build order. To detect
> how bad the problem is for in-tree packages, Yann posted a series [2] that allows to
> detect it. The idea is that we will turn that detection into an error when PPS is
> merged.
> 
> 
> > And I think rsync has enough power to handle such parallel changes in this situation.
> but after the successful modify there is still problem. for example, rsync make the
> content as 'TXXT' and this result is suitable thread t2. But the thread t1 wants to
> 'EXIT'.
> 
>  Actually, I checked, and rsync has no protection at all against parallel execution.
> E.g. it will first test if a directory exists already and then create it; if another rsync is
> at the same time creating that directory, one of the directory creations will fail. So
> all the rsync's into the same (per-package) directory have to be serialised.
> 
> > I know this probability is minimal. But what should we do if such a conflict arises?
> 
>  As I said: error out.
> 
> 
> > Q2: toolchain HOST_DIR
> > I use  external custom toolchain. 'BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y'
> > line 66 of pkg-toolchain-external.mk66 in toolchain/toolchain-external/ is
> 'TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR = $(HOST_DIR)/opt/ext-toolchain '.
> > this $(HOST_DIR) is also 'output/host/' rather than 'output/per-package/....'.
> 
>  Do you have an actual example of that? The toolchain is just a package, so in all
> places where this variable is used, HOST_DIR should expand into
> output/per-package/.... The except is the definition of TOOLCHAIN_EXTERNAL_BIN,
> but that was fixed in patch 3/4.
> 
> 
>  So, for your further testing, can you try without CCACHE and without patches that
> attempt to fix it? Thanks to your testing we found out that ccache is problematic,
> and it will take a bit of time to fix that. As you have noticed, fixing it correctly is not
> trivial :-)
> 
> 
>  Regards,
>  Arnout
> 
> 
> [1] https://elinux.org/Buildroot:DeveloperDaysELCE2017#Top-level_parallel_build
> [2] http://patchwork.ozlabs.org/project/buildroot/list/?series=10725
> 
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Arnout Vandecappelle]

On 03-11-17 17:06, Thomas Petazzoni wrote:
[snip]
> What remains to be done?
> ========================
> 
>  - Completely get rid of the global HOST_DIR, TARGET_DIR and
>    STAGING_DIR definitions, so that nothing uses them, and they really
>    become per-package variables.

 I don't think that that's needed.


>  - While the toolchain sysroot, pkg-config paths and host RPATHs are
>    properly handled, there are quite certainly a lot of other places
>    that have absolute paths that won't work well with per-package SDK,
>    and that need to be adjusted.
> 
>  - Perhaps use a temporary per-package SDK and target directory when
>    building a package, and rename it to the final name once the
>    package has finished building. This would allow to detect
>    situations where the per-package SDK directory of package A
>    contains absolute paths referring to the per-package SDK directory
>    of package B. Such absolute paths are wrong, but we currently don't
>    see them because the per-package SDK directory for package B still
>    exists.


 Add an instrumentation hook:

	! grep -r $(PER_PACKAGE_DIR) $(HOST_DIR) $(TARGET_DIR)

That will immediately point to the culprit package. Maybe you want to avoid
grepping through gigabytes of data over and over again so perhaps use the same
approach as check-bin-arch.


 But actually,

> 
>  - Many other issues that I'm sure people will report.
> 
> Comparison with Gustavo's patch series
> ======================================
> 
> Gustavo Zacarias did a previous implementation of this, available in
> http://repo.or.cz/buildroot-gz.git/shortlog/refs/heads/pps3-next. Compared
> to Gustavo's patch series, this patch series:
> 

[snip]
>  - I haven't integrated Gustavo patches that move from $(shell ...) to
>    using backticks to delay the evaluation of
>    STAGING_DIR/HOST_DIR/TARGET_DIR:
> 
>    http://repo.or.cz/buildroot-gz.git/commitdiff/0c11c60865f1445830643bb404f92c20d563260c
>    http://repo.or.cz/buildroot-gz.git/commitdiff/207d2248ec8542293b6201b5c86f971021a3a591
>    http://repo.or.cz/buildroot-gz.git/commitdiff/7f6a69819fbb176e29a1c3a53b4a76efe87dad15
>    http://repo.or.cz/buildroot-gz.git/commitdiff/3328a9745eee555f5e5ef7df835afc26612ecd7b
>    http://repo.or.cz/buildroot-gz.git/commitdiff/45a9d8c2aa6f3c0d2d8ed989d843890025faa3e8

 Independently of the PPS feature, these are useful patches I think. Can you
submit them?


>  - I haven't looked at Qt specific issues, such as the ones fixed by
>    Gustavo in:
> 
>    http://repo.or.cz/buildroot-gz.git/commitdiff/643e982e635f4386ccefcda9da4b84536af84d04

 Yeah, and something similar for CMake as well...

 I'm a bit worried about python and friends. They may embed paths in all kinds
of weird places. Like sysconfigdata has paths to compilers and stuff.

 But now I think about it a little more: is there really a problem if e.g. a .la
file refers to a library in one of the other per-package dirs? The library does
exist there after all... The only thing we need to avoid is that a package
installs something in a different package's per-package dir. You may of course
end up with references to indirect dependencies, but they are still dependencies
that are tracked by Buildroot so they're OK.

 So how about this: at the end of the package build, we make the per-package dir
readonly. An evil package can of course still make it readwrite again before
installing something, but it's not very likely.

 Cleaning up all references to other per-package dirs is only really needed if
we want to make a tarball of the directory to save as a seed for a later build.
But we're not there yet by a long shot, so let's just ignore that use case.


 Regards,
 Arnout

> 
>  - I am not doing all the .la files, *-config, *.prl, etc. tweaks that
>    Gustavo is doing in:
> 
>    http://repo.or.cz/buildroot-gz.git/commitdiff/3f7b227b4c8e4514df6e5d54f88fcfb3a3a4bae0
> 
>    It is part of the "remaining absolute paths that need to be fixed"
>    item that I mentionned above.

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [RFCv1 0/4] Per-package SDK and target directories

[Thomas Petazzoni]

Hello,

Here is a first iteration of the per-package SDK and target directory
implementation. This is definitely not ready for merging, but it
already works enough to give it some exposure. I'm sending it early so
that:

 - Interested people can take the patches, try them on their
   configurations, and report me the problems that they face.

 - Interested people start reviewing the patch series will it is still
   small and simple, because I'm sure it will grow quite a bit before
   it gets to a point where it can be merged.

It works well enough to build a complete toolchain and a system with
Busybox, Dropbear, a Python interpreter, strace, with the result
booting under Qemu.

If you're interested in testing it, you can find the patch series at:

  http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=ppsh

Note: in the following discussion, we use the terminology "SDK" to
refer to the combination of HOST_DIR and STAGING_DIR, i.e the
cross-compiler and its sysroot with target libraries/headers, and a
bunch of other native programs. Since STAGING_DIR is included inside
HOST_DIR, the SDK directory is in essence just HOST_DIR.

What is per-package SDK and target?
===================================

Currently, output/host (SDK) and output/target are global directories:
all packages use files installed by other packages in those global
directories, and in turn into those global directories.

The idea of per-package SDK and target is that instead of having such
global directories, we would have one SDK directory and one target
directory per-package, containing just what this package needs
(dependencies), and in which the package installs its files.

Why do we want per-package SDK and target?
==========================================

There are two main motivations for per-package SDK and target
directories, which are related:

 1. Since the SDK directory is global, a package can currently see
    libraries/headers that it has not explicitly expressed a
    dependency on. So package A may not have a dependency on package
    B, but if package B happens to have been installed before, and
    package A "configure" script automaticatically detects a library
    installed by package B, it will use it. We have added tons of
    conditional dependencies in Buildroot to make such situations less
    problematic, but it is an endless fight, as upstream developers
    constantly add more optional dependencies to their software.

    Using per-package SDK, a given package uses as its SDK a directory
    that only contains the libraries/headers from packages explicitly
    listed in its dependencies. So it cannot mistakenly see a
    library/header installed by another package.

 2. Top-level parallel build (building multiple packages in parallel)
    is currently not enabled because we have global SDK and target
    directories.

    Let's imagine package A configure script auto-detects a library
    provided by package B, but Buildroot does not encode this
    dependency in package A's .mk file. Package A and package B are
    therefore totally independent from a build dependency point of
    view.

    Without top-level parallel build (current situation), you have a
    guarantee on the build order: either A is always built before B,
    or B is always built before A. So at least, every build gives the
    same result: A is built with B's functionality or without it, but
    it is consistent accross rebuilds.

    If you enable top-level parallel build, because A and B are
    totally independent, you can get A built before B or B built
    before A depending on the build scheduling. This can change at
    every build! This means that for a given configuration, A will
    sometimes be built with B's functionality, sometimes not. Totally
    unacceptable.

    And of course, this extends to case where package B gets installed
    while package A is being configured. So package A configure script
    will see some parts of B being installed, other parts not
    installed, causing all sort of weird and difficult to debug race
    conditions.

    By having per-package SDK directories, we ensure that package A
    will only see the dependencies its has explicitly expressed, and
    that no other package gets installed in parallel in the same SDK
    directory.

How is it implemented?
======================

The basic idea is pretty simple:

 - For each package, we have:

   output/per-package/<package>/host, which is the specific SDK
   directory for <package>.

   output/per-package/<package>/target, which is the specific target
   directory for <package>.

 - Before <package> is configured, we copy into its specific SDK and
   target directories the SDK and target directories from its direct
   dependencies. This basically populates its SDK with the
   cross-compiler and all libraries that <package> needs. This copy is
   done using rsync with hard links, so it is very fast and cheap from
   a storage consumption point of view.

 - While <package> is being built, HOST_DIR, STAGING_DIR and
   TARGET_DIR are tweaked to point to the current <package> specific
   SDK and target directories. I.e, HOST_DIR no longer points to
   output/host, but to output/per-package/<package>/host/.

And this is basically enough to get things working!

There are of course a few additional things to do:

 - At the end of the build, we still want a global target directory
   (to generate the filesystem images) and a global SDK. Therefore, in
   target-finalize, we assemble such global directories by copying the
   per-package SDK and target directories from all packages listed in
   the $(PACKAGES) variable.

 - We need to fix our pkg-config wrapper script so that it uses
   relative path instead of absolute paths. This allows the wrapper
   script to work regardless of which per-package SDK directory it is
   installed in.

 - We need to move away from using absolute RPATH for host binaries,
   and use LD_LIBRARY_PATH pointing to the current per-package SDK
   directory.

What remains to be done?
========================

 - Completely get rid of the global HOST_DIR, TARGET_DIR and
   STAGING_DIR definitions, so that nothing uses them, and they really
   become per-package variables.

 - While the toolchain sysroot, pkg-config paths and host RPATHs are
   properly handled, there are quite certainly a lot of other places
   that have absolute paths that won't work well with per-package SDK,
   and that need to be adjusted.

 - Perhaps use a temporary per-package SDK and target directory when
   building a package, and rename it to the final name once the
   package has finished building. This would allow to detect
   situations where the per-package SDK directory of package A
   contains absolute paths referring to the per-package SDK directory
   of package B. Such absolute paths are wrong, but we currently don't
   see them because the per-package SDK directory for package B still
   exists.

 - Many other issues that I'm sure people will report.

Comparison with Gustavo's patch series
======================================

Gustavo Zacarias did a previous implementation of this, available in
http://repo.or.cz/buildroot-gz.git/shortlog/refs/heads/pps3-next. Compared
to Gustavo's patch series, this patch series:

 - Implement per-package SDK and target directories, while Gustavo was
   only doing per-package staging.

 - Gustavo had several patches to pass BR_TOOLCHAIN_SYSROOT to the
   toolchain wrapper to pass the location of the current sysroot.

   This is no longer needed, because 1/ we're doing a full per-package
   SDK directory rather than per-package staging, which means the
   sysroot is always at the same relative location compared to the
   cross-compiler binary and 2/ the toolchain wrapper derives the
   sysroot location relatively to the wrapper location. Thanks to
   this, the following patches from Gustavo are (I believe) not
   needed:

   http://repo.or.cz/buildroot-gz.git/commitdiff/1ef6e798064649726d396bcb581ddbe4573c2460
   http://repo.or.cz/buildroot-gz.git/commitdiff/d15f6c6917b555bbbbbf97c292fad4db9d4007d4
   http://repo.or.cz/buildroot-gz.git/commitdiff/95900b65c29a010d85a769c9c6374cf3835f2169
   http://repo.or.cz/buildroot-gz.git/commitdiff/7ed53a5437ab09b85bf6d1953f6ff6049dfcc114

 - Gustavo had a patch to make our pkg-config wrapper script look at
   BR_TOOLCHAIN_SYSROOT, in order to find the
   /usr/{lib,share}/pkgconfig folders of the current sysroot. Just
   like for the toolchain-wrapper, such a change is no longer needed,
   and a simple change to our pkg-config script to use relative paths
   is sufficient.

 - I haven't integrated Gustavo patches that move from $(shell ...) to
   using backticks to delay the evaluation of
   STAGING_DIR/HOST_DIR/TARGET_DIR:

   http://repo.or.cz/buildroot-gz.git/commitdiff/0c11c60865f1445830643bb404f92c20d563260c
   http://repo.or.cz/buildroot-gz.git/commitdiff/207d2248ec8542293b6201b5c86f971021a3a591
   http://repo.or.cz/buildroot-gz.git/commitdiff/7f6a69819fbb176e29a1c3a53b4a76efe87dad15
   http://repo.or.cz/buildroot-gz.git/commitdiff/3328a9745eee555f5e5ef7df835afc26612ecd7b
   http://repo.or.cz/buildroot-gz.git/commitdiff/45a9d8c2aa6f3c0d2d8ed989d843890025faa3e8

 - I haven't looked at Qt specific issues, such as the ones fixed by
   Gustavo in:

   http://repo.or.cz/buildroot-gz.git/commitdiff/643e982e635f4386ccefcda9da4b84536af84d04

 - I am not doing all the .la files, *-config, *.prl, etc. tweaks that
   Gustavo is doing in:

   http://repo.or.cz/buildroot-gz.git/commitdiff/3f7b227b4c8e4514df6e5d54f88fcfb3a3a4bae0

   It is part of the "remaining absolute paths that need to be fixed"
   item that I mentionned above.

Best regards,

Thomas


Thomas Petazzoni (4):
  pkgconf: use relative path to STAGING_DIR instead of absolute path
  core: change host RPATH handling
  toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
  core: implement per-package SDK and target

 Makefile                                           | 19 +++++++--
 package/Makefile.in                                |  2 +-
 package/pkg-generic.mk                             | 45 ++++++++++++++++++----
 package/pkgconf/pkgconf.mk                         |  4 +-
 support/scripts/fix-rpath                          | 15 +++++---
 .../toolchain-external/pkg-toolchain-external.mk   |  2 +-
 6 files changed, 67 insertions(+), 20 deletions(-)

-- 
2.13.6