* [PATCH] populate_sdk_base: use xz -T instead of pixz
@ 2018-04-09 15:19 Ross Burton
2018-04-09 17:46 ` Khem Raj
2018-04-09 20:13 ` Andre McCurdy
0 siblings, 2 replies; 9+ messages in thread
From: Ross Burton @ 2018-04-09 15:19 UTC (permalink / raw)
To: openembedded-core
xz has native support for threaded compression now and SDK creation was the only
part of oe-core which is using pixz instead of xz.
Not only does this remove pixz-native from the SDK dependencies, but in my
limited testing xz -T0 is slightly faster and produces smaller archives than
pixz for the same input.
Signed-off-by: Ross Burton <ross.burton@intel.com>
---
meta/classes/populate_sdk_base.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 77ec8aaec27..79984d7914f 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
-SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
+SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
# We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
@@ -225,7 +225,7 @@ fakeroot tar_sdk() {
# Package it up
mkdir -p ${SDKDEPLOYDIR}
cd ${SDK_OUTPUT}/${SDKPATH}
- tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
+ tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
}
fakeroot create_shar() {
--
2.11.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 15:19 [PATCH] populate_sdk_base: use xz -T instead of pixz Ross Burton
@ 2018-04-09 17:46 ` Khem Raj
2018-04-09 18:29 ` Burton, Ross
2018-04-09 20:13 ` Andre McCurdy
1 sibling, 1 reply; 9+ messages in thread
From: Khem Raj @ 2018-04-09 17:46 UTC (permalink / raw)
To: Ross Burton; +Cc: Patches and discussions about the oe-core layer
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
> xz has native support for threaded compression now and SDK creation was the only
> part of oe-core which is using pixz instead of xz.
>
> Not only does this remove pixz-native from the SDK dependencies, but in my
> limited testing xz -T0 is slightly faster and produces smaller archives than
> pixz for the same input.
>
this is using nativesdk-xz ?
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
> meta/classes/populate_sdk_base.bbclass | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
> index 77ec8aaec27..79984d7914f 100644
> --- a/meta/classes/populate_sdk_base.bbclass
> +++ b/meta/classes/populate_sdk_base.bbclass
> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>
> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>
> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
> # Package it up
> mkdir -p ${SDKDEPLOYDIR}
> cd ${SDK_OUTPUT}/${SDKPATH}
> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
> }
>
> fakeroot create_shar() {
> --
> 2.11.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 17:46 ` Khem Raj
@ 2018-04-09 18:29 ` Burton, Ross
2018-04-09 18:44 ` Khem Raj
0 siblings, 1 reply; 9+ messages in thread
From: Burton, Ross @ 2018-04-09 18:29 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).
Ross
On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote:
> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>> xz has native support for threaded compression now and SDK creation was the only
>> part of oe-core which is using pixz instead of xz.
>>
>> Not only does this remove pixz-native from the SDK dependencies, but in my
>> limited testing xz -T0 is slightly faster and produces smaller archives than
>> pixz for the same input.
>>
>
> this is using nativesdk-xz ?
>
>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>> ---
>> meta/classes/populate_sdk_base.bbclass | 4 ++--
>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
>> index 77ec8aaec27..79984d7914f 100644
>> --- a/meta/classes/populate_sdk_base.bbclass
>> +++ b/meta/classes/populate_sdk_base.bbclass
>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>>
>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>>
>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>> # Package it up
>> mkdir -p ${SDKDEPLOYDIR}
>> cd ${SDK_OUTPUT}/${SDKPATH}
>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>> }
>>
>> fakeroot create_shar() {
>> --
>> 2.11.0
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 18:29 ` Burton, Ross
@ 2018-04-09 18:44 ` Khem Raj
2018-04-09 19:52 ` Burton, Ross
0 siblings, 1 reply; 9+ messages in thread
From: Khem Raj @ 2018-04-09 18:44 UTC (permalink / raw)
To: Burton, Ross; +Cc: Patches and discussions about the oe-core layer
On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross <ross.burton@intel.com> wrote:
> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).
>
OK then we need to make sure that all supported distros will have the
threaded xz version
> Ross
>
> On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote:
>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>>> xz has native support for threaded compression now and SDK creation was the only
>>> part of oe-core which is using pixz instead of xz.
>>>
>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>> pixz for the same input.
>>>
>>
>> this is using nativesdk-xz ?
>>
>>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>>> ---
>>> meta/classes/populate_sdk_base.bbclass | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
>>> index 77ec8aaec27..79984d7914f 100644
>>> --- a/meta/classes/populate_sdk_base.bbclass
>>> +++ b/meta/classes/populate_sdk_base.bbclass
>>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>>>
>>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
>>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>>>
>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>> # Package it up
>>> mkdir -p ${SDKDEPLOYDIR}
>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>> }
>>>
>>> fakeroot create_shar() {
>>> --
>>> 2.11.0
>>>
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 18:44 ` Khem Raj
@ 2018-04-09 19:52 ` Burton, Ross
0 siblings, 0 replies; 9+ messages in thread
From: Burton, Ross @ 2018-04-09 19:52 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
xz-native isn't in ASSUME_PROVIDED, we built it.
Ross
On 9 April 2018 at 19:44, Khem Raj <raj.khem@gmail.com> wrote:
> On Mon, Apr 9, 2018 at 11:29 AM, Burton, Ross <ross.burton@intel.com> wrote:
>> No, it's SDK generation, so xz-native (see SDK_DEPENDS in the first hunk).
>>
>
> OK then we need to make sure that all supported distros will have the
> threaded xz version
>
>> Ross
>>
>> On 9 April 2018 at 18:46, Khem Raj <raj.khem@gmail.com> wrote:
>>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>>>> xz has native support for threaded compression now and SDK creation was the only
>>>> part of oe-core which is using pixz instead of xz.
>>>>
>>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>>> pixz for the same input.
>>>>
>>>
>>> this is using nativesdk-xz ?
>>>
>>>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>>>> ---
>>>> meta/classes/populate_sdk_base.bbclass | 4 ++--
>>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
>>>> index 77ec8aaec27..79984d7914f 100644
>>>> --- a/meta/classes/populate_sdk_base.bbclass
>>>> +++ b/meta/classes/populate_sdk_base.bbclass
>>>> @@ -46,7 +46,7 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
>>>> TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
>>>>
>>>> SDK_RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_HOST_TASK}"
>>>> -SDK_DEPENDS = "virtual/fakeroot-native pixz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>>>> +SDK_DEPENDS = "virtual/fakeroot-native xz-native cross-localedef-native ${MLPREFIX}qemuwrapper-cross"
>>>> SDK_DEPENDS_append_libc-glibc = " nativesdk-glibc-locale"
>>>>
>>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>>> # Package it up
>>>> mkdir -p ${SDKDEPLOYDIR}
>>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>> }
>>>>
>>>> fakeroot create_shar() {
>>>> --
>>>> 2.11.0
>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 15:19 [PATCH] populate_sdk_base: use xz -T instead of pixz Ross Burton
2018-04-09 17:46 ` Khem Raj
@ 2018-04-09 20:13 ` Andre McCurdy
2018-04-09 20:44 ` Burton, Ross
1 sibling, 1 reply; 9+ messages in thread
From: Andre McCurdy @ 2018-04-09 20:13 UTC (permalink / raw)
To: Ross Burton; +Cc: OE Core mailing list
On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
> xz has native support for threaded compression now and SDK creation was the only
> part of oe-core which is using pixz instead of xz.
>
> Not only does this remove pixz-native from the SDK dependencies, but in my
> limited testing xz -T0 is slightly faster and produces smaller archives than
> pixz for the same input.
>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>
> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
> # Package it up
> mkdir -p ${SDKDEPLOYDIR}
> cd ${SDK_OUTPUT}/${SDKPATH}
> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
Since -T 0 will use all available CPUs (ie any attempts the user may
have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
will be ignored), perhaps it's worth adding something like
"--memlimit=70%" to try to give some protection for environments with
lots of CPUs but not so much DRAM?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 20:13 ` Andre McCurdy
@ 2018-04-09 20:44 ` Burton, Ross
2018-04-09 21:20 ` Khem Raj
0 siblings, 1 reply; 9+ messages in thread
From: Burton, Ross @ 2018-04-09 20:44 UTC (permalink / raw)
To: Andre McCurdy; +Cc: OE Core mailing list
On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote:
> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>> xz has native support for threaded compression now and SDK creation was the only
>> part of oe-core which is using pixz instead of xz.
>>
>> Not only does this remove pixz-native from the SDK dependencies, but in my
>> limited testing xz -T0 is slightly faster and produces smaller archives than
>> pixz for the same input.
>>
>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>> ---
>>
>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>> # Package it up
>> mkdir -p ${SDKDEPLOYDIR}
>> cd ${SDK_OUTPUT}/${SDKPATH}
>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>
> Since -T 0 will use all available CPUs (ie any attempts the user may
> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
> will be ignored), perhaps it's worth adding something like
> "--memlimit=70%" to try to give some protection for environments with
> lots of CPUs but not so much DRAM?
There's a few places where -T0 is passed already: tar.xz image
creation and opkg creation, so I guess we need to centralise this
somewhere too.
Ross
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 20:44 ` Burton, Ross
@ 2018-04-09 21:20 ` Khem Raj
2018-04-09 22:04 ` Burton, Ross
0 siblings, 1 reply; 9+ messages in thread
From: Khem Raj @ 2018-04-09 21:20 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE Core mailing list
On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross <ross.burton@intel.com> wrote:
> On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote:
>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>>> xz has native support for threaded compression now and SDK creation was the only
>>> part of oe-core which is using pixz instead of xz.
>>>
>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>> pixz for the same input.
>>>
>>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>>> ---
>>>
>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>> # Package it up
>>> mkdir -p ${SDKDEPLOYDIR}
>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>
>> Since -T 0 will use all available CPUs (ie any attempts the user may
>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
>> will be ignored), perhaps it's worth adding something like
>> "--memlimit=70%" to try to give some protection for environments with
>> lots of CPUs but not so much DRAM?
>
> There's a few places where -T0 is passed already: tar.xz image
> creation and opkg creation, so I guess we need to centralise this
> somewhere too.
>
May be use XZ_OPT env bariable and param to -T could use the bitbake
calculated number of CPUs
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] populate_sdk_base: use xz -T instead of pixz
2018-04-09 21:20 ` Khem Raj
@ 2018-04-09 22:04 ` Burton, Ross
0 siblings, 0 replies; 9+ messages in thread
From: Burton, Ross @ 2018-04-09 22:04 UTC (permalink / raw)
To: Khem Raj; +Cc: OE Core mailing list
On 9 April 2018 at 22:20, Khem Raj <raj.khem@gmail.com> wrote:
> On Mon, Apr 9, 2018 at 1:44 PM, Burton, Ross <ross.burton@intel.com> wrote:
>> On 9 April 2018 at 21:13, Andre McCurdy <armccurdy@gmail.com> wrote:
>>> On Mon, Apr 9, 2018 at 8:19 AM, Ross Burton <ross.burton@intel.com> wrote:
>>>> xz has native support for threaded compression now and SDK creation was the only
>>>> part of oe-core which is using pixz instead of xz.
>>>>
>>>> Not only does this remove pixz-native from the SDK dependencies, but in my
>>>> limited testing xz -T0 is slightly faster and produces smaller archives than
>>>> pixz for the same input.
>>>>
>>>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>>>> ---
>>>>
>>>> # We want the MULTIARCH_TARGET_SYS to point to the TUNE_PKGARCH, not PACKAGE_ARCH as it
>>>> @@ -225,7 +225,7 @@ fakeroot tar_sdk() {
>>>> # Package it up
>>>> mkdir -p ${SDKDEPLOYDIR}
>>>> cd ${SDK_OUTPUT}/${SDKPATH}
>>>> - tar ${SDKTAROPTS} -cf - . | pixz > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>> + tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz
>>>
>>> Since -T 0 will use all available CPUs (ie any attempts the user may
>>> have made to limit parallelism via BB_NUMBER_THREADS or PARALLEL_MAKE
>>> will be ignored), perhaps it's worth adding something like
>>> "--memlimit=70%" to try to give some protection for environments with
>>> lots of CPUs but not so much DRAM?
>>
>> There's a few places where -T0 is passed already: tar.xz image
>> creation and opkg creation, so I guess we need to centralise this
>> somewhere too.
>>
>
> May be use XZ_OPT env bariable and param to -T could use the bitbake
> calculated number of CPUs
-T0 is use as many threads as there are cores, which is the same as
the bitbake function. Re-using BB_NUMBER_THREADS might work.
Ross
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-04-09 22:05 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-09 15:19 [PATCH] populate_sdk_base: use xz -T instead of pixz Ross Burton
2018-04-09 17:46 ` Khem Raj
2018-04-09 18:29 ` Burton, Ross
2018-04-09 18:44 ` Khem Raj
2018-04-09 19:52 ` Burton, Ross
2018-04-09 20:13 ` Andre McCurdy
2018-04-09 20:44 ` Burton, Ross
2018-04-09 21:20 ` Khem Raj
2018-04-09 22:04 ` Burton, Ross
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.