* [PATCH 0/3] Fix xz compression command and optimize compression time
@ 2012-07-12 18:13 Andrei Gherzan
2012-07-12 18:13 ` [PATCH 1/3] image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed data to file Andrei Gherzan
` (4 more replies)
0 siblings, 5 replies; 15+ messages in thread
From: Andrei Gherzan @ 2012-07-12 18:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Koen Kooi
Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
-T threads, --threads=threads
Specify the number of worker threads to use. Setting threads to
a special value 0 makes xz use as many threads as there are CPU
cores on the system. The actual number of threads can be less
than threads if the input file is not big enough for threading
with the given settings or if using more threads would exceed
the memory usage limit.
-------------------
Memory: 7.8 GiB
Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
File to compress: 1.9G
xz -f -k -e -9 --check=crc32 (current configuration)
Memory: 673Mb
real 6m37.170s
user 6m35.389s
sys 0m0.884s
Compressed file size: 3.4Mb
xz -f -k -e -9 -T 4 --check=crc32
Memory: Cannot allocate memory
xz -f -k -e -8 -T 4 --check=crc32
Memory: 1.8G
real 3m24.462s
user 12m8.502s
sys 0m2.180s
Compressed file size: 3.4Mb
xz -f -k -T 4 --check=crc32 (-e defaults to -6)
Memory: 471Mb
real 1m9.265s
user 4m8.972s
sys 0m0.944s
Compressed file size: 3.4Mb
So my conclusion would be to use the default -e -6 with -T 0.
The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ag/xz
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
Andrei Gherzan (3):
image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
data to file
image_types.bbclass: Add XZ variable to set number of threads to be
used while compressing
image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
meta/classes/image_types.bbclass | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--
1.7.9.5
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH 1/3] image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed data to file 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan @ 2012-07-12 18:13 ` Andrei Gherzan 2012-07-12 18:13 ` [PATCH 2/3] image_types.bbclass: Add XZ variable to set number of threads to be used while compressing Andrei Gherzan ` (3 subsequent siblings) 4 siblings, 0 replies; 15+ messages in thread From: Andrei Gherzan @ 2012-07-12 18:13 UTC (permalink / raw) To: openembedded-core Having -c modifier makes xz to output the compressed data to stdout. In this way the needed data will be in the do_rootfs log. Redirect data to ${IMAGE_NAME}.rootfs.${type}.xz . Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> --- meta/classes/image_types.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index 5734edf..0e79820 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -236,7 +236,7 @@ COMPRESSIONTYPES = "gz bz2 lzma xz" COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}.rootfs.${type}" COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}.rootfs.${type} > ${IMAGE_NAME}.rootfs.${type}.gz" COMPRESS_CMD_bz2 = "bzip2 -f -k ${IMAGE_NAME}.rootfs.${type}" -COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}.rootfs.${type}" +COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}.rootfs.${type} > ${IMAGE_NAME}.rootfs.${type}.xz" COMPRESS_DEPENDS_lzma = "xz-native" COMPRESS_DEPENDS_gz = "" COMPRESS_DEPENDS_bz2 = "" -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 2/3] image_types.bbclass: Add XZ variable to set number of threads to be used while compressing 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan 2012-07-12 18:13 ` [PATCH 1/3] image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed data to file Andrei Gherzan @ 2012-07-12 18:13 ` Andrei Gherzan 2012-07-12 18:13 ` [PATCH 3/3] image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 Andrei Gherzan ` (2 subsequent siblings) 4 siblings, 0 replies; 15+ messages in thread From: Andrei Gherzan @ 2012-07-12 18:13 UTC (permalink / raw) To: openembedded-core Default this variable to 0. This will make xz use as many threads as there are CPU cores on the system. Signed-off-by: Koen Kooi <koen@dominion.thruhere.net> Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> --- meta/classes/image_types.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index 0e79820..b19ad3b 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -124,6 +124,7 @@ def imagetypes_getdepends(d): XZ_COMPRESSION_LEVEL ?= "-e -9" XZ_INTEGRITY_CHECK ?= "crc32" +XZ_THREADS ?= "-T 0" IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.jffs2 -n ${EXTRA_IMAGECMD}" IMAGE_CMD_sum.jffs2 = "${IMAGE_CMD_jffs2} && sumtool -i ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.jffs2 \ @@ -236,7 +237,7 @@ COMPRESSIONTYPES = "gz bz2 lzma xz" COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}.rootfs.${type}" COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}.rootfs.${type} > ${IMAGE_NAME}.rootfs.${type}.gz" COMPRESS_CMD_bz2 = "bzip2 -f -k ${IMAGE_NAME}.rootfs.${type}" -COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}.rootfs.${type} > ${IMAGE_NAME}.rootfs.${type}.xz" +COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}.rootfs.${type} > ${IMAGE_NAME}.rootfs.${type}.xz" COMPRESS_DEPENDS_lzma = "xz-native" COMPRESS_DEPENDS_gz = "" COMPRESS_DEPENDS_bz2 = "" -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 3/3] image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan 2012-07-12 18:13 ` [PATCH 1/3] image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed data to file Andrei Gherzan 2012-07-12 18:13 ` [PATCH 2/3] image_types.bbclass: Add XZ variable to set number of threads to be used while compressing Andrei Gherzan @ 2012-07-12 18:13 ` Andrei Gherzan 2012-07-12 18:58 ` [PATCH 0/3] Fix xz compression command and optimize compression time Koen Kooi 2012-07-17 16:27 ` Saul Wold 4 siblings, 0 replies; 15+ messages in thread From: Andrei Gherzan @ 2012-07-12 18:13 UTC (permalink / raw) To: openembedded-core Having XZ_COMPRESSION_LEVEL on -e -9 and -T 0 will make xz eat more than 6Gb memory. Reduce this to -6 to make xz to use about 471Mb on the tested machine. Signed-off-by: Koen Kooi <koen@dominion.thruhere.net> Signed-off-by: Andrei Gherzan <andrei@gherzan.ro> --- meta/classes/image_types.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index b19ad3b..727d8d6 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -122,7 +122,7 @@ def imagetypes_getdepends(d): return depstr -XZ_COMPRESSION_LEVEL ?= "-e -9" +XZ_COMPRESSION_LEVEL ?= "-e -6" XZ_INTEGRITY_CHECK ?= "crc32" XZ_THREADS ?= "-T 0" -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan ` (2 preceding siblings ...) 2012-07-12 18:13 ` [PATCH 3/3] image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 Andrei Gherzan @ 2012-07-12 18:58 ` Koen Kooi 2012-07-12 19:20 ` McClintock Matthew-B29882 2012-07-12 19:22 ` Saul Wold 2012-07-17 16:27 ` Saul Wold 4 siblings, 2 replies; 15+ messages in thread From: Koen Kooi @ 2012-07-12 18:58 UTC (permalink / raw) To: openembedded oe-core layer Any volunteers to test this on a system with >4 real cores? Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven: > Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. > > -T threads, --threads=threads > Specify the number of worker threads to use. Setting threads to > a special value 0 makes xz use as many threads as there are CPU > cores on the system. The actual number of threads can be less > than threads if the input file is not big enough for threading > with the given settings or if using more threads would exceed > the memory usage limit. > > ------------------- > > Memory: 7.8 GiB > Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 > > File to compress: 1.9G > > xz -f -k -e -9 --check=crc32 (current configuration) > Memory: 673Mb > real 6m37.170s > user 6m35.389s > sys 0m0.884s > Compressed file size: 3.4Mb > > xz -f -k -e -9 -T 4 --check=crc32 > Memory: Cannot allocate memory > > xz -f -k -e -8 -T 4 --check=crc32 > Memory: 1.8G > real 3m24.462s > user 12m8.502s > sys 0m2.180s > Compressed file size: 3.4Mb > > xz -f -k -T 4 --check=crc32 (-e defaults to -6) > Memory: 471Mb > real 1m9.265s > user 4m8.972s > sys 0m0.944s > Compressed file size: 3.4Mb > > So my conclusion would be to use the default -e -6 with -T 0. > > The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: > > upstream_tracking: update boost (2012-07-09 17:21:38 +0100) > > are available in the git repository at: > > git://git.yoctoproject.org/poky-contrib ag/xz > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz > > Andrei Gherzan (3): > image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed > data to file > image_types.bbclass: Add XZ variable to set number of threads to be > used while compressing > image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 > > meta/classes/image_types.bbclass | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > -- > 1.7.9.5 > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 18:58 ` [PATCH 0/3] Fix xz compression command and optimize compression time Koen Kooi @ 2012-07-12 19:20 ` McClintock Matthew-B29882 2012-07-12 19:22 ` Saul Wold 1 sibling, 0 replies; 15+ messages in thread From: McClintock Matthew-B29882 @ 2012-07-12 19:20 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Thu, Jul 12, 2012 at 1:58 PM, Koen Kooi <koen@dominion.thruhere.net> wrote: > Any volunteers to test this on a system with >4 real cores? How do you get the memory usage bit? -M > > Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven: > >> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. >> >> -T threads, --threads=threads >> Specify the number of worker threads to use. Setting threads to >> a special value 0 makes xz use as many threads as there are CPU >> cores on the system. The actual number of threads can be less >> than threads if the input file is not big enough for threading >> with the given settings or if using more threads would exceed >> the memory usage limit. >> >> ------------------- >> >> Memory: 7.8 GiB >> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 >> >> File to compress: 1.9G >> >> xz -f -k -e -9 --check=crc32 (current configuration) >> Memory: 673Mb >> real 6m37.170s >> user 6m35.389s >> sys 0m0.884s >> Compressed file size: 3.4Mb >> >> xz -f -k -e -9 -T 4 --check=crc32 >> Memory: Cannot allocate memory >> >> xz -f -k -e -8 -T 4 --check=crc32 >> Memory: 1.8G >> real 3m24.462s >> user 12m8.502s >> sys 0m2.180s >> Compressed file size: 3.4Mb >> >> xz -f -k -T 4 --check=crc32 (-e defaults to -6) >> Memory: 471Mb >> real 1m9.265s >> user 4m8.972s >> sys 0m0.944s >> Compressed file size: 3.4Mb >> >> So my conclusion would be to use the default -e -6 with -T 0. >> >> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: >> >> upstream_tracking: update boost (2012-07-09 17:21:38 +0100) >> >> are available in the git repository at: >> >> git://git.yoctoproject.org/poky-contrib ag/xz >> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz >> >> Andrei Gherzan (3): >> image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed >> data to file >> image_types.bbclass: Add XZ variable to set number of threads to be >> used while compressing >> image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 >> >> meta/classes/image_types.bbclass | 5 +++-- >> 1 file changed, 3 insertions(+), 2 deletions(-) >> >> -- >> 1.7.9.5 >> > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 18:58 ` [PATCH 0/3] Fix xz compression command and optimize compression time Koen Kooi 2012-07-12 19:20 ` McClintock Matthew-B29882 @ 2012-07-12 19:22 ` Saul Wold 2012-07-12 19:59 ` Andrei Gherzan 2012-07-13 6:45 ` Koen Kooi 1 sibling, 2 replies; 15+ messages in thread From: Saul Wold @ 2012-07-12 19:22 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: Koen Kooi On 07/12/2012 11:58 AM, Koen Kooi wrote: > Any volunteers to test this on a system with >4 real cores? > Koen, Does OE-Core or Poky have an image setup for using .xz by default? I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running. I was about to ask what msm asked about the Memory info. Sau! > Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven: > >> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. >> >> -T threads, --threads=threads >> Specify the number of worker threads to use. Setting threads to >> a special value 0 makes xz use as many threads as there are CPU >> cores on the system. The actual number of threads can be less >> than threads if the input file is not big enough for threading >> with the given settings or if using more threads would exceed >> the memory usage limit. >> >> ------------------- >> >> Memory: 7.8 GiB >> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 >> >> File to compress: 1.9G >> >> xz -f -k -e -9 --check=crc32 (current configuration) >> Memory: 673Mb >> real 6m37.170s >> user 6m35.389s >> sys 0m0.884s >> Compressed file size: 3.4Mb >> >> xz -f -k -e -9 -T 4 --check=crc32 >> Memory: Cannot allocate memory >> >> xz -f -k -e -8 -T 4 --check=crc32 >> Memory: 1.8G >> real 3m24.462s >> user 12m8.502s >> sys 0m2.180s >> Compressed file size: 3.4Mb >> >> xz -f -k -T 4 --check=crc32 (-e defaults to -6) >> Memory: 471Mb >> real 1m9.265s >> user 4m8.972s >> sys 0m0.944s >> Compressed file size: 3.4Mb >> >> So my conclusion would be to use the default -e -6 with -T 0. >> >> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: >> >> upstream_tracking: update boost (2012-07-09 17:21:38 +0100) >> >> are available in the git repository at: >> >> git://git.yoctoproject.org/poky-contrib ag/xz >> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz >> >> Andrei Gherzan (3): >> image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed >> data to file >> image_types.bbclass: Add XZ variable to set number of threads to be >> used while compressing >> image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 >> >> meta/classes/image_types.bbclass | 5 +++-- >> 1 file changed, 3 insertions(+), 2 deletions(-) >> >> -- >> 1.7.9.5 >> > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 19:22 ` Saul Wold @ 2012-07-12 19:59 ` Andrei Gherzan 2012-07-12 20:14 ` Khem Raj 2012-07-13 6:45 ` Koen Kooi 1 sibling, 1 reply; 15+ messages in thread From: Andrei Gherzan @ 2012-07-12 19:59 UTC (permalink / raw) To: Saul Wold; +Cc: Koen Kooi, Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 581 bytes --] On Thu, Jul 12, 2012 at 10:22 PM, Saul Wold <sgw@linux.intel.com> wrote: > On 07/12/2012 11:58 AM, Koen Kooi wrote: > >> Any volunteers to test this on a system with >4 real cores? >> >> > Koen, > > Does OE-Core or Poky have an image setup for using .xz by default? > > I will run it on a machine I have, just want to make sure I am doing the > same tests that Andrei is running. > > I was about to ask what msm asked about the Memory info. > > That field in results is not from "time". It's from gnome-system-monitor. Sorry for being a little confusing. @g [-- Attachment #2: Type: text/html, Size: 1040 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 19:59 ` Andrei Gherzan @ 2012-07-12 20:14 ` Khem Raj 0 siblings, 0 replies; 15+ messages in thread From: Khem Raj @ 2012-07-12 20:14 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Thu, Jul 12, 2012 at 12:59 PM, Andrei Gherzan <andrei@gherzan.ro> wrote: > On Thu, Jul 12, 2012 at 10:22 PM, Saul Wold <sgw@linux.intel.com> wrote: >> >> On 07/12/2012 11:58 AM, Koen Kooi wrote: >>> >>> Any volunteers to test this on a system with >4 real cores? >>> >> >> Koen, >> >> Does OE-Core or Poky have an image setup for using .xz by default? >> >> I will run it on a machine I have, just want to make sure I am doing the >> same tests that Andrei is running. >> >> I was about to ask what msm asked about the Memory info. use htop > That field in results is not from "time". It's from gnome-system-monitor. > Sorry for being a little confusing. > > @g > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 19:22 ` Saul Wold 2012-07-12 19:59 ` Andrei Gherzan @ 2012-07-13 6:45 ` Koen Kooi 2012-07-13 8:19 ` Andrea Adami 1 sibling, 1 reply; 15+ messages in thread From: Koen Kooi @ 2012-07-13 6:45 UTC (permalink / raw) To: Saul Wold; +Cc: Patches and discussions about the oe-core layer Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven: > On 07/12/2012 11:58 AM, Koen Kooi wrote: >> Any volunteers to test this on a system with >4 real cores? >> > > Koen, > > Does OE-Core or Poky have an image setup for using .xz by default? No, and as you can see from Andrei's patches, it would have never worked :) regards, Koen > > I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running. > > I was about to ask what msm asked about the Memory info. > > > Sau! > >> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven: >> >>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. >>> >>> -T threads, --threads=threads >>> Specify the number of worker threads to use. Setting threads to >>> a special value 0 makes xz use as many threads as there are CPU >>> cores on the system. The actual number of threads can be less >>> than threads if the input file is not big enough for threading >>> with the given settings or if using more threads would exceed >>> the memory usage limit. >>> >>> ------------------- >>> >>> Memory: 7.8 GiB >>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 >>> >>> File to compress: 1.9G >>> >>> xz -f -k -e -9 --check=crc32 (current configuration) >>> Memory: 673Mb >>> real 6m37.170s >>> user 6m35.389s >>> sys 0m0.884s >>> Compressed file size: 3.4Mb >>> >>> xz -f -k -e -9 -T 4 --check=crc32 >>> Memory: Cannot allocate memory >>> >>> xz -f -k -e -8 -T 4 --check=crc32 >>> Memory: 1.8G >>> real 3m24.462s >>> user 12m8.502s >>> sys 0m2.180s >>> Compressed file size: 3.4Mb >>> >>> xz -f -k -T 4 --check=crc32 (-e defaults to -6) >>> Memory: 471Mb >>> real 1m9.265s >>> user 4m8.972s >>> sys 0m0.944s >>> Compressed file size: 3.4Mb >>> >>> So my conclusion would be to use the default -e -6 with -T 0. >>> >>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: >>> >>> upstream_tracking: update boost (2012-07-09 17:21:38 +0100) >>> >>> are available in the git repository at: >>> >>> git://git.yoctoproject.org/poky-contrib ag/xz >>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz >>> >>> Andrei Gherzan (3): >>> image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed >>> data to file >>> image_types.bbclass: Add XZ variable to set number of threads to be >>> used while compressing >>> image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 >>> >>> meta/classes/image_types.bbclass | 5 +++-- >>> 1 file changed, 3 insertions(+), 2 deletions(-) >>> >>> -- >>> 1.7.9.5 >>> >> >> >> _______________________________________________ >> Openembedded-core mailing list >> Openembedded-core@lists.openembedded.org >> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >> >> > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-13 6:45 ` Koen Kooi @ 2012-07-13 8:19 ` Andrea Adami 2012-07-13 9:44 ` Andrei Gherzan 0 siblings, 1 reply; 15+ messages in thread From: Andrea Adami @ 2012-07-13 8:19 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net> wrote: > > Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven: > >> On 07/12/2012 11:58 AM, Koen Kooi wrote: >>> Any volunteers to test this on a system with >4 real cores? >>> >> >> Koen, >> >> Does OE-Core or Poky have an image setup for using .xz by default? > > No, and as you can see from Andrei's patches, it would have never worked :) > > regards, > > Koen > This leads to a similar remark for lzma: The xz/lzma have been added with commit 38334ac and at that time both where using XZ_COMPRESSION_LEVEL. The code has been refactored afterwards and what happened is that lzma now defaults to compression 7 (the default) while xz is way too high (default is 6). There is a well hidden pitfall with that, being that any image, including the initramfs, will use that compression factor. As you can notice in that (old) man xz [2] the memory requirements are very high and what did happen to me was that the cpio.lzma could not be decompressed thus kernel could not boot (we override now the value setting "-e2" in our BSP). Adding more threads will multiply that figures. My 2 cents Andrea [1] http://cgit.openembedded.org/openembedded-core/log/meta/classes/image_types.bbclass [2] http://www.linuxcertif.com/man/1/xz >> >> I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running. >> >> I was about to ask what msm asked about the Memory info. >> >> >> Sau! >> >>> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven: >>> >>>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. >>>> >>>> -T threads, --threads=threads >>>> Specify the number of worker threads to use. Setting threads to >>>> a special value 0 makes xz use as many threads as there are CPU >>>> cores on the system. The actual number of threads can be less >>>> than threads if the input file is not big enough for threading >>>> with the given settings or if using more threads would exceed >>>> the memory usage limit. >>>> >>>> ------------------- >>>> >>>> Memory: 7.8 GiB >>>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 >>>> >>>> File to compress: 1.9G >>>> >>>> xz -f -k -e -9 --check=crc32 (current configuration) >>>> Memory: 673Mb >>>> real 6m37.170s >>>> user 6m35.389s >>>> sys 0m0.884s >>>> Compressed file size: 3.4Mb >>>> >>>> xz -f -k -e -9 -T 4 --check=crc32 >>>> Memory: Cannot allocate memory >>>> >>>> xz -f -k -e -8 -T 4 --check=crc32 >>>> Memory: 1.8G >>>> real 3m24.462s >>>> user 12m8.502s >>>> sys 0m2.180s >>>> Compressed file size: 3.4Mb >>>> >>>> xz -f -k -T 4 --check=crc32 (-e defaults to -6) >>>> Memory: 471Mb >>>> real 1m9.265s >>>> user 4m8.972s >>>> sys 0m0.944s >>>> Compressed file size: 3.4Mb >>>> >>>> So my conclusion would be to use the default -e -6 with -T 0. >>>> >>>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: >>>> >>>> upstream_tracking: update boost (2012-07-09 17:21:38 +0100) >>>> >>>> are available in the git repository at: >>>> >>>> git://git.yoctoproject.org/poky-contrib ag/xz >>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz >>>> >>>> Andrei Gherzan (3): >>>> image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed >>>> data to file >>>> image_types.bbclass: Add XZ variable to set number of threads to be >>>> used while compressing >>>> image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 >>>> >>>> meta/classes/image_types.bbclass | 5 +++-- >>>> 1 file changed, 3 insertions(+), 2 deletions(-) >>>> >>>> -- >>>> 1.7.9.5 >>>> >>> >>> >>> _______________________________________________ >>> Openembedded-core mailing list >>> Openembedded-core@lists.openembedded.org >>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core >>> >>> >> > > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-13 8:19 ` Andrea Adami @ 2012-07-13 9:44 ` Andrei Gherzan 2012-07-13 9:54 ` Andrea Adami 0 siblings, 1 reply; 15+ messages in thread From: Andrei Gherzan @ 2012-07-13 9:44 UTC (permalink / raw) To: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 1635 bytes --] On Fri, Jul 13, 2012 at 11:19 AM, Andrea Adami <andrea.adami@gmail.com>wrote: > On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net> > wrote: > > > > Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven: > > > >> On 07/12/2012 11:58 AM, Koen Kooi wrote: > >>> Any volunteers to test this on a system with >4 real cores? > >>> > >> > >> Koen, > >> > >> Does OE-Core or Poky have an image setup for using .xz by default? > > > > No, and as you can see from Andrei's patches, it would have never worked > :) > > > > regards, > > > > Koen > > > > This leads to a similar remark for lzma: > > The xz/lzma have been added with commit 38334ac and at that time both > where using XZ_COMPRESSION_LEVEL. > > The code has been refactored afterwards and what happened is that lzma > now defaults to compression 7 (the default) while xz is way too high > (default is 6). > > Nope. It's default value is -9. > There is a well hidden pitfall with that, being that any image, > including the initramfs, will use that compression factor. > > As you can notice in that (old) man xz [2] the memory requirements > What old version? > are very high and what did happen to me was that the cpio.lzma could > not be decompressed thus kernel could not boot (we override now the > value setting "-e2" in our BSP). > > Without threads, memory demand was around 700mb. So it should be ok for most of machines. Adding more threads will multiply that figures. > > Indeed. With 8GB memory at my disposal i couldn't compress an image with -e -9 and -T 0 (-T 4)/ @g [-- Attachment #2: Type: text/html, Size: 2615 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-13 9:44 ` Andrei Gherzan @ 2012-07-13 9:54 ` Andrea Adami 2012-07-13 9:59 ` Andrei Gherzan 0 siblings, 1 reply; 15+ messages in thread From: Andrea Adami @ 2012-07-13 9:54 UTC (permalink / raw) To: Patches and discussions about the oe-core layer On Fri, Jul 13, 2012 at 11:44 AM, Andrei Gherzan <andrei@gherzan.ro> wrote: > On Fri, Jul 13, 2012 at 11:19 AM, Andrea Adami <andrea.adami@gmail.com> > wrote: >> >> On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen@dominion.thruhere.net> >> wrote: >> > >> > Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven: >> > >> >> On 07/12/2012 11:58 AM, Koen Kooi wrote: >> >>> Any volunteers to test this on a system with >4 real cores? >> >>> >> >> >> >> Koen, >> >> >> >> Does OE-Core or Poky have an image setup for using .xz by default? >> > >> > No, and as you can see from Andrei's patches, it would have never worked >> > :) >> > >> > regards, >> > >> > Koen >> > >> >> This leads to a similar remark for lzma: >> >> The xz/lzma have been added with commit 38334ac and at that time both >> where using XZ_COMPRESSION_LEVEL. >> >> The code has been refactored afterwards and what happened is that lzma >> now defaults to compression 7 (the default) while xz is way too high >> (default is 6). >> > > Nope. It's default value is -9. > Well, in oe-core maybe. Upstream they are more conservative, see that man. "The default preset level in LZMA Utils is -7 while in XZ Utils it is -6, so both use 8 MiB dictionary by default." Cheers Andrea >> >> There is a well hidden pitfall with that, being that any image, >> including the initramfs, will use that compression factor. >> >> As you can notice in that (old) man xz [2] the memory requirements > > > What old version? > >> >> are very high and what did happen to me was that the cpio.lzma could >> not be decompressed thus kernel could not boot (we override now the >> value setting "-e2" in our BSP). >> > > Without threads, memory demand was around 700mb. So it should be ok for most > of machines. > >> Adding more threads will multiply that figures. >> > > Indeed. With 8GB memory at my disposal i couldn't compress an image with -e > -9 and -T 0 (-T 4)/ > > @g > > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-13 9:54 ` Andrea Adami @ 2012-07-13 9:59 ` Andrei Gherzan 0 siblings, 0 replies; 15+ messages in thread From: Andrei Gherzan @ 2012-07-13 9:59 UTC (permalink / raw) To: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 274 bytes --] > > > Well, in oe-core maybe. > Upstream they are more conservative, see that man. > > "The default preset level in LZMA Utils is -7 while in XZ Utils it is > -6, so both use 8 MiB dictionary by default." > > So you were talking about upstream. Ok got that. @g [-- Attachment #2: Type: text/html, Size: 505 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] Fix xz compression command and optimize compression time 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan ` (3 preceding siblings ...) 2012-07-12 18:58 ` [PATCH 0/3] Fix xz compression command and optimize compression time Koen Kooi @ 2012-07-17 16:27 ` Saul Wold 4 siblings, 0 replies; 15+ messages in thread From: Saul Wold @ 2012-07-17 16:27 UTC (permalink / raw) To: Patches and discussions about the oe-core layer; +Cc: Koen Kooi On 07/12/2012 11:13 AM, Andrei Gherzan wrote: > Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha. > > -T threads, --threads=threads > Specify the number of worker threads to use. Setting threads to > a special value 0 makes xz use as many threads as there are CPU > cores on the system. The actual number of threads can be less > than threads if the input file is not big enough for threading > with the given settings or if using more threads would exceed > the memory usage limit. > > ------------------- > > Memory: 7.8 GiB > Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4 > > File to compress: 1.9G > > xz -f -k -e -9 --check=crc32 (current configuration) > Memory: 673Mb > real 6m37.170s > user 6m35.389s > sys 0m0.884s > Compressed file size: 3.4Mb > > xz -f -k -e -9 -T 4 --check=crc32 > Memory: Cannot allocate memory > > xz -f -k -e -8 -T 4 --check=crc32 > Memory: 1.8G > real 3m24.462s > user 12m8.502s > sys 0m2.180s > Compressed file size: 3.4Mb > > xz -f -k -T 4 --check=crc32 (-e defaults to -6) > Memory: 471Mb > real 1m9.265s > user 4m8.972s > sys 0m0.944s > Compressed file size: 3.4Mb > > So my conclusion would be to use the default -e -6 with -T 0. > > The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669: > > upstream_tracking: update boost (2012-07-09 17:21:38 +0100) > > are available in the git repository at: > > git://git.yoctoproject.org/poky-contrib ag/xz > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz > > Andrei Gherzan (3): > image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed > data to file > image_types.bbclass: Add XZ variable to set number of threads to be > used while compressing > image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 > > meta/classes/image_types.bbclass | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > Merged into OE-Core Thanks Sau! ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-07-17 16:39 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-07-12 18:13 [PATCH 0/3] Fix xz compression command and optimize compression time Andrei Gherzan 2012-07-12 18:13 ` [PATCH 1/3] image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed data to file Andrei Gherzan 2012-07-12 18:13 ` [PATCH 2/3] image_types.bbclass: Add XZ variable to set number of threads to be used while compressing Andrei Gherzan 2012-07-12 18:13 ` [PATCH 3/3] image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6 Andrei Gherzan 2012-07-12 18:58 ` [PATCH 0/3] Fix xz compression command and optimize compression time Koen Kooi 2012-07-12 19:20 ` McClintock Matthew-B29882 2012-07-12 19:22 ` Saul Wold 2012-07-12 19:59 ` Andrei Gherzan 2012-07-12 20:14 ` Khem Raj 2012-07-13 6:45 ` Koen Kooi 2012-07-13 8:19 ` Andrea Adami 2012-07-13 9:44 ` Andrei Gherzan 2012-07-13 9:54 ` Andrea Adami 2012-07-13 9:59 ` Andrei Gherzan 2012-07-17 16:27 ` Saul Wold
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.