[RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)

* [RFC PATCH] populate_sdk_base: add support for zstd compression (and default to it)
@ 2021-10-15 17:06 Alexander Kanavin
  2021-10-15 17:29 ` [OE-core] " Richard Purdie
  0 siblings, 1 reply; 7+ messages in thread
From: Alexander Kanavin @ 2021-10-15 17:06 UTC (permalink / raw)
  To: openembedded-core; +Cc: Alexander Kanavin

This allows significantly faster do_populate_sdk() (5 min -> 3 min for
the sato sdk on my machine) and decompression, at the cost of SDK file
taking up more disk space.

If that's a significant concern, it's possible to increase the zstd
compression level or even go back to xz compression.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes/populate_sdk_base.bbclass | 10 ++++++++--
 meta/files/toolchain-shar-extract.sh   |  9 ++++-----
 2 files changed, 12 insertions(+), 7 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 49e166e697..92a23e2325 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -50,17 +50,23 @@ TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"
 
 # Default archived SDK's suffix
-SDK_ARCHIVE_TYPE ?= "tar.xz"
+SDK_ARCHIVE_TYPE ?= "tar.zst"
 SDK_XZ_COMPRESSION_LEVEL ?= "-9"
 SDK_XZ_OPTIONS ?= "${XZ_DEFAULTS} ${SDK_XZ_COMPRESSION_LEVEL}"
 
-# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip and tar.xz
+SDK_ZSTD_COMPRESSION_LEVEL ?= ""
+SDK_ZSTD_OPTIONS ?= "-T${ZSTD_THREADS} ${SDK_ZSTD_COMPRESSION_LEVEL}"
+
+# To support different sdk type according to SDK_ARCHIVE_TYPE, now support zip, tar.zst and tar.xz
 python () {
     if d.getVar('SDK_ARCHIVE_TYPE') == 'zip':
        d.setVar('SDK_ARCHIVE_DEPENDS', 'zip-native')
        # SDK_ARCHIVE_CMD used to generate archived sdk ${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} from input dir ${SDK_OUTPUT}/${SDKPATH} to output dir ${SDKDEPLOYDIR}
        # recommand to cd into input dir first to avoid archive with buildpath
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; zip -r -y ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE} .')
+    elif d.getVar('SDK_ARCHIVE_TYPE') == 'tar.zst':
+       d.setVar('SDK_ARCHIVE_DEPENDS', 'zstd-native')
+       d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | zstd ${SDK_ZSTD_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
     else:
        d.setVar('SDK_ARCHIVE_DEPENDS', 'xz-native')
        d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}')
diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 4386b985bb..4b8892af79 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -51,11 +51,6 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
 	fi
 fi
 
-if ! xz -V > /dev/null 2>&1; then
-	echo "Error: xz is required for installation of this SDK, please install it first"
-	exit 1
-fi
-
 SDK_BUILD_PATH="@SDKPATH@"
 DEFAULT_INSTALL_DIR="@SDKPATHINSTALL@"
 SUDO_EXEC=""
@@ -121,6 +116,8 @@ if [ "$listcontents" = "1" ] ; then
         else
             rm sdk.zip && exit 1
         fi
+    elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then
+        tail -n +$payload_offset "$0"| tar tv --zstd || exit 1
     else
         tail -n +$payload_offset "$0"| tar tvJ || exit 1
     fi
@@ -249,6 +246,8 @@ if [ @SDK_ARCHIVE_TYPE@ = "zip" ]; then
     else
         rm sdk.zip && exit 1
     fi
+elif [ @SDK_ARCHIVE_TYPE@ = "tar.zst" ]; then
+    tail -n +$payload_offset "$0"| $SUDO_EXEC tar mx -I "pzstd -p 8" -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 else
     tail -n +$payload_offset "$0"| $SUDO_EXEC tar mxJ -C $target_sdk_dir --checkpoint=.2500 $EXTRA_TAR_OPTIONS || exit 1
 fi
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread