[isar-cip-core][RFC v1 0/6] Integrate Delta Update with rdiff_image and delta handler

[isar-cip-core][RFC v1 0/6] Integrate Delta Update with rdiff_image and delta handler

[Adithya Balakumar]

These series of patches enable the support for delta update for the root file system 
with swupdate's rdiff_image and delta handler(zchunk). For now, I send this as an RFC, 
as this needs to be reviewed by a broader audience.
 
A couple notes about the implementation:
- In this patchset, Delta update is only supported for the root file system.
- The newly added delta-update.bbclass handles the creation of the delta update artifacts.
- The update handler type i.e rdiff_image or delta handler is set in the variable "DELTA_UPDATE_TYPE" 
in the newly added delta-udpate.yml file along some variables required for each handler.
- The creation of delta artifact for the rdiff_image handler requires a reference image 
(against which the delta is computed). Currently the reference image (.wic) file is passed to the 
build system by path to the file from the project's root directory in the variable DELTA_RDIFF_REF_IMAGE 
in the delta-udpate.yml file.

Adithya Balakumar (6):
  conf: Use bookworm backports for enabling delta handler in swupdate
  swupdate-handler-roundrobin: Increase revision for Delta Handler
    support
  swupdate: Enable Delta handler in swupdate
  Add Delta update support with rdiff_image and delta handler
  swupdate.bbclass: Generate swu for delta updates
  doc/README.swupdate.md: Update steps to test Delta software Update

 classes/delta_update.bbclass                  | 90 ++++++++++++++++++
 classes/swupdate.bbclass                      | 35 ++++++-
 conf/distro/cip-core-bookworm.conf            |  3 +
 conf/distro/debian-bookworm-backports.list    |  1 +
 ...eferences.swupdate-bookworm-backports.conf | 12 +++
 doc/README.swupdate.md                        | 95 +++++++++++++++++++
 kas/opt/delta-update.yml                      | 21 ++++
 .../swupdate-handler-roundrobin_0.1.bb        |  2 +-
 recipes-core/swupdate/swupdate_2023.05.bb     |  3 +
 9 files changed, 259 insertions(+), 3 deletions(-)
 create mode 100644 classes/delta_update.bbclass
 create mode 100644 conf/distro/debian-bookworm-backports.list
 create mode 100644 conf/distro/preferences.swupdate-bookworm-backports.conf
 create mode 100644 kas/opt/delta-update.yml

-- 
2.39.2

[isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[Adithya Balakumar]

Delta handler requires zchunk > 1.3, but the required version is only
available in bookworm backports

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 conf/distro/cip-core-bookworm.conf                   |  3 +++
 conf/distro/debian-bookworm-backports.list           |  1 +
 .../preferences.swupdate-bookworm-backports.conf     | 12 ++++++++++++
 3 files changed, 16 insertions(+)
 create mode 100644 conf/distro/debian-bookworm-backports.list
 create mode 100644 conf/distro/preferences.swupdate-bookworm-backports.conf

diff --git a/conf/distro/cip-core-bookworm.conf b/conf/distro/cip-core-bookworm.conf
index 9bcf559..e0a4f7f 100644
--- a/conf/distro/cip-core-bookworm.conf
+++ b/conf/distro/cip-core-bookworm.conf
@@ -12,6 +12,9 @@
 require conf/distro/debian-bookworm.conf
 require cip-core-common.inc
 
+DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-backports.list"
+DISTRO_APT_PREFERENCES:append:swupdate = " conf/distro/preferences.swupdate-bookworm-backports.conf"
+
 PREFERRED_VERSION_swupdate ?= "2023.05%"
 PREFERRED_VERSION_linux-cip ?= "6.1.%"
 PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
diff --git a/conf/distro/debian-bookworm-backports.list b/conf/distro/debian-bookworm-backports.list
new file mode 100644
index 0000000..c4873b4
--- /dev/null
+++ b/conf/distro/debian-bookworm-backports.list
@@ -0,0 +1 @@
+deb http://deb.debian.org/debian bookworm-backports main contrib non-free
diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf b/conf/distro/preferences.swupdate-bookworm-backports.conf
new file mode 100644
index 0000000..b6efffc
--- /dev/null
+++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
@@ -0,0 +1,12 @@
+Package: libzck-dev
+Pin: release n=bookworm-backports
+Pin-Priority: 801
+
+Package: libzck1
+Pin: release n=bookworm-backports
+Pin-Priority: 801
+
+Package: zchunk
+Pin: release n=bookworm-backports
+Pin-Priority: 801
+
-- 
2.39.2

[isar-cip-core][RFC v1 2/6] swupdate-handler-roundrobin: Increase revision for Delta Handler support

[Adithya Balakumar]

This revision allows the use of delta handler in swupdate for zchunk based
delta updates

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 .../swupdate-handler-roundrobin_0.1.bb                          | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb b/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
index b39278e..41eb06c 100644
--- a/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
+++ b/recipes-core/swupdate-handler-roundrobin/swupdate-handler-roundrobin_0.1.bb
@@ -13,7 +13,7 @@ inherit dpkg-raw
 PROVIDES = "swupdate-handlers"
 
 SRC_URI += "git://gitlab.com/cip-project/cip-sw-updates/swupdate-handler-roundrobin.git;protocol=https;destsuffix=swupdate-handler-roundrobin;name=swupdate-handler-roundrobin;nobranch=1"
-SRCREV_swupdate-handler-roundrobin ?= "bb35127231ec08a67f79a7584ccfc0cada88cc4e"
+SRCREV_swupdate-handler-roundrobin ?= "e6936b8c29a20cb6a6893faae57e12499ebbc342"
 
 SWUPDATE_LUASCRIPT = "swupdate-handler-roundrobin/swupdate_handlers_roundrobin.lua"
 
-- 
2.39.2

[isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

[Adithya Balakumar]

Enable delta handler in swupdate for bookworm and disable cross compile
for arm and arm64 on bookworm to avoid build issues

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 recipes-core/swupdate/swupdate_2023.05.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/recipes-core/swupdate/swupdate_2023.05.bb b/recipes-core/swupdate/swupdate_2023.05.bb
index d1fe1fa..7a22947 100644
--- a/recipes-core/swupdate/swupdate_2023.05.bb
+++ b/recipes-core/swupdate/swupdate_2023.05.bb
@@ -36,10 +36,13 @@ SRC_URI:append:bullseye = " file://0004-d-swupdate-www.install-Fix-path-for-debi
 DEB_BUILD_PROFILES += "pkg.swupdate.suricattalua"
 # add suricatta wfx
 DEB_BUILD_PROFILES += "pkg.swupdate.suricattawfx"
+# add delta update build profile for bookworm
+DEB_BUILD_PROFILES:append:bookworm = " pkg.swupdate.delta"
 
 # Disable cross for arm and arm64 on bullseye
 # with cross compile we have a unsat-dependency to dh-nodejs on arm/arm64
 ISAR_CROSS_COMPILE:bullseye = "0"
+ISAR_CROSS_COMPILE:bookworm = "0"
 
 # use backport build profile for bullseye
 DEB_BUILD_PROFILES:append:bullseye = " pkg.swupdate.bpo"
-- 
2.39.2

[isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Adithya Balakumar]

swupdate supports delta updates with rdiff_image and delta(zchunk)
handler. This change adds support to use either of the handler
for creating delta update artifacts. The type of handler can be set
in the delta-update.yml file along with additional variables needed
for the chosen handler.

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 classes/delta_update.bbclass | 90 ++++++++++++++++++++++++++++++++++++
 kas/opt/delta-update.yml     | 21 +++++++++
 2 files changed, 111 insertions(+)
 create mode 100644 classes/delta_update.bbclass
 create mode 100644 kas/opt/delta-update.yml

diff --git a/classes/delta_update.bbclass b/classes/delta_update.bbclass
new file mode 100644
index 0000000..8d47081
--- /dev/null
+++ b/classes/delta_update.bbclass
@@ -0,0 +1,90 @@
+#
+# CIP Core, generic profile
+#
+# Copyright (c) Toshiba Corporation 2024
+#
+# Authors:
+#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
+
+FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
+IMAGE_SRC_URI:delta_update += " ${@ "file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
+
+do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
+
+extract_root_fs() {
+    INPUT_ARTIFACT=$1
+    OUTPUT_ARTIFACT=$2
+    SECTOR_SIZE=512
+    ROOTFS_LABEL="primary"
+    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ name {print $2}' | head -n 1)
+    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE) / $(expr 1024 \* 1024))"
+    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
+}
+
+do_delta_update_sw_description_template() {
+   # Since the sw-description file for delta update would not be very different from that of a 
+   # normal update, copy sw-description.tmpl as sw-description-delta.tmpl and add delta update 
+   # specific properties
+   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
+   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
+   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
+   sed -i '/configfilecheck =/a \                      \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
+}
+addtask do_delta_update_sw_description_template before do_transform_template after do_generate_image_uuid
+
+IMAGE_CMD:delta_update() {
+    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [ "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
+        bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
+    fi
+    
+    DELTA_ARTIFACT_SWU=""
+
+    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
+	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
+
+    	# Create .zck file
+    	${SUDO_CHROOT} /bin/zck \
+		--output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
+		-u --chunk-hash-type sha256 \
+		${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw
+
+    	# Calculate size of zck header
+    	HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
+    
+    	# Extract the zck header
+    	${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" status=none
+
+        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
+
+    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
+        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
+            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
+        fi
+
+	# Extracting root partition from image1
+	extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"  "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
+	
+	# Extracting root partition from image2
+	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"  "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"	
+
+	rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
+	# create signature file with rdiff
+	${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
+		${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
+
+        # create delta file with the signature file
+	${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
+		${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
+
+	DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
+    fi
+        # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
+        ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
+}
+
+addtask do_image_delta_update before do_image_swu after do_image_wic
diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
new file mode 100644
index 0000000..c7a3893
--- /dev/null
+++ b/kas/opt/delta-update.yml
@@ -0,0 +1,21 @@
+#
+# CIP Core, generic profile
+#
+# Copyright (c) Toshiba Corporation 2024
+#
+# Authors:
+#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+header:
+  version: 14
+
+local_conf_header:
+  delta-update: |
+    IMAGE_CLASSES:append = " delta_update"
+    IMAGE_FSTYPES:append = " delta_update"
+    DELTA_UPDATE_TYPE = ""
+    DELTA_ZCK_URL = ""
+    DELTA_RDIFF_REF_IMAGE = "" 
-- 
2.39.2

[isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya Balakumar]

This allows the creation of a swu file for delta update. The swu file
for delta update is only created when delta-update.yml file is included
at build time.

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
index 2c69892..2964024 100644
--- a/classes/swupdate.bbclass
+++ b/classes/swupdate.bbclass
@@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
 SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
 SWU_BOOTLOADER ??= "ebg"
 SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-${SWU_BOOTLOADER}"
+SWU_DESCRIPITION_FILE_DELTA_UPDATE ??= "${SWU_DESCRIPTION_FILE}-delta"
+SWU_DELTA_UPDATE_ARTIFACT = "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
+SWU_DELTA_UPDATE_ARTIFACT_TYPE = "delta_update${@get_swu_compression_type(d)}"
 
 SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
 SWU_DESCRIPTION_FILE ?= "sw-description"
-SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
+SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME} ${SWU_DELTA_UPDATE_ARTIFACT}"
 SWU_SIGNED ??= ""
 SWU_SIGNATURE_EXT ?= "sig"
 SWU_SIGNATURE_TYPE ?= "cms"
@@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
 SWU_BUILDCHROOT_IMAGE_FILE ?= "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
 
 IMAGE_TYPEDEP:swu = "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
+IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if d.getVar('DELTA_UPDATE_TYPE') else ''}"
 IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
 IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
 IMAGE_INSTALL += "${@'swupdate-certificates' if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
@@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = "file://${SWU_DESCRIPTION_FILE}.tmpl"
 IMAGE_SRC_URI:swu += "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
 IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
 IMAGE_TEMPLATE_FILES:swu += "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
+IMAGE_TEMPLATE_FILES:delta_update += "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
 IMAGE_TEMPLATE_VARS:swu = " \
     SWU_ROOTFS_PARTITION_NAME \
     TARGET_IMAGE_UUID \
@@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
     SWU_FILE_NODES \
     SWU_BOOTLOADER_FILE_NODE \
     SWU_SCRIPTS_NODE \
+    SWU_DELTA_UPDATE_ARTIFACT \
+    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
     "
 
 # TARGET_IMAGE_UUID needs to be generated before completing the template
@@ -136,6 +143,22 @@ python add_scripts(){
     d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
 }
 
+SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
+python add_swu_delta_update_properties() {
+    delta_type = d.getVar('DELTA_UPDATE_TYPE')
+    swu_delta_update_properties = ""
+    if delta_type == "rdiff":
+        swu_delta_update_properties =  'chainhandler = "rdiff_image";'
+    elif delta_type == "zchunk":
+        zck_url = d.getVar('DELTA_ZCK_URL')
+        swu_delta_update_properties = f"""
+                        chainhandler = "delta";
+                        url = "{zck_url}";
+                        zckloglevel = "error";
+        """
+    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES', swu_delta_update_properties)
+}
+
 # convert between swupdate compressor name and imagetype extension
 def get_swu_compression_type(d):
     swu_ct = d.getVar('SWU_COMPRESSION_TYPE')
@@ -154,13 +177,21 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}/recipes-core/images/swu"
 
 do_image_swu[depends] += "${PN}:do_transform_template"
 do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
-do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-${SWU_BOOTLOADER}"
+do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
 IMAGE_CMD:swu() {
     rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
+    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
+        rm -rf ${WORKDIR}/swu-delta
+    fi
+
     cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}' '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
     if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
         cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
     fi
+    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ]; then
+        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
+    fi
+
 
     for swu_file in "${WORKDIR}"/swu*; do
         swu_file_base=$(basename $swu_file)
-- 
2.39.2

[isar-cip-core][RFC v1 6/6] doc/README.swupdate.md: Update steps to test Delta software Update

[Adithya Balakumar]

This change includes steps to verify Delta Software Update with
rdiff_image and delta handler

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 doc/README.swupdate.md | 95 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)

diff --git a/doc/README.swupdate.md b/doc/README.swupdate.md
index cf1bcfb..4003243 100644
--- a/doc/README.swupdate.md
+++ b/doc/README.swupdate.md
@@ -324,6 +324,101 @@ user variables:
 
 
 ```
+# Building and testing the CIP Core image for Delta Software Update
+
+Set up `kas-container` as described in the [top-level README](../README.md), and then proceed with the following steps.
+
+First build an image using the following command:
+```
+host$ ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/rt.yml
+```
+Currently, Delta Software Update is supported only for root file system.
+Now, to verify a delta update, build an image with some modifications to the root file system (ex: Add additional packages).
+For this example, add the packages vim and nano (since the packages are not already included in the base image) to the image. This can be done by modifying the `recipes-core/images/cip-core-image.bb` file as shown below.
+```
+diff --git a/recipes-core/images/cip-core-image.bb b/recipes-core/images/cip-core-image.bb
+index 0ec7220..f61ce23 100644
+--- a/recipes-core/images/cip-core-image.bb
++++ b/recipes-core/images/cip-core-image.bb
+@@ -14,6 +14,7 @@ inherit image
+ ISAR_RELEASE_CMD = "git -C ${LAYERDIR_cip-core} describe --tags --dirty --always --match 'v[0-9].[0-9]*'"
+ DESCRIPTION = "CIP Core image"
+
++IMAGE_PREINSTALL += "vim nano"
+ IMAGE_INSTALL += "customizations"
+
+ CIP_IMAGE_OPTIONS ?= ""
+```
+## Delta Software Update using rdiff_image handler
+
+Creating a delta update file for rdiff_image handler requires a reference image (against which the delta is computed). In this case, the image built in the previous section can be used as the reference image.
+Copy the reference image to the project directory. And set the variable `DELTA_RDIFF_REF_IMAGE` with the path to the reference image and set the variable `DELTA_UPDATE_TYPE` to `rdiff`.
+
+```
+header:
+  version: 14
+
+local_conf_header:
+  delta-update: |
+    IMAGE_CLASSES:append = " delta_update"
+    DELTA_UPDATE_TYPE = "rdiff"
+    DELTA_ZCK_URL = ""
+    DELTA_RDIFF_REF_IMAGE = "path/to/image/from/project/root/directory.wic"
+
+```
+Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
+```
+KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
+```
+Run the below command to start first image
+```
+host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
+```
+Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
+```
+host$ cd image2/tmp/deploy/images/qemu-amd64/
+host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
+```
+
+### Delta Software Update Verification
+
+Follow the steps mentioned in the section [SWUpdate verification](#swupdate-verification) for verification.
+
+Since our target was to update the root file system by adding a few packages (vim and nano in this example), Check if the packages are available after the update by running either the `vim` command or `nano` command.
+
+## Delta Software Update using delta handler
+
+For Delta update with zchunk, set the variable `DELTA_ZCK_URL` with the URL of the zck file that is hosted in a http server and set the variable `DELTA_UPDATE_TYPE` to `zchunk`.
+```
+header:
+  version: 14
+
+local_conf_header:
+  delta-update: |
+    IMAGE_CLASSES:append = " delta_update"
+    DELTA_UPDATE_TYPE = "zchunk"
+    DELTA_ZCK_URL = "<zck file url>"
+    DELTA_RDIFF_REF_IMAGE = ""
+
+```
+Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
+```
+KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
+```
+Run the below command to start first image
+```
+host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
+```
+Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
+```
+host$ cd image2/tmp/deploy/images/qemu-amd64/
+host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
+```
+The `cip-core-image-cip-core-bookworm-qemu-amd64.zck` file must be hosted in a server.
+
+### Delta Software Update Verification
+
+Follow the steps mentioned in the section [Delta Software Update Verification](#delta-software-update-verification) for verification.
 
 # Building and testing the CIP Core image for BBB
 
-- 
2.39.2

Re: [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[MOESSBAUER, Felix]

On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> Delta handler requires zchunk > 1.3, but the required version is only
> available in bookworm backports
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  conf/distro/cip-core-bookworm.conf                   |  3 +++
>  conf/distro/debian-bookworm-backports.list           |  1 +
>  .../preferences.swupdate-bookworm-backports.conf     | 12
> ++++++++++++
>  3 files changed, 16 insertions(+)
>  create mode 100644 conf/distro/debian-bookworm-backports.list
>  create mode 100644 conf/distro/preferences.swupdate-bookworm-
> backports.conf
> 
> diff --git a/conf/distro/cip-core-bookworm.conf b/conf/distro/cip-
> core-bookworm.conf
> index 9bcf559..e0a4f7f 100644
> --- a/conf/distro/cip-core-bookworm.conf
> +++ b/conf/distro/cip-core-bookworm.conf
> @@ -12,6 +12,9 @@
>  require conf/distro/debian-bookworm.conf
>  require cip-core-common.inc
>  
> +DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-
> backports.list"
> +DISTRO_APT_PREFERENCES:append:swupdate = "
> conf/distro/preferences.swupdate-bookworm-backports.conf"
> +
>  PREFERRED_VERSION_swupdate ?= "2023.05%"
>  PREFERRED_VERSION_linux-cip ?= "6.1.%"
>  PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
> diff --git a/conf/distro/debian-bookworm-backports.list
> b/conf/distro/debian-bookworm-backports.list
> new file mode 100644
> index 0000000..c4873b4
> --- /dev/null
> +++ b/conf/distro/debian-bookworm-backports.list
> @@ -0,0 +1 @@
> +deb http://deb.debian.org/debian bookworm-backports main contrib
> non-free
> diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf
> b/conf/distro/preferences.swupdate-bookworm-backports.conf
> new file mode 100644
> index 0000000..b6efffc
> --- /dev/null
> +++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
> @@ -0,0 +1,12 @@
> +Package: libzck-dev
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801
> +
> +Package: libzck1
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801

Hi!

Just put all packages in one line, e.g.

Package: libzck-dev libzck1 zchunk

Felix

> +
> +Package: zchunk
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801
> +

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[MOESSBAUER, Felix]

On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk)
> handler. This change adds support to use either of the handler
> for creating delta update artifacts. The type of handler can be set
> in the delta-update.yml file along with additional variables needed
> for the chosen handler.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/delta_update.bbclass | 90

Hi,

please name this class delta-update.bbclass to keep the naming style
similar with the rest of the project.

> ++++++++++++++++++++++++++++++++++++
>  kas/opt/delta-update.yml     | 21 +++++++++
>  2 files changed, 111 insertions(+)
>  create mode 100644 classes/delta_update.bbclass
>  create mode 100644 kas/opt/delta-update.yml
> 
> diff --git a/classes/delta_update.bbclass
> b/classes/delta_update.bbclass
> new file mode 100644
> index 0000000..8d47081
> --- /dev/null
> +++ b/classes/delta_update.bbclass
> @@ -0,0 +1,90 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"

Conditionally appending with += overwrites already set weak defaults.
You might want to use .= " fdisk zchunk rdiff" instead.

> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +IMAGE_SRC_URI:delta_update += " ${@ "

Same here.

> file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if

I guess this could be simplified and made more portable by referencing
the DELTA_RDIFF_REF_IMAGE directly (e.g. from ${DEPLOY_DIR_IMAGE}). The
problem with the FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}" is
that it only works if the delta image is also from cip-core, but
usually cip-core is just referenced in a downstream layer and used
there.


> d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and
> d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
> +
> +do_image_delta_update[cleandirs] +=
> "${WORKDIR}/delta_interim_artifacts"
> +
> +extract_root_fs() {
> +    INPUT_ARTIFACT=$1
> +    OUTPUT_ARTIFACT=$2
> +    SECTOR_SIZE=512
> +    ROOTFS_LABEL="primary"
> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o

Can't we simply use sfdisk to query this information instead of all the
awk stuff?

> Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~
> name {print $2}' | head -n 1)
> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE)
> / $(expr 1024 \* 1024))"
> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M
> skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
> +}

Why copy exactly 1GB?

Instead of extracting that here, it might be simpler to just store the
.rdiff file (of the rootfs) with each image build and use that.

> +
> +do_delta_update_sw_description_template() {
> +   # Since the sw-description file for delta update would not be
> very different from that of a 
> +   # normal update, copy sw-description.tmpl as sw-description-
> delta.tmpl and add delta update 
> +   # specific properties
> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl
> "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g'
> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +   sed -i '/configfilecheck =/a \                     
> \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}'
> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +}

@Quirin: IIRC you recently refactored the creation of the sw-
description file. This might simplify things here.

> +addtask do_delta_update_sw_description_template before
> do_transform_template after do_generate_image_uuid
> +
> +IMAGE_CMD:delta_update() {
> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [
> "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE
> (rdiff/zchunk)"
> +    fi
> +    
> +    DELTA_ARTIFACT_SWU=""
> +
> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
> "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
> +
> +       # Create .zck file
> +       ${SUDO_CHROOT} /bin/zck \
> +               --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> +               -u --chunk-hash-type sha256 \
> +               ${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.
> raw
> +
> +       # Calculate size of zck header
> +       HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v
> ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':'
> -f 2)"
> +    
> +       # Extract the zck header
> +       ${SUDO_CHROOT} /bin/dd
> if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
> of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE"
> status=none
> +
> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide
> the required files as artifacts to this recipe"
> +        fi
> +
> +       # Extracting root partition from image1
> +       extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}" 
> "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"

We only need that to compute the rdiff signature, right?
As these images might be quite big, better remove the old one before
extracting the new one. By that we save ~half of temporary space.

> +       
> +       # Extracting root partition from image2
> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" 
> "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw" 
> +
> +       rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> +       # create signature file with rdiff
> +       ${SUDO_CHROOT} /usr/bin/rdiff signature
> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
> +               ${WORKDIR}/delta_interim_artifacts/old-image-
> rootfs.sig
> +
> +        # create delta file with the signature file
> +       ${SUDO_CHROOT} /usr/bin/rdiff delta
> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> +               ${WORKDIR}/delta_interim_artifacts/new-image-
> rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> +       DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +    fi
> +        # create a symbolic link as IMAGE_CMD expects a
> *.delta_update file in deploy image directory
> +        ln -sf ${DELTA_ARTIFACT_SWU}
> ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
> new file mode 100644
> index 0000000..c7a3893
> --- /dev/null
> +++ b/kas/opt/delta-update.yml
> @@ -0,0 +1,21 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    IMAGE_FSTYPES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = ""
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "" 

Please also extra-weak define the DELTA_* variables in the
delta_update.bbclass (to always have them defined and for documentation
reasons).

Best regards,
Felix

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[MOESSBAUER, Felix]

On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> This allows the creation of a swu file for delta update. The swu file
> for delta update is only created when delta-update.yml file is
> included
> at build time.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
> 
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
> index 2c69892..2964024 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>  SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>  SWU_BOOTLOADER ??= "ebg"
>  SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-
> ${SWU_BOOTLOADER}"
> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=

Hi, do we really need a dedicated file / variable for that?

> "${SWU_DESCRIPTION_FILE}-delta"
> +SWU_DELTA_UPDATE_ARTIFACT =
> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
> "delta_update${@get_swu_compression_type(d)}"
>  
>  SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>  SWU_DESCRIPTION_FILE ?= "sw-description"
> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
> ${SWU_DELTA_UPDATE_ARTIFACT}"

Does this mean the delta artifact is always added here?

>  SWU_SIGNED ??= ""
>  SWU_SIGNATURE_EXT ?= "sig"
>  SWU_SIGNATURE_TYPE ?= "cms"
> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>  SWU_BUILDCHROOT_IMAGE_FILE ?=
> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>  
>  IMAGE_TYPEDEP:swu =
> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>  IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>  IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>  IMAGE_INSTALL += "${@'swupdate-certificates' if
> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>  IMAGE_SRC_URI:swu +=
> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>  IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>  IMAGE_TEMPLATE_FILES:swu +=
> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
> +IMAGE_TEMPLATE_FILES:delta_update +=
> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>  IMAGE_TEMPLATE_VARS:swu = " \
>      SWU_ROOTFS_PARTITION_NAME \
>      TARGET_IMAGE_UUID \
> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>      SWU_FILE_NODES \
>      SWU_BOOTLOADER_FILE_NODE \
>      SWU_SCRIPTS_NODE \
> +    SWU_DELTA_UPDATE_ARTIFACT \
> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>      "
>  
>  # TARGET_IMAGE_UUID needs to be generated before completing the
> template
> @@ -136,6 +143,22 @@ python add_scripts(){
>      d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>  }
>  
> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
> +python add_swu_delta_update_properties() {
> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
> +    swu_delta_update_properties = ""
> +    if delta_type == "rdiff":
> +        swu_delta_update_properties =  'chainhandler =
> "rdiff_image";'
> +    elif delta_type == "zchunk":
> +        zck_url = d.getVar('DELTA_ZCK_URL')
> +        swu_delta_update_properties = f"""
> +                        chainhandler = "delta";
> +                        url = "{zck_url}";
> +                        zckloglevel = "error";
> +        """
> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
> swu_delta_update_properties)
> +}
> +
>  # convert between swupdate compressor name and imagetype extension
>  def get_swu_compression_type(d):
>      swu_ct = d.getVar('SWU_COMPRESSION_TYPE')
> @@ -154,13 +177,21 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
> core}/recipes-core/images/swu"
>  
>  do_image_swu[depends] += "${PN}:do_transform_template"
>  do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
> ${SWU_BOOTLOADER}"
> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>  IMAGE_CMD:swu() {
>      rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
> +        rm -rf ${WORKDIR}/swu-delta

Why is this required if we already clear the ${WORKDIR}/swu-delta in
cleandirs?

Felix

> 
> +    fi
> +
>      cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>      if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
>          cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>      fi
> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
> then
> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
> +    fi
> +
>  
>      for swu_file in "${WORKDIR}"/swu*; do
>          swu_file_base=$(basename $swu_file)

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[Jan Kiszka]

On 07.03.24 08:46, Adithya Balakumar wrote:
> Delta handler requires zchunk > 1.3, but the required version is only
> available in bookworm backports
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  conf/distro/cip-core-bookworm.conf                   |  3 +++
>  conf/distro/debian-bookworm-backports.list           |  1 +
>  .../preferences.swupdate-bookworm-backports.conf     | 12 ++++++++++++
>  3 files changed, 16 insertions(+)
>  create mode 100644 conf/distro/debian-bookworm-backports.list
>  create mode 100644 conf/distro/preferences.swupdate-bookworm-backports.conf
> 
> diff --git a/conf/distro/cip-core-bookworm.conf b/conf/distro/cip-core-bookworm.conf
> index 9bcf559..e0a4f7f 100644
> --- a/conf/distro/cip-core-bookworm.conf
> +++ b/conf/distro/cip-core-bookworm.conf
> @@ -12,6 +12,9 @@
>  require conf/distro/debian-bookworm.conf
>  require cip-core-common.inc
>  
> +DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-backports.list"
> +DISTRO_APT_PREFERENCES:append:swupdate = " conf/distro/preferences.swupdate-bookworm-backports.conf"
> +
>  PREFERRED_VERSION_swupdate ?= "2023.05%"
>  PREFERRED_VERSION_linux-cip ?= "6.1.%"
>  PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
> diff --git a/conf/distro/debian-bookworm-backports.list b/conf/distro/debian-bookworm-backports.list
> new file mode 100644
> index 0000000..c4873b4
> --- /dev/null
> +++ b/conf/distro/debian-bookworm-backports.list
> @@ -0,0 +1 @@
> +deb http://deb.debian.org/debian bookworm-backports main contrib non-free
> diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf b/conf/distro/preferences.swupdate-bookworm-backports.conf
> new file mode 100644
> index 0000000..b6efffc
> --- /dev/null
> +++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
> @@ -0,0 +1,12 @@
> +Package: libzck-dev
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801
> +
> +Package: libzck1
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801
> +
> +Package: zchunk
> +Pin: release n=bookworm-backports
> +Pin-Priority: 801
> +

Backports are bad when it comes to use in production (they receive
best-effort updates only). It this the only options?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

[Jan Kiszka]

On 07.03.24 08:46, Adithya Balakumar wrote:
> Enable delta handler in swupdate for bookworm and disable cross compile
> for arm and arm64 on bookworm to avoid build issues
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  recipes-core/swupdate/swupdate_2023.05.bb | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/recipes-core/swupdate/swupdate_2023.05.bb b/recipes-core/swupdate/swupdate_2023.05.bb
> index d1fe1fa..7a22947 100644
> --- a/recipes-core/swupdate/swupdate_2023.05.bb
> +++ b/recipes-core/swupdate/swupdate_2023.05.bb
> @@ -36,10 +36,13 @@ SRC_URI:append:bullseye = " file://0004-d-swupdate-www.install-Fix-path-for-debi
>  DEB_BUILD_PROFILES += "pkg.swupdate.suricattalua"
>  # add suricatta wfx
>  DEB_BUILD_PROFILES += "pkg.swupdate.suricattawfx"
> +# add delta update build profile for bookworm
> +DEB_BUILD_PROFILES:append:bookworm = " pkg.swupdate.delta"
>  

What about sid?

>  # Disable cross for arm and arm64 on bullseye
>  # with cross compile we have a unsat-dependency to dh-nodejs on arm/arm64
>  ISAR_CROSS_COMPILE:bullseye = "0"
> +ISAR_CROSS_COMPILE:bookworm = "0"

This is unfortunate. Why that?

>  
>  # use backport build profile for bullseye
>  DEB_BUILD_PROFILES:append:bullseye = " pkg.swupdate.bpo"

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Jan Kiszka]

On 07.03.24 08:46, Adithya Balakumar wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk)
> handler. This change adds support to use either of the handler
> for creating delta update artifacts. The type of handler can be set
> in the delta-update.yml file along with additional variables needed
> for the chosen handler.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/delta_update.bbclass | 90 ++++++++++++++++++++++++++++++++++++
>  kas/opt/delta-update.yml     | 21 +++++++++
>  2 files changed, 111 insertions(+)
>  create mode 100644 classes/delta_update.bbclass
>  create mode 100644 kas/opt/delta-update.yml
> 
> diff --git a/classes/delta_update.bbclass b/classes/delta_update.bbclass
> new file mode 100644
> index 0000000..8d47081
> --- /dev/null
> +++ b/classes/delta_update.bbclass
> @@ -0,0 +1,90 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +IMAGE_SRC_URI:delta_update += " ${@ "file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
> +
> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
> +
> +extract_root_fs() {
> +    INPUT_ARTIFACT=$1
> +    OUTPUT_ARTIFACT=$2
> +    SECTOR_SIZE=512
> +    ROOTFS_LABEL="primary"
> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ name {print $2}' | head -n 1)
> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE) / $(expr 1024 \* 1024))"
> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
> +}
> +
> +do_delta_update_sw_description_template() {
> +   # Since the sw-description file for delta update would not be very different from that of a 
> +   # normal update, copy sw-description.tmpl as sw-description-delta.tmpl and add delta update 
> +   # specific properties
> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +   sed -i '/configfilecheck =/a \                      \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +}
> +addtask do_delta_update_sw_description_template before do_transform_template after do_generate_image_uuid
> +
> +IMAGE_CMD:delta_update() {
> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [ "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
> +    fi
> +    
> +    DELTA_ARTIFACT_SWU=""
> +
> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
> +
> +    	# Create .zck file
> +    	${SUDO_CHROOT} /bin/zck \
> +		--output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> +		-u --chunk-hash-type sha256 \
> +		${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw
> +
> +    	# Calculate size of zck header
> +    	HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
> +    
> +    	# Extract the zck header
> +    	${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" status=none
> +
> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
> +        fi
> +
> +	# Extracting root partition from image1
> +	extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"  "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
> +	
> +	# Extracting root partition from image2
> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"  "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"	
> +
> +	rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> +	# create signature file with rdiff
> +	${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
> +		${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
> +
> +        # create delta file with the signature file
> +	${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> +		${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> +	DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +    fi
> +        # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> +        ln -sf ${DELTA_ARTIFACT_SWU} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
> new file mode 100644
> index 0000000..c7a3893
> --- /dev/null
> +++ b/kas/opt/delta-update.yml
> @@ -0,0 +1,21 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    IMAGE_FSTYPES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = ""
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "" 

What prevents making this the default for generating update images?
People could still add their own options or local configs if they want
to deviate. Would also simplify patch 6 further.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 6/6] doc/README.swupdate.md: Update steps to test Delta software Update

[Jan Kiszka]

On 07.03.24 08:46, Adithya Balakumar wrote:
> This change includes steps to verify Delta Software Update with
> rdiff_image and delta handler
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  doc/README.swupdate.md | 95 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 95 insertions(+)
> 
> diff --git a/doc/README.swupdate.md b/doc/README.swupdate.md
> index cf1bcfb..4003243 100644
> --- a/doc/README.swupdate.md
> +++ b/doc/README.swupdate.md
> @@ -324,6 +324,101 @@ user variables:
>  
>  
>  ```
> +# Building and testing the CIP Core image for Delta Software Update
> +
> +Set up `kas-container` as described in the [top-level README](../README.md), and then proceed with the following steps.
> +
> +First build an image using the following command:
> +```
> +host$ ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/rt.yml
> +```
> +Currently, Delta Software Update is supported only for root file system.
> +Now, to verify a delta update, build an image with some modifications to the root file system (ex: Add additional packages).
> +For this example, add the packages vim and nano (since the packages are not already included in the base image) to the image. This can be done by modifying the `recipes-core/images/cip-core-image.bb` file as shown below.
> +```
> +diff --git a/recipes-core/images/cip-core-image.bb b/recipes-core/images/cip-core-image.bb
> +index 0ec7220..f61ce23 100644
> +--- a/recipes-core/images/cip-core-image.bb
> ++++ b/recipes-core/images/cip-core-image.bb
> +@@ -14,6 +14,7 @@ inherit image
> + ISAR_RELEASE_CMD = "git -C ${LAYERDIR_cip-core} describe --tags --dirty --always --match 'v[0-9].[0-9]*'"
> + DESCRIPTION = "CIP Core image"
> +
> ++IMAGE_PREINSTALL += "vim nano"
> + IMAGE_INSTALL += "customizations"
> +
> + CIP_IMAGE_OPTIONS ?= ""
> +```

Do not duplicate the description how to generate v1 and v2 images,
compared to the existing "Building and testing the CIP Core image", that
is just confusing. Describe the delta-update specifics only, likely the
bits below.

> +## Delta Software Update using rdiff_image handler
> +
> +Creating a delta update file for rdiff_image handler requires a reference image (against which the delta is computed). In this case, the image built in the previous section can be used as the reference image.
> +Copy the reference image to the project directory. And set the variable `DELTA_RDIFF_REF_IMAGE` with the path to the reference image and set the variable `DELTA_UPDATE_TYPE` to `rdiff`.
> +
> +```
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = "rdiff"
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "path/to/image/from/project/root/directory.wic"
> +
> +```

I don't want to fiddle with such files for a first test unless I really
have to. I would like to have a default case ideally without any
editing, just by rebuilding an image differently, maybe providing v1 to
that build under a specific path/filename, and let the second build
generate the delta image automatically.

Jan

> +Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
> +```
> +Run the below command to start first image
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [SWUpdate verification](#swupdate-verification) for verification.
> +
> +Since our target was to update the root file system by adding a few packages (vim and nano in this example), Check if the packages are available after the update by running either the `vim` command or `nano` command.
> +
> +## Delta Software Update using delta handler
> +
> +For Delta update with zchunk, set the variable `DELTA_ZCK_URL` with the URL of the zck file that is hosted in a http server and set the variable `DELTA_UPDATE_TYPE` to `zchunk`.
> +```
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = "zchunk"
> +    DELTA_ZCK_URL = "<zck file url>"
> +    DELTA_RDIFF_REF_IMAGE = ""
> +
> +```
> +Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
> +```
> +Run the below command to start first image
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +The `cip-core-image-cip-core-bookworm-qemu-amd64.zck` file must be hosted in a server.
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [Delta Software Update Verification](#delta-software-update-verification) for verification.
>  
>  # Building and testing the CIP Core image for BBB
>  

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 0/6] Integrate Delta Update with rdiff_image and delta handler

[Jan Kiszka]

On 07.03.24 08:46, Adithya Balakumar wrote:
> These series of patches enable the support for delta update for the root file system 
> with swupdate's rdiff_image and delta handler(zchunk). For now, I send this as an RFC, 
> as this needs to be reviewed by a broader audience.
>  
> A couple notes about the implementation:
> - In this patchset, Delta update is only supported for the root file system.
> - The newly added delta-update.bbclass handles the creation of the delta update artifacts.
> - The update handler type i.e rdiff_image or delta handler is set in the variable "DELTA_UPDATE_TYPE" 
> in the newly added delta-udpate.yml file along some variables required for each handler.
> - The creation of delta artifact for the rdiff_image handler requires a reference image 
> (against which the delta is computed). Currently the reference image (.wic) file is passed to the 
> build system by path to the file from the project's root directory in the variable DELTA_RDIFF_REF_IMAGE 
> in the delta-udpate.yml file.
> 
> Adithya Balakumar (6):
>   conf: Use bookworm backports for enabling delta handler in swupdate
>   swupdate-handler-roundrobin: Increase revision for Delta Handler
>     support
>   swupdate: Enable Delta handler in swupdate
>   Add Delta update support with rdiff_image and delta handler
>   swupdate.bbclass: Generate swu for delta updates
>   doc/README.swupdate.md: Update steps to test Delta software Update
> 
>  classes/delta_update.bbclass                  | 90 ++++++++++++++++++
>  classes/swupdate.bbclass                      | 35 ++++++-
>  conf/distro/cip-core-bookworm.conf            |  3 +
>  conf/distro/debian-bookworm-backports.list    |  1 +
>  ...eferences.swupdate-bookworm-backports.conf | 12 +++
>  doc/README.swupdate.md                        | 95 +++++++++++++++++++
>  kas/opt/delta-update.yml                      | 21 ++++
>  .../swupdate-handler-roundrobin_0.1.bb        |  2 +-
>  recipes-core/swupdate/swupdate_2023.05.bb     |  3 +
>  9 files changed, 259 insertions(+), 3 deletions(-)
>  create mode 100644 classes/delta_update.bbclass
>  create mode 100644 conf/distro/debian-bookworm-backports.list
>  create mode 100644 conf/distro/preferences.swupdate-bookworm-backports.conf
>  create mode 100644 kas/opt/delta-update.yml
> 

Thanks a lot for sharing! This is great step forward, even if we still
have some remarks how to make things even more user friendly.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Gylstorff Quirin]

On 3/8/24 11:52 AM, Moessbauer, Felix (T CED OES-DE) wrote:
> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>> handler. This change adds support to use either of the handler
>> for creating delta update artifacts. The type of handler can be set
>> in the delta-update.yml file along with additional variables needed
>> for the chosen handler.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   classes/delta_update.bbclass | 90
> 
> Hi,
> 
> please name this class delta-update.bbclass to keep the naming style
> similar with the rest of the project.
> 
>> ++++++++++++++++++++++++++++++++++++
>>   kas/opt/delta-update.yml     | 21 +++++++++
>>   2 files changed, 111 insertions(+)
>>   create mode 100644 classes/delta_update.bbclass
>>   create mode 100644 kas/opt/delta-update.yml
>>
>> diff --git a/classes/delta_update.bbclass
>> b/classes/delta_update.bbclass
>> new file mode 100644
>> index 0000000..8d47081
>> --- /dev/null
>> +++ b/classes/delta_update.bbclass
>> @@ -0,0 +1,90 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024
>> +#
>> +# Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
> 
> Conditionally appending with += overwrites already set weak defaults.
> You might want to use .= " fdisk zchunk rdiff" instead.
> 
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +IMAGE_SRC_URI:delta_update += " ${@ "
> 
> Same here.
> 
>> file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if
> 
> I guess this could be simplified and made more portable by referencing
> the DELTA_RDIFF_REF_IMAGE directly (e.g. from ${DEPLOY_DIR_IMAGE}). The
> problem with the FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}" is
> that it only works if the delta image is also from cip-core, but
> usually cip-core is just referenced in a downstream layer and used
> there.
> 
> 
>> d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and
>> d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
>> +
>> +do_image_delta_update[cleandirs] +=
>> "${WORKDIR}/delta_interim_artifacts"
>> +
>> +extract_root_fs() {
>> +    INPUT_ARTIFACT=$1
>> +    OUTPUT_ARTIFACT=$2
>> +    SECTOR_SIZE=512
>> +    ROOTFS_LABEL="primary"
>> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o
> 
> Can't we simply use sfdisk to query this information instead of all the
> awk stuff?
> 
>> Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~
>> name {print $2}' | head -n 1)
>> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE)
>> / $(expr 1024 \* 1024))"
>> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M
>> skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
>> +}Why do we do this extraction from the WIC file. We established in the 
issue that the verity/squashfs file is working. see 
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/99#note_1798945281

> 
> Why copy exactly 1GB?
> 
> Instead of extracting that here, it might be simpler to just store the
> .rdiff file (of the rootfs) with each image build and use that.
> 
>> +
>> +do_delta_update_sw_description_template() {
>> +   # Since the sw-description file for delta update would not be
>> very different from that of a
>> +   # normal update, copy sw-description.tmpl as sw-description-
>> delta.tmpl and add delta update
>> +   # specific properties
>> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
>> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl
>> "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
>> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g'
>> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +   sed -i '/configfilecheck =/a \
>> \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}'
>> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +}
> 
> @Quirin: IIRC you recently refactored the creation of the sw-
> description file. This might simplify things here.

Please don't use sed but adapt the sw-description Variables
to allow both variants.

> 
>> +addtask do_delta_update_sw_description_template before
>> do_transform_template after do_generate_image_uuid
>> +
>> +IMAGE_CMD:delta_update() {
>> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [
>> "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
>> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE
>> (rdiff/zchunk)"
>> +    fi
>> +
>> +    DELTA_ARTIFACT_SWU=""
>> +
>> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then
use the case statement.
>> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
>> "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
>> +
>> +       # Create .zck file
>> +       ${SUDO_CHROOT} /bin/zck \
>> +               --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
>> +               -u --chunk-hash-type sha256 \
>> +               ${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.
>> raw
>> +
>> +       # Calculate size of zck header
>> +       HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v
>> ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':'
>> -f 2)"
>> +
>> +       # Extract the zck header
>> +       ${SUDO_CHROOT} /bin/dd
>> if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
>> of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE"
>> status=none
>> +
>> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header

>> +
>> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
>> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
>> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide
>> the required files as artifacts to this recipe"
>> +        fi
>> +
>> +       # Extracting root partition from image1
>> +       extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"
>> "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
> 
> We only need that to compute the rdiff signature, right?
> As these images might be quite big, better remove the old one before
> extracting the new one. By that we save ~half of temporary space.
> 
>> +
>> +       # Extracting root partition from image2
>> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
>> "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"
>> +
>> +       rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
>> +       # create signature file with rdiff
>> +       ${SUDO_CHROOT} /usr/bin/rdiff signature
>> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
>> +               ${WORKDIR}/delta_interim_artifacts/old-image-
>> rootfs.sig
>> +
>> +        # create delta file with the signature file
>> +       ${SUDO_CHROOT} /usr/bin/rdiff delta
>> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
>> +               ${WORKDIR}/delta_interim_artifacts/new-image-
>> rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
>> +
>> +       DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
Can you use seperate functions for rdiff and zchunk?
>> +    fi
>> +        # create a symbolic link as IMAGE_CMD expects a
>> *.delta_update file in deploy image directory
>> +        ln -sf ${DELTA_ARTIFACT_SWU}
>> ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
>> +}
>> +
>> +addtask do_image_delta_update before do_image_swu after do_image_wic
>> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
>> new file mode 100644
>> index 0000000..c7a3893
>> --- /dev/null
>> +++ b/kas/opt/delta-update.yml
>> @@ -0,0 +1,21 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024
>> +#
>> +# Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +header:
>> +  version: 14
>> +
>> +local_conf_header:
>> +  delta-update: |
>> +    IMAGE_CLASSES:append = " delta_update"
>> +    IMAGE_FSTYPES:append = " delta_update"
>> +    DELTA_UPDATE_TYPE = ""
>> +    DELTA_ZCK_URL = ""
>> +    DELTA_RDIFF_REF_IMAGE = ""
> 
> Please also extra-weak define the DELTA_* variables in the
> delta_update.bbclass (to always have them defined and for documentation
> reasons).
> 
> Best regards,
> Felix
> 

Re: [cip-dev] [isar-cip-core][RFC v1 6/6] doc/README.swupdate.md: Update steps to test Delta software Update

[Gylstorff Quirin]

On 3/7/24 8:45 AM, Adithya Balakumar via lists.cip-project.org wrote:
> This change includes steps to verify Delta Software Update with
> rdiff_image and delta handler
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>   doc/README.swupdate.md | 95 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 95 insertions(+)
> 
> diff --git a/doc/README.swupdate.md b/doc/README.swupdate.md
> index cf1bcfb..4003243 100644
> --- a/doc/README.swupdate.md
> +++ b/doc/README.swupdate.md
> @@ -324,6 +324,101 @@ user variables:
>   
>   
>   ```
> +# Building and testing the CIP Core image for Delta Software Update
> +
> +Set up `kas-container` as described in the [top-level README](../README.md), and then proceed with the following steps.
> +
> +First build an image using the following command:
> +```
> +host$ ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/rt.yml
> +```
> +Currently, Delta Software Update is supported only for root file system.
> +Now, to verify a delta update, build an image with some modifications to the root file system (ex: Add additional packages).
> +For this example, add the packages vim and nano (since the packages are not already included in the base image) to the image. This can be done by modifying the `recipes-core/images/cip-core-image.bb` file as shown below.
> +```
> +diff --git a/recipes-core/images/cip-core-image.bb b/recipes-core/images/cip-core-image.bb
> +index 0ec7220..f61ce23 100644
> +--- a/recipes-core/images/cip-core-image.bb
> ++++ b/recipes-core/images/cip-core-image.bb
> +@@ -14,6 +14,7 @@ inherit image
> + ISAR_RELEASE_CMD = "git -C ${LAYERDIR_cip-core} describe --tags --dirty --always --match 'v[0-9].[0-9]*'"
> + DESCRIPTION = "CIP Core image"
> +
> ++IMAGE_PREINSTALL += "vim nano"
> + IMAGE_INSTALL += "customizations"
> +
> + CIP_IMAGE_OPTIONS ?= ""
> +```
> +## Delta Software Update using rdiff_image handler
> +
> +Creating a delta update file for rdiff_image handler requires a reference image (against which the delta is computed). In this case, the image built in the previous section can be used as the reference image.
> +Copy the reference image to the project directory. And set the variable `DELTA_RDIFF_REF_IMAGE` with the path to the reference image and set the variable `DELTA_UPDATE_TYPE` to `rdiff`.
> +
> +```
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = "rdiff"
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "path/to/image/from/project/root/directory.wic"

In my opinion `DELTA_RDIFF_REF_IMAGE` should allow an url to an server. 
This would allow to setup automatic builds.

Quirin

> +
> +```
> +Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
> +```
> +Run the below command to start first image
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [SWUpdate verification](#swupdate-verification) for verification.
> +
> +Since our target was to update the root file system by adding a few packages (vim and nano in this example), Check if the packages are available after the update by running either the `vim` command or `nano` command.
> +
> +## Delta Software Update using delta handler
> +
> +For Delta update with zchunk, set the variable `DELTA_ZCK_URL` with the URL of the zck file that is hosted in a http server and set the variable `DELTA_UPDATE_TYPE` to `zchunk`.
> +```
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = "zchunk"
> +    DELTA_ZCK_URL = "<zck file url>"
> +    DELTA_RDIFF_REF_IMAGE = ""
> +
> +```
> +Run the below command to build the image with the modification as shown above and with the root file system modification as described in previous section.
> +```
> +KAS_BUILD_DIR=image2 ./kas-container build kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml:kas/opt/delta-update.yml
> +```
> +Run the below command to start first image
> +```
> +host$ DISTRO_RELEASE=bookworm SWUPDATE_BOOT=y ./start-qemu.sh amd64
> +```
> +Copy `cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu` file from `image2/tmp/deploy/images/qemu-amd64/` folder into the running system:
> +```
> +host$ cd image2/tmp/deploy/images/qemu-amd64/
> +host$ scp -P 22222 ./cip-core-image-cip-core-bookworm-qemu-amd64-delta.swu root@localhost:
> +```
> +The `cip-core-image-cip-core-bookworm-qemu-amd64.zck` file must be hosted in a server.
> +
> +### Delta Software Update Verification
> +
> +Follow the steps mentioned in the section [Delta Software Update Verification](#delta-software-update-verification) for verification.
>   
>   # Building and testing the CIP Core image for BBB
>   
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15258): https://lists.cip-project.org/g/cip-dev/message/15258
> Mute This Topic: https://lists.cip-project.org/mt/104783343/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

RE: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Adithya.Balakumar]

From: MOESSBAUER, Felix <felix.moessbauer@siemens.com> 
Sent: Friday, March 8, 2024 4:22 PM
To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; quirin.gylstorff@siemens.com; Kiszka, Jan <jan.kiszka@siemens.com>
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk) 
> handler. This change adds support to use either of the handler for 
> creating delta update artifacts. The type of handler can be set in the 
> delta-update.yml file along with additional variables needed for the 
> chosen handler.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/delta_update.bbclass | 90

>Hi,

>please name this class delta-update.bbclass to keep the naming style similar with the rest of the project.

> ++++++++++++++++++++++++++++++++++++
>  kas/opt/delta-update.yml     | 21 +++++++++
>  2 files changed, 111 insertions(+)
>  create mode 100644 classes/delta_update.bbclass
>  create mode 100644 kas/opt/delta-update.yml
> 
> diff --git a/classes/delta_update.bbclass 
> b/classes/delta_update.bbclass new file mode 100644 index 
> 0000000..8d47081
> --- /dev/null
> +++ b/classes/delta_update.bbclass
> @@ -0,0 +1,90 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"

>Conditionally appending with += overwrites already set weak defaults.
>You might want to use .= " fdisk zchunk rdiff" instead.

> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +IMAGE_SRC_URI:delta_update += " ${@ "

>Same here.

> file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if

>I guess this could be simplified and made more portable by referencing the DELTA_RDIFF_REF_IMAGE directly (e.g. from ${DEPLOY_DIR_IMAGE}). The problem with the FILESEXTRAPATHS:append = ":${LAYERDIR_cip->core}" is that it only works if the delta image is also from cip-core, but usually cip-core is just referenced in a downstream layer and used there.

[Adithya Balakumar] 
Hi Felix,

I understand the drawback of using FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}" that it only looks for the delta image placed in the isar-cip-core layer.
As an example you point to referencing the DELTA_RDIFF_REF_IMAGE directly from ${DEPLOY_DIR_IMAGE} but as per my understanding ${DEPLOY_DIR_IMAGE} points to the current build's image deploy directory. 

So, Could you please suggest a suitable way to provide the delta image in the case of rdiff.

> d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and
> d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
> +
> +do_image_delta_update[cleandirs] +=
> "${WORKDIR}/delta_interim_artifacts"
> +
> +extract_root_fs() {
> +    INPUT_ARTIFACT=$1
> +    OUTPUT_ARTIFACT=$2
> +    SECTOR_SIZE=512
> +    ROOTFS_LABEL="primary"
> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o

>Can't we simply use sfdisk to query this information instead of all the awk stuff?

> Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ 
> name {print $2}' | head -n 1)
> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE)
> / $(expr 1024 \* 1024))"
> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M
> skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
> +}

>Why copy exactly 1GB?

>Instead of extracting that here, it might be simpler to just store the .rdiff file (of the rootfs) with each image build and use that.[Adithya Balakumar] 

[Adithya Balakumar]  Here, I am copying 1GB from the wic image because that is the size of the rootfs partition. 

However, Quirin left a remark stating that the .squashfs file (or .verity file) could be used to create the delta artifact. Using the squashfs or verity file to create the delta will eliminate need for extracting the rootfs from the wic image and will simplify many things in this recipe. Would like to understand your opinion on this.

Thanks and Regards,
Adithya

> +
> +do_delta_update_sw_description_template() {
> +   # Since the sw-description file for delta update would not be
> very different from that of a
> +   # normal update, copy sw-description.tmpl as sw-description-
> delta.tmpl and add delta update
> +   # specific properties
> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl
> "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g'
> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +   sed -i '/configfilecheck =/a \
> \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}'
> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +}

>@Quirin: IIRC you recently refactored the creation of the sw- description file. This might simplify things here.

> +addtask do_delta_update_sw_description_template before
> do_transform_template after do_generate_image_uuid
> +
> +IMAGE_CMD:delta_update() {
> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [
> "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE
> (rdiff/zchunk)"
> +    fi
> +    
> +    DELTA_ARTIFACT_SWU=""
> +
> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then
> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
> "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
> +
> +       # Create .zck file
> +       ${SUDO_CHROOT} /bin/zck \
> +               --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> +               -u --chunk-hash-type sha256 \
> +               ${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.
> raw
> +
> +       # Calculate size of zck header
> +       HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v
> ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':'
> -f 2)"
> +    
> +       # Extract the zck header
> +       ${SUDO_CHROOT} /bin/dd
> if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
> of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE"
> status=none
> +
> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide
> the required files as artifacts to this recipe"
> +        fi
> +
> +       # Extracting root partition from image1
> +       extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"
> "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"

>We only need that to compute the rdiff signature, right?
>As these images might be quite big, better remove the old one before
>extracting the new one. By that we save ~half of temporary space.

> +       
> +       # Extracting root partition from image2
> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" 
> "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw" 
> +
> +       rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> +       # create signature file with rdiff
> +       ${SUDO_CHROOT} /usr/bin/rdiff signature
> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
> +               ${WORKDIR}/delta_interim_artifacts/old-image-
> rootfs.sig
> +
> +        # create delta file with the signature file
> +       ${SUDO_CHROOT} /usr/bin/rdiff delta
> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> +               ${WORKDIR}/delta_interim_artifacts/new-image-
> rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> +       DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +    fi
> +        # create a symbolic link as IMAGE_CMD expects a
> *.delta_update file in deploy image directory
> +        ln -sf ${DELTA_ARTIFACT_SWU}
> ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
> new file mode 100644
> index 0000000..c7a3893
> --- /dev/null
> +++ b/kas/opt/delta-update.yml
> @@ -0,0 +1,21 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024
> +#
> +# Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    IMAGE_FSTYPES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = ""
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "" 

>Please also extra-weak define the DELTA_* variables in the
>delta_update.bbclass (to always have them defined and for documentation
>reasons).

>Best regards,
>Felix

-- 
>Siemens AG, Technology
>Linux Expert Center

RE: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya.Balakumar]

-----Original Message-----
From: MOESSBAUER, Felix <felix.moessbauer@siemens.com> 
Sent: Friday, March 8, 2024 4:27 PM
To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan <jan.kiszka@siemens.com>
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> This allows the creation of a swu file for delta update. The swu file 
> for delta update is only created when delta-update.yml file is 
> included at build time.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
> 
> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass index 
> 2c69892..2964024 100644
> --- a/classes/swupdate.bbclass
> +++ b/classes/swupdate.bbclass
> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>  SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>  SWU_BOOTLOADER ??= "ebg"
>  SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}- 
> ${SWU_BOOTLOADER}"
> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=

Hi, do we really need a dedicated file / variable for that?

[Adithya Balakumar] 
Hi Felix,

The sw-description file for a delta update is slightly different in terms of the image artifact inlcuded in the swu file and some additional delta update 
specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.

Would like to understand if there is a better approach.

> "${SWU_DESCRIPTION_FILE}-delta"
> +SWU_DELTA_UPDATE_ARTIFACT =
> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
> "delta_update${@get_swu_compression_type(d)}"
>  
>  SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>  SWU_DESCRIPTION_FILE ?= "sw-description"
> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
> ${SWU_DELTA_UPDATE_ARTIFACT}"

Does this mean the delta artifact is always added here?
[Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.

>  SWU_SIGNED ??= ""
>  SWU_SIGNATURE_EXT ?= "sig"
>  SWU_SIGNATURE_TYPE ?= "cms"
> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>  SWU_BUILDCHROOT_IMAGE_FILE ?=
> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>  
>  IMAGE_TYPEDEP:swu =
> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>  IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>  IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>  IMAGE_INSTALL += "${@'swupdate-certificates' if
> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>  IMAGE_SRC_URI:swu +=
> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>  IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>  IMAGE_TEMPLATE_FILES:swu +=
> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
> +IMAGE_TEMPLATE_FILES:delta_update +=
> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>  IMAGE_TEMPLATE_VARS:swu = " \
>      SWU_ROOTFS_PARTITION_NAME \
>      TARGET_IMAGE_UUID \
> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>      SWU_FILE_NODES \
>      SWU_BOOTLOADER_FILE_NODE \
>      SWU_SCRIPTS_NODE \
> +    SWU_DELTA_UPDATE_ARTIFACT \
> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>      "
>  
>  # TARGET_IMAGE_UUID needs to be generated before completing the 
> template @@ -136,6 +143,22 @@ python add_scripts(){
>      d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>  }
>  
> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
> +python add_swu_delta_update_properties() {
> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
> +    swu_delta_update_properties = ""
> +    if delta_type == "rdiff":
> +        swu_delta_update_properties =  'chainhandler =
> "rdiff_image";'
> +    elif delta_type == "zchunk":
> +        zck_url = d.getVar('DELTA_ZCK_URL')
> +        swu_delta_update_properties = f"""
> +                        chainhandler = "delta";
> +                        url = "{zck_url}";
> +                        zckloglevel = "error";
> +        """
> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
> swu_delta_update_properties)
> +}
> +
>  # convert between swupdate compressor name and imagetype extension
>  def get_swu_compression_type(d):
>      swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 @@ 
> FILESEXTRAPATHS:append = ":${LAYERDIR_cip- 
> core}/recipes-core/images/swu"
>  
>  do_image_swu[depends] += "${PN}:do_transform_template"
>  do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
> ${SWU_BOOTLOADER}"
> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>  IMAGE_CMD:swu() {
>      rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
> +        rm -rf ${WORKDIR}/swu-delta

Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?

Felix

[Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled. 
But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.

Thanks and Regards,
Adithya

> 
> +    fi
> +
>      cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>      if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
>          cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>      fi
> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
> then
> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
> +    fi
> +
>  
>      for swu_file in "${WORKDIR}"/swu*; do
>          swu_file_base=$(basename $swu_file)

--
>Siemens AG, Technology
>Linux Expert Center

RE: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Adithya.Balakumar]

-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Friday, March 8, 2024 5:15 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

On 07.03.24 08:46, Adithya Balakumar wrote:
> swupdate supports delta updates with rdiff_image and delta(zchunk) 
> handler. This change adds support to use either of the handler for 
> creating delta update artifacts. The type of handler can be set in the 
> delta-update.yml file along with additional variables needed for the 
> chosen handler.
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  classes/delta_update.bbclass | 90 ++++++++++++++++++++++++++++++++++++
>  kas/opt/delta-update.yml     | 21 +++++++++
>  2 files changed, 111 insertions(+)
>  create mode 100644 classes/delta_update.bbclass  create mode 100644 
> kas/opt/delta-update.yml
> 
> diff --git a/classes/delta_update.bbclass 
> b/classes/delta_update.bbclass new file mode 100644 index 
> 0000000..8d47081
> --- /dev/null
> +++ b/classes/delta_update.bbclass
> @@ -0,0 +1,90 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
> +
> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> +IMAGE_SRC_URI:delta_update += " ${@ "file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
> +
> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
> +
> +extract_root_fs() {
> +    INPUT_ARTIFACT=$1
> +    OUTPUT_ARTIFACT=$2
> +    SECTOR_SIZE=512
> +    ROOTFS_LABEL="primary"
> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ name {print $2}' | head -n 1)
> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE) / $(expr 1024 \* 1024))"
> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M 
> +skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none }
> +
> +do_delta_update_sw_description_template() {
> +   # Since the sw-description file for delta update would not be very different from that of a 
> +   # normal update, copy sw-description.tmpl as sw-description-delta.tmpl and add delta update 
> +   # specific properties
> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +   sed -i '/configfilecheck =/a \                      \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
> +}
> +addtask do_delta_update_sw_description_template before 
> +do_transform_template after do_generate_image_uuid
> +
> +IMAGE_CMD:delta_update() {
> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [ "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
> +    fi
> +    
> +    DELTA_ARTIFACT_SWU=""
> +
> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
> +
> +    	# Create .zck file
> +    	${SUDO_CHROOT} /bin/zck \
> +		--output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
> +		-u --chunk-hash-type sha256 \
> +		${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw
> +
> +    	# Calculate size of zck header
> +    	HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
> +    
> +    	# Extract the zck header
> +    	${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck 
> + of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" 
> + status=none
> +
> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
> +
> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
> +        fi
> +
> +	# Extracting root partition from image1
> +	extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"  "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
> +	
> +	# Extracting root partition from image2
> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"  "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"	
> +
> +	rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
> +	# create signature file with rdiff
> +	${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
> +		${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
> +
> +        # create delta file with the signature file
> +	${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
> +		${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw 
> +${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
> +
> +	DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
> +    fi
> +        # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
> +        ln -sf ${DELTA_ARTIFACT_SWU} 
> +${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
> +}
> +
> +addtask do_image_delta_update before do_image_swu after do_image_wic
> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml new 
> file mode 100644 index 0000000..c7a3893
> --- /dev/null
> +++ b/kas/opt/delta-update.yml
> @@ -0,0 +1,21 @@
> +#
> +# CIP Core, generic profile
> +#
> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> +#
> +# SPDX-License-Identifier: MIT
> +#
> +
> +header:
> +  version: 14
> +
> +local_conf_header:
> +  delta-update: |
> +    IMAGE_CLASSES:append = " delta_update"
> +    IMAGE_FSTYPES:append = " delta_update"
> +    DELTA_UPDATE_TYPE = ""
> +    DELTA_ZCK_URL = ""
> +    DELTA_RDIFF_REF_IMAGE = "" 

What prevents making this the default for generating update images?
People could still add their own options or local configs if they want to deviate. Would also simplify patch 6 further.

Jan

[Adithya Balakumar] 
Hi Jan,

Apologies, I would like to understand your remarks more clearly.
Are you suggesting to have sensible default values for the delta-update variables in the delta-update.yml file or to not have these variables in a seperate .yml file at all?

Thanks and Regards,
Adithya

--
>Siemens AG, Technology
>Linux Expert Center

RE: [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

[Adithya.Balakumar]

-----Original Message-----
From: Jan Kiszka <jan.kiszka@siemens.com> 
Sent: Friday, March 8, 2024 5:06 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

On 07.03.24 08:46, Adithya Balakumar wrote:
> Enable delta handler in swupdate for bookworm and disable cross 
> compile for arm and arm64 on bookworm to avoid build issues
> 
> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
> ---
>  recipes-core/swupdate/swupdate_2023.05.bb | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/recipes-core/swupdate/swupdate_2023.05.bb 
> b/recipes-core/swupdate/swupdate_2023.05.bb
> index d1fe1fa..7a22947 100644
> --- a/recipes-core/swupdate/swupdate_2023.05.bb
> +++ b/recipes-core/swupdate/swupdate_2023.05.bb
> @@ -36,10 +36,13 @@ SRC_URI:append:bullseye = " 
> file://0004-d-swupdate-www.install-Fix-path-for-debi
>  DEB_BUILD_PROFILES += "pkg.swupdate.suricattalua"
>  # add suricatta wfx
>  DEB_BUILD_PROFILES += "pkg.swupdate.suricattawfx"
> +# add delta update build profile for bookworm 
> +DEB_BUILD_PROFILES:append:bookworm = " pkg.swupdate.delta"
>  

What about sid?

[Adithya Balakumar]
Hi Jan,
Apologies, Will handle this in v2. 

>  # Disable cross for arm and arm64 on bullseye  # with cross compile 
> we have a unsat-dependency to dh-nodejs on arm/arm64  
> ISAR_CROSS_COMPILE:bullseye = "0"
> +ISAR_CROSS_COMPILE:bookworm = "0"

This is unfortunate. Why that?

[Adithya Balakumar]
This is because of build failures of swupdate with delta handler enabled in arm64/arm architectures due to dependency on libzck-dev package.
Have also raised this as an issue in isar-cip-core: https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/93

Thanks and Regards,
Adithya

>  
>  # use backport build profile for bullseye  
> DEB_BUILD_PROFILES:append:bullseye = " pkg.swupdate.bpo"

>Jan

--
>Siemens AG, Technology
>Linux Expert Center

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Jan Kiszka]

On 13.03.24 10:20, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com> 
> Sent: Friday, March 8, 2024 5:15 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler
> 
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk) 
>> handler. This change adds support to use either of the handler for 
>> creating delta update artifacts. The type of handler can be set in the 
>> delta-update.yml file along with additional variables needed for the 
>> chosen handler.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>  classes/delta_update.bbclass | 90 ++++++++++++++++++++++++++++++++++++
>>  kas/opt/delta-update.yml     | 21 +++++++++
>>  2 files changed, 111 insertions(+)
>>  create mode 100644 classes/delta_update.bbclass  create mode 100644 
>> kas/opt/delta-update.yml
>>
>> diff --git a/classes/delta_update.bbclass 
>> b/classes/delta_update.bbclass new file mode 100644 index 
>> 0000000..8d47081
>> --- /dev/null
>> +++ b/classes/delta_update.bbclass
>> @@ -0,0 +1,90 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +IMAGE_SRC_URI:delta_update += " ${@ "file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
>> +
>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>> +
>> +extract_root_fs() {
>> +    INPUT_ARTIFACT=$1
>> +    OUTPUT_ARTIFACT=$2
>> +    SECTOR_SIZE=512
>> +    ROOTFS_LABEL="primary"
>> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ name {print $2}' | head -n 1)
>> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE) / $(expr 1024 \* 1024))"
>> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M 
>> +skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none }
>> +
>> +do_delta_update_sw_description_template() {
>> +   # Since the sw-description file for delta update would not be very different from that of a 
>> +   # normal update, copy sw-description.tmpl as sw-description-delta.tmpl and add delta update 
>> +   # specific properties
>> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
>> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
>> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +   sed -i '/configfilecheck =/a \                      \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +}
>> +addtask do_delta_update_sw_description_template before 
>> +do_transform_template after do_generate_image_uuid
>> +
>> +IMAGE_CMD:delta_update() {
>> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [ "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
>> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
>> +    fi
>> +    
>> +    DELTA_ARTIFACT_SWU=""
>> +
>> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
>> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
>> +
>> +    	# Create .zck file
>> +    	${SUDO_CHROOT} /bin/zck \
>> +		--output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
>> +		-u --chunk-hash-type sha256 \
>> +		${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw
>> +
>> +    	# Calculate size of zck header
>> +    	HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
>> +    
>> +    	# Extract the zck header
>> +    	${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck 
>> + of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" 
>> + status=none
>> +
>> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
>> +
>> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
>> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
>> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
>> +        fi
>> +
>> +	# Extracting root partition from image1
>> +	extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"  "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
>> +	
>> +	# Extracting root partition from image2
>> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"  "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"	
>> +
>> +	rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
>> +	# create signature file with rdiff
>> +	${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
>> +		${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
>> +
>> +        # create delta file with the signature file
>> +	${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
>> +		${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw 
>> +${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
>> +
>> +	DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
>> +    fi
>> +        # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
>> +        ln -sf ${DELTA_ARTIFACT_SWU} 
>> +${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
>> +}
>> +
>> +addtask do_image_delta_update before do_image_swu after do_image_wic
>> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml new 
>> file mode 100644 index 0000000..c7a3893
>> --- /dev/null
>> +++ b/kas/opt/delta-update.yml
>> @@ -0,0 +1,21 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +header:
>> +  version: 14
>> +
>> +local_conf_header:
>> +  delta-update: |
>> +    IMAGE_CLASSES:append = " delta_update"
>> +    IMAGE_FSTYPES:append = " delta_update"
>> +    DELTA_UPDATE_TYPE = ""
>> +    DELTA_ZCK_URL = ""
>> +    DELTA_RDIFF_REF_IMAGE = "" 
> 
> What prevents making this the default for generating update images?
> People could still add their own options or local configs if they want to deviate. Would also simplify patch 6 further.
> 
> Jan
> 
> [Adithya Balakumar] 
> Hi Jan,
> 
> Apologies, I would like to understand your remarks more clearly.
> Are you suggesting to have sensible default values for the delta-update variables in the delta-update.yml file or to not have these variables in a seperate .yml file at all?
> 

No, I would like to have an interface that comes without the need for a
delta-update.yml. As sketched elsewhere: If you build an swupdate image
AND provide artifact(s) of a previous version at some to-be-defined
spot, isar-cip-core should automatically also generate the delta update
artifacts, along with full updates swus.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Gylstorff Quirin]

On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
> -----Original Message-----
> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
> Sent: Friday, March 8, 2024 4:27 PM
> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan <jan.kiszka@siemens.com>
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates
> 
> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>> This allows the creation of a swu file for delta update. The swu file
>> for delta update is only created when delta-update.yml file is
>> included at build time.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>   1 file changed, 33 insertions(+), 2 deletions(-)
>>
>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass index
>> 2c69892..2964024 100644
>> --- a/classes/swupdate.bbclass
>> +++ b/classes/swupdate.bbclass
>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>   SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>   SWU_BOOTLOADER ??= "ebg"
>>   SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-
>> ${SWU_BOOTLOADER}"
>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
> 
> Hi, do we really need a dedicated file / variable for that?
> 
> [Adithya Balakumar]
> Hi Felix,
> 
> The sw-description file for a delta update is slightly different in terms of the image artifact inlcuded in the swu file and some additional delta update
> specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
> 
> Would like to understand if there is a better approach.

A seperate sw-description file is not ideal, we should take the same 
abroach as with scripts and other optional or malible fields by writing
the content to variable and use the template mechanism of ISAR. The 
sw-description can look like this:

```
      images: ({
-            filename = "${SWU_ROOTFS_PARTITION_NAME}";
+            filename = "${SWU_IMAGE_ARTIFACT}";
              device = 
"C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
              type = "roundrobin";
              ${SWU_COMPRESSION_NODE}
-            properties: {
-                        subtype = "image";
-                        configfilecheck = 
"/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
-            };
+            ${SWU_IMAGE_PROPERTIES}
              sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
      });
```

an alternative would be generate the whole swu with the Variable[flag] 
syntax, e.g. 
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/generate-sw-description/classes/sw-description-generator.bbclass?ref_type=heads

Quirin

> 
>> "${SWU_DESCRIPTION_FILE}-delta"
>> +SWU_DELTA_UPDATE_ARTIFACT =
>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>> "delta_update${@get_swu_compression_type(d)}"
>>   
>>   SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>   SWU_DESCRIPTION_FILE ?= "sw-description"
>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>> ${SWU_DELTA_UPDATE_ARTIFACT}"
> 
> Does this mean the delta artifact is always added here?
> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
> 
>>   SWU_SIGNED ??= ""
>>   SWU_SIGNATURE_EXT ?= "sig"
>>   SWU_SIGNATURE_TYPE ?= "cms"
>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>   SWU_BUILDCHROOT_IMAGE_FILE ?=
>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>   
>>   IMAGE_TYPEDEP:swu =
>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>   IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>   IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>   IMAGE_INSTALL += "${@'swupdate-certificates' if
>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>   IMAGE_SRC_URI:swu +=
>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>   IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>   IMAGE_TEMPLATE_FILES:swu +=
>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>> +IMAGE_TEMPLATE_FILES:delta_update +=
>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>   IMAGE_TEMPLATE_VARS:swu = " \
>>       SWU_ROOTFS_PARTITION_NAME \
>>       TARGET_IMAGE_UUID \
>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>       SWU_FILE_NODES \
>>       SWU_BOOTLOADER_FILE_NODE \
>>       SWU_SCRIPTS_NODE \
>> +    SWU_DELTA_UPDATE_ARTIFACT \
>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>       "
>>   
>>   # TARGET_IMAGE_UUID needs to be generated before completing the
>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>       d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>   }
>>   
>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>> +python add_swu_delta_update_properties() {
>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>> +    swu_delta_update_properties = ""
>> +    if delta_type == "rdiff":
>> +        swu_delta_update_properties =  'chainhandler =
>> "rdiff_image";'
>> +    elif delta_type == "zchunk":
>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>> +        swu_delta_update_properties = f"""
>> +                        chainhandler = "delta";
>> +                        url = "{zck_url}";
>> +                        zckloglevel = "error";
>> +        """
>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>> swu_delta_update_properties)
>> +}
>> +
>>   # convert between swupdate compressor name and imagetype extension
>>   def get_swu_compression_type(d):
>>       swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 @@
>> FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
>> core}/recipes-core/images/swu"
>>   
>>   do_image_swu[depends] += "${PN}:do_transform_template"
>>   do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>> ${SWU_BOOTLOADER}"
>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>   IMAGE_CMD:swu() {
>>       rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>> +        rm -rf ${WORKDIR}/swu-delta
> 
> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
> 
> Felix
> 
> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
> 
> Thanks and Regards,
> Adithya
> 
>>
>> +    fi
>> +
>>       cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>       if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; then
>>           cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>       fi
>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>> then
>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>> +    fi
>> +
>>   
>>       for swu_file in "${WORKDIR}"/swu*; do
>>           swu_file_base=$(basename $swu_file)
> 
> --
>> Siemens AG, Technology
>> Linux Expert Center
> 
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15284): https://lists.cip-project.org/g/cip-dev/message/15284
> Mute This Topic: https://lists.cip-project.org/mt/104783341/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Gylstorff Quirin]

On 3/13/24 7:11 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> 
> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
> Sent: Friday, March 8, 2024 4:22 PM
> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; quirin.gylstorff@siemens.com; Kiszka, Jan <jan.kiszka@siemens.com>
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler
> 
> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk)
>> handler. This change adds support to use either of the handler for
>> creating delta update artifacts. The type of handler can be set in the
>> delta-update.yml file along with additional variables needed for the
>> chosen handler.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   classes/delta_update.bbclass | 90
> 
>> Hi,
> 
>> please name this class delta-update.bbclass to keep the naming style similar with the rest of the project.
> 
>> ++++++++++++++++++++++++++++++++++++
>>   kas/opt/delta-update.yml     | 21 +++++++++
>>   2 files changed, 111 insertions(+)
>>   create mode 100644 classes/delta_update.bbclass
>>   create mode 100644 kas/opt/delta-update.yml
>>
>> diff --git a/classes/delta_update.bbclass
>> b/classes/delta_update.bbclass new file mode 100644 index
>> 0000000..8d47081
>> --- /dev/null
>> +++ b/classes/delta_update.bbclass
>> @@ -0,0 +1,90 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
> 
>> Conditionally appending with += overwrites already set weak defaults.
>> You might want to use .= " fdisk zchunk rdiff" instead.
> 
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +IMAGE_SRC_URI:delta_update += " ${@ "
> 
>> Same here.
> 
>> file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if
> 
>> I guess this could be simplified and made more portable by referencing the DELTA_RDIFF_REF_IMAGE directly (e.g. from ${DEPLOY_DIR_IMAGE}). The problem with the FILESEXTRAPATHS:append = ":${LAYERDIR_cip->core}" is that it only works if the delta image is also from cip-core, but usually cip-core is just referenced in a downstream layer and used there.
> 
> [Adithya Balakumar]
> Hi Felix,
> 
> I understand the drawback of using FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}" that it only looks for the delta image placed in the isar-cip-core layer.
> As an example you point to referencing the DELTA_RDIFF_REF_IMAGE directly from ${DEPLOY_DIR_IMAGE} but as per my understanding ${DEPLOY_DIR_IMAGE} points to the current build's image deploy directory.
> 
> So, Could you please suggest a suitable way to provide the delta image in the case of rdiff.
> 
>> d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and
>> d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
>> +
>> +do_image_delta_update[cleandirs] +=
>> "${WORKDIR}/delta_interim_artifacts"
>> +
>> +extract_root_fs() {
>> +    INPUT_ARTIFACT=$1
>> +    OUTPUT_ARTIFACT=$2
>> +    SECTOR_SIZE=512
>> +    ROOTFS_LABEL="primary"
>> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o
> 
>> Can't we simply use sfdisk to query this information instead of all the awk stuff?
> 
>> Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~
>> name {print $2}' | head -n 1)
>> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE)
>> / $(expr 1024 \* 1024))"
>> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M
>> skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
>> +}
> 
>> Why copy exactly 1GB?
> 
>> Instead of extracting that here, it might be simpler to just store the .rdiff file (of the rootfs) with each image build and use that.[Adithya Balakumar]
> 
> [Adithya Balakumar]  Here, I am copying 1GB from the wic image because that is the size of the rootfs partition.
> 
> However, Quirin left a remark stating that the .squashfs file (or .verity file) could be used to create the delta artifact. Using the squashfs or verity file to create the delta will eliminate need for extracting the rootfs from the wic image and will simplify many things in this recipe. Would like to understand your opinion on this.

Also you can use the `WIC_DEPLOY_PARTITIONS = "1"` to generate the 
seperate wic partitions. That is much more robust then some
dd magic.

Quirin
> 
> Thanks and Regards,
> Adithya
> 
>> +
>> +do_delta_update_sw_description_template() {
>> +   # Since the sw-description file for delta update would not be
>> very different from that of a
>> +   # normal update, copy sw-description.tmpl as sw-description-
>> delta.tmpl and add delta update
>> +   # specific properties
>> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
>> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl
>> "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
>> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g'
>> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +   sed -i '/configfilecheck =/a \
>> \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}'
>> ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +}
> 
>> @Quirin: IIRC you recently refactored the creation of the sw- description file. This might simplify things here.
> 
>> +addtask do_delta_update_sw_description_template before
>> do_transform_template after do_generate_image_uuid
>> +
>> +IMAGE_CMD:delta_update() {
>> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [
>> "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
>> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE
>> (rdiff/zchunk)"
>> +    fi
>> +
>> +    DELTA_ARTIFACT_SWU=""
>> +
>> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then
>> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
>> "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
>> +
>> +       # Create .zck file
>> +       ${SUDO_CHROOT} /bin/zck \
>> +               --output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
>> +               -u --chunk-hash-type sha256 \
>> +               ${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.
>> raw
>> +
>> +       # Calculate size of zck header
>> +       HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v
>> ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':'
>> -f 2)"
>> +
>> +       # Extract the zck header
>> +       ${SUDO_CHROOT} /bin/dd
>> if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
>> of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE"
>> status=none
>> +
>> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
>> +
>> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
>> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
>> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide
>> the required files as artifacts to this recipe"
>> +        fi
>> +
>> +       # Extracting root partition from image1
>> +       extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"
>> "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
> 
>> We only need that to compute the rdiff signature, right?
>> As these images might be quite big, better remove the old one before
>> extracting the new one. By that we save ~half of temporary space.
> 
>> +
>> +       # Extracting root partition from image2
>> +       extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"
>> "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"
>> +
>> +       rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
>> +       # create signature file with rdiff
>> +       ${SUDO_CHROOT} /usr/bin/rdiff signature
>> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
>> +               ${WORKDIR}/delta_interim_artifacts/old-image-
>> rootfs.sig
>> +
>> +        # create delta file with the signature file
>> +       ${SUDO_CHROOT} /usr/bin/rdiff delta
>> ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
>> +               ${WORKDIR}/delta_interim_artifacts/new-image-
>> rootfs.raw ${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
>> +
>> +       DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
>> +    fi
>> +        # create a symbolic link as IMAGE_CMD expects a
>> *.delta_update file in deploy image directory
>> +        ln -sf ${DELTA_ARTIFACT_SWU}
>> ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
>> +}
>> +
>> +addtask do_image_delta_update before do_image_swu after do_image_wic
>> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml
>> new file mode 100644
>> index 0000000..c7a3893
>> --- /dev/null
>> +++ b/kas/opt/delta-update.yml
>> @@ -0,0 +1,21 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024
>> +#
>> +# Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +header:
>> +  version: 14
>> +
>> +local_conf_header:
>> +  delta-update: |
>> +    IMAGE_CLASSES:append = " delta_update"
>> +    IMAGE_FSTYPES:append = " delta_update"
>> +    DELTA_UPDATE_TYPE = ""
>> +    DELTA_ZCK_URL = ""
>> +    DELTA_RDIFF_REF_IMAGE = ""
> 
>> Please also extra-weak define the DELTA_* variables in the
>> delta_update.bbclass (to always have them defined and for documentation
>> reasons).
> 
>> Best regards,
>> Felix
> 

Re: [cip-dev] [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

[Gylstorff Quirin]

On 3/8/24 12:36 PM, Jan Kiszka via lists.cip-project.org wrote:
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> Enable delta handler in swupdate for bookworm and disable cross compile
>> for arm and arm64 on bookworm to avoid build issues
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   recipes-core/swupdate/swupdate_2023.05.bb | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/recipes-core/swupdate/swupdate_2023.05.bb b/recipes-core/swupdate/swupdate_2023.05.bb
>> index d1fe1fa..7a22947 100644
>> --- a/recipes-core/swupdate/swupdate_2023.05.bb
>> +++ b/recipes-core/swupdate/swupdate_2023.05.bb
>> @@ -36,10 +36,13 @@ SRC_URI:append:bullseye = " file://0004-d-swupdate-www.install-Fix-path-for-debi
>>   DEB_BUILD_PROFILES += "pkg.swupdate.suricattalua"
>>   # add suricatta wfx
>>   DEB_BUILD_PROFILES += "pkg.swupdate.suricattawfx"
>> +# add delta update build profile for bookworm
>> +DEB_BUILD_PROFILES:append:bookworm = " pkg.swupdate.delta"
>>   
> 
> What about sid?
> 
>>   # Disable cross for arm and arm64 on bullseye
>>   # with cross compile we have a unsat-dependency to dh-nodejs on arm/arm64
>>   ISAR_CROSS_COMPILE:bullseye = "0"
>> +ISAR_CROSS_COMPILE:bookworm = "0"
> 
> This is unfortunate. Why that?
> 

I tested swupdate 2023.05 on bookworm with armhf and arm64 it currently 
needs no backports.

My two cents are this is an issue with backports.

Quirin

>>   
>>   # use backport build profile for bullseye
>>   DEB_BUILD_PROFILES:append:bullseye = " pkg.swupdate.bpo"
> 
> Jan
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15265): https://lists.cip-project.org/g/cip-dev/message/15265
> Mute This Topic: https://lists.cip-project.org/mt/104783337/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

RE: [cip-dev] [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Adithya.Balakumar]

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Jan Kiszka via lists.cip-project.org
Sent: Wednesday, March 13, 2024 5:59 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

On 13.03.24 10:20, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Friday, March 8, 2024 5:15 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; 
> cip-dev@lists.cip-project.org
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with 
> rdiff_image and delta handler
> 
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> swupdate supports delta updates with rdiff_image and delta(zchunk) 
>> handler. This change adds support to use either of the handler for 
>> creating delta update artifacts. The type of handler can be set in 
>> the delta-update.yml file along with additional variables needed for 
>> the chosen handler.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>  classes/delta_update.bbclass | 90 ++++++++++++++++++++++++++++++++++++
>>  kas/opt/delta-update.yml     | 21 +++++++++
>>  2 files changed, 111 insertions(+)
>>  create mode 100644 classes/delta_update.bbclass  create mode 100644 
>> kas/opt/delta-update.yml
>>
>> diff --git a/classes/delta_update.bbclass 
>> b/classes/delta_update.bbclass new file mode 100644 index
>> 0000000..8d47081
>> --- /dev/null
>> +++ b/classes/delta_update.bbclass
>> @@ -0,0 +1,90 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
>> +
>> +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
>> +IMAGE_SRC_URI:delta_update += " ${@ "file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
>> +
>> +do_image_delta_update[cleandirs] += "${WORKDIR}/delta_interim_artifacts"
>> +
>> +extract_root_fs() {
>> +    INPUT_ARTIFACT=$1
>> +    OUTPUT_ARTIFACT=$2
>> +    SECTOR_SIZE=512
>> +    ROOTFS_LABEL="primary"
>> +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}" '$0 ~ name {print $2}' | head -n 1)
>> +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \* $SECTOR_SIZE) / $(expr 1024 \* 1024))"
>> +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M 
>> +skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none }
>> +
>> +do_delta_update_sw_description_template() {
>> +   # Since the sw-description file for delta update would not be very different from that of a 
>> +   # normal update, copy sw-description.tmpl as sw-description-delta.tmpl and add delta update 
>> +   # specific properties
>> +   SWU_DESCRIPTION_DELTA_UPDATE="sw-description-delta"
>> +   cp "${WORKDIR}/${SWU_DESCRIPTION_FILE}".tmpl "${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}".tmpl
>> +   sed -i 's/\SWU_ROOTFS_PARTITION_NAME/SWU_DELTA_UPDATE_ARTIFACT/g' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +   sed -i '/configfilecheck =/a \                      \'$'{SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES}' ${WORKDIR}/${SWU_DESCRIPTION_DELTA_UPDATE}.tmpl
>> +}
>> +addtask do_delta_update_sw_description_template before 
>> +do_transform_template after do_generate_image_uuid
>> +
>> +IMAGE_CMD:delta_update() {
>> +    if [ "${DELTA_UPDATE_TYPE}" != "zchunk" ] && [ "${DELTA_UPDATE_TYPE}" != "rdiff" ]; then
>> +        bbfatal "You must set a valid DELTA_UPDATE_TYPE (rdiff/zchunk)"
>> +    fi
>> +    
>> +    DELTA_ARTIFACT_SWU=""
>> +
>> +    if [ "${DELTA_UPDATE_TYPE}" = "zchunk" ]; then 
>> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic" "${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw"
>> +
>> +    	# Create .zck file
>> +    	${SUDO_CHROOT} /bin/zck \
>> +		--output ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck \
>> +		-u --chunk-hash-type sha256 \
>> +		${WORKDIR}/delta_interim_artifacts/${IMAGE_FULLNAME}.raw
>> +
>> +    	# Calculate size of zck header
>> +    	HSIZE="$(${SUDO_CHROOT} /bin/zck_read_header -v ${PP_DEPLOY}/${IMAGE_FULLNAME}.zck | grep "Header size" | cut -d ':' -f 2)"
>> +    
>> +    	# Extract the zck header
>> +    	${SUDO_CHROOT} /bin/dd if="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck
>> + of="${PP_DEPLOY}/${IMAGE_FULLNAME}".zck.header bs=1 count="$HSIZE" 
>> + status=none
>> +
>> +        DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.zck.header
>> +
>> +    elif [ "${DELTA_UPDATE_TYPE}" = "rdiff" ]; then
>> +        if [ -z "${DELTA_RDIFF_REF_IMAGE}" ]; then
>> +            bbfatal "You must set DELTA_RDIFF_REF_IMAGE and provide the required files as artifacts to this recipe"
>> +        fi
>> +
>> +	# Extracting root partition from image1
>> +	extract_root_fs "${WORKDIR}/${DELTA_RDIFF_REF_IMAGE}"  "${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw"
>> +	
>> +	# Extracting root partition from image2
>> +	extract_root_fs "${PP_DEPLOY}/${IMAGE_FULLNAME}.wic"  "${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw"	
>> +
>> +	rm -f ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta
>> +	# create signature file with rdiff
>> +	${SUDO_CHROOT} /usr/bin/rdiff signature ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.raw \
>> +		${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig
>> +
>> +        # create delta file with the signature file
>> +	${SUDO_CHROOT} /usr/bin/rdiff delta ${WORKDIR}/delta_interim_artifacts/old-image-rootfs.sig \
>> +		${WORKDIR}/delta_interim_artifacts/new-image-rootfs.raw
>> +${PP_DEPLOY}/${IMAGE_FULLNAME}.delta
>> +
>> +	DELTA_ARTIFACT_SWU=${IMAGE_FULLNAME}.delta
>> +    fi
>> +        # create a symbolic link as IMAGE_CMD expects a *.delta_update file in deploy image directory
>> +        ln -sf ${DELTA_ARTIFACT_SWU} 
>> +${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.delta_update
>> +}
>> +
>> +addtask do_image_delta_update before do_image_swu after do_image_wic
>> diff --git a/kas/opt/delta-update.yml b/kas/opt/delta-update.yml new 
>> file mode 100644 index 0000000..c7a3893
>> --- /dev/null
>> +++ b/kas/opt/delta-update.yml
>> @@ -0,0 +1,21 @@
>> +#
>> +# CIP Core, generic profile
>> +#
>> +# Copyright (c) Toshiba Corporation 2024 # # Authors:
>> +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
>> +#
>> +# SPDX-License-Identifier: MIT
>> +#
>> +
>> +header:
>> +  version: 14
>> +
>> +local_conf_header:
>> +  delta-update: |
>> +    IMAGE_CLASSES:append = " delta_update"
>> +    IMAGE_FSTYPES:append = " delta_update"
>> +    DELTA_UPDATE_TYPE = ""
>> +    DELTA_ZCK_URL = ""
>> +    DELTA_RDIFF_REF_IMAGE = "" 
> 
> What prevents making this the default for generating update images?
> People could still add their own options or local configs if they want to deviate. Would also simplify patch 6 further.
> 
> Jan
> 
> [Adithya Balakumar]
> Hi Jan,
> 
> Apologies, I would like to understand your remarks more clearly.
> Are you suggesting to have sensible default values for the delta-update variables in the delta-update.yml file or to not have these variables in a seperate .yml file at all?
> 

>No, I would like to have an interface that comes without the need for a delta-update.yml. As sketched elsewhere: If you build an swupdate image AND provide artifact(s) of a previous version at some to-be-defined >spot, isar-cip-core should automatically also generate the delta update artifacts, along with full updates swus.

>Jan

[Adithya Balakumar] 
Hi Jan,

Thanks for the clarification. 

In this case, as you already pointed out in the README update, we could have a default value for DELTA_UPDATE_TYPE and DELTA_RDIFF_REF_IMAGE.
ex: "rdiff" is the default type and "image-v1/<artifact-name>" could be the default path for the previous artifact. Ofcourse these default values can be changed anytime the user wishes to.

Would this approach be acceptable?

Regards,
Adithya

--
>Siemens AG, Technology
>Linux Expert Center

RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya.Balakumar]

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Wednesday, March 13, 2024 10:46 PM
To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates



On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
> -----Original Message-----
> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
> Sent: Friday, March 8, 2024 4:27 PM
> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) 
> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan 
> <jan.kiszka@siemens.com>
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate 
> swu for delta updates
> 
> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>> This allows the creation of a swu file for delta update. The swu file 
>> for delta update is only created when delta-update.yml file is 
>> included at build time.
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>   1 file changed, 33 insertions(+), 2 deletions(-)
>>
>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass 
>> index
>> 2c69892..2964024 100644
>> --- a/classes/swupdate.bbclass
>> +++ b/classes/swupdate.bbclass
>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>   SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>   SWU_BOOTLOADER ??= "ebg"
>>   SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}- 
>> ${SWU_BOOTLOADER}"
>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
> 
> Hi, do we really need a dedicated file / variable for that?
> 
> [Adithya Balakumar]
> Hi Felix,
> 
> The sw-description file for a delta update is slightly different in 
> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
> 
> Would like to understand if there is a better approach.

A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:

```
      images: ({
-            filename = "${SWU_ROOTFS_PARTITION_NAME}";
+            filename = "${SWU_IMAGE_ARTIFACT}";
              device =
"C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
              type = "roundrobin";
              ${SWU_COMPRESSION_NODE}
-            properties: {
-                        subtype = "image";
-                        configfilecheck = 
"/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
-            };
+            ${SWU_IMAGE_PROPERTIES}
              sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
      });
```

an alternative would be generate the whole swu with the Variable[flag] syntax, e.g. 
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/generate-sw-description/classes/sw-description-generator.bbclass?ref_type=heads

Quirin

[Adithya Balakumar] 
Hi Quirin,

Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.

As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.

Please correct me if my understanding is wrong.

Regards,
Adithya


> 
>> "${SWU_DESCRIPTION_FILE}-delta"
>> +SWU_DELTA_UPDATE_ARTIFACT =
>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>> "delta_update${@get_swu_compression_type(d)}"
>>   
>>   SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>   SWU_DESCRIPTION_FILE ?= "sw-description"
>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>> ${SWU_DELTA_UPDATE_ARTIFACT}"
> 
> Does this mean the delta artifact is always added here?
> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
> 
>>   SWU_SIGNED ??= ""
>>   SWU_SIGNATURE_EXT ?= "sig"
>>   SWU_SIGNATURE_TYPE ?= "cms"
>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>   SWU_BUILDCHROOT_IMAGE_FILE ?=
>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>   
>>   IMAGE_TYPEDEP:swu =
>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>   IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>   IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>   IMAGE_INSTALL += "${@'swupdate-certificates' if
>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = 
>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>   IMAGE_SRC_URI:swu +=
>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>   IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>   IMAGE_TEMPLATE_FILES:swu +=
>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>> +IMAGE_TEMPLATE_FILES:delta_update +=
>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>   IMAGE_TEMPLATE_VARS:swu = " \
>>       SWU_ROOTFS_PARTITION_NAME \
>>       TARGET_IMAGE_UUID \
>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>       SWU_FILE_NODES \
>>       SWU_BOOTLOADER_FILE_NODE \
>>       SWU_SCRIPTS_NODE \
>> +    SWU_DELTA_UPDATE_ARTIFACT \
>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>       "
>>   
>>   # TARGET_IMAGE_UUID needs to be generated before completing the 
>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>       d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>   }
>>   
>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>> +python add_swu_delta_update_properties() {
>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>> +    swu_delta_update_properties = ""
>> +    if delta_type == "rdiff":
>> +        swu_delta_update_properties =  'chainhandler =
>> "rdiff_image";'
>> +    elif delta_type == "zchunk":
>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>> +        swu_delta_update_properties = f"""
>> +                        chainhandler = "delta";
>> +                        url = "{zck_url}";
>> +                        zckloglevel = "error";
>> +        """
>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>> swu_delta_update_properties)
>> +}
>> +
>>   # convert between swupdate compressor name and imagetype extension
>>   def get_swu_compression_type(d):
>>       swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 @@ 
>> FILESEXTRAPATHS:append = ":${LAYERDIR_cip- 
>> core}/recipes-core/images/swu"
>>   
>>   do_image_swu[depends] += "${PN}:do_transform_template"
>>   do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
>> ${SWU_BOOTLOADER}"
>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>   IMAGE_CMD:swu() {
>>       rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>> +        rm -rf ${WORKDIR}/swu-delta
> 
> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
> 
> Felix
> 
> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
> 
> Thanks and Regards,
> Adithya
> 
>>
>> +    fi
>> +
>>       cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>       if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; 
>> then
>>           cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>       fi
>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>> then
>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>> +    fi
>> +
>>   
>>       for swu_file in "${WORKDIR}"/swu*; do
>>           swu_file_base=$(basename $swu_file)
> 
> --
>> Siemens AG, Technology
>> Linux Expert Center
> 
> 
> 
> 
> 
> 

RE: [cip-dev] [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate

[Adithya.Balakumar]

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Thursday, March 14, 2024 12:27 PM
To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 3/6] swupdate: Enable Delta handler in swupdate



On 3/8/24 12:36 PM, Jan Kiszka via lists.cip-project.org wrote:
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> Enable delta handler in swupdate for bookworm and disable cross 
>> compile for arm and arm64 on bookworm to avoid build issues
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   recipes-core/swupdate/swupdate_2023.05.bb | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/recipes-core/swupdate/swupdate_2023.05.bb 
>> b/recipes-core/swupdate/swupdate_2023.05.bb
>> index d1fe1fa..7a22947 100644
>> --- a/recipes-core/swupdate/swupdate_2023.05.bb
>> +++ b/recipes-core/swupdate/swupdate_2023.05.bb
>> @@ -36,10 +36,13 @@ SRC_URI:append:bullseye = " file://0004-d-swupdate-www.install-Fix-path-for-debi
>>   DEB_BUILD_PROFILES += "pkg.swupdate.suricattalua"
>>   # add suricatta wfx
>>   DEB_BUILD_PROFILES += "pkg.swupdate.suricattawfx"
>> +# add delta update build profile for bookworm 
>> +DEB_BUILD_PROFILES:append:bookworm = " pkg.swupdate.delta"
>>   
> 
> What about sid?
> 
>>   # Disable cross for arm and arm64 on bullseye
>>   # with cross compile we have a unsat-dependency to dh-nodejs on arm/arm64
>>   ISAR_CROSS_COMPILE:bullseye = "0"
>> +ISAR_CROSS_COMPILE:bookworm = "0"
> 
> This is unfortunate. Why that?
> 

I tested swupdate 2023.05 on bookworm with armhf and arm64 it currently needs no backports.

My two cents are this is an issue with backports.

Quirin

[Adithya Balakumar] 
Hi Quirin,

This was because of build failures of swupdate with delta handler enabled in arm64/arm architectures due to dependency on libzck-dev package (a backported package).
Have also raised this as an issue in isar-cip-core: https://gitlab.com/cip-project/cip-core/isar-cip-core/-/issues/93

Regards,
Adithya

>>   
>>   # use backport build profile for bullseye
>>   DEB_BUILD_PROFILES:append:bullseye = " pkg.swupdate.bpo"
> 
> Jan
> 
> 
> 
> 
> 

Re: [cip-dev] [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[Gylstorff Quirin]

On 3/8/24 12:35 PM, Jan Kiszka via lists.cip-project.org wrote:
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> Delta handler requires zchunk > 1.3, but the required version is only
>> available in bookworm backports
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   conf/distro/cip-core-bookworm.conf                   |  3 +++
>>   conf/distro/debian-bookworm-backports.list           |  1 +
>>   .../preferences.swupdate-bookworm-backports.conf     | 12 ++++++++++++
>>   3 files changed, 16 insertions(+)
>>   create mode 100644 conf/distro/debian-bookworm-backports.list
>>   create mode 100644 conf/distro/preferences.swupdate-bookworm-backports.conf
>>
>> diff --git a/conf/distro/cip-core-bookworm.conf b/conf/distro/cip-core-bookworm.conf
>> index 9bcf559..e0a4f7f 100644
>> --- a/conf/distro/cip-core-bookworm.conf
>> +++ b/conf/distro/cip-core-bookworm.conf
>> @@ -12,6 +12,9 @@
>>   require conf/distro/debian-bookworm.conf
>>   require cip-core-common.inc
>>   
>> +DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-backports.list"
>> +DISTRO_APT_PREFERENCES:append:swupdate = " conf/distro/preferences.swupdate-bookworm-backports.conf"
>> +
>>   PREFERRED_VERSION_swupdate ?= "2023.05%"
>>   PREFERRED_VERSION_linux-cip ?= "6.1.%"
>>   PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
>> diff --git a/conf/distro/debian-bookworm-backports.list b/conf/distro/debian-bookworm-backports.list
>> new file mode 100644
>> index 0000000..c4873b4
>> --- /dev/null
>> +++ b/conf/distro/debian-bookworm-backports.list
>> @@ -0,0 +1 @@
>> +deb http://deb.debian.org/debian bookworm-backports main contrib non-free
>> diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf b/conf/distro/preferences.swupdate-bookworm-backports.conf
>> new file mode 100644
>> index 0000000..b6efffc
>> --- /dev/null
>> +++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
>> @@ -0,0 +1,12 @@
>> +Package: libzck-dev
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
>> +Package: libzck1
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
>> +Package: zchunk
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
> 
> Backports are bad when it comes to use in production (they receive
> best-effort updates only). It this the only options?
> 

As we need only backports for zchunk i would propose that we support 
only rdiff based image for distros <= bookworm. The zchunk support can
be activiated for trixie and later.

Quirin

> Jan
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15264): https://lists.cip-project.org/g/cip-dev/message/15264
> Mute This Topic: https://lists.cip-project.org/mt/104783336/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Gylstorff Quirin]

On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Wednesday, March 13, 2024 10:46 PM
> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates
> 
> 
> 
> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>> -----Original Message-----
>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>> Sent: Friday, March 8, 2024 4:27 PM
>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR)
>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan
>> <jan.kiszka@siemens.com>
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate
>> swu for delta updates
>>
>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>> This allows the creation of a swu file for delta update. The swu file
>>> for delta update is only created when delta-update.yml file is
>>> included at build time.
>>>
>>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>>> ---
>>>    classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>    1 file changed, 33 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
>>> index
>>> 2c69892..2964024 100644
>>> --- a/classes/swupdate.bbclass
>>> +++ b/classes/swupdate.bbclass
>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>    SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>    SWU_BOOTLOADER ??= "ebg"
>>>    SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-
>>> ${SWU_BOOTLOADER}"
>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>
>> Hi, do we really need a dedicated file / variable for that?
>>
>> [Adithya Balakumar]
>> Hi Felix,
>>
>> The sw-description file for a delta update is slightly different in
>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>
>> Would like to understand if there is a better approach.
> 
> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
> 
> ```
>        images: ({
> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
> +            filename = "${SWU_IMAGE_ARTIFACT}";
>                device =
> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>                type = "roundrobin";
>                ${SWU_COMPRESSION_NODE}
> -            properties: {
> -                        subtype = "image";
> -                        configfilecheck =
> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
> -            };
> +            ${SWU_IMAGE_PROPERTIES}
>                sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>        });
> ```
> 
> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/generate-sw-description/classes/sw-description-generator.bbclass?ref_type=heads
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.

What would be the use case to have both a regular and a delta update?

If you have multiple swus the server has the additional responsibility 
to select the correct swu.

In the first version of the delta update I would use the either delta 
update or full update.

Quirin

> 
> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
> 
> Please correct me if my understanding is wrong.
> 
> Regards,
> Adithya
> 
> 
>>
>>> "${SWU_DESCRIPTION_FILE}-delta"
>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>> "delta_update${@get_swu_compression_type(d)}"
>>>    
>>>    SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>    SWU_DESCRIPTION_FILE ?= "sw-description"
>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>
>> Does this mean the delta artifact is always added here?
>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>
>>>    SWU_SIGNED ??= ""
>>>    SWU_SIGNATURE_EXT ?= "sig"
>>>    SWU_SIGNATURE_TYPE ?= "cms"
>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>    SWU_BUILDCHROOT_IMAGE_FILE ?=
>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>    
>>>    IMAGE_TYPEDEP:swu =
>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>    IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGE_INSTALL += "${@'swupdate-certificates' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_SRC_URI:swu +=
>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu +=
>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>    IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_ROOTFS_PARTITION_NAME \
>>>        TARGET_IMAGE_UUID \
>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_FILE_NODES \
>>>        SWU_BOOTLOADER_FILE_NODE \
>>>        SWU_SCRIPTS_NODE \
>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>        "
>>>    
>>>    # TARGET_IMAGE_UUID needs to be generated before completing the
>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>        d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>    }
>>>    
>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>> +python add_swu_delta_update_properties() {
>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>> +    swu_delta_update_properties = ""
>>> +    if delta_type == "rdiff":
>>> +        swu_delta_update_properties =  'chainhandler =
>>> "rdiff_image";'
>>> +    elif delta_type == "zchunk":
>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>> +        swu_delta_update_properties = f"""
>>> +                        chainhandler = "delta";
>>> +                        url = "{zck_url}";
>>> +                        zckloglevel = "error";
>>> +        """
>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>> swu_delta_update_properties)
>>> +}
>>> +
>>>    # convert between swupdate compressor name and imagetype extension
>>>    def get_swu_compression_type(d):
>>>        swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 @@
>>> FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
>>> core}/recipes-core/images/swu"
>>>    
>>>    do_image_swu[depends] += "${PN}:do_transform_template"
>>>    do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>> ${SWU_BOOTLOADER}"
>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>    IMAGE_CMD:swu() {
>>>        rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>> +        rm -rf ${WORKDIR}/swu-delta
>>
>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>
>> Felix
>>
>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>
>> Thanks and Regards,
>> Adithya
>>
>>>
>>> +    fi
>>> +
>>>        cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>        if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ];
>>> then
>>>            cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>        fi
>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>> then
>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>> +    fi
>>> +
>>>    
>>>        for swu_file in "${WORKDIR}"/swu*; do
>>>            swu_file_base=$(basename $swu_file)
>>
>> --
>>> Siemens AG, Technology
>>> Linux Expert Center
>>
>>
>>
>>
>>
>>

RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya.Balakumar]

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Thursday, March 14, 2024 2:57 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates



On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On 
> Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Wednesday, March 13, 2024 10:46 PM
> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
> jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: 
> Generate swu for delta updates
> 
> 
> 
> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>> -----Original Message-----
>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>> Sent: Friday, March 8, 2024 4:27 PM
>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) 
>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan 
>> <jan.kiszka@siemens.com>
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG 
>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate 
>> swu for delta updates
>>
>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>> This allows the creation of a swu file for delta update. The swu 
>>> file for delta update is only created when delta-update.yml file is 
>>> included at build time.
>>>
>>> Signed-off-by: Adithya Balakumar 
>>> <Adithya.Balakumar@toshiba-tsip.com>
>>> ---
>>>    classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>    1 file changed, 33 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass 
>>> index
>>> 2c69892..2964024 100644
>>> --- a/classes/swupdate.bbclass
>>> +++ b/classes/swupdate.bbclass
>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>    SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>    SWU_BOOTLOADER ??= "ebg"
>>>    SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}- 
>>> ${SWU_BOOTLOADER}"
>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>
>> Hi, do we really need a dedicated file / variable for that?
>>
>> [Adithya Balakumar]
>> Hi Felix,
>>
>> The sw-description file for a delta update is slightly different in 
>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>
>> Would like to understand if there is a better approach.
> 
> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
> 
> ```
>        images: ({
> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
> +            filename = "${SWU_IMAGE_ARTIFACT}";
>                device =
> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>                type = "roundrobin";
>                ${SWU_COMPRESSION_NODE}
> -            properties: {
> -                        subtype = "image";
> -                        configfilecheck =
> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
> -            };
> +            ${SWU_IMAGE_PROPERTIES}
>                sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>        });
> ```
> 
> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/gene
> rate-sw-description/classes/sw-description-generator.bbclass?ref_type=
> heads
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.

What would be the use case to have both a regular and a delta update?

If you have multiple swus the server has the additional responsibility to select the correct swu.

In the first version of the delta update I would use the either delta update or full update.

Quirin

[Adithya Balakumar] 
Hi Quirin,

I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.

I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.

Would like your take on this.

Thanks and Regards,
Adithya

> 
> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
> 
> Please correct me if my understanding is wrong.
> 
> Regards,
> Adithya
> 
> 
>>
>>> "${SWU_DESCRIPTION_FILE}-delta"
>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>> "delta_update${@get_swu_compression_type(d)}"
>>>    
>>>    SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>    SWU_DESCRIPTION_FILE ?= "sw-description"
>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>
>> Does this mean the delta artifact is always added here?
>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>
>>>    SWU_SIGNED ??= ""
>>>    SWU_SIGNATURE_EXT ?= "sig"
>>>    SWU_SIGNATURE_TYPE ?= "cms"
>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>    SWU_BUILDCHROOT_IMAGE_FILE ?=
>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>    
>>>    IMAGE_TYPEDEP:swu =
>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>    IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>    IMAGE_INSTALL += "${@'swupdate-certificates' if
>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = 
>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_SRC_URI:swu +=
>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>    IMAGE_TEMPLATE_FILES:swu +=
>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>    IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_ROOTFS_PARTITION_NAME \
>>>        TARGET_IMAGE_UUID \
>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>        SWU_FILE_NODES \
>>>        SWU_BOOTLOADER_FILE_NODE \
>>>        SWU_SCRIPTS_NODE \
>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>        "
>>>    
>>>    # TARGET_IMAGE_UUID needs to be generated before completing the 
>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>        d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>    }
>>>    
>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>> +python add_swu_delta_update_properties() {
>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>> +    swu_delta_update_properties = ""
>>> +    if delta_type == "rdiff":
>>> +        swu_delta_update_properties =  'chainhandler =
>>> "rdiff_image";'
>>> +    elif delta_type == "zchunk":
>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>> +        swu_delta_update_properties = f"""
>>> +                        chainhandler = "delta";
>>> +                        url = "{zck_url}";
>>> +                        zckloglevel = "error";
>>> +        """
>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>> swu_delta_update_properties)
>>> +}
>>> +
>>>    # convert between swupdate compressor name and imagetype extension
>>>    def get_swu_compression_type(d):
>>>        swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21 
>>> @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip- 
>>> core}/recipes-core/images/swu"
>>>    
>>>    do_image_swu[depends] += "${PN}:do_transform_template"
>>>    do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
>>> ${SWU_BOOTLOADER}"
>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>    IMAGE_CMD:swu() {
>>>        rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>> +        rm -rf ${WORKDIR}/swu-delta
>>
>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>
>> Felix
>>
>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>
>> Thanks and Regards,
>> Adithya
>>
>>>
>>> +    fi
>>> +
>>>        cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>        if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; 
>>> then
>>>            cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>        fi
>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>> then
>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>> +    fi
>>> +
>>>    
>>>        for swu_file in "${WORKDIR}"/swu*; do
>>>            swu_file_base=$(basename $swu_file)
>>
>> --
>>> Siemens AG, Technology
>>> Linux Expert Center
>>
>>
>>
>>
>>
>>

Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Gylstorff Quirin]

On 3/15/24 7:45 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Thursday, March 14, 2024 2:57 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates
> 
> 
> 
> On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>> -----Original Message-----
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
>> Behalf Of Quirin Gylstorff via lists.cip-project.org
>> Sent: Wednesday, March 13, 2024 10:46 PM
>> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com;
>> jan.kiszka@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>> Generate swu for delta updates
>>
>>
>>
>> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>> -----Original Message-----
>>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>>> Sent: Friday, March 8, 2024 4:27 PM
>>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR)
>>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan
>>> <jan.kiszka@siemens.com>
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate
>>> swu for delta updates
>>>
>>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>>> This allows the creation of a swu file for delta update. The swu
>>>> file for delta update is only created when delta-update.yml file is
>>>> included at build time.
>>>>
>>>> Signed-off-by: Adithya Balakumar
>>>> <Adithya.Balakumar@toshiba-tsip.com>
>>>> ---
>>>>     classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>>     1 file changed, 33 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
>>>> index
>>>> 2c69892..2964024 100644
>>>> --- a/classes/swupdate.bbclass
>>>> +++ b/classes/swupdate.bbclass
>>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>>     SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>>     SWU_BOOTLOADER ??= "ebg"
>>>>     SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-
>>>> ${SWU_BOOTLOADER}"
>>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>>
>>> Hi, do we really need a dedicated file / variable for that?
>>>
>>> [Adithya Balakumar]
>>> Hi Felix,
>>>
>>> The sw-description file for a delta update is slightly different in
>>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>>
>>> Would like to understand if there is a better approach.
>>
>> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
>>
>> ```
>>         images: ({
>> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
>> +            filename = "${SWU_IMAGE_ARTIFACT}";
>>                 device =
>> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>>                 type = "roundrobin";
>>                 ${SWU_COMPRESSION_NODE}
>> -            properties: {
>> -                        subtype = "image";
>> -                        configfilecheck =
>> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
>> -            };
>> +            ${SWU_IMAGE_PROPERTIES}
>>                 sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>>         });
>> ```
>>
>> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/gene
>> rate-sw-description/classes/sw-description-generator.bbclass?ref_type=
>> heads
>>
>> Quirin
>>
>> [Adithya Balakumar]
>> Hi Quirin,
>>
>> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.
> 
> What would be the use case to have both a regular and a delta update?
> 
> If you have multiple swus the server has the additional responsibility to select the correct swu.
> 
> In the first version of the delta update I would use the either delta update or full update.
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.

Ok I see, then we should use a seperate sw-description file, similar to 
the bootloader update case. , with the necessary VARIABLES and still 
make the properties of the orginial sw-description modifiable. Also the 
functions and setting of variables
should then be provided by your delta-update.bbclass like the 
efibootguard class.

Quirin

> 
> I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.
> 
> Would like your take on this.
> 
> Thanks and Regards,
> Adithya
> 
>>
>> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
>>
>> Please correct me if my understanding is wrong.
>>
>> Regards,
>> Adithya
>>
>>
>>>
>>>> "${SWU_DESCRIPTION_FILE}-delta"
>>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>>> "delta_update${@get_swu_compression_type(d)}"
>>>>     
>>>>     SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>>     SWU_DESCRIPTION_FILE ?= "sw-description"
>>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>>
>>> Does this mean the delta artifact is always added here?
>>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>>
>>>>     SWU_SIGNED ??= ""
>>>>     SWU_SIGNATURE_EXT ?= "sig"
>>>>     SWU_SIGNATURE_TYPE ?= "cms"
>>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>>     SWU_BUILDCHROOT_IMAGE_FILE ?=
>>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>>     
>>>>     IMAGE_TYPEDEP:swu =
>>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>>     IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>     IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>     IMAGE_INSTALL += "${@'swupdate-certificates' if
>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
>>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>>     IMAGE_SRC_URI:swu +=
>>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>     IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>>     IMAGE_TEMPLATE_FILES:swu +=
>>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>>     IMAGE_TEMPLATE_VARS:swu = " \
>>>>         SWU_ROOTFS_PARTITION_NAME \
>>>>         TARGET_IMAGE_UUID \
>>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>>         SWU_FILE_NODES \
>>>>         SWU_BOOTLOADER_FILE_NODE \
>>>>         SWU_SCRIPTS_NODE \
>>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>>         "
>>>>     
>>>>     # TARGET_IMAGE_UUID needs to be generated before completing the
>>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>>         d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>>     }
>>>>     
>>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>>> +python add_swu_delta_update_properties() {
>>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>>> +    swu_delta_update_properties = ""
>>>> +    if delta_type == "rdiff":
>>>> +        swu_delta_update_properties =  'chainhandler =
>>>> "rdiff_image";'
>>>> +    elif delta_type == "zchunk":
>>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>>> +        swu_delta_update_properties = f"""
>>>> +                        chainhandler = "delta";
>>>> +                        url = "{zck_url}";
>>>> +                        zckloglevel = "error";
>>>> +        """
>>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>>> swu_delta_update_properties)
>>>> +}
>>>> +
>>>>     # convert between swupdate compressor name and imagetype extension
>>>>     def get_swu_compression_type(d):
>>>>         swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 +177,21
>>>> @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
>>>> core}/recipes-core/images/swu"
>>>>     
>>>>     do_image_swu[depends] += "${PN}:do_transform_template"
>>>>     do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>> ${SWU_BOOTLOADER}"
>>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>>     IMAGE_CMD:swu() {
>>>>         rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>>> +        rm -rf ${WORKDIR}/swu-delta
>>>
>>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>>
>>> Felix
>>>
>>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>>
>>> Thanks and Regards,
>>> Adithya
>>>
>>>>
>>>> +    fi
>>>> +
>>>>         cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>>         if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ];
>>>> then
>>>>             cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>>         fi
>>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>>> then
>>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>>> +    fi
>>>> +
>>>>     
>>>>         for swu_file in "${WORKDIR}"/swu*; do
>>>>             swu_file_base=$(basename $swu_file)
>>>
>>> --
>>>> Siemens AG, Technology
>>>> Linux Expert Center
>>>
>>>
>>>
>>>
>>>
>>>

RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya.Balakumar]

-----Original Message-----
From: Gylstorff Quirin <quirin.gylstorff@siemens.com> 
Sent: Friday, March 15, 2024 2:12 PM
To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates



On 3/15/24 7:45 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> -----Original Message-----
> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On 
> Behalf Of Quirin Gylstorff via lists.cip-project.org
> Sent: Thursday, March 14, 2024 2:57 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; 
> cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
> jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: 
> Generate swu for delta updates
> 
> 
> 
> On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>> -----Original Message-----
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>> On Behalf Of Quirin Gylstorff via lists.cip-project.org
>> Sent: Wednesday, March 13, 2024 10:46 PM
>> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
>> jan.kiszka@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG 
>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>> Generate swu for delta updates
>>
>>
>>
>> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>> -----Original Message-----
>>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>>> Sent: Friday, March 8, 2024 4:27 PM
>>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) 
>>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan 
>>> <jan.kiszka@siemens.com>
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate 
>>> swu for delta updates
>>>
>>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>>> This allows the creation of a swu file for delta update. The swu 
>>>> file for delta update is only created when delta-update.yml file is 
>>>> included at build time.
>>>>
>>>> Signed-off-by: Adithya Balakumar
>>>> <Adithya.Balakumar@toshiba-tsip.com>
>>>> ---
>>>>     classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>>     1 file changed, 33 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass 
>>>> index
>>>> 2c69892..2964024 100644
>>>> --- a/classes/swupdate.bbclass
>>>> +++ b/classes/swupdate.bbclass
>>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>>     SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>>     SWU_BOOTLOADER ??= "ebg"
>>>>     SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}- 
>>>> ${SWU_BOOTLOADER}"
>>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>>
>>> Hi, do we really need a dedicated file / variable for that?
>>>
>>> [Adithya Balakumar]
>>> Hi Felix,
>>>
>>> The sw-description file for a delta update is slightly different in 
>>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>>
>>> Would like to understand if there is a better approach.
>>
>> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
>>
>> ```
>>         images: ({
>> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
>> +            filename = "${SWU_IMAGE_ARTIFACT}";
>>                 device =
>> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>>                 type = "roundrobin";
>>                 ${SWU_COMPRESSION_NODE}
>> -            properties: {
>> -                        subtype = "image";
>> -                        configfilecheck =
>> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
>> -            };
>> +            ${SWU_IMAGE_PROPERTIES}
>>                 sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>>         });
>> ```
>>
>> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/gen
>> e 
>> rate-sw-description/classes/sw-description-generator.bbclass?ref_type
>> =
>> heads
>>
>> Quirin
>>
>> [Adithya Balakumar]
>> Hi Quirin,
>>
>> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.
> 
> What would be the use case to have both a regular and a delta update?
> 
> If you have multiple swus the server has the additional responsibility to select the correct swu.
> 
> In the first version of the delta update I would use the either delta update or full update.
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.

Ok I see, then we should use a seperate sw-description file, similar to the bootloader update case. , with the necessary VARIABLES and still make the properties of the orginial sw-description modifiable. Also the functions and setting of variables should then be provided by your delta-update.bbclass like the efibootguard class.

Quirin

[Adithya Balakumar] 
Hi Quirin,

When you say a separate sw-description file, do you mean a separate .tmpl file for delta update (ex: sw-description-delta.tmpl) ?

There would only be a few changes in the sw-description for the delta update case when compared to the complete update case. 

Hence in the first patch, I created a separate sw-description file (copied the existing sw-description.tmpl as sw-description-delta.tmpl) for delta update in the recipe itself and only modified/added certain variables in the new .tmpl file with the sed command and let the ISAR template system create the final sw-description file.

Regards,
Adithya

> 
> I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.
> 
> Would like your take on this.
> 
> Thanks and Regards,
> Adithya
> 
>>
>> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
>>
>> Please correct me if my understanding is wrong.
>>
>> Regards,
>> Adithya
>>
>>
>>>
>>>> "${SWU_DESCRIPTION_FILE}-delta"
>>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>>> "delta_update${@get_swu_compression_type(d)}"
>>>>     
>>>>     SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>>     SWU_DESCRIPTION_FILE ?= "sw-description"
>>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>>
>>> Does this mean the delta artifact is always added here?
>>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>>
>>>>     SWU_SIGNED ??= ""
>>>>     SWU_SIGNATURE_EXT ?= "sig"
>>>>     SWU_SIGNATURE_TYPE ?= "cms"
>>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>>     SWU_BUILDCHROOT_IMAGE_FILE ?=
>>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>>     
>>>>     IMAGE_TYPEDEP:swu =
>>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>>     IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>     IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>     IMAGE_INSTALL += "${@'swupdate-certificates' if
>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = 
>>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>>     IMAGE_SRC_URI:swu +=
>>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>     IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>>     IMAGE_TEMPLATE_FILES:swu +=
>>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>>     IMAGE_TEMPLATE_VARS:swu = " \
>>>>         SWU_ROOTFS_PARTITION_NAME \
>>>>         TARGET_IMAGE_UUID \
>>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>>         SWU_FILE_NODES \
>>>>         SWU_BOOTLOADER_FILE_NODE \
>>>>         SWU_SCRIPTS_NODE \
>>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>>         "
>>>>     
>>>>     # TARGET_IMAGE_UUID needs to be generated before completing the 
>>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>>         d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>>     }
>>>>     
>>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>>> +python add_swu_delta_update_properties() {
>>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>>> +    swu_delta_update_properties = ""
>>>> +    if delta_type == "rdiff":
>>>> +        swu_delta_update_properties =  'chainhandler =
>>>> "rdiff_image";'
>>>> +    elif delta_type == "zchunk":
>>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>>> +        swu_delta_update_properties = f"""
>>>> +                        chainhandler = "delta";
>>>> +                        url = "{zck_url}";
>>>> +                        zckloglevel = "error";
>>>> +        """
>>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>>> swu_delta_update_properties)
>>>> +}
>>>> +
>>>>     # convert between swupdate compressor name and imagetype extension
>>>>     def get_swu_compression_type(d):
>>>>         swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13 
>>>> +177,21 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip- 
>>>> core}/recipes-core/images/swu"
>>>>     
>>>>     do_image_swu[depends] += "${PN}:do_transform_template"
>>>>     do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
>>>> ${SWU_BOOTLOADER}"
>>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>>     IMAGE_CMD:swu() {
>>>>         rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>>> +        rm -rf ${WORKDIR}/swu-delta
>>>
>>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>>
>>> Felix
>>>
>>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>>
>>> Thanks and Regards,
>>> Adithya
>>>
>>>>
>>>> +    fi
>>>> +
>>>>         cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>>         if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ]; 
>>>> then
>>>>             cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>>         fi
>>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>>> then
>>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>>> +    fi
>>>> +
>>>>     
>>>>         for swu_file in "${WORKDIR}"/swu*; do
>>>>             swu_file_base=$(basename $swu_file)
>>>
>>> --
>>>> Siemens AG, Technology
>>>> Linux Expert Center
>>>
>>>
>>>
>>>
>>>
>>>

RE: [cip-dev] [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[Adithya.Balakumar]

-----Original Message-----
From: Gylstorff Quirin <quirin.gylstorff@siemens.com> 
Sent: Thursday, March 14, 2024 1:26 PM
To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate



On 3/8/24 12:35 PM, Jan Kiszka via lists.cip-project.org wrote:
> On 07.03.24 08:46, Adithya Balakumar wrote:
>> Delta handler requires zchunk > 1.3, but the required version is only 
>> available in bookworm backports
>>
>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>> ---
>>   conf/distro/cip-core-bookworm.conf                   |  3 +++
>>   conf/distro/debian-bookworm-backports.list           |  1 +
>>   .../preferences.swupdate-bookworm-backports.conf     | 12 ++++++++++++
>>   3 files changed, 16 insertions(+)
>>   create mode 100644 conf/distro/debian-bookworm-backports.list
>>   create mode 100644 
>> conf/distro/preferences.swupdate-bookworm-backports.conf
>>
>> diff --git a/conf/distro/cip-core-bookworm.conf 
>> b/conf/distro/cip-core-bookworm.conf
>> index 9bcf559..e0a4f7f 100644
>> --- a/conf/distro/cip-core-bookworm.conf
>> +++ b/conf/distro/cip-core-bookworm.conf
>> @@ -12,6 +12,9 @@
>>   require conf/distro/debian-bookworm.conf
>>   require cip-core-common.inc
>>   
>> +DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-backports.list"
>> +DISTRO_APT_PREFERENCES:append:swupdate = " conf/distro/preferences.swupdate-bookworm-backports.conf"
>> +
>>   PREFERRED_VERSION_swupdate ?= "2023.05%"
>>   PREFERRED_VERSION_linux-cip ?= "6.1.%"
>>   PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
>> diff --git a/conf/distro/debian-bookworm-backports.list 
>> b/conf/distro/debian-bookworm-backports.list
>> new file mode 100644
>> index 0000000..c4873b4
>> --- /dev/null
>> +++ b/conf/distro/debian-bookworm-backports.list
>> @@ -0,0 +1 @@
>> +deb http://deb.debian.org/debian bookworm-backports main contrib 
>> +non-free
>> diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf 
>> b/conf/distro/preferences.swupdate-bookworm-backports.conf
>> new file mode 100644
>> index 0000000..b6efffc
>> --- /dev/null
>> +++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
>> @@ -0,0 +1,12 @@
>> +Package: libzck-dev
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
>> +Package: libzck1
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
>> +Package: zchunk
>> +Pin: release n=bookworm-backports
>> +Pin-Priority: 801
>> +
> 
> Backports are bad when it comes to use in production (they receive 
> best-effort updates only). It this the only options?
> 

As we need only backports for zchunk i would propose that we support only rdiff based image for distros <= bookworm. The zchunk support can be activiated for trixie and later.

Quirin

[Adithya Balakumar] 
Hi Quirin,

Since we currently support creating sid images in isar-cip-core, would it be ok to enable zchunk support only for sid images currently?

Regards,
Adithya

> Jan
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15264): 
> https://lists.cip-project.org/g/cip-dev/message/15264
> Mute This Topic: https://lists.cip-project.org/mt/104783336/1753640
> Group Owner: cip-dev+owner@lists.cip-project.org
> Unsubscribe: 
> https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/14052693
> 26/xyzzy [quirin.gylstorff@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [cip-dev] [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate

[Gylstorff Quirin]

On 3/18/24 6:02 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> 
> 
> -----Original Message-----
> From: Gylstorff Quirin <quirin.gylstorff@siemens.com>
> Sent: Thursday, March 14, 2024 1:26 PM
> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 1/6] conf: Use bookworm backports for enabling delta handler in swupdate
> 
> 
> 
> On 3/8/24 12:35 PM, Jan Kiszka via lists.cip-project.org wrote:
>> On 07.03.24 08:46, Adithya Balakumar wrote:
>>> Delta handler requires zchunk > 1.3, but the required version is only
>>> available in bookworm backports
>>>
>>> Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
>>> ---
>>>    conf/distro/cip-core-bookworm.conf                   |  3 +++
>>>    conf/distro/debian-bookworm-backports.list           |  1 +
>>>    .../preferences.swupdate-bookworm-backports.conf     | 12 ++++++++++++
>>>    3 files changed, 16 insertions(+)
>>>    create mode 100644 conf/distro/debian-bookworm-backports.list
>>>    create mode 100644
>>> conf/distro/preferences.swupdate-bookworm-backports.conf
>>>
>>> diff --git a/conf/distro/cip-core-bookworm.conf
>>> b/conf/distro/cip-core-bookworm.conf
>>> index 9bcf559..e0a4f7f 100644
>>> --- a/conf/distro/cip-core-bookworm.conf
>>> +++ b/conf/distro/cip-core-bookworm.conf
>>> @@ -12,6 +12,9 @@
>>>    require conf/distro/debian-bookworm.conf
>>>    require cip-core-common.inc
>>>    
>>> +DISTRO_APT_SOURCES:append:swupdate = " conf/distro/debian-bookworm-backports.list"
>>> +DISTRO_APT_PREFERENCES:append:swupdate = " conf/distro/preferences.swupdate-bookworm-backports.conf"
>>> +
>>>    PREFERRED_VERSION_swupdate ?= "2023.05%"
>>>    PREFERRED_VERSION_linux-cip ?= "6.1.%"
>>>    PREFERRED_VERSION_linux-cip-rt ?= "6.1.%"
>>> diff --git a/conf/distro/debian-bookworm-backports.list
>>> b/conf/distro/debian-bookworm-backports.list
>>> new file mode 100644
>>> index 0000000..c4873b4
>>> --- /dev/null
>>> +++ b/conf/distro/debian-bookworm-backports.list
>>> @@ -0,0 +1 @@
>>> +deb http://deb.debian.org/debian bookworm-backports main contrib
>>> +non-free
>>> diff --git a/conf/distro/preferences.swupdate-bookworm-backports.conf
>>> b/conf/distro/preferences.swupdate-bookworm-backports.conf
>>> new file mode 100644
>>> index 0000000..b6efffc
>>> --- /dev/null
>>> +++ b/conf/distro/preferences.swupdate-bookworm-backports.conf
>>> @@ -0,0 +1,12 @@
>>> +Package: libzck-dev
>>> +Pin: release n=bookworm-backports
>>> +Pin-Priority: 801
>>> +
>>> +Package: libzck1
>>> +Pin: release n=bookworm-backports
>>> +Pin-Priority: 801
>>> +
>>> +Package: zchunk
>>> +Pin: release n=bookworm-backports
>>> +Pin-Priority: 801
>>> +
>>
>> Backports are bad when it comes to use in production (they receive
>> best-effort updates only). It this the only options?
>>
> 
> As we need only backports for zchunk i would propose that we support only rdiff based image for distros <= bookworm. The zchunk support can be activiated for trixie and later.
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> Since we currently support creating sid images in isar-cip-core, would it be ok to enable zchunk support only for sid images currently?

sure that also would work.

Quirin
> 
> Regards,
> Adithya
> 
>> Jan
>>
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#15264):
>> https://lists.cip-project.org/g/cip-dev/message/15264
>> Mute This Topic: https://lists.cip-project.org/mt/104783336/1753640
>> Group Owner: cip-dev+owner@lists.cip-project.org
>> Unsubscribe:
>> https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/14052693
>> 26/xyzzy [quirin.gylstorff@siemens.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Gylstorff Quirin]

On 3/18/24 5:59 AM, Adithya Balakumar via lists.cip-project.org wrote:
> 
> 
> -----Original Message-----
> From: Gylstorff Quirin <quirin.gylstorff@siemens.com>
> Sent: Friday, March 15, 2024 2:12 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates
> 
> 
> 
> On 3/15/24 7:45 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>> -----Original Message-----
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On
>> Behalf Of Quirin Gylstorff via lists.cip-project.org
>> Sent: Thursday, March 14, 2024 2:57 PM
>> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>;
>> cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com;
>> jan.kiszka@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting)
>> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>> Generate swu for delta updates
>>
>>
>>
>> On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>>> -----Original Message-----
>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org>
>>> On Behalf Of Quirin Gylstorff via lists.cip-project.org
>>> Sent: Wednesday, March 13, 2024 10:46 PM
>>> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com;
>>> jan.kiszka@siemens.com
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>>> Generate swu for delta updates
>>>
>>>
>>>
>>> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>>> -----Original Message-----
>>>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>>>> Sent: Friday, March 8, 2024 4:27 PM
>>>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR)
>>>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan
>>>> <jan.kiszka@siemens.com>
>>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
>>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate
>>>> swu for delta updates
>>>>
>>>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>>>> This allows the creation of a swu file for delta update. The swu
>>>>> file for delta update is only created when delta-update.yml file is
>>>>> included at build time.
>>>>>
>>>>> Signed-off-by: Adithya Balakumar
>>>>> <Adithya.Balakumar@toshiba-tsip.com>
>>>>> ---
>>>>>      classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>>>      1 file changed, 33 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass
>>>>> index
>>>>> 2c69892..2964024 100644
>>>>> --- a/classes/swupdate.bbclass
>>>>> +++ b/classes/swupdate.bbclass
>>>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>>>      SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>>>      SWU_BOOTLOADER ??= "ebg"
>>>>>      SWU_DESCRIPITION_FILE_BOOTLOADER ??= "${SWU_DESCRIPTION_FILE}-
>>>>> ${SWU_BOOTLOADER}"
>>>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>>>
>>>> Hi, do we really need a dedicated file / variable for that?
>>>>
>>>> [Adithya Balakumar]
>>>> Hi Felix,
>>>>
>>>> The sw-description file for a delta update is slightly different in
>>>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>>>
>>>> Would like to understand if there is a better approach.
>>>
>>> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
>>>
>>> ```
>>>          images: ({
>>> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
>>> +            filename = "${SWU_IMAGE_ARTIFACT}";
>>>                  device =
>>> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>>>                  type = "roundrobin";
>>>                  ${SWU_COMPRESSION_NODE}
>>> -            properties: {
>>> -                        subtype = "image";
>>> -                        configfilecheck =
>>> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
>>> -            };
>>> +            ${SWU_IMAGE_PROPERTIES}
>>>                  sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>>>          });
>>> ```
>>>
>>> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
>>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/gen
>>> e
>>> rate-sw-description/classes/sw-description-generator.bbclass?ref_type
>>> =
>>> heads
>>>
>>> Quirin
>>>
>>> [Adithya Balakumar]
>>> Hi Quirin,
>>>
>>> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.
>>
>> What would be the use case to have both a regular and a delta update?
>>
>> If you have multiple swus the server has the additional responsibility to select the correct swu.
>>
>> In the first version of the delta update I would use the either delta update or full update.
>>
>> Quirin
>>
>> [Adithya Balakumar]
>> Hi Quirin,
>>
>> I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.
> 
> Ok I see, then we should use a seperate sw-description file, similar to the bootloader update case. , with the necessary VARIABLES and still make the properties of the orginial sw-description modifiable. Also the functions and setting of variables should then be provided by your delta-update.bbclass like the efibootguard class.
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> When you say a separate sw-description file, do you mean a separate .tmpl file for delta update (ex: sw-description-delta.tmpl) ?

Yes.

Quirin

> 
> There would only be a few changes in the sw-description for the delta update case when compared to the complete update case.
> 
> Hence in the first patch, I created a separate sw-description file (copied the existing sw-description.tmpl as sw-description-delta.tmpl) for delta update in the recipe itself and only modified/added certain variables in the new .tmpl file with the sed command and let the ISAR template system create the final sw-description file.



> 
> Regards,
> Adithya
> 
>>
>> I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.
>>
>> Would like your take on this.
>>
>> Thanks and Regards,
>> Adithya
>>
>>>
>>> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
>>>
>>> Please correct me if my understanding is wrong.
>>>
>>> Regards,
>>> Adithya
>>>
>>>
>>>>
>>>>> "${SWU_DESCRIPTION_FILE}-delta"
>>>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>>>> "delta_update${@get_swu_compression_type(d)}"
>>>>>      
>>>>>      SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>>>      SWU_DESCRIPTION_FILE ?= "sw-description"
>>>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>>>
>>>> Does this mean the delta artifact is always added here?
>>>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>>>
>>>>>      SWU_SIGNED ??= ""
>>>>>      SWU_SIGNATURE_EXT ?= "sig"
>>>>>      SWU_SIGNATURE_TYPE ?= "cms"
>>>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>>>      SWU_BUILDCHROOT_IMAGE_FILE ?=
>>>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>>>      
>>>>>      IMAGE_TYPEDEP:swu =
>>>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>>>      IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>>      IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>>      IMAGE_INSTALL += "${@'swupdate-certificates' if
>>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu =
>>>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>>>      IMAGE_SRC_URI:swu +=
>>>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>>      IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>>>      IMAGE_TEMPLATE_FILES:swu +=
>>>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>>>      IMAGE_TEMPLATE_VARS:swu = " \
>>>>>          SWU_ROOTFS_PARTITION_NAME \
>>>>>          TARGET_IMAGE_UUID \
>>>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>>>          SWU_FILE_NODES \
>>>>>          SWU_BOOTLOADER_FILE_NODE \
>>>>>          SWU_SCRIPTS_NODE \
>>>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>>>          "
>>>>>      
>>>>>      # TARGET_IMAGE_UUID needs to be generated before completing the
>>>>> template @@ -136,6 +143,22 @@ python add_scripts(){
>>>>>          d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>>>      }
>>>>>      
>>>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>>>> +python add_swu_delta_update_properties() {
>>>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>>>> +    swu_delta_update_properties = ""
>>>>> +    if delta_type == "rdiff":
>>>>> +        swu_delta_update_properties =  'chainhandler =
>>>>> "rdiff_image";'
>>>>> +    elif delta_type == "zchunk":
>>>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>>>> +        swu_delta_update_properties = f"""
>>>>> +                        chainhandler = "delta";
>>>>> +                        url = "{zck_url}";
>>>>> +                        zckloglevel = "error";
>>>>> +        """
>>>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>>>> swu_delta_update_properties)
>>>>> +}
>>>>> +
>>>>>      # convert between swupdate compressor name and imagetype extension
>>>>>      def get_swu_compression_type(d):
>>>>>          swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13
>>>>> +177,21 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
>>>>> core}/recipes-core/images/swu"
>>>>>      
>>>>>      do_image_swu[depends] += "${PN}:do_transform_template"
>>>>>      do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>>> ${SWU_BOOTLOADER}"
>>>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>>>      IMAGE_CMD:swu() {
>>>>>          rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>>>> +        rm -rf ${WORKDIR}/swu-delta
>>>>
>>>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>>>
>>>> Felix
>>>>
>>>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>>>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>>>
>>>> Thanks and Regards,
>>>> Adithya
>>>>
>>>>>
>>>>> +    fi
>>>>> +
>>>>>          cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>>>          if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' ];
>>>>> then
>>>>>              cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>>>          fi
>>>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>>>> then
>>>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>>>> +    fi
>>>>> +
>>>>>      
>>>>>          for swu_file in "${WORKDIR}"/swu*; do
>>>>>              swu_file_base=$(basename $swu_file)
>>>>
>>>> --
>>>>> Siemens AG, Technology
>>>>> Linux Expert Center
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>> Links: You receive all messages sent to this group.
>>>> View/Reply Online (#15334): https://lists.cip-project.org/g/cip-dev/message/15334
>>>> Mute This Topic: https://lists.cip-project.org/mt/104783341/1753640
>>>> Group Owner: cip-dev+owner@lists.cip-project.org
>>>> Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129121/1753640/1405269326/xyzzy [quirin.gylstorff@siemens.com]
>>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>>

RE: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates

[Adithya.Balakumar]

-----Original Message-----
From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Quirin Gylstorff via lists.cip-project.org
Sent: Monday, March 18, 2024 3:05 PM
To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; jan.kiszka@siemens.com
Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate swu for delta updates



On 3/18/24 5:59 AM, Adithya Balakumar via lists.cip-project.org wrote:
> 
> 
> -----Original Message-----
> From: Gylstorff Quirin <quirin.gylstorff@siemens.com>
> Sent: Friday, March 15, 2024 2:12 PM
> To: balakumar adithya(ＴＳＩＰ TEUR) <Adithya.Balakumar@toshiba-tsip.com>; 
> cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
> jan.kiszka@siemens.com
> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ) 
> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) 
> <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ 
> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass: 
> Generate swu for delta updates
> 
> 
> 
> On 3/15/24 7:45 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>> -----Original Message-----
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>> On Behalf Of Quirin Gylstorff via lists.cip-project.org
>> Sent: Thursday, March 14, 2024 2:57 PM
>> To: balakumar adithya(ＴＳＩＰ TEUR) 
>> <Adithya.Balakumar@toshiba-tsip.com>;
>> cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
>> jan.kiszka@siemens.com
>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG 
>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>> Generate swu for delta updates
>>
>>
>>
>> On 3/14/24 8:41 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
>>> -----Original Message-----
>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>>> On Behalf Of Quirin Gylstorff via lists.cip-project.org
>>> Sent: Wednesday, March 13, 2024 10:46 PM
>>> To: cip-dev@lists.cip-project.org; felix.moessbauer@siemens.com; 
>>> jan.kiszka@siemens.com
>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>> Subject: Re: [cip-dev] [isar-cip-core][RFC v1 5/6] swupdate.bbclass:
>>> Generate swu for delta updates
>>>
>>>
>>>
>>> On 3/13/24 8:03 AM, Adithya Balakumar via lists.cip-project.org wrote:
>>>> -----Original Message-----
>>>> From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
>>>> Sent: Friday, March 8, 2024 4:27 PM
>>>> To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR) 
>>>> <Adithya.Balakumar@toshiba-tsip.com>; Kiszka, Jan 
>>>> <jan.kiszka@siemens.com>
>>>> Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting) 
>>>> <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
>>>> <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
>>>> Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
>>>> ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
>>>> Subject: Re: [isar-cip-core][RFC v1 5/6] swupdate.bbclass: Generate 
>>>> swu for delta updates
>>>>
>>>> On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
>>>>> This allows the creation of a swu file for delta update. The swu 
>>>>> file for delta update is only created when delta-update.yml file 
>>>>> is included at build time.
>>>>>
>>>>> Signed-off-by: Adithya Balakumar
>>>>> <Adithya.Balakumar@toshiba-tsip.com>
>>>>> ---
>>>>>      classes/swupdate.bbclass | 35 +++++++++++++++++++++++++++++++++--
>>>>>      1 file changed, 33 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/classes/swupdate.bbclass b/classes/swupdate.bbclass 
>>>>> index
>>>>> 2c69892..2964024 100644
>>>>> --- a/classes/swupdate.bbclass
>>>>> +++ b/classes/swupdate.bbclass
>>>>> @@ -26,10 +26,13 @@ SWU_EBG_UPDATE ?= ""
>>>>>      SWU_EFI_BOOT_DEVICE ?= "/dev/disk/by-uuid/4321-DCBA"
>>>>>      SWU_BOOTLOADER ??= "ebg"
>>>>>      SWU_DESCRIPITION_FILE_BOOTLOADER ??= 
>>>>> "${SWU_DESCRIPTION_FILE}- ${SWU_BOOTLOADER}"
>>>>> +SWU_DESCRIPITION_FILE_DELTA_UPDATE ??=
>>>>
>>>> Hi, do we really need a dedicated file / variable for that?
>>>>
>>>> [Adithya Balakumar]
>>>> Hi Felix,
>>>>
>>>> The sw-description file for a delta update is slightly different in 
>>>> terms of the image artifact inlcuded in the swu file and some additional delta update specific properties included in the sw-description file. Hence I chose to create a seperate sw-description file for delta update.
>>>>
>>>> Would like to understand if there is a better approach.
>>>
>>> A seperate sw-description file is not ideal, we should take the same abroach as with scripts and other optional or malible fields by writing the content to variable and use the template mechanism of ISAR. The sw-description can look like this:
>>>
>>> ```
>>>          images: ({
>>> -            filename = "${SWU_ROOTFS_PARTITION_NAME}";
>>> +            filename = "${SWU_IMAGE_ARTIFACT}";
>>>                  device =
>>> "C:BOOT0:linux.efi->${ABROOTFS_PART_UUID_A},C:BOOT1:linux.efi->${ABROOTFS_PART_UUID_B}";
>>>                  type = "roundrobin";
>>>                  ${SWU_COMPRESSION_NODE}
>>> -            properties: {
>>> -                        subtype = "image";
>>> -                        configfilecheck =
>>> "/etc/os-release@not_match@IMAGE_UUID=${TARGET_IMAGE_UUID}";
>>> -            };
>>> +            ${SWU_IMAGE_PROPERTIES}
>>>                  sha256 = "${SWU_ROOTFS_PARTITION_NAME}-sha256";
>>>          });
>>> ```
>>>
>>> an alternative would be generate the whole swu with the Variable[flag] syntax, e.g.
>>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/blob/feat/ge
>>> n
>>> e
>>> rate-sw-description/classes/sw-description-generator.bbclass?ref_typ
>>> e
>>> =
>>> heads
>>>
>>> Quirin
>>>
>>> [Adithya Balakumar]
>>> Hi Quirin,
>>>
>>> Thanks for the quick response. I understand the approach you are suggesting. But in a case where I have to create a swu file for both regular and a delta update, I would need two different sw-description files right. Each with different filename in the images section and some additional properties for delta update in the properties section.
>>
>> What would be the use case to have both a regular and a delta update?
>>
>> If you have multiple swus the server has the additional responsibility to select the correct swu.
>>
>> In the first version of the delta update I would use the either delta update or full update.
>>
>> Quirin
>>
>> [Adithya Balakumar]
>> Hi Quirin,
>>
>> I understand your opinion. But my thought was to have delta update swu as an optional / additional artifact apart from the complete update swu. Delta update artifact (atleast for rdiff) needs a reference artifact without which a delta artifact cannot be created. So in a case where a reference artifact is not provided, a delta artifact will not be created. But we could still create a complete update swu anyway.
> 
> Ok I see, then we should use a seperate sw-description file, similar to the bootloader update case. , with the necessary VARIABLES and still make the properties of the orginial sw-description modifiable. Also the functions and setting of variables should then be provided by your delta-update.bbclass like the efibootguard class.
> 
> Quirin
> 
> [Adithya Balakumar]
> Hi Quirin,
> 
> When you say a separate sw-description file, do you mean a separate .tmpl file for delta update (ex: sw-description-delta.tmpl) ?

Yes.

Quirin

[Adithya Balakumar] 
Ok, Understood. I was concerned about duplication since the sw-description for delta update will have only two changes when compared to sw-description for a complete update:
- artifact name to be included in the swu 
- some additional delta update specific properties 

Regards,
Adithya

> 
> There would only be a few changes in the sw-description for the delta update case when compared to the complete update case.
> 
> Hence in the first patch, I created a separate sw-description file (copied the existing sw-description.tmpl as sw-description-delta.tmpl) for delta update in the recipe itself and only modified/added certain variables in the new .tmpl file with the sed command and let the ISAR template system create the final sw-description file.



> 
> Regards,
> Adithya
> 
>>
>> I am trying to address concerns on usability as in a user not having to meddle with a yml to enable a delta update atleast for a initial test. One suggestion was to have a default case without editing any yml files. If we were to build a swupdate image AND provide the artifact of a previous version at some defined spot, the second build should generate the delta image automatically along with full update swus.
>>
>> Would like your take on this.
>>
>> Thanks and Regards,
>> Adithya
>>
>>>
>>> As per my understanding of the template mechanism of ISAR, the do_transform_template task generates an output file for every .tmpl file provided in the IMAGE_TEMPLATE_FILES variable.
>>>
>>> Please correct me if my understanding is wrong.
>>>
>>> Regards,
>>> Adithya
>>>
>>>
>>>>
>>>>> "${SWU_DESCRIPTION_FILE}-delta"
>>>>> +SWU_DELTA_UPDATE_ARTIFACT =
>>>>> "${SWU_ROOTFS_NAME}.delta_update${@get_swu_compression_type(d)}"
>>>>> +SWU_DELTA_UPDATE_ARTIFACT_TYPE =
>>>>> "delta_update${@get_swu_compression_type(d)}"
>>>>>      
>>>>>      SWU_IMAGE_FILE ?= "${IMAGE_FULLNAME}"
>>>>>      SWU_DESCRIPTION_FILE ?= "sw-description"
>>>>> -SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}"
>>>>> +SWU_ADDITIONAL_FILES ?= "linux.efi ${SWU_ROOTFS_PARTITION_NAME}
>>>>> ${SWU_DELTA_UPDATE_ARTIFACT}"
>>>>
>>>> Does this mean the delta artifact is always added here?
>>>> [Adithya Balakumar]  Apologies, I should have appended ${SWU_DELTA_UPDATE_ARTIFACT} conditionally if delta update is enabled.
>>>>
>>>>>      SWU_SIGNED ??= ""
>>>>>      SWU_SIGNATURE_EXT ?= "sig"
>>>>>      SWU_SIGNATURE_TYPE ?= "cms"
>>>>> @@ -37,6 +40,7 @@ SWU_SIGNATURE_TYPE ?= "cms"
>>>>>      SWU_BUILDCHROOT_IMAGE_FILE ?= 
>>>>> "${@os.path.basename(d.getVar('SWU_IMAGE_FILE'))}"
>>>>>      
>>>>>      IMAGE_TYPEDEP:swu =
>>>>> "${SWU_ROOTFS_TYPE}${@get_swu_compression_type(d)}"
>>>>> +IMAGE_TYPEDEP:swu += "${@ '${SWU_DELTA_UPDATE_ARTIFACT_TYPE}' if
>>>>> d.getVar('DELTA_UPDATE_TYPE') else ''}"
>>>>>      IMAGER_BUILD_DEPS:swu += "${@'swupdate-certificates-key' if
>>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>>      IMAGER_INSTALL:swu += "cpio ${@'openssl swupdate-certificates-key'
>>>>> if bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>>      IMAGE_INSTALL += "${@'swupdate-certificates' if
>>>>> bb.utils.to_boolean(d.getVar('SWU_SIGNED')) else ''}"
>>>>> @@ -46,6 +50,7 @@ IMAGE_SRC_URI:swu = 
>>>>> "file://${SWU_DESCRIPTION_FILE}.tmpl"
>>>>>      IMAGE_SRC_URI:swu +=
>>>>> "file://${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>>      IMAGE_TEMPLATE_FILES:swu = "${SWU_DESCRIPTION_FILE}.tmpl"
>>>>>      IMAGE_TEMPLATE_FILES:swu +=
>>>>> "${SWU_DESCRIPITION_FILE_BOOTLOADER}.tmpl"
>>>>> +IMAGE_TEMPLATE_FILES:delta_update +=
>>>>> "${SWU_DESCRIPITION_FILE_DELTA_UPDATE}.tmpl"
>>>>>      IMAGE_TEMPLATE_VARS:swu = " \
>>>>>          SWU_ROOTFS_PARTITION_NAME \
>>>>>          TARGET_IMAGE_UUID \
>>>>> @@ -58,6 +63,8 @@ IMAGE_TEMPLATE_VARS:swu = " \
>>>>>          SWU_FILE_NODES \
>>>>>          SWU_BOOTLOADER_FILE_NODE \
>>>>>          SWU_SCRIPTS_NODE \
>>>>> +    SWU_DELTA_UPDATE_ARTIFACT \
>>>>> +    SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES \
>>>>>          "
>>>>>      
>>>>>      # TARGET_IMAGE_UUID needs to be generated before completing 
>>>>> the template @@ -136,6 +143,22 @@ python add_scripts(){
>>>>>          d.appendVar('SWU_SCRIPTS_NODE', swu_scripts_node)
>>>>>      }
>>>>>      
>>>>> +SWU_EXTEND_SW_DESCRIPTION += "add_swu_delta_update_properties"
>>>>> +python add_swu_delta_update_properties() {
>>>>> +    delta_type = d.getVar('DELTA_UPDATE_TYPE')
>>>>> +    swu_delta_update_properties = ""
>>>>> +    if delta_type == "rdiff":
>>>>> +        swu_delta_update_properties =  'chainhandler =
>>>>> "rdiff_image";'
>>>>> +    elif delta_type == "zchunk":
>>>>> +        zck_url = d.getVar('DELTA_ZCK_URL')
>>>>> +        swu_delta_update_properties = f"""
>>>>> +                        chainhandler = "delta";
>>>>> +                        url = "{zck_url}";
>>>>> +                        zckloglevel = "error";
>>>>> +        """
>>>>> +    d.setVar('SWU_DELTA_UPDATE_ADDITIONAL_PROPERTIES',
>>>>> swu_delta_update_properties)
>>>>> +}
>>>>> +
>>>>>      # convert between swupdate compressor name and imagetype extension
>>>>>      def get_swu_compression_type(d):
>>>>>          swu_ct = d.getVar('SWU_COMPRESSION_TYPE') @@ -154,13
>>>>> +177,21 @@ FILESEXTRAPATHS:append = ":${LAYERDIR_cip-
>>>>> core}/recipes-core/images/swu"
>>>>>      
>>>>>      do_image_swu[depends] += "${PN}:do_transform_template"
>>>>>      do_image_swu[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>>> -do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu- 
>>>>> ${SWU_BOOTLOADER}"
>>>>> +do_image_swu[cleandirs] += "${WORKDIR}/swu ${WORKDIR}/swu-
>>>>> ${SWU_BOOTLOADER} ${WORKDIR}/swu-delta"
>>>>>      IMAGE_CMD:swu() {
>>>>>          rm -f '${DEPLOY_DIR_IMAGE}/${SWU_IMAGE_FILE}'*.swu
>>>>> +    if [ -z "${DELTA_UPDATE_TYPE}" ]; then
>>>>> +        rm -rf ${WORKDIR}/swu-delta
>>>>
>>>> Why is this required if we already clear the ${WORKDIR}/swu-delta in cleandirs?
>>>>
>>>> Felix
>>>>
>>>> [Adithya Balakumar] Apologies. I should add swu-delta to do_image_swu[cleandirs] only if a delta update is enabled.
>>>> But even with the above change, in a case where we build with delta update enabled and without cleaning the build directory if we build again without enabling delta update, the swu-delta directory is retained in the WORKDIR.
>>>>
>>>> Thanks and Regards,
>>>> Adithya
>>>>
>>>>>
>>>>> +    fi
>>>>> +
>>>>>          cp '${WORKDIR}/${SWU_DESCRIPTION_FILE}'
>>>>> '${WORKDIR}/swu/${SWU_DESCRIPTION_FILE}'
>>>>>          if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}' 
>>>>> ]; then
>>>>>              cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_BOOTLOADER}'
>>>>> '${WORKDIR}/swu-${SWU_BOOTLOADER}/${SWU_DESCRIPTION_FILE}'
>>>>>          fi
>>>>> +    if [ -f '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}' ];
>>>>> then
>>>>> +        cp '${WORKDIR}/${SWU_DESCRIPITION_FILE_DELTA_UPDATE}'
>>>>> '${WORKDIR}/swu-delta/${SWU_DESCRIPTION_FILE}'
>>>>> +    fi
>>>>> +
>>>>>      
>>>>>          for swu_file in "${WORKDIR}"/swu*; do
>>>>>              swu_file_base=$(basename $swu_file)
>>>>
>>>> --
>>>>> Siemens AG, Technology
>>>>> Linux Expert Center
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 
>>>>

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[MOESSBAUER, Felix]

On Wed, 2024-03-13 at 18:19 +0100, Gylstorff Quirin wrote:
> 
> 
> On 3/13/24 7:11 AM, Adithya.Balakumar@toshiba-tsip.com wrote:
> > 
> > From: MOESSBAUER, Felix <felix.moessbauer@siemens.com>
> > Sent: Friday, March 8, 2024 4:22 PM
> > To: cip-dev@lists.cip-project.org; balakumar adithya(ＴＳＩＰ TEUR)
> > <Adithya.Balakumar@toshiba-tsip.com>; quirin.gylstorff@siemens.com;
> > Kiszka, Jan <jan.kiszka@siemens.com>
> > Cc: kunijadar shivanand(ＴＳＩＰ TMIEC ODG Porting)
> > <Shivanand.Kunijadar@toshiba-tsip.com>; ashrith sai(ＴＳＩＰ)
> > <Sai.Sathujoda@toshiba-tsip.com>; dinesh kumar(ＴＳＩＰ TMIEC ODG
> > Porting) <dinesh.kumar@toshiba-tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ
> > ○ＤＩＧ□ＭＰＳ○ＭＰ４) <kazuhiro3.hayashi@toshiba.co.jp>
> > Subject: Re: [isar-cip-core][RFC v1 4/6] Add Delta update support
> > with rdiff_image and delta handler
> > 
> > On Thu, 2024-03-07 at 13:16 +0530, Adithya Balakumar wrote:
> > > swupdate supports delta updates with rdiff_image and
> > > delta(zchunk)
> > > handler. This change adds support to use either of the handler
> > > for
> > > creating delta update artifacts. The type of handler can be set
> > > in the
> > > delta-update.yml file along with additional variables needed for
> > > the
> > > chosen handler.
> > > 
> > > Signed-off-by: Adithya Balakumar
> > > <Adithya.Balakumar@toshiba-tsip.com>
> > > ---
> > >   classes/delta_update.bbclass | 90
> > 
> > > Hi,
> > 
> > > please name this class delta-update.bbclass to keep the naming
> > > style similar with the rest of the project.
> > 
> > > ++++++++++++++++++++++++++++++++++++
> > >   kas/opt/delta-update.yml     | 21 +++++++++
> > >   2 files changed, 111 insertions(+)
> > >   create mode 100644 classes/delta_update.bbclass
> > >   create mode 100644 kas/opt/delta-update.yml
> > > 
> > > diff --git a/classes/delta_update.bbclass
> > > b/classes/delta_update.bbclass new file mode 100644 index
> > > 0000000..8d47081
> > > --- /dev/null
> > > +++ b/classes/delta_update.bbclass
> > > @@ -0,0 +1,90 @@
> > > +#
> > > +# CIP Core, generic profile
> > > +#
> > > +# Copyright (c) Toshiba Corporation 2024 # # Authors:
> > > +#  Adithya Balakumar <adithya.balakumar@toshiba-tsip.com>
> > > +#
> > > +# SPDX-License-Identifier: MIT
> > > +#
> > > +
> > > +IMAGER_INSTALL:delta_update += "fdisk zchunk rdiff"
> > 
> > > Conditionally appending with += overwrites already set weak
> > > defaults.
> > > You might want to use .= " fdisk zchunk rdiff" instead.
> > 
> > > +
> > > +FILESEXTRAPATHS:append = ":${LAYERDIR_cip-core}"
> > > +IMAGE_SRC_URI:delta_update += " ${@ "
> > 
> > > Same here.
> > 
> > > file://"+d.getVar('DELTA_RDIFF_REF_IMAGE') if
> > 
> > > I guess this could be simplified and made more portable by
> > > referencing the DELTA_RDIFF_REF_IMAGE directly (e.g. from
> > > ${DEPLOY_DIR_IMAGE}). The problem with the FILESEXTRAPATHS:append
> > > = ":${LAYERDIR_cip->core}" is that it only works if the delta
> > > image is also from cip-core, but usually cip-core is just
> > > referenced in a downstream layer and used there.
> > 
> > [Adithya Balakumar]
> > Hi Felix,
> > 
> > I understand the drawback of using FILESEXTRAPATHS:append =
> > ":${LAYERDIR_cip-core}" that it only looks for the delta image
> > placed in the isar-cip-core layer.
> > As an example you point to referencing the DELTA_RDIFF_REF_IMAGE
> > directly from ${DEPLOY_DIR_IMAGE} but as per my understanding
> > ${DEPLOY_DIR_IMAGE} points to the current build's image deploy
> > directory.
> > 
> > So, Could you please suggest a suitable way to provide the delta
> > image in the case of rdiff.
> > 
> > > d.getVar('DELTA_UPDATE_TYPE') == 'rdiff' and
> > > d.getVar('DELTA_RDIFF_REF_IMAGE') else '' }"
> > > +
> > > +do_image_delta_update[cleandirs] +=
> > > "${WORKDIR}/delta_interim_artifacts"
> > > +
> > > +extract_root_fs() {
> > > +    INPUT_ARTIFACT=$1
> > > +    OUTPUT_ARTIFACT=$2
> > > +    SECTOR_SIZE=512
> > > +    ROOTFS_LABEL="primary"
> > > +    ROOTFS_START_SECTOR=$(${SUDO_CHROOT} /sbin/fdisk -l -o
> > 
> > > Can't we simply use sfdisk to query this information instead of
> > > all the awk stuff?
> > 
> > > Name,Start "${INPUT_ARTIFACT}" | awk -v name="${ROOTFS_LABEL}"
> > > '$0 ~
> > > name {print $2}' | head -n 1)
> > > +    SKIP_VALUE="$(expr $(expr $ROOTFS_START_SECTOR \*
> > > $SECTOR_SIZE)
> > > / $(expr 1024 \* 1024))"
> > > +    ${SUDO_CHROOT} /bin/dd if=${INPUT_ARTIFACT} bs=1M
> > > skip=$SKIP_VALUE count=1024 of=${OUTPUT_ARTIFACT} status=none
> > > +}
> > 
> > > Why copy exactly 1GB?
> > 
> > > Instead of extracting that here, it might be simpler to just
> > > store the .rdiff file (of the rootfs) with each image build and
> > > use that.[Adithya Balakumar]
> > 
> > [Adithya Balakumar]  Here, I am copying 1GB from the wic image
> > because that is the size of the rootfs partition.
> > 
> > However, Quirin left a remark stating that the .squashfs file (or
> > .verity file) could be used to create the delta artifact. Using the
> > squashfs or verity file to create the delta will eliminate need for
> > extracting the rootfs from the wic image and will simplify many
> > things in this recipe. Would like to understand your opinion on
> > this.
> 
> Also you can use the `WIC_DEPLOY_PARTITIONS = "1"` to generate the 
> seperate wic partitions. That is much more robust then some
> dd magic.

Yes, that is possible, but please try to avoid this. Partitions can be
quite large and WIC_DEPLOY_PARTITONS=1 deploys all partitions. Better
just use the squashfs or verity blob.

Felix

> 
> Quirin
> > 
> > Thanks and Regards,
> > Adithya
> > 
> > 
> > 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Wang, Qi]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]

One suggestion for rdiff delta update:
It should be more resource efficient to create delta image from the rdiff signature of old image, instead of from the whole old image file.
So that user doesn't need to save the whole old image file, just need to save the small signature file.
At least we can provide this option for user to choose.

[-- Attachment #2: Type: text/html, Size: 366 bytes --]

Re: [isar-cip-core][RFC v1 4/6] Add Delta update support with rdiff_image and delta handler

[Wang, Qi]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

using fixed URL for downloading zchunk file might not be a good idea, e.g. user might not know the IP address / domain of the webserver when generating the delta image.
one possible way is to change the configuration of delta update handler, to support relative url (or other parameters), and combine the url when downloading.

[-- Attachment #2: Type: text/html, Size: 333 bytes --]